Flowio

FCS Semantics for FlowIO

Use this reference when the task depends on what values or metadata mean, not just how to call the API.

Scope

FlowIO reads FCS 2.0, 3.0, and 3.1 and writes a constrained FCS 3.1 representation. It is a file-format library, not a complete flow-cytometry analysis system.

Before downstream biological interpretation, decide whether the workflow also requires:

• Spillover compensation

• Logicle, biexponential, or arcsinh transformation

• Quality-control filtering

• Singlet, viability, or population gating

• Batch correction or normalization

FlowIO does none of these. Its optional preprocessing is limited to scaling defined by FCS acquisition metadata.

File Segments

An FCS dataset can contain:

• HEADER: Version and byte offsets for other segments.

• TEXT: Required and optional keyword/value metadata.

• DATA: Event values.

• ANALYSIS: Optional keyword/value results.

FlowIO exposes these through header, text, events, and analysis.

FCS 3.1 deprecated storing multiple datasets in one file through $NEXTDATA, but FlowIO can read legacy multi-dataset files with read_multiple_data_sets().

TEXT Keyword Normalization

The FCS standard treats keyword names as case-insensitive. Standard keywords are written with $, but FlowIO normalizes parsed keys:

1. Strip the leading $.

2. Convert the key to lowercase.

3. Keep the value as a string.

Examples:

• $DATE becomes text["date"]

• $P1N becomes text["p1n"]

• $SPILLOVER becomes text["spillover"]

• $NEXTDATA becomes text["nextdata"]

Use:

date = flow.text.get("date")
spill = flow.text.get("spillover", flow.text.get("spill"))
nextdata = int(flow.text.get("nextdata", "0"))

Avoid:

date = flow.text.get("$DATE")  # Always misses in FlowIO's normalized mapping.

Custom, non-standard keys are also lowercased. The normalized mapping may contain subject, sample, operator, institution, instrument serial-number, and free-text fields. Treat an unrestricted metadata dump as potentially identifying.

FlowIO 1.4.0 implements the first step by removing every $ character from the decoded segment before splitting keys and values. A literal $ inside a value is therefore lost ("a$b" becomes "ab"). Do not use the parsed mapping as a lossless metadata archive.

Parameters, Labels, and Indices

FCS parameter numbers begin at 1:

• P1N, P2N, ... are required primary labels.

• P1S, P2S, ... are optional descriptive/stain labels.

• PnR records a parameter range.

• PnE records logarithmic amplification information.

• PnG records gain.

FlowIO presents these through two index systems:

• flow.channels[1], flow.channels[2], ... use one-based parameter numbers.

• NumPy columns and all *_indices attributes use zero-based indices.

When reporting a channel, label both values if ambiguity matters:

for array_index, pnn in enumerate(flow.pnn_labels):
    parameter_number = array_index + 1
    pns = flow.pns_labels[array_index]
    print(parameter_number, array_index, pnn, pns)

PnS is optional. FlowIO inserts "" for missing PnS entries so that pns_labels and pnn_labels have the same length.

FlowIO infers scatter, fluorescence, and time indices from channel labels. Treat these as conveniences, not a substitute for checking the instrument panel. Vendor-specific labels may not classify as expected.

null_channel_list accepts PnN labels and excludes matching channels from the derived fluorescence/scatter/time index lists. It does not remove columns from the event data. flow.null_channels stores the supplied labels unchanged; it does not contain zero-based indices and can retain labels that did not match a channel.

Encoded and Preprocessed Event Values

Encoded representation

flow.events is a decoded flattened one-dimensional sequence, ordered by event and then channel. It is usually an array.array; mixed-width integer channels use a Python list:

event_1_channel_1, event_1_channel_2, ..., event_2_channel_1, ...

flow.as_array(preprocess=False) reshapes that representation to:

(event_count, channel_count)

For integer DATA, FlowIO applies the PnR range mask during decoding, before as_array(). Values with high bits outside that range do not retain those bits in flow.events or preprocess=False; preserve the original file for byte fidelity. The number of array rows is inferred from DATA and may disagree with $TOT. Verify shape/counts explicitly.

The returned NumPy array is float64, even when the encoded file uses integer or single-precision event data.

FlowIO preprocessing

flow.as_array(preprocess=True) performs the following operations.

Time scaling

When a time channel and nonempty timestep keyword exist:

time_scaled = time_encoded * timestep

An empty or whitespace-only timestep is treated as 1.0 in FlowIO 1.4.0.

Logarithmically stored channels

For PnE (decades, log_zero) and PnR range:

linear_value = 10 ** (decades * encoded_value / range) * log_zero

The conversion is applied when decades > 0. FlowIO replaces a zero log_zero with 1.0 when decades is nonzero.

Gain

For PnG gain:

gain_scaled = value / gain

FlowIO skips division when gain is zero or one. For a recognized non-null channel named Time (case-insensitive), FlowIO forces the parsed channel gain to 1.0, regardless of TEXT PnG; timestep scaling still applies.

The operations are metadata-driven. Bad metadata can therefore produce bad scaled values even when the DATA bytes were parsed correctly.

What preprocessing does not do

FlowIO does not:

• Parse and apply $SPILL/$SPILLOVER compensation

• Perform logicle, biexponential, hyperlog, or arcsinh transformation

• Normalize between files

• Identify acquisition anomalies

• Remove debris, doublets, or dead cells

• Gate populations

For compensation/transformation/gating, use a higher-level package such as FlowKit and preserve a record of the matrix and transform parameters.

Spillover Metadata

FlowIO can expose or write a spillover keyword as a string. It does not validate the matrix scientifically or apply it.

A standard spillover string begins with:

1. Number of compensated fluorescence parameters

2. Matching PnN labels

3. Flattened matrix values

Items are comma-delimited with no newline characters.

Before passing the matrix downstream:

• Confirm the labels exactly map to the event columns.

• Confirm the matrix dimensions match the declared parameter count.

• Confirm whether values are a compensation matrix or spillover matrix as expected by the downstream API.

• Keep the uncompensated values and original metadata for provenance.

Offset Semantics

FCS 3.0/3.1 DATA offsets can appear in HEADER and TEXT. FlowIO normally uses TEXT offsets and checks that HEADER agrees.

Some files have known defects:

• The last DATA byte is reported as exclusive rather than inclusive, creating an off-by-one error.

• HEADER and TEXT report different offsets.

• Large FCS 3.1 files put zero in HEADER DATA offsets when a segment extends beyond the eight-digit HEADER limit and store the real offsets in TEXT.

FlowIO accounts for the FCS 3.1 large-file rule. Do not use relaxed flags merely because a file is large.

Recovery options:

• ignore_offset_error=True: tolerate the documented off-by-one case.

• ignore_offset_discrepancy=True: use TEXT despite a HEADER/TEXT mismatch.

• use_header_offsets=True: use HEADER and suppress the mismatch error.

Each option changes which bytes are interpreted as events. Use one only when the file's provenance or vendor behavior justifies it, and validate event count, channel distributions, and known controls afterward.

Memory Model

Normal FlowData construction reads the full DATA segment into memory. as_array() then allocates a second float64 representation.

Approximate additional memory for the 2-D array is:

event_count * channel_count * 8 bytes

The estimate assumes $TOT agrees with DATA; check the actual loaded length before allocating the array. This excludes the original event array, Python objects, temporary arrays, and downstream DataFrames.

Use only_text=True for inventory. FlowIO 1.4.0 does not expose chunked, streaming, lazy, or memory-mapped event reads. If a file does not fit safely in memory, use a different parser/workflow or process it in a resource-limited environment; do not claim FlowIO can chunk it.

only_text=True skips DATA loading but still parses ANALYSIS. Also prefer paths over open handles: FlowData closes caller-provided handles, and the multi-dataset helper cannot reuse a closed handle for its second dataset in FlowIO 1.4.0.

Writer Representation

create_fcs() and write_fcs() produce:

• FCS 3.1

• List mode

• Single-precision 32-bit float event values

• Little-endian byte order

• No ANALYSIS segment

• A single dataset

The float32 representation has roughly 6-7 decimal digits of precision. Round-trip comparisons should use tolerances rather than exact equality.

Passing a NumPy array makes create_fcs() allocate an array('f') copy. An existing array('f') is written directly, so retain that representation when large writer memory is a concern.

create_fcs() needs flattened event data. Flatten a two-dimensional array in C order:

flat = events_2d.astype("float32", copy=False).ravel(order="C")

Validate:

if events_2d.ndim != 2:
    raise ValueError("expected events x channels")
if events_2d.shape[1] != len(channel_names):
    raise ValueError("channel label count mismatch")

Required keywords and channel interpretation fields are generated by FlowIO. Do not rely on metadata_dict to override $PAR, $TOT, $MODE, $DATATYPE, PnB, or PnN.

For floating-point sources, write_fcs() can keep encoded values while dropping PnG and timestep. This changes the interpretation returned by as_array(preprocess=True). Validate both encoded and metadata-scaled values, not just event counts.

Scientific Provenance

For any converted or rewritten file, record:

• Source file name and checksum

• FlowIO version

• FCS version and source $DATATYPE

• Whether preprocess was true or false

• Any relaxed offset option and its justification

• Channel order and label mapping

• Metadata removed, added, or renamed

• Whether compensation or another transform occurred elsewhere

• Output representation and float32 precision

• Round-trip validation results