X12 Format

Clinker reads and writes ANSI ASC X12 interchanges alongside CSV, JSON, XML, fixed-width, and EDIFACT. An X12 interchange is a finite file with a three-tier envelope: an ISA..IEA interchange wraps one or more GS..GE functional groups, and each functional group wraps one or more ST..SE transaction sets. The reader streams one segment at a time and the writer reconstructs the three envelope tiers around emitted records.

The three tiers surface as nested document-context levels: the ISA interchange becomes the file-level $doc document, and each GS group and ST set opens a nested level whose $doc sections layer over the enclosing tiers. A body record therefore sees every enclosing tier’s fields through one $doc.<section>.<field> lookup.

Delimiters and the ISA header

Unlike EDIFACT’s optional UNA service-string advice, X12 declares its delimiters in a fixed-length 106-byte ISA header. Three delimiter bytes live at structural positions within it:

The reader reads these three bytes from the header rather than assuming a fixed delimiter set, so an interchange that uses */:/~, |/^/newline, or any other producer-chosen delimiters parses correctly. The ISA13 interchange control number is located as the 13th element of the header split on the discovered element separator — structurally, not by an absolute byte offset — so producer padding quirks do not misalign it.

On output, an X12 sink that echoes the header via interchange_from_doc also adopts the source’s discovered delimiter set, so a reconstructed interchange keeps the exact element separator, sub-element separator, and segment terminator bytes it arrived with (see Writing X12). The literal interchange option keeps the writer’s */:/~ defaults.

No escape character

X12 has no release/escape character (EDIFACT’s ? has no X12 equivalent). A data value that contains a delimiter byte is therefore unrepresentable. On output the writer rejects any element value carrying the element separator or the segment terminator with a precise error rather than silently corrupting the interchange; re-encode the value or choose delimiters the data does not contain.

The sub-element (component) separator inside an element (e.g. the : in a composite A:B:C) is kept as part of the element’s text and is not split — the positional element model works above component resolution, so a composite element round-trips unchanged.

Newlines between segments

Some producers insert CR/LF after each segment terminator for readability. Those bytes are insignificant and are stripped between segments; CR/LF that appears inside an element is preserved.

Record shape

Each non-service segment becomes one record under a fixed positional schema:

The reader does not stamp the enclosing functional-group control number as a column; it surfaces through $doc.functional_group.e06 (see Envelope sections over the three tiers). An optional group_ref column drives multi-group output — project it from that $doc field when reconstructing several functional groups (see Writing X12).

Service segments (ISA, IEA, GS, GE, SE) are consumed by the reader to drive the envelope and validation — they are never emitted as body records. The ST segment that opens a transaction set is emitted as a body record (its seg_id is ST), carrying the set reference and type.

The number of eNN columns is controlled by the source max_elements option (default 32). A segment carrying more data elements than that is rejected with guidance rather than silently truncated. Absent trailing elements read as null.

nodes:
  - type: source
    name: orders
    config:
      name: orders
      type: x12
      glob: ./inbox/*.x12
      options:
        max_elements: 48      # widen the positional schema for exotic segments
        encoding: iso-8859-1  # decode body element text as Latin-1
      schema:
        - { name: seg_id, type: string }
        - { name: set_ref, type: string }
        - { name: e01, type: string }

Character set

X12 carries no in-band element that names the body character repertoire (unlike EDIFACT’s UNB syntax identifier). Element text is therefore decoded through the charset the source declares in its encoding option, defaulting to UTF-8. The ISA header’s control fields are ASCII, so the declared charset affects only the body element text.

Decoding stays streaming and per-element — the interchange is never buffered whole. An interchange whose body carries Latin-1 high bytes (for example accented characters in free-text name or address fields) parses without error once the source declares encoding: iso-8859-1, and the decoded element text matches the source bytes under Latin-1.

A source that omits the option and meets non-UTF-8 bytes fails explicitly (“segment is not valid UTF-8”) rather than corrupting the data silently; a source that declares an unsupported encoding fails at startup with a precise error naming the value. On output, set the same encoding on the X12 sink so the round-trip is byte-faithful; a character the chosen charset cannot represent (for example a non-Latin-1 codepoint under iso-8859-1) is rejected rather than emitted truncated.

Envelope sections over the three tiers

The interchange header ISA is extractable as a file-level document envelope section, exposing its positional elements to CXL as $doc.<section>.<field>. Use the segment extract rule with the section field names matching the positional keys e01, e02, …:

envelope:
  sections:
    interchange:
      extract: { segment: "ISA" }
      fields:
        e13: string          # interchange control number (ISA13)

The GS functional group and the ST transaction set surface automatically as the nested $doc sections functional_group and transaction_set, each keyed by positional eNN elements — no envelope declaration is needed for them. A Transform on any body record can read all three tiers at once:

emit isa13 = $doc.interchange.e13       # interchange control number
emit gs06  = $doc.functional_group.e06  # group control number (GS06)
emit st02  = $doc.transaction_set.e02   # set control number (ST02)

Naming and typing the nested levels

The ISA header is declared through envelope: because a bounded pre-scan resolves it before any body streams. The GS group and ST set exist only mid-file, so they cannot be declared the same way — instead, name them and give them a typed field schema under the X12 source’s options. The reader applies the declaration each time it crosses a group or set boundary, so $doc.<your-name>.<field> exposes the level’s elements under the name you chose, coerced to the types you declared — the same way a declared ISA field is typed and coerced:

type: x12
options:
  group_section:
    name: functional_group   # your choice — the engine reserves no name
    fields:
      e01: string            # GS01 functional identifier code
      e06: int               # GS06 group control number
  set_section:
    name: transaction_set    # your choice
    fields:
      e01: int               # ST01 transaction-set identifier code
      e02: string            # ST02 set control number

emit functional_id = $doc.functional_group.e01   # typed string
emit group_control = $doc.functional_group.e06    # typed int
emit txn_type      = $doc.transaction_set.e01     # typed int

The two levels are declared independently — name one, both, or neither. A declared field schema is the contract: only the elements it lists surface in the typed section, and an element the wire carries but the schema omits is absent from $doc. An element that cannot coerce to its declared type (declaring the alphabetic GS01 code as an int, say) fails the run with a precise error. Omit a level’s declaration and it keeps its default name (functional_group / transaction_set) keyed by untyped positional eNN strings, unchanged from before this option existed.

Only the ISA header is extractable through the envelope: block. Trailer segments (SE, GE, IEA) arrive after the body they close and cannot become $doc fields without buffering the whole interchange — their control counts are instead validated inline by the reader (see below). A segment extract naming any tag other than ISA, or an xml_path / json_pointer extract against an X12 source, is rejected at startup.

Control-count validation

The reader validates the structural integrity claims carried in the trailers as they arrive, failing the run on a mismatch (a truncation or corruption signal):

• SE segment count (SE01) — must equal the number of segments in the transaction set, counting the ST and SE themselves.
• SE set control number (SE02) — must echo the opening ST02.
• GE transaction-set count (GE01) — must equal the number of ST sets in the functional group.
• GE group control number (GE02) — must echo the GS06.
• IEA functional-group count (IEA01) — must equal the number of GS groups in the interchange.
• IEA control number (IEA02) — must echo the ISA13.

A missing IEA at end of input is a truncation error; content after the IEA trailer is rejected.

Routing a count mismatch to the DLQ

By default a SE/GE/IEA count mismatch aborts the run. A source declaring dlq_granularity: document instead dead-letters the whole interchange / file to the DLQ — the file’s records become a structural_validation trigger plus document_rejected collaterals, and no record of the malformed file reaches the sink. The count is only known at the trailer, after the body has streamed, so the rejection lands at the sink boundary (no record is written out), not literally before the first record. The grain is the whole file: an SE-level mismatch rejects the entire interchange, not just that transaction set. The control-number echo mismatches (SE02/GE02/IEA02) and every other corruption (truncation, post-trailer content) always abort, even under the opt-in. See Malformed envelopes.

Writing X12

An X12 Output node reconstructs the three-tier envelope around emitted records. Records map by the same positional columns (seg_id, set_ref, set_type, eNN, and the optional group_ref); trailing null/empty elements are trimmed so no fabricated delimiters appear, and a column the writer does not recognize is an error (project the record to the X12 columns first). Engine-internal $-namespaced columns are excluded automatically.

nodes:
  - type: output
    name: out
    input: messages
    config:
      name: out
      type: x12
      path: ./out/result.x12
      options:
        interchange:
          ["00", "          ", "00", "          ", "ZZ", "SENDER         ",
           "ZZ", "RECEIVER       ", "240101", "1200", "U", "00401",
           "000000001", "0", "P", ":"]
        group_header: ["PO", "SENDER", "RECEIVER", "20240101", "1200", "1", "X", "004010"]
        set_type: "850"
        segment_newline: true

Output options:

Consecutive records are grouped into ST..SE transaction sets on set_ref transitions and into GS..GE functional groups on group_ref transitions — the group discriminator is the outer-tier analog of set_ref. The writer recomputes the SE segment count, the per-group GE transaction-set count, and the IEA functional-group count, and echoes the set, group, and interchange control numbers, so the output passes its own count validation on re-read.

Multiple functional groups

A real interchange can carry several functional groups — purchase orders (PO) and invoices (IN) in one ISA..IEA, each in its own GS..GE with a distinct functional identifier code. Project a group_ref column and the writer opens a fresh GS..GE group every time its value changes, echoing that value as the group control number (GS06/GE02) and recomputing each group’s own GE01 transaction-set count. With no group_ref column the whole stream collapses to one functional group, byte-identical to the single-group shape, so the column is opt-in. To split a source interchange into the same groups it arrived in, project group_ref from $doc.functional_group.e06:

nodes:
  - type: transform
    name: regroup
    input: orders
    config:
      cxl: |
        emit seg_id    = seg_id
        emit group_ref = $doc.functional_group.e06   # functional group control number (GS06)
        emit set_ref   = set_ref
        emit set_type  = set_type
        emit e01       = e01
        # … remaining eNN columns …
  - type: output
    name: out
    input: regroup
    config:
      name: out
      type: x12
      path: ./out/result.x12
      options:
        interchange_from_doc: interchange
        group_header: ["PO", "SENDER", "RECEIVER", "20240101", "1200", "1", "X", "004010"]

Records that share a group_ref value must arrive consecutively, exactly as records sharing a set_ref must: the writer streams and closes a group the moment the discriminator changes, so an interleaved stream would reopen a group it already closed. Sort upstream by group_ref (then set_ref) when the record order does not already guarantee it.

interchange_from_doc echoes the header from a record’s document context. That context is populated by a source’s ISA envelope section (declare a segment: "ISA" envelope section on the source) and travels with every body record through the pipeline — including to a sink that sits directly downstream of the source with no intervening Transform. The reader stashes the complete, ordered ISA element list together with the delimiter set it discovered from the header, so the reconstructed header is faithful and the whole output interchange — header, envelopes, and body — is emitted with the original element separator, sub-element separator, and segment terminator rather than the writer’s */:/~ defaults. Supply interchange literal elements instead when the records have no source ISA section to echo; that path keeps the default delimiters.

Limitations

• Charset. Element text is decoded through the source’s encoding option (UTF-8 by default, or ISO-8859-1). An unsupported or unconfigured-but-non-UTF-8 repertoire is rejected explicitly rather than silently corrupted (see Character set above).
• No escape character. X12 has no release mechanism, so a data value that contains a delimiter byte is rejected on output rather than corrupting the interchange.
• Consecutive grouping. Both GS..GE functional groups (group_ref) and ST..SE transaction sets (set_ref) close the moment their discriminator changes, so records sharing a value must arrive together; sort upstream when the source order does not already guarantee it.
• Output splitting. An interchange is a single ISA..IEA envelope and cannot be divided across files. An x12 output combined with a split: block is rejected at config-validation time (diagnostic E338) rather than emitting a structurally corrupt interchange.