EDIFACT Format

Clinker reads and writes UN/EDIFACT interchanges alongside CSV, JSON, XML, and fixed-width. An interchange is a finite file: it opens with an optional UNA service-string advice and a mandatory UNB header, wraps one or more UNH..UNT messages, and closes with a UNZ trailer. The reader streams one segment at a time and the writer reconstructs the envelope around emitted records. The reader decodes release-escape sequences into clean data values and the writer re-escapes them on output, so a reader → writer → reader round-trip preserves the data values and the envelope control references.

Delimiters and the UNA service string

Each segment is terminated by the segment terminator; within a segment, data elements split on the element separator and components on the component separator. A release character escapes a delimiter that occurs as literal data.

When the file begins with a 9-byte UNA prefix, its six service characters override the defaults in this fixed order: component, element, decimal, release, repetition, terminator. When UNA is absent, the syntax Level-A defaults apply:

UNA is optional — a parser that requires it would fail on the common no-UNA interchange, so Clinker assumes Level-A when it is absent.

Release character

The release character (default ?) marks the following byte as literal data rather than a delimiter: ?+ is a literal + inside an element, ?' is a literal apostrophe (not a terminator), and ?? is a literal ?. The reader decodes these sequences into clean data values, so a downstream CSV/JSON sink, a CXL string comparison, or a $doc field sees O'BRIEN, never the wire form O?'BRIEN. The writer re-escapes on output: any element value that carries the element separator, the segment terminator, or the release character is release-escaped automatically, so a value computed by a Transform or sourced from CSV — never EDIFACT-escaped to begin with — does not corrupt the interchange. A reader → writer → reader round-trip therefore preserves the data values exactly.

The component separator inside an element (e.g. the : in the composite UNOA:1) is kept as part of the element’s text and is not escaped — the positional element model works above component resolution, so a composite element round-trips unchanged. A literal colon in free-text data is the one ambiguity this introduces: because components are not split into separate fields, a : in a value re-reads as a component boundary. Repeating elements ride inside one element string intact and are likewise never truncated to their first repetition.

Newlines between segments

Some producers insert CR/LF after each segment terminator for readability. Those bytes are insignificant and are stripped between segments; CR/LF that appears inside an element is preserved.

Record shape

Each non-service segment becomes one record under a fixed positional schema:

Service segments (UNB, UNZ, UNH, UNT) are consumed by the reader to drive envelope state and validation — they are never emitted as body records. The UNH segment that opens a message is emitted as a body record (its seg_id is UNH), carrying the message reference in msg_ref and the message-type composite in msg_type, with its full positional element list also stamped onto e01, e02, … — so any UNH element past the message type (a common access reference, a message subset identification, and so on) is available as e03 onward and is reconstructed on write.

The number of eNN columns is controlled by the source max_elements option (default 32). A segment carrying more data elements than that is rejected with guidance rather than silently truncated. Absent trailing elements read as null.

nodes:
  - type: source
    name: orders
    config:
      name: orders
      type: edifact
      glob: ./inbox/*.edi
      options:
        max_elements: 48      # widen the positional schema for exotic segments
      schema:
        - { name: seg_id, type: string }
        - { name: msg_ref, type: string }
        - { name: e01, type: string }

Envelope sections over UNB

The interchange header UNB is extractable as a document envelope section, exposing its positional elements to CXL as $doc.<section>.<field>. Use the segment extract rule with the section field names matching the positional keys e01, e02, …:

envelope:
  sections:
    interchange:
      extract: { segment: "UNB" }
      fields:
        e05: string          # interchange control reference (UNB element 5)

A Transform can then read $doc.interchange.e05 on every body record.

Only the UNB header is extractable as an envelope section. Trailer segments (UNT, UNZ) arrive after the body and cannot become $doc fields without buffering the whole interchange — their control counts are instead validated inline by the reader (see below). A segment extract naming any tag other than UNB, or an xml_path / json_pointer extract against an EDIFACT source, is rejected at startup.

Control-count validation

The reader validates the structural integrity claims carried in the trailers as they arrive, failing the run on a mismatch (a truncation or corruption signal):

• UNT segment count — must equal the actual number of segments in the message, counting the UNH and UNT themselves.
• UNT message reference — must echo the opening UNH reference.
• UNZ message count — must equal the actual number of UNH messages in the interchange.
• UNZ control reference — must echo the UNB control reference.

Clinker locates the UNB control reference correctly even when the header carries an empty optional element or transmits the date/time of preparation as two separate parts rather than one combined element, so such interchanges still validate and round-trip with the trailer echoing the correct reference.

A missing UNZ at end of input is a truncation error; content after the UNZ trailer is rejected — including a lone stray release character with no following segment, which is treated as unterminated (truncated) content rather than silently dropped.

Routing a count mismatch to the DLQ

By default a UNT/UNZ count mismatch aborts the run. A source declaring dlq_granularity: document instead dead-letters the whole interchange / file to the DLQ — the file’s records become a structural_validation trigger plus document_rejected collaterals, and no record of the malformed file reaches the sink. The count is only known at the trailer, after the body has streamed, so the rejection lands at the sink boundary (no record is written out), not literally before the first record. The grain is the whole file. The control-reference echo mismatches and every other corruption (truncation, post-trailer content) always abort, even under the opt-in. See Malformed envelopes.

Writing EDIFACT

An EDIFACT Output node reconstructs the envelope around emitted records. Records map by the same positional columns (seg_id, msg_ref, msg_type, eNN); trailing null/empty elements are trimmed so no fabricated delimiters appear, and a column the writer does not recognize is an error (project the record to the EDIFACT columns first). Engine-internal $-namespaced columns are excluded automatically.

nodes:
  - type: output
    name: out
    input: messages
    config:
      name: out
      type: edifact
      path: ./out/result.edi
      options:
        interchange: ["UNOA:1", "SENDER", "RECEIVER", "240101:1200", "REF1"]
        message_type: "ORDERS:D:96A:UN"
        write_una: false
        segment_newline: true

Output options:

Consecutive records are grouped into UNH..UNT messages on msg_ref transitions. The writer recomputes the UNT segment count and UNZ message count, and echoes the message and interchange control references, so the output passes its own count validation on re-read.

interchange_from_doc echoes the header from a record’s document context. That context is populated by a source’s UNB envelope section (declare a segment: "UNB" envelope section on the source) and travels with every body record through the pipeline — including to a sink that sits directly downstream of the source with no intervening Transform. The reader stashes the complete, ordered UNB element list (empty middle elements included), so the reconstructed header is faithful even when a middle element is empty and the user declares only the fields they care about. Supply interchange literal elements instead when the records have no source UNB section to echo.

Character set

EDIFACT names its body character repertoire in-band: the UNB header’s syntax identifier (data element S001, component 1 — the UNOA in UNOA:1) declares the repertoire for the whole interchange. There is therefore no encoding option on an EDIFACT source or sink; the reader discovers the repertoire from the UNB and the writer re-derives it from the UNB it emits, so a read → write round-trip is byte-faithful without any configuration.

Decoding stays streaming and per-segment — the interchange is never buffered whole. The syntax identifier is itself ASCII, so it is read straight from the raw UNB bytes before any text is decoded; the UNB and every body segment are then decoded through the negotiated repertoire. The UNB’s own sender and recipient identification elements may legitimately carry non-ASCII text under UNOC or UNOY, and they decode (and re-encode on output) under that repertoire too — so a UNOC interchange whose header or body carries Latin-1 high bytes (for example accented characters in a party name) parses without error, surfaces the correct text in $doc.UNB.*, and round-trips byte-for-byte.

The repertoire is enforced loudly. A UNOA/UNOB interchange whose body carries a high byte fails (“outside the ASCII repertoire”) rather than silently reinterpreting it; a UNOY interchange with invalid UTF-8 fails (“not valid UTF-8”); and a UNB declaring an unsupported syntax level (UNOD..UNOX) fails at startup with a precise error naming the level, never falling back to a guessed encoding or substituting replacement characters. On output the writer encodes element text through the same repertoire the UNB declares; a character the repertoire cannot represent (a non-ASCII character under UNOA/UNOB, or a codepoint above U+00FF under UNOC) is rejected rather than emitted truncated.

Limitations

• Functional groups. A single UNB..UNZ interchange is supported; UNG/UNE functional-group segments are rejected with a precise error.
• Output splitting. An interchange is a single UNB..UNZ envelope and cannot be divided across files. An edifact output combined with a split: block is rejected at config-validation time (diagnostic E323) rather than emitting a structurally corrupt interchange.
• Rare degenerate headers. A few unusual header shapes — those that combine an empty date/time slot with a date-only date/time where the control reference normally sits — may not round-trip byte-for-byte on re-emit. Conformant headers and ordinary variations are unaffected.