Output Nodes

Output nodes write processed records to files. They are the terminal nodes of a pipeline – every pipeline path must end at an output (or records are silently dropped).

Basic structure

- type: output
  name: result
  input: transform_node
  config:
    name: output_stage
    type: csv
    path: "./output/result.csv"

The type: field selects the output format: csv, json, xml, fixed_width, edifact, x12, hl7, or swift. The edifact, x12, and swift writers reconstruct one interchange/message envelope around emitted records; the hl7 writer re-emits HL7 v2 segments and optionally wraps them in batch/file envelopes. See EDIFACT Format, X12 Format, HL7 v2 Format, and SWIFT MT Format.

Structured single-writer outputs (edifact, x12, hl7, and swift) accept one concrete document grain per output file. A multi-file source or multi-input merge feeding one of these outputs is rejected instead of being silently written as one merged envelope. To write multiple structured documents, consolidate them deliberately with an Envelope node first or route each document to a separate output path.

Direct broadcast to several outputs

Several Output nodes may name the same input. This is a broadcast: every Output receives every upstream record, regardless of node declaration order. The run report counts one write per sink, so five input records feeding a CSV and a JSON Output produce records_written: 10.

Use a Route node when outputs should receive different subsets. Writing a field such as _route does not select a destination; it is an ordinary output column unless a Route condition explicitly reads it.

Field control

Output nodes can either pass every upstream field through to the writer or restrict output to the fields the upstream transform explicitly emitted. Several options control which fields appear and how they are named.

Unmapped input field passthrough

    include_unmapped: false    # Default: true

When true (the default), every field on an input record that the upstream transform did not explicitly emit still passes through to the output unchanged. This includes fields the source’s on_unmapped: auto_widen policy absorbed into the per-record $widened sidecar map – their contents expand back to top-level columns at the sink.

When false, only fields named by an emit statement in the upstream transform appear in the output. The $widened sidecar slot is stripped and undeclared input fields are dropped.

When true, how a carried-along column reaches the writer depends on the output format. Self-describing formats (JSON / NDJSON / XML) write each record’s own keys. A CSV output widens its header to the union of every record’s columns when it can materialize the batch, and otherwise — on a bounded-memory streaming path (a Merge, a fused Transform, a single-branch Route, a streaming-strategy Aggregate, or the probe side of a hash-build-probe Combine feeding the output), or an envelope-reconstructing path — fails loudly with a SchemaDrift error rather than dropping a column it cannot fit under its already-committed header. A fixed-width output has no room for an undeclared column and likewise raises SchemaDrift. See Auto-Widen & Schema Drift → Schema drift across records.

Migration notice

The default flipped from false to true in a recent release (see issue #90). Pipelines that relied on the previous behavior – where output records contained only the fields explicitly emitted upstream – must now set include_unmapped: false explicitly to restore that shape.

The flag composes independently with include_correlation_keys: true – see below. See Auto-Widen & Schema Drift -> Output controls for the full specification and cross-format flow examples.

Worked example

Suppose the upstream source emits records with order_id, customer_id, amount, and region, and a transform that emits only one derived field:

- type: transform
  name: classify
  input: orders
  config:
    cxl: |
      emit amount_bucket = if amount >= 1000 then "high" else "low"

With include_unmapped: true (the default), each output record carries order_id, customer_id, amount, region, and amount_bucket. With include_unmapped: false, each output record carries only amount_bucket. The transform’s CXL is unchanged in both cases – the Output node decides the field set.

Include correlation-key shadow columns

    include_correlation_keys: true    # Default: false

When a source declares a correlation_key:, the engine tracks correlation-group identity on hidden columns that are stripped from output by default. Set include_correlation_keys: true to surface them in the writer output — typically for debugging correlation-group routing or auditing DLQ behavior. See Correlation Keys.

include_correlation_keys does not surface auto-widened columns – include_unmapped is the separate flag for that. The two are independent: each, both, or neither can be set.

Nested columns and non-JSON writers

The CSV, XML, fixed-width, EDIFACT, X12, and HL7 writers can only write flat scalar columns; a nested value reaching one of them fails with an UnserializableMapValue error. JSON writes nested values natively. The usual cause is an auto-widened column reaching a non-JSON writer with include_unmapped: false — see Auto-Widen & Schema Drift for the fix.

Field mapping

mapping: declares the columns the file carries – which columns, under what names, in what order – without changing upstream CXL. It is a sequence, one item per output column:

    mapping:
      - order_id                  # carried through under its own name
      - sold_to: customer_id      # written as `sold_to`, read from `customer_id`
      - contact_email: customer_email
      - channel
      - sku

Two item shapes:

• A bare column name emits that column unchanged. This is the common case, and it costs one line naming the column once.
• A single-key pair renames. The output name is on the left, the source column on the right – the same side the bare form names. Reading an item left to right always tells you what appears in the file first.

The renames are the only items carrying a colon, so in a wide output they are found by scanning for structure rather than by comparing two names per line.

Order and selection

Declaration order is the output column order. Listed columns are written first, in the order the block declares them, whatever order they arrive in.

include_unmapped governs everything the block does not list. With include_unmapped: true (the default) unlisted columns are appended after the declared ones, in their existing relative order. With include_unmapped: false they are dropped, so the block becomes the complete statement of the output:

    include_unmapped: false
    mapping:
      - department
      - surname: last_name
      - first_name

Given upstream columns first_name, last_name, department, that writes exactly department,surname,first_name.

Every record carries every declared column. When a record does not supply an item’s source column, that column is still written, empty. The file’s shape follows the block, not the data — so a stream whose records differ in shape (a multi-record-type source, a column arriving through auto_widen, a composition body’s open row) still produces one stable column set in declaration order, rather than one that depends on which record happened to arrive first.

One upstream column may feed two output columns – - sku and - item_code: sku – because names must be unique on the output side, not the source side. Declaring the same output name twice is rejected (E364): a file cannot carry two columns under one header.

For the same reason, an output name that include_unmapped: true would also carry through is rejected. If upstream already has a sold_to column, writing - sold_to: customer_id under include_unmapped: true would put two sold_to columns in the file and readers would resolve the wrong one. Rename the mapped column, exclude the upstream one, or set include_unmapped: false.

Where the compiler cannot enumerate the upstream columns, the same collision reaches the run. The mapped value wins – the block is your explicit statement of what the file carries – and the displaced upstream column is named in a W366 warning at the end of the run. Applying one of the three fixes above silences it.

Diagnostics

A mapping: item naming a column that does not exist at that point in the pipeline is rejected at compile time (E365), with the available column list and a did you mean when the name is a near miss. Nothing is renamed silently.

The compiler cannot always see the column set. Inside a composition body the rows are open by construction, and under on_unmapped: auto_widen a column can reach the sink through the sidecar without being declared anywhere. There an item naming an unknown column compiles even when its name resembles a declared column: spelling similarity cannot prove that a dynamic field is absent. W365 reports it after the run if no written record supplied it.

What catches the rest is the end of the run: if no record supplied an item’s source column, that item wrote an empty column in every row, and the run reports it as W365, naming the column to correct. An item some records supply and others do not is a sparse column, not a mistake, and is not reported.

Both W365 and W366 are advisory. They print to standard error when the run finishes and do not change the exit code – the file is written and readable either way, and by the time a stream ends the run’s other outputs have already been flushed.

A column absent from the source’s schema: reaches the sink only through the auto_widen sidecar, which is expanded to top-level columns only under include_unmapped: true. A mapping: item may name such a column when that flag is set; under include_unmapped: false it cannot resolve and is rejected at compile time.

An empty block – mapping: {} or mapping: [] – is rejected (E364): it declares an output with no columns. To write every upstream column, remove the mapping: key rather than emptying it.

Writing the block as a YAML map instead of a sequence is rejected (E364); the message prints your own block already rewritten. Run clinker explain --code E364 for the migration, and read the direction note there before pasting: releases before this one documented output_name: source_field but executed the reverse, so the rewrite swaps each pair’s two sides to preserve what the pipeline was actually writing.

Excluding fields

Remove specific fields from output:

    exclude: [internal_id, _debug_flag, temp_calc]

exclude: matches incoming column names, and runs before mapping:. Two consequences:

• The columns that survive keep their relative order. Upstream a, b, c, d with exclude: [b] writes a, c, d.
• Naming a column that a mapping: item also produces is not a conflict – the exclusion removes the upstream column of that name and leaves the mapped one standing. That is the fix for the two-columns-under-one-header collision above: - sold_to: customer_id with exclude: [sold_to] writes one sold_to column, carrying customer_id’s value.

Excluding a column a mapping: item reads is a different matter, and is rejected (E364): the exclusion removes the column before the item can read it, so the item could never resolve.

Header control (CSV)

    include_header: true      # Default: true

Set to false to omit the CSV header row.

Null handling

    preserve_nulls: false     # Default: false

When false, null values are written as empty strings. When true, nulls are preserved in the output format’s native null representation (e.g., null in JSON).

Rounding decimals to a declared scale

An Output node’s optional schema: may declare a column type: decimal with a scale. A decimal value landing in that column is rounded to the declared number of fractional places on write, using banker’s rounding — the same boundary contract a decimal source column applies on read.

    schema:
      - { name: dept,    type: string }
      - { name: total,   type: decimal, scale: 2 }
      - { name: average, type: decimal, scale: 2 }

Decimals compute at full precision inside the pipeline (division and avg keep every digit), so a declared output scale is how you pin a computed result to fixed places at the sink: avg(amount) over 1.00, 1.00, 2.00 writes 1.33 into a scale: 2 column, while sum(amount) — already at scale 2 — stays 4.00. This works for every format (CSV, JSON, fixed-width); an output column with no declared scale, or an output with no schema: block at all, keeps the full-precision value. Only decimal values in decimal-declared columns are affected — no other type is coerced. See Decimal — arithmetic rules for the full boundary-contract model.

The same rounding applies to an Output node declared inside a composition body. When its schema: names an external .schema.yaml file, the path resolves relative to the composition file’s own directory (not the invoking pipeline’s).

Output format options

CSV

- type: output
  name: csv_out
  input: processed
  config:
    name: csv_out
    type: csv
    path: "./output/result.csv"
    options:
      delimiter: "|"

delimiter is a single byte on the wire, so it must be exactly one ASCII character (for example ,, |, or \t). An empty, multi-character, or non-ASCII value is rejected at plan validation rather than silently truncated to its first byte.

JSON

- type: output
  name: json_out
  input: processed
  config:
    name: json_out
    type: json
    path: "./output/result.json"
    options:
      format: ndjson           # array | ndjson
      pretty: true             # Pretty-print JSON

• array (default) – writes a single JSON array containing all records.
• ndjson – writes one JSON object per line.

JSON numbers cannot represent non-finite floats; a record carrying NaN or an infinity fails the write with a JSON error instead of silently becoming null. See JSON Format.

XML

- type: output
  name: xml_out
  input: processed
  config:
    name: xml_out
    type: xml
    path: "./output/result.xml"
    options:
      root_element: "data"
      record_element: "row"
      attribute_prefix: "@"    # emit @-prefixed fields as XML attributes

Fields whose final path segment carries the attribute_prefix (default @, matching the XML source option) are emitted as XML attributes of their enclosing element, so attribute fields read from an XML source round-trip. See XML Format for details.

Fixed-width

- type: output
  name: fw_out
  input: processed
  config:
    name: fw_out
    type: fixed_width
    path: "./output/result.dat"
    schema: "./schemas/output.schema.yaml"
    options:
      line_separator: crlf

Fixed-width output requires a format schema defining field positions and widths. Fields land at their declared byte ranges with gaps space-filled — see Fixed-Width Format for the layout semantics.

EDIFACT

- type: output
  name: edi_out
  input: messages
  config:
    name: edi_out
    type: edifact
    path: "./out/result.edi"
    options:
      interchange: ["UNOA:1", "SENDER", "RECEIVER", "240101:1200", "REF1"]
      message_type: "ORDERS:D:96A:UN"
      write_una: false
      segment_newline: true

The EDIFACT writer reconstructs the interchange envelope around emitted records, recomputing the UNT/UNZ control counts and echoing the control references, and release-escapes any element data that carries a service character. The UNB header comes from interchange (literal elements) or interchange_from_doc (echoed from a $doc section). An interchange is a single envelope, so an edifact output cannot be combined with a split: block — the combination is rejected at config-validation time (E323). See EDIFACT Format for the full option reference, the record schema, and the round-trip semantics.

HL7 v2

- type: output
  name: hl7_out
  input: messages
  config:
    name: hl7_out
    type: hl7
    path: "./out/result.hl7"
    options:
      file_header: ["^~\\&", "LAB", "HOSP", "EHR", "HOSP", "20240102", "FILE7"]
      batch_header: ["^~\\&", "LAB", "HOSP", "EHR", "HOSP", "20240102", "BATCH3"]
      segment_newline: true

The HL7 writer re-emits the MSH and body segments from the record stream, escaping any field data that carries a delimiter character (| → \F\, ^ → \S\, and so on). When a file_header (or file_header_from_doc) or batch_header is configured the writer wraps the messages in an FHS..FTS file or BHS..BTS batch and recomputes the closing BTS/FTS counts. A batch/file envelope is a single structure, so an hl7 output cannot be combined with a split: block — the combination is rejected at config-validation time (E339). See HL7 v2 Format for the full option reference, the record schema, the MSH off-by-one, and the round-trip semantics.

Sort order

Sort records before writing:

    sort_order:
      - { field: "name", order: asc }
      - { field: "amount", order: desc, null_order: last }

• first – nulls sort before all non-null values.
• last – nulls sort after all non-null values.
• drop – records with null sort keys are excluded from output.

Shorthand: a bare string defaults to ascending with nulls last:

    sort_order:
      - "name"
      - { field: "amount", order: desc }

File splitting

Split output into multiple files based on record count, byte size, or group boundaries:

- type: output
  name: split_output
  input: processed
  config:
    name: split_output
    type: csv
    path: "./output/result.csv"
    split:
      max_records: 10000
      max_bytes: 10485760           # 10 MB
      group_key: "department"       # Never split mid-group
      naming: "{stem}_{seq:04}.{ext}"
      repeat_header: true           # Repeat CSV header in each file
      oversize_group: warn          # warn | error | allow

Split configuration fields

At least one of max_records or max_bytes should be specified for splitting to have any effect.

For formats whose output wraps the whole file in framing – a JSON array or an XML root element – each split file is a complete, independently valid document: the framing is closed at rotation and reopened for the next file.

Oversize group policies

• warn (default) – log a warning and allow the oversized file.
• error – stop the pipeline.
• allow – silently allow the oversized file.

When group_key is set, the split point is the first group boundary after the threshold is reached (greedy). Without group_key, files are split at the exact limit.

Streaming writes after an interleave Merge

When a single Output sits directly after a Merge with mode: interleave whose inputs are all Sources, records are written to disk as they arrive rather than being buffered until the merge finishes. This keeps memory flat and lets a slow writer naturally pace the upstream readers.

- type: source
  name: src_a
  config: { type: csv, path: a.csv, schema: ... }
- type: source
  name: src_b
  config: { type: csv, path: b.csv, schema: ... }
- type: merge
  name: merged
  inputs: [src_a, src_b]
  config:
    mode: interleave        # required
- type: output
  name: out
  input: merged
  config:
    name: out
    type: csv
    path: out.csv

This is automatic — there is no setting to enable it. It applies only to this exact shape: one interleave Merge of Sources feeding one non-splitting Output, in a pipeline without correlation keys. Any other topology buffers as usual, with identical output either way.

Complete example

- type: output
  name: department_reports
  input: enriched_employees
  config:
    name: department_reports
    type: csv
    path: "./output/employees.csv"
    # `include_unmapped: false` makes the mapping the whole output: these four
    # columns, in this order, and nothing else. Without it every unlisted
    # upstream column would still be appended after them, and an `exclude:`
    # would be needed to keep any of them out.
    include_unmapped: false
    mapping:
      - "Employee ID": employee_id
      - "Full Name": display_name
      - department
      - "Annual Salary": salary
    include_header: true
    sort_order:
      - { field: "department", order: asc }
      - { field: "display_name", order: asc }
    split:
      max_records: 5000
      group_key: "department"
      naming: "employees_{seq:03}.csv"
      repeat_header: true