CSV Format

CSV is the default file format and the most common Clinker input. The reader decodes each line into a record whose fields are matched positionally (or by header name) against the source’s declared schema:; the writer reverses the process. CSV pairs with the file transport — see Source Nodes for the transport / format split and the schema rules every source shares.

- type: source
  name: orders
  config:
    name: orders
    type: csv
    path: "./data/orders.csv"
    schema:
      - { name: order_id, type: int }
      - { name: customer_id, type: int }
      - { name: amount, type: float }
      - { name: order_date, type: date }
    options:
      delimiter: ","         # default ","
      quote_char: "\""       # default "\""
      has_header: true        # default true
      encoding: "utf-8"      # default "utf-8"

Options

All CSV options are optional. With no options: block, Clinker uses standard RFC 4180 defaults.

delimiter and quote_char are each a single byte on the wire, so each must be exactly one ASCII character. An empty, multi-character, or non-ASCII value (for example "||" or "→") is rejected at plan validation — it is never silently truncated to its first byte.

Encoding

The reader decodes every field through the source’s declared encoding:

• utf-8 (the default) is strict — a byte sequence that is not valid UTF-8 fails the run loudly rather than substituting replacement characters, so a mis-declared encoding is caught instead of silently corrupting data.
• iso-8859-1 (Latin-1; also spelled latin-1 or latin1) maps each byte 0xNN to codepoint U+00NN, so high bytes such as 0xE9 (é) from legacy exports decode correctly.

An unsupported encoding is rejected at startup with a precise error naming the value and the supported set — it is never silently ignored.

Multi-record CSV sources (those whose schema: is a map with a records: list, described below) are decoded as UTF-8 only. Declaring a non-UTF-8 encoding on such a source is rejected at startup; split the file into a single-schema CSV source if it needs a non-UTF-8 charset.

Header handling

With has_header: true, the header row’s names bind input columns to the schema: entries — column order in the file may differ from the schema. With has_header: false, binding is strictly positional, so the schema order must match the file’s column order.

Input columns the schema does not name are governed by the source’s on_unmapped policy, the same as every other format.

Multi-value cells (split_values)

A CSV cell holds one string, but that string may pack several values behind a delimiter (1,a;b;c). Declare the column multiple: true and add a split_values entry naming the field and its delimiter, and the reader parses the cell into an array:

- type: source
  name: orders
  config:
    name: orders
    type: csv
    path: ./orders.csv
    split_values:
      - { field: tags, delimiter: ";" }
    schema:
      - { name: order_id, type: string }
      - { name: tags, type: string, multiple: true }

tags reads as ["a", "b", "c"]. An empty cell is an empty array; a cell with no delimiter is a one-element array; each element is coerced to the column’s declared type:. A quoted cell is unquoted first, so a delimiter inside the quotes is not a boundary. A multiple: true column with no covering split_values entry is rejected at compile (E361). The entry is read only on a single-schema source, not the multi-record reader below. See split_values in the Source reference for the full grammar.

Writing CSV

On output, the writer emits one row per record with cells in the output schema’s column order — the same order as the header row — regardless of how an upstream node ordered the record’s fields. An output-schema column the record does not carry emits an empty cell, the same as an explicit null; with include_unmapped: false a record field the output schema does not name is not written. See Output Nodes for header control, field mapping, and null handling.

Writing multi-value cells (join_values)

A multiple: field is joined into one delimited cell on write — the write-side inverse of split_values. The default needs no configuration: values join with ;, and a value that itself contains the delimiter is a hard error rather than a cell that would split back wrongly.

- type: output
  name: report
  input: orders
  config:
    name: report
    type: csv
    path: ./out/report.csv
    join_values:
      - tags                              # delimiter ";", on_conflict: error
      - { field: notes, delimiter: "|", on_conflict: escape, escape: "\\" }

A field with no join_values entry still joins, with the defaults. An entry overrides, per field:

• delimiter — the separator written between values (default ;).
• on_conflict — what to do when a value contains the delimiter:
  • error (default) — dead-letter the record, naming the field and the offending value, rather than emit a cell that splits back wrongly. This is what makes a defaulted delimiter safe. Under error_handling.strategy: continue, the offending record goes to the dead-letter queue (category multi_value_join_collision) and the run continues; under fail_fast it aborts.
  • escape — prefix each delimiter (and each escape character) inside a value with escape (default \), so a matching split_values escape: recovers the original. Lossless. delimiter and escape must each be a single character.
  • encode_json — encode the whole field as an embedded JSON array, recovered by a matching split_values json: true. Preserves every value’s text exactly, including ones carrying the delimiter, quotes, or newlines — nothing is lost or mis-split. (A decimal/date/datetime element serializes as its JSON string form and reads back as a string, re-typed by the column’s declared type:, the same round trip every CSV cell takes.)

An empty field emits an empty cell — and, under the delimited policies (error, escape), a single empty-string value [""] emits an empty cell too, which reads back as zero values: the delimited encoding cannot tell an empty field from one empty value. Use encode_json when that distinction matters. A single non-empty value emits that value with no delimiter. The joined cell is quoted by the normal CSV rules when it contains the field delimiter, a quote, or a newline. Declaring join_values on a non-CSV output is rejected at compile (E362).

Round trip. on_conflict: escape and encode_json are recovered exactly by a matching source split_values entry:

# write side
join_values:
  - { field: tags, on_conflict: escape, escape: "\\" }
# read side (a later pipeline)
split_values:
  - { field: tags, escape: "\\" }

Header widening under auto-widen

When auto_widen is in effect and the Output leaves include_unmapped at its default of true, different records can carry different carried-along columns. The header must still be shared by every row, so Clinker widens it to the union of every record’s columns in first-seen order: a column that first appears on a later record still gets its own header slot, and the earlier rows write an empty cell for it. This pre-scan runs on the buffered output path, where the record batch is materialized.

An output that streams under a bounded-memory budget cannot pre-scan the whole batch: a CSV output fused directly after a Merge/Transform, a single-branch Route, a streaming-strategy Aggregate, or the probe side of a hash-build-probe Combine, or one reconstructing an envelope (which suppresses the shared header and streams a headerless body), commits its columns to the first record. A later record carrying a column that first record lacked then fails the run with a SchemaDrift error naming the column, rather than silently dropping it. Declare the column in the source or output schema: so every record carries it, or route to a self-describing format (JSON / NDJSON / XML). A reconstruct_envelope CSV output therefore requires a stable body shape — every record must carry the same columns.

Multi-record files (header / trailer / body)

Some CSV exports interleave multiple record types in one file — a header row, many body rows, and a trailer row — each distinguished by a discriminator column. Declare these with a map-form schema: carrying a discriminator: and a records: list, instead of the single column-list schema:. Each record type names its tag (the discriminator value that identifies it) and its own columns:; the discriminator field must sit at the same column in every type (usually the first). The reader derives the runtime superset schema (a lead record_type column plus the union of every record type’s columns) automatically.

- type: source
  name: payments
  config:
    name: payments
    type: csv
    path: "./data/payments.csv"
    schema:                                     # one multi-record schema (map form)
      discriminator: { field: rec_type }        # the physical column carrying the type tag
      records:
        - { id: header,  tag: H, columns: [ { name: rec_type, type: string }, { name: batch_id, type: string } ] }
        - { id: detail,  tag: D, columns: [ { name: rec_type, type: string }, { name: id, type: int }, { name: amount, type: int } ] }
        - { id: trailer, tag: T, columns: [ { name: rec_type, type: string }, { name: count, type: int } ] }
      structure:
        - { record: trailer, count: count }     # validate T's count against the body count
    envelope:
      sections:
        head:
          extract: { record_type: H }          # the H record type surfaces as $doc.head.*
          fields:
            batch_id: string

The reader streams one record per line on a single superset schema whose lead record_type column carries the matched type’s id. A downstream Route discriminates on that column. Rows of different record types may carry different column counts (ragged rows) — the reader validates the column count per record type, not file-wide. A textual column-header row is skipped when has_header is true (the default), so a leading record_type,name,amount line is not mistaken for a record of an unknown type. Each declared field honors its own type / trim / pad, the same as a single-record CSV field.

• Header rows declared as an envelope: section via the record_type extract surface as $doc.<section>.* and are excluded from the body stream (see Envelopes & Document Context).
• Trailer rows named by a structure: constraint are validated as they stream — the declared count field is checked against the actual body-record count at document close — and excluded from the body stream. A declared trailer that never appears is an incomplete-document error; a body row after the trailer is rejected as content past the document close.
• Blank lines (empty or whitespace-only, common after concatenation) are skipped rather than parsed.
• An unknown discriminator value (a tag no records: entry declares) is a structural-integrity failure, classified separately from a trailer count mismatch: it aborts the run by default, or under dlq_granularity: document condemns the whole file to the dead-letter sink and the run continues.