The envelope
around your records

The Source's envelope: block lists the sections to pull out of each file, where each one is, and which of its fields to keep, with their types.

- type: source
  name: payments
  config:
    name: payments
    type: json
    glob: ./payments-*.json
    options:
      record_path: records
    envelope:
      sections:
        :
          extract: { json_pointer: "/BatchInfo" }
          fields:
            batch_id: string
            run_date: date
            operator: string
        :
          extract: { json_pointer: "/Summary" }
          fields:
            total: int
    schema:
      - { name: id, type: string }
      - { name: region, type: string }
      - { name: amount, type: int }

• The names are yours. The engine reserves none. Rename a section on the right and watch the CXL change with it.
• The trailer is readable from the first record. JSON and XML sources read the declared sections before any record is streamed, so a total at the bottom of the file is already known on row one.
• Typos fail early. Reading a section or field the envelope: block doesn't declare is rejected when the pipeline is checked (E341), instead of quietly giving null.
• Missing data is null. A declared field or section that a particular file doesn't contain reads as null.

When a Source reads several files (with glob: or paths:), each file is its own document with its own section values. A record's $doc.* always comes from the file it was read from.

Document boundaries travel with the records. A grouped or global Aggregate gives one set of results per document: it finishes and emits a document's groups when that document ends, then starts afresh for the next. Twelve monthly files through one Aggregate give twelve monthly roll-ups, not one yearly total.

- type: aggregate
  name: by_region
  input: payments
  config:
    group_by: [region]
    cxl: |
      emit region = region
      emit total  = sum(amount)

The reference page covers how an Aggregate treats documents arriving through a Merge of several Sources, or after a Combine.

By default only the failing record is dead-lettered. When a half-processed file is worse than none, for example an interchange or a batch with a control total, set dlq_granularity: document on the Source to reject the whole file instead.

- type: source
  name: payments
  config:
    …
    dlq_granularity: 
error_handling:
  strategy: continue      # required

• The first failing record is the root cause, with its real error category.
• Every other record of that file that reached a Sink, or failed too, is written as document_rejected, pointing back at the root cause through _cxl_dlq_trigger_id.
• No record read from a rejected file is written by any Sink. Other files carry on untouched.
• The document is the whole file: for X12 or EDIFACT, the whole interchange, not one transaction set.