One bad line,
the whole order goes

An orders file has one row per order line. Order A has three lines, B has two and C has one, and the last two lines have no order id. The pipeline reads the file and checks each line's quantity; a quantity that isn't a whole number fails the check.

- type: source
  name: orders
  config:
    correlation_key: order_id
    …
- type: transform
  name: validate
  input: orders
  config:
    cxl: |
      emit q = qty.to_int()   # "BAD" fails here

With correlation_key: order_id, a failing line takes the rest of its order with it:

• The failing line is written to the DLQ as a trigger, with the real error category.
• The order's other lines are written as correlated. They did nothing wrong; they were condemned by their order.
• Other orders are not affected.
• A line with an empty key is a group of one. It never takes another line with it.

The group is fixed when the row is read

The engine copies the key value into a hidden column as soon as the Source reads the row. Later steps can rename, rewrite or anonymise order_id and the row stays in its original group.

The hidden columns never appear in your output. To see them while debugging, set include_correlation_keys: true on a Sink. They show up as $ck.order_id.

Rows arrive sorted by the key

To keep each group's rows together, the engine sorts the Source's rows by the key (empty keys first), keeping file order within a key. If the Source already declares a sort_order that starts with the key, nothing is re-sorted.

You can see this in section 01: with the key on, the output lists the empty-key line first, then B, then C. With the key off, it follows the file.

An Aggregate turns many lines into one total, so “turn the whole group away” needs a rule. The engine looks at your group_by. You never choose this setting yourself:

group_by includes the key → strict

Each total belongs to exactly one order. If any line of the order fails, the order's total is dead-lettered as correlated, even though the engine did compute it from the clean lines.

group_by leaves the key out → retraction

Now one order spreads across several totals: order A has lines in both ENG and HR. Throwing away every total an order touched would void whole departments over one bad line. Instead the engine takes out only the failing line and recomputes the totals from the lines that remain, so the clean lines of a failed order still count.

A check after the Aggregate can also fail a total. Under retraction, that total is dead-lettered as a trigger and drops out of the output. The lines that fed it are not dead-lettered, and their orders are untouched. One exception today: those lines are also taken out of any other relaxed Aggregate on the same input, so a sibling total that did not fail loses them too (#1391).