Nodes
A Clinker pipeline is a single flat nodes: list. Every entry carries a
type: discriminator that selects a node kind — the unified node
taxonomy. There is no separate “join section” or “filter section”:
records flow through one homogeneous graph of typed nodes, wired together
by input: / inputs:.
This part documents the nine record-processing node kinds; a tenth,
Composition, is a call-site that inlines a
reusable sub-pipeline and is covered under Pipelines.
The pages in this part are ordered the way data flows through a DAG — a record enters at a Source, fans through the record-level and combining nodes, and leaves at an Output:
| Node | Role | Arity | Streaming vs blocking |
|---|---|---|---|
| Source | Reads records from a file or network cursor; the entry point. | 0 → 1 | Streaming |
| Transform | Record-level CXL projection, filter, and lookup. | 1 → 1 | Streaming |
| Route | Predicate-based fan-out into named branches. | 1 → N | Streaming |
| Merge | Streamwise concatenation of inputs that share a schema. | N → 1 | Streaming |
| Combine | N-ary record combining with mixed predicates (equi + range + arbitrary CXL). | N → 1 | Blocking (build side) |
| Aggregate | Grouped or windowed reduction. | 1 → 1 | Blocking (or streaming when sorted) |
| Reshape | Pivot / unpivot between wide and long record shapes. | 1 → 1 | Blocking |
| Cull | Per-correlation-group removal on a group-level predicate, with a removed_to side-output port. | 1 → 2 | Blocking |
| Envelope | Frames a body stream into per-document documents; a composable framing stage. | 1 → 1 | Streaming |
| Output | Writes records to a sink; the exit point. | 1 → 0 | Streaming |
Streaming vs blocking
Stateless nodes (Transform, Route, Merge, the Combine probe side, Output) evaluate records one at a time without accumulating per-record state. Blocking nodes (Aggregate, sort, the grace-hash Combine build side) accumulate state inside the RSS budget and spill to disk rather than OOM the process. The Streaming vs. Blocking Stages page in the Operations Guide is the full memory model.
Wiring and naming
Every node needs a unique name: (no dots — the dot is reserved for port
syntax). Single-input nodes use input:; Merge and Combine use inputs:.
Route branches are consumed downstream as route_name.port. The
Pipeline YAML Structure page covers the full
wiring grammar, optional fields (description:, _notes:), and strict
parsing rules.