Some stages pass records along as they arrive; others have to collect their whole input first. Which one a stage is decides how much memory it needs. Pick a pipeline shape and flip its settings, and every stage shows its class, why, and the --explain lines you would see.
Only two shapes: a Source → Transform → Sink chain, and an interleave Merge without an interleave_seed whose inputs are all Sources. They pull records off the reader and pass each batch on, so their memory stays at one batch (pipeline.batch_size, default 2048) however big the input is.
A Route with one wired branch, any Merge, a streaming Aggregate, a hash Combine, or a range Combine's output. Each hands its output straight to one consumer instead of copying it into a buffer, but still builds its own result first.
The output goes into a buffer between stages. The buffer counts against memory.limit and spills to disk under pressure, so materialized is not unbounded: it is held, not lost.
streaming Aggregate, or a hash or range Combine)? And does its output go to exactly one consumer that can take one (a Sink, the input side of an Aggregate that isn't time-windowed, or the driver side of a hash Combine)? A no to either, two consumers, or a window all mean a buffer. In Source → Transform → Transform → Sink, only the first Transform streams.Separately, some stages block by nature: a hash Aggregate needs every group member before it can emit, a sort needs all its input, and a Combine builds its whole lookup side first. They hold that state within the memory budget and spill when it gets tight. A blocking stage can still have a streamed input, as the Aggregate in “Source → Transform → Aggregate” shows.
A correlation_key on any Source makes the engine hold rows until each group's outcome is known, so no stage streams to a consumer anywhere in the pipeline. Sinks still show streaming in --explain, although the correlation commit writes their rows (tracked in #1318).
dlq_granularity: document on any Source does the same, and every Sink becomes materialized: it keeps each open file's records until the file's verdict. A Source read live by a single Transform or by an unseeded interleave Merge still hands its records straight over.
A Sink with sort_order, split, or a per-source-file path takes no stream; the stage before it keeps a buffer.
A Transform with an analytic_window needs its whole input to build the window. A stage feeding two consumers, like a Transform read by two Sinks or a Route with two branches, gives each its own buffer.
pipeline.batch_size sets how many records cross a stream at a time. It caps memory only for the one-batch shapes; for the others it sets the slice size. It never changes the output.