Every run ends with one line: how many rows were read, how many made it out, how many copies were written, how many went to the dead-letter file. The numbers often do not add up, and that is not a bug. Pick a pipeline below, see what happened to every row, and see which number each one landed in, or why it is in none.
Here 45 ok + 1 dlq = 46, so 4 rows are in neither. Nothing on this line says where they went. Section 02 shows the usual answers: a filter, a distinct, an Aggregate folding rows together, a Combine that skipped a row, or a Route branch nothing reads.
The line is a log message, so it appears on the terminal by default (log level info). When --machine or another option uses standard output for data, log lines move to standard error. A run whose output could not be published prints an error instead of this line.
Clinker counts rows a filter rejects and rows a distinct drops, but shows neither count on the summary line or in the metrics file (#1352).
To check: preview a sample with --dry-run -n 100, which reads the first 100 rows of each Source and prints the rows that reach each output, and compare them with those input rows.
A branch with no node reading it is allowed. Its rows are dropped and counted nowhere. In an exclusive Route a row stops at the first branch whose condition is true, even when nothing reads that branch. A Cull's removed_to port that nothing reads drops its rows the same way (#1351).
--explain marks an unread Route branch (unconsumed); it does not mark an unread removed_to port. Wire either one to a Sink to keep those rows.
With on_miss: skip, a driver row with no match is dropped and counted nowhere (#1352).
Use on_miss: null_fields to keep it with empty lookup fields, or error to stop the run.
Rows of the other inputs (the lookup side) are read, so they are in total, but they never count as ok themselves. The output row counts as its driver row (#1352).
The metrics file's per_source_record_counts shows how many rows each Source read.
An Aggregate turns many rows into one per group. ok counts one row for each group written, not every row that went into it (#1352).
Compare total with the number of output rows, not with ok.
A Sink sort field with null_order: drop leaves those rows out. This one is reported: an extra line, N record(s) excluded by null_order: drop, and records_null_dropped in the metrics file.
Use null_order: last (the default) to keep them.
Whether every row read should end up in exactly one reported count, so that the summary line always adds up, is an open design question (#1352). Until it is settled, the numbers behave as this page shows.
A Combine's own filter and distinct: rows they reject inside a Combine's cxl: body are counted only when the Combine runs as an in-memory hash join, and not under the other join strategies (#1347). Since filtered and duplicate counts are not shown anyway, this changes nothing you can see today.
| Number | Summary line | Metrics file | CXL during the run |
|---|---|---|---|
| Rows read, from every Source | N total | records_total, and per_source_record_counts by Source | $pipeline.total_count reads 0 today |
| Rows that reached an output | N ok | records_ok | $pipeline.ok_count reads 0 today |
| Copies written to all Sinks | N written | records_written | not available |
| Dead-letter entries | N dlq, and the exit code is 2 | records_dlq, and per_source_dlq_counts by Source | $pipeline.dlq_count reads 0 today |
Dropped by null_order: drop | its own line, only when above 0 | records_null_dropped | not available |
Rejected by filter | not shown | not shown | $pipeline.filtered_count reads 0 today |
Dropped by distinct | not shown | not shown | $pipeline.distinct_count reads 0 today |
The metrics file is written only when metrics are enabled (--metrics-spool-dir, CLINKER_METRICS_SPOOL_DIR, or pipeline.metrics.spool_dir); see Metrics & Monitoring. The $pipeline.*_count members exist in CXL but are never updated during a run, so they always read 0 (#1346). A preview run (--dry-run -n) reads only part of each Source and prints no summary line.