How it works
NAME
internals - what happens between a byte arriving and an exit code leaving
PIPELINE
Stage 3 is the only interesting one. For each record it walks the object and emits a flat set of path, type pairs: order.items[].sku and string, and so on. Array elements are merged into a single element shape as they are walked, so an array of a million objects costs the same as an array of two.
Stage 4 folds those sets into one. Folding two types for the same path produces either the same type, a widened type, or a union. Nothing else is possible, which is what keeps the whole program bounded and boring.
TYPE LATTICE
Seven types, one widening edge, no others:
null bool int → float string object array
int widens to float because a producer that starts emitting 1.5 where it used to emit 1 has not broken any consumer that was parsing numbers. Every other change of type produces a union, and a union is a narrowing event: string becoming string|int means half your consumers now have a branch they did not have.
Classification by syntax, not by value: 1 is int, 1.0 and 1e3 are float. The decoder does not convert numbers to a machine type at all, so integers beyond the usual 53-bit limit keep their precision and still classify as int. The consequence for round numbers written in exponent form is listed under limitations.
null is tracked as a member of the union like any other type. That is why nullability is its own class rather than a flag: string to string|null and string to string|int are both unions, but one of them breaks consumers in a very specific and very common way, and it deserves its own line in the report.
CLASSIFICATION
| Symbol | Class | Condition | Default |
|---|---|---|---|
| + | new_fields | path observed, not in lock | warn |
| - | removed_fields | path in lock, not observed | fail |
| ~ | type_widening | int in lock, float observed | warn |
| ! | type_narrowing | any other type replacement or union | fail |
| ! | nullability | null joins a union that lacked it | fail |
| ~ | depth_exceeded | a subtree was not expanded | warn |
A single path produces at most one difference. If a field is both widened and made nullable, it is reported once, as nullability, because that is the more severe of the two and reporting both would double-count in the summary.
DETERMINISM
- Paths are sorted bytewise before the comparison, so report order never depends on map iteration.
- Unions are sorted by a fixed type order, not alphabetically, so
string|nullis always written that way round. - JSON output has sorted keys and a trailing newline, and diffs cleanly between two runs.
- Given the same input bytes, the same lock and the same version, the output is byte-identical. This is checked in the test suite by running every fixture twice and comparing.
- Nothing in the report depends on the clock, the hostname, the locale or the working directory. The only timestamp driftpipe writes is the one in a new lock.
COST
State is proportional to the number of distinct paths, not to the number of records. One path costs about 300 bytes including its type union and its name, so ten thousand distinct paths is roughly 3 MB of live heap, and a stream of fifty million records costs the same as a stream of fifty.
# 48213 records, 31 paths, one core of a 2022 laptop$time driftpipe check --input events.ndjson --sample 0driftpipe 0.9.4 48213 records 31 paths 182msreal 0m0.191s user 0m0.172s sys 0m0.024s
That is about 265 000 records per second on one core, decoding included, and it is entirely bounded by the JSON decoder. There is no parallel mode: a second core would buy 1.8 times a number that is already smaller than the time the surrounding pipeline spends starting up.
WHY NO DAEMON
The obvious next feature is a long-running mode that watches a topic and reports drift as it happens. It is not planned, for a reason that is worth stating: the moment this becomes a service it needs a configuration reload story, a health endpoint, a metrics endpoint, an alerting integration and somebody on call for it. A tool that exits is auditable, pinnable and trivially correct; a daemon is a system. Run it in a nightly job against yesterday's dump and you get the same signal for none of that cost.