driftpipev0.9.4docs for Sign in

How it works

NAME

internals - what happens between a byte arriving and an exit code leaving

PIPELINE

The seven stages of a driftpipe check Bytes are read, decoded into records, reduced to a path and type shape, merged into one observed schema, compared with the lock file, classified by severity, and reported with an exit code. 1 read2 decode3 shape4 merge5 compare6 classify7 reportschema.lock file, stdin, gzipone record at a timepath -> type, per recordobserved schema, O(paths)severity per differencetext, json, github + exitcommitted baseline records are never retained
Stages 1 to 4 see records; stages 5 to 7 see only the merged shape. The lock enters at stage 5, which is why a malformed lock is reported after the stream has been read, not before.

Stage 3 is the only interesting one. For each record it walks the object and emits a flat set of path, type pairs: order.items[].sku and string, and so on. Array elements are merged into a single element shape as they are walked, so an array of a million objects costs the same as an array of two.

Stage 4 folds those sets into one. Folding two types for the same path produces either the same type, a widened type, or a union. Nothing else is possible, which is what keeps the whole program bounded and boring.

TYPE LATTICE

Seven types, one widening edge, no others:

null   bool   int → float   string   object   array

int widens to float because a producer that starts emitting 1.5 where it used to emit 1 has not broken any consumer that was parsing numbers. Every other change of type produces a union, and a union is a narrowing event: string becoming string|int means half your consumers now have a branch they did not have.

Classification by syntax, not by value: 1 is int, 1.0 and 1e3 are float. The decoder does not convert numbers to a machine type at all, so integers beyond the usual 53-bit limit keep their precision and still classify as int. The consequence for round numbers written in exponent form is listed under limitations.

null is tracked as a member of the union like any other type. That is why nullability is its own class rather than a flag: string to string|null and string to string|int are both unions, but one of them breaks consumers in a very specific and very common way, and it deserves its own line in the report.

CLASSIFICATION

The complete set. There are six classes and there will not be a seventh without a lock format bump.
SymbolClassConditionDefault
+new_fieldspath observed, not in lockwarn
-removed_fieldspath in lock, not observedfail
~type_wideningint in lock, float observedwarn
!type_narrowingany other type replacement or unionfail
!nullabilitynull joins a union that lacked itfail
~depth_exceededa subtree was not expandedwarn

A single path produces at most one difference. If a field is both widened and made nullable, it is reported once, as nullability, because that is the more severe of the two and reporting both would double-count in the summary.

DETERMINISM

  • Paths are sorted bytewise before the comparison, so report order never depends on map iteration.
  • Unions are sorted by a fixed type order, not alphabetically, so string|null is always written that way round.
  • JSON output has sorted keys and a trailing newline, and diffs cleanly between two runs.
  • Given the same input bytes, the same lock and the same version, the output is byte-identical. This is checked in the test suite by running every fixture twice and comparing.
  • Nothing in the report depends on the clock, the hostname, the locale or the working directory. The only timestamp driftpipe writes is the one in a new lock.

COST

State is proportional to the number of distinct paths, not to the number of records. One path costs about 300 bytes including its type union and its name, so ten thousand distinct paths is roughly 3 MB of live heap, and a stream of fifty million records costs the same as a stream of fifty.

# 48213 records, 31 paths, one core of a 2022 laptop
$ time driftpipe check --input events.ndjson --sample 0
driftpipe 0.9.4   48213 records   31 paths   182ms
real 0m0.191s   user 0m0.172s   sys 0m0.024s

That is about 265 000 records per second on one core, decoding included, and it is entirely bounded by the JSON decoder. There is no parallel mode: a second core would buy 1.8 times a number that is already smaller than the time the surrounding pipeline spends starting up.

WHY NO DAEMON

The obvious next feature is a long-running mode that watches a topic and reports drift as it happens. It is not planned, for a reason that is worth stating: the moment this becomes a service it needs a configuration reload story, a health endpoint, a metrics endpoint, an alerting integration and somebody on call for it. A tool that exits is auditable, pinnable and trivially correct; a daemon is a system. Run it in a nightly job against yesterday's dump and you get the same signal for none of that cost.