driftpipev0.9.4docs for Sign in

Limitations

NAME

limitations - the list we would want to read before adopting a tool

Everything here is a deliberate boundary or a known cost, not a bug waiting for a release. Items marked not planned have been considered and rejected; items marked open we would fix given a clean design.

KNOWN LIMITATIONS

The same list as the sections below, as structured data you can filter. It is generated from /api/limitations.json (14 items); the prose sections after it give the reasoning in full.

  • input Only JSON records not planned

    Newline-delimited JSON, a JSON array of objects, concatenated objects, optionally gzipped - nothing else. Columnar and binary formats carry their own schema in the file header, so comparing two headers is a different and much easier program than this one.

  • input No CSV not planned

    CSV has no types and no nesting; structural drift in a CSV is a changed header line, which diff already handles better than a type lattice would.

  • input No network input not planned

    A file path or standard input only. Pipe a consumer into it for live data, which keeps credentials, offsets and retries in the tool that already knows about them.

  • sampling The sample is the first N records open - use --sample 0 where correctness matters

    This is head bias and it is the sharpest edge in the tool: a field that appears only after the cut is invisible, and a dump sorted by anything correlated with the schema will hide drift.

  • structure Arrays are collapsed to one element shape not planned

    items[0] and items[7] are the same path and their types are merged, so a tuple-shaped array is reported as a union across all positions.

  • structure Depth is limited to 8 by default open

    Below the limit a subtree is reported once as depth_exceeded and not expanded, so drift inside it is invisible; raising --max-depth costs memory linearly in distinct paths.

  • structure Keys are compared literally not planned

    userId, userid and user_id are three unrelated paths; there is no case folding, because a producer that changes the case of a key has changed the key.

  • structure Duplicate keys in one object: last wins, silently open

    This matches the decoder's behaviour and the duplication itself is not reported; detecting it needs a custom decoding path and a major version.

  • types Numbers are classified by syntax not planned

    1 is int; 1.0 and 1e3 are float, even though both are mathematically integral, so a switch from 1000 to 1e3 is reported as widening.

  • types Strings are opaque not planned

    A timestamp, a UUID, an email and a base64 blob are all string; format inference is a guess, and a guess ends up frozen in a committed lock.

  • types No value-level checks of any kind not planned

    No ranges, enums, regular expressions, uniqueness, cross-field constraints or row counts - values are never read, which is what lets a lock be committed publicly.

  • interop The lock is not JSON Schema, and conversion is lossy open

    Outgoing, the lock cannot express required faithfully; incoming, oneOf, patternProperties, conditionals and formats have no representation here.

  • interop No plugins, no scripting, no rule language not planned

    Six classes, three severities and a path pattern syntax; a policy that does not fit reads --format json with the tool of your choice.

INPUT

Only JSON records. not planned

Newline-delimited JSON, a JSON array of objects, concatenated objects, optionally gzipped. No Parquet, no Avro, no Protobuf, no XML. Columnar and binary formats carry their own schema in the file header, so comparing two headers is a different and much easier program than this one, and it should be a different program.

No CSV. not planned

CSV has no types and no nesting. Structural drift in a CSV is a changed header line, which diff already handles better than a type lattice would.

No network input. not planned

A file path or standard input. Pipe a consumer into it if you need live data; that keeps credentials, offsets and retries in the tool that already knows about them.

SAMPLING

The sample is the first N records, not a random N. open

This is head bias and it is the sharpest edge in the tool. A field that appears only in records written after the cut is invisible to the check, and a dump sorted by anything correlated with the schema - time, tenant, event type - will hide drift from a sampled run. Reservoir sampling would fix the bias and break byte-determinism, which we value more; a seeded shuffle would restore both at the cost of a flag nobody would set correctly.

The practical answer is --sample 0 in any job where correctness matters more than seconds, and the default of 5000 only for the fast pull-request check against a committed fixture. A 48 000 record file costs 182 ms to read in full, so the default is a convenience, not a necessity.

STRUCTURE

Arrays are collapsed to one element shape. not planned

items[0] and items[7] are the same path, and their types are merged. A tuple-shaped array, where position two is always a string and position three always an object, is therefore reported as a union across all positions. Positional arrays in a record stream are rare and usually a mistake; tracking them would double the lock format for that rarity.

Depth is limited to 8 by default. open

Below the limit a subtree is reported once as depth_exceeded and not expanded, so drift inside it is invisible. Raising --max-depth costs memory linearly in the number of distinct paths that appear, which for a deeply nested document can be a surprising multiple.

Keys are compared literally. not planned

userId, userid and user_id are three unrelated paths. There is no case folding and no normalisation, because a producer that changes the case of a key has changed the key, and pretending otherwise would hide a real break.

Duplicate keys in one object: last wins, silently. open

This matches the decoder's behaviour, and the duplication itself is not reported. Detecting it is possible but needs a custom decoding path, which is the kind of change that deserves a major version.

TYPES

Numbers are classified by syntax. not planned

1 is int; 1.0 and 1e3 are float, even though both are mathematically integral. A producer that switches from 1000 to 1e3 will be reported as widening. The alternative - parsing every number to decide - costs more than the whole decode loop and changes the meaning of the lock for the sake of a case nobody has reported in two years.

Strings are opaque. not planned

A timestamp, a UUID, an email address and a base64 blob are all string. Format inference is a guess, guesses end up in a lock file, and a lock file that says date-time because the first 5000 values happened to parse is worse than one that says string.

No value-level checks of any kind. not planned

No ranges, no enums, no regular expressions, no uniqueness, no cross-field constraints, no row counts. Values are never read. This is the property that lets a lock file be committed to a public repository without leaking anything about the data.

INTEROPERABILITY

The lock is not JSON Schema, and conversion is lossy. open

Going out, the lock has no way to express required faithfully: a path that was present in all 41 160 sampled records might still be optional in the producer's contract, and we have no evidence either way. Going in, almost everything a schema can say - oneOf, patternProperties, conditionals, formats - has no representation here. A one-directional exporter that emits a deliberately weak schema is a plausible future addition; an importer is not.

No plugins, no scripting, no rule language. not planned

Six classes, three severities, a path pattern syntax. If your policy does not fit in that, read --format json with the tool of your choice; that output is stable and documented, and you will get a better result than any embedded expression language we could ship.

PLATFORM

Symlinks are not followed.

On any platform, for both the input and the lock. A symlinked lock file is read, but snapshot --force replaces the link rather than writing through it.

Windows paths work; Windows line endings in the stream do not matter.

Backslash paths and drive letters are accepted on the command line, and carriage returns are tolerated in newline-delimited input. This was broken until 0.9.3, which is the one bug in this list that has already been fixed.