driftpipev0.9.4docs for Sign in

Config & lock file

NAME

config - .driftpipe.yml, the rules map, and what a lock file contains

CONFIG FILE

Loaded from the working directory if present. Everything in it has a flag equivalent, so the file is a convenience and never a requirement; a repository with no config behaves exactly as the documented defaults say.

.driftpipe.yml
# loaded from the working directory unless --config says otherwise
version: 1                      # config format version, not the tool version

input: fixtures/orders.ndjson    # same as --input; "-" is stdin
schema: schema.lock              # same as --schema
sample: 20000                    # 0 reads the whole stream
max_depth: 10                    # deeper paths become depth_exceeded

rules:                           # per class: allow | warn | fail
  new_fields:     allow          # path in the stream, absent from the lock
  removed_fields: fail           # path in the lock, absent from the stream
  type_widening:  warn           # int -> float, and nothing else
  type_narrowing: fail           # float -> int, string -> int, replacements
  nullability:    fail           # null where there was none before
  depth_exceeded: warn           # nesting below max_depth

ignore:                          # path patterns, syntax as in --ignore
  - meta.received_at
  - order.items[].debug
  - "*.trace_id"
  - internal.**

output:
  format: text                   # text | json | github
  color: auto                    # auto | always | never
  quiet: false

An unknown key is a usage error, not a warning: type_widenning: warn exits 1 and names the key. A config typo that silently disabled a rule would be the worst possible failure mode for a tool whose whole job is to notice things.

THE RULES MAP

What each severity does. Only fail affects the exit status.
SeverityPrintedCounted in summaryExit 3
allownonono
warnyesyesno
failyesyesyes

--fail-on is shorthand for a whole map and replaces it rather than merging into it. --fail-on any sets all six classes to fail; --fail-on breaking sets removed, narrowing and nullability to fail and the other three to warn. That second mapping is also the built-in default, which is why a tool run with no configuration at all still fails on a dropped field.

PRECEDENCE

  1. Command line flag.
  2. Environment variable, where one exists for that setting.
  3. Config file.
  4. Built-in default.

Resolution is per setting, not per source: a config file that sets sample and ignore keeps both when you override only --sample on the command line. The one exception is the pair rules and --fail-on, described above, where the flag wins wholesale.

To see what was actually resolved, before reading a single record:

$ driftpipe check --config .driftpipe.yml --print-config
input          fixtures/orders.ndjson     (config)
schema         schema.lock                (config)
sample         5000                       (flag)
max_depth      10                         (config)
rules.new_fields      allow               (config)
rules.removed_fields  fail                (config)
rules.type_widening   warn                (default)
ignore         4 patterns                 (config)
output.format  text                       (default)

LOCK FILE

JSON, sorted keys, newline at the end, intended to be committed and to be read in a pull request diff. Current format version is 3.

schema.lock
{
  "driftpipe": 3,
  "created": "2026-09-30T08:14:02Z",
  "note": "orders v2 fixture, nightly dump 2026-09-29",
  "records": 41160,
  "sample": 0,
  "max_depth": 8,
  "paths": {
    "meta.received_at": "string",
    "order.customer_id": "string",
    "order.id": "string",
    "order.items[].qty": "int",
    "order.items[].sku": "string",
    "order.items[].unit_price": "int",
    "order.legacy_ref": "string",
    "order.total": "int"
    // 22 further paths omitted here for brevity
  }
}

A type is either one of null bool int float string object array or a sorted union written with |, as in string|null. Nothing else appears in the paths map: no counts, no examples, no sample values. That is a deliberate privacy property - a lock file can be committed to a public repository without leaking anything about the records it was built from.

READING A LOCK

$ driftpipe explain --path order
schema.lock   format 3   created 2026-09-30T08:14:02Z
note: orders v2 fixture, nightly dump 2026-09-29
41160 records, depth limit 8

order
├── customer_id      string
├── id               string
├── legacy_ref       string
├── total            int
└── items[]
    ├── qty          int
    ├── sku          string
    └── unit_price   int

8 of 30 paths shown

Older locks are read without complaint; a lock from a newer format version exits 4 rather than guessing. Format 2 locks are still accepted and are upgraded in place on the next snapshot --force, as noted in the changelog under 0.8.0.