Nightly cron
NAME
cron - a scheduled run over yesterday's whole dump
A nightly job watches real data rather than a fixture, so it reads the full stream with --sample 0, uses the breaking threshold, and keeps a vanished dump distinct from a genuine schema change.
WRAPPER SCRIPT
ci/schema-drift.sh#!/bin/sh# usage: schema-drift.sh INPUT [THRESHOLD]set -eu INPUT="$1" THRESHOLD="${2:-breaking}" rc=0 driftpipe check \ --input "$INPUT" \ --schema schema.lock \ --sample 0 \ --fail-on "$THRESHOLD" \ --format json \ > drift.json || rc=$? case "$rc" in 0) echo "schema unchanged" ;; 3) echo "drift above threshold $THRESHOLD, see drift.json" >&2; exit 1 ;; *) echo "driftpipe failed with status $rc" >&2; exit "$rc" ;; esac
SCHEDULE
crontab# 04:17 every day, over the dump written overnight17 4 * * * cd /srv/pipe && ci/schema-drift.sh "dumps/$(date -d yesterday +\%F).ndjson.gz"
WHAT IT CATCHES
Separating exit 3 from everything else is the whole point of the wrapper. A dump that failed to land exits 2 and must fail the run loudly as an infrastructure fault, not quietly as a clean schema; folding both into “non-zero” throws that difference away exactly when it matters.
Reading the full stream matters more here than anywhere else, because overnight dumps are frequently sorted by time or tenant, and a sampled run would let a late-appearing field hide behind the cut.
GOTCHAS
- A systemd timer with
Persistent=trueis the better host if the machine sleeps, since a missed cron tick is simply skipped. - Rotate or date the
drift.jsonoutput if you keep a history; the script overwrites it each run. - Keep the threshold at
breaking: a nightly job on live data withanyis the textbook way to train everyone to ignore it.