driftpipev0.9.4docs for Sign in

Nightly cron

NAME

cron - a scheduled run over yesterday's whole dump

A nightly job watches real data rather than a fixture, so it reads the full stream with --sample 0, uses the breaking threshold, and keeps a vanished dump distinct from a genuine schema change.

WRAPPER SCRIPT

ci/schema-drift.sh
#!/bin/sh
# usage: schema-drift.sh INPUT [THRESHOLD]
set -eu
INPUT="$1"
THRESHOLD="${2:-breaking}"

rc=0
driftpipe check \
  --input "$INPUT" \
  --schema schema.lock \
  --sample 0 \
  --fail-on "$THRESHOLD" \
  --format json \
  > drift.json || rc=$?

case "$rc" in
  0) echo "schema unchanged" ;;
  3) echo "drift above threshold $THRESHOLD, see drift.json" >&2; exit 1 ;;
  *) echo "driftpipe failed with status $rc" >&2; exit "$rc" ;;
esac

SCHEDULE

crontab
# 04:17 every day, over the dump written overnight
17 4 * * *  cd /srv/pipe && ci/schema-drift.sh "dumps/$(date -d yesterday +\%F).ndjson.gz"

WHAT IT CATCHES

Separating exit 3 from everything else is the whole point of the wrapper. A dump that failed to land exits 2 and must fail the run loudly as an infrastructure fault, not quietly as a clean schema; folding both into “non-zero” throws that difference away exactly when it matters.

Reading the full stream matters more here than anywhere else, because overnight dumps are frequently sorted by time or tenant, and a sampled run would let a late-appearing field hide behind the cut.

GOTCHAS

  • A systemd timer with Persistent=true is the better host if the machine sleeps, since a missed cron tick is simply skipped.
  • Rotate or date the drift.json output if you keep a history; the script overwrites it each run.
  • Keep the threshold at breaking: a nightly job on live data with any is the textbook way to train everyone to ignore it.