Runbook: the end-to-end pipeline test

make e2e

One command, about a minute, and it exercises every edge of the architecture in one pass. Nothing is needed of the machine beyond a checkout: no state directory, no vendored release, no network — the run seals every Python child process it starts to loopback and proves the seal bites before the first stage.

What it does

It builds a workspace under dist/e2e/work/, exports this checkout’s tracked tree into a fresh one-commit git repository there, empties that copy’s canon, graph and one atom, and then rebuilds all three from the fixture sources:

Stage	What it establishes
ingest	all four adapters poll their recorded feeds and capture; the scope crawl writes the regime’s Expressions from the fixture Formex
gates	every staged Expression met its source’s coverage and reference thresholds, the degraded gazette scan is held for a person, and every captured payload re-hashes to the digest it is stored under
consolidate	the amending act is applied — and the canon and SOURCES.lock this run produced are the committed ones, byte for byte
atomize	the fixture model reads the freshly consolidated Article 5a and proposes FIXREG-Art5a-Ob1 onto a branch of a protected bare remote
review-merge	an unreviewed merge is refused; then the CI identity publishes the checks, a code owner of every changed path approves the exact commit, and the merge restores the atom the corpus holds
graph	the edge list derived from the rebuilt canon and atoms is byte-identical to the committed one
release	fixreg@2025.04 is built from the merge commit and signed into a registry
verify	all five cryptographic checks pass, offline
pull	the release is verified before it is unpacked, into a directory that did not exist
serve	that pulled copy answers the 50 replay queries

Then it follows one answer’s citation home: the q009 response’s atom id → the atom’s provenance in the bundle → the unit version in force → that unit’s transform version and source_hash → the SOURCES.lock entry pinning it → and the sha256 fixtures/MANIFEST.yaml records for fixtures/sources/eur-lex/fixreg.formex.xml. Every hop is read out of an artifact the run produced; the last comparison is against the manifest, read at run time.

What it writes

Everything lands in dist/e2e/ (gitignored), and the CI job publishes it as the e2e-report artifact whether the run went green or red:

• run.jsonl — the log, one JSON object per line in the schema of spec/schemas/log-record.schema.json. Also written to standard error as the run goes, which is where a log belongs; standard output carries the sentences. This file is what step 2 of the task is checked against: the stage names are read back out of it and compared with the ten, in order.
• report.json and report.md — the stage timings, the five steps’ verdicts, the citation chain and the replay shortfall.
• work/ — the whole workspace, left in place on purpose. A failed run is one you want to walk into: the repository it built, the canon it wrote, the registry it signed and the release it vendored are all still there.

Reading a failure

The last lines of stdout are the five steps with their verdicts, and the process exits 1 when a step failed and 2 when a stage could not be completed at all (the reports are written either way). Then:

• a stage failed — the error names the command and quotes everything it said. Re-run just that command against dist/e2e/work/ to iterate; the workspace is exactly what the stage saw.
• step 3 short of 50/50 — the shortfall lists the queries and the atom ids their answers did not cite. That is a retrieval or applicability question, not a pipeline one; tooling/tests/test_serve_parity.py holds the measured floor.
• step 5 over budget — read report.json’s stage timings before touching config/perf-budgets.yaml. Raising a limit to make a red build green is the one use of that file that is never legitimate; the limit is there because a stage that started redoing work is the thing worth catching.

The budget

e2e-pipeline in config/perf-budgets.yaml, in seconds, wall-clock. make e2e checks the run it has just made against it, and make bench measures it on the reference runner in the nightly job — one reviewed number, two gates.

make bench runs the pipeline with --measure, which reports the duration and exits 0 as long as all ten stages ran. A replay query the serving plane cannot yet answer is a make e2e failure, and a benchmark that went red for it would be putting a red number next to a machine that was not slow.