Runbook — the CI/CD pipeline

Audience: anyone changing .github/workflows/, and whoever administers the repository. Trigger: a workflow needs a new job, a pin needs moving, or a job went red. Time: minutes.

The pipeline is the release plane’s automation: it is what decides that a pull request may merge and that a tag becomes a signed, published release. It is reviewed like code, and it is tested like code — tooling/tests/test_ci_workflows.py runs the workflows offline through tooling/tests/workflow_harness.py and fails if any of what follows stops being true.

---

The three workflows

File	Fires on	Jobs
pr.yml	every pull request (and workflow_call)	lint, typecheck, test, check-canon, conformance
main.yml	push to main, and nightly at 04:37 UTC	the five above again, via gates, then e2e and bench
release.yml	push of a <regime>@<version> tag	build → sign → eval → publish → attach

What is enforced by a machine, and what is not

Rule	Enforced by
the four required contexts exist and are named exactly as the policy says	test_ci_workflows.py, mechanically
every run: step of lint, check-canon, conformance and all five release stages succeeds	the harness, by running them
the e2e job runs make e2e and publishes dist/e2e/ whether it went green or red	test_ci_workflows.py, mechanically — the pipeline itself is run by make e2e, never from inside make test; see e2e.md
a canon violation on a branch fails the check-canon job	the harness, against a seeded fixture
the eval stage scores the built artifacts against config/eval.yaml’s floors	the harness, by running openregs eval --gate for real
a red eval leaves publish and attach unrun	the harness, against the degraded build fixtures/patches/disable-exempts-expansion.patch produces
the publish stage builds both packages and puts them in a local registry	the harness, by running openregs release publish --mode dry-run for real
a live publish with no credential fails rather than skipping the upload	the harness, by running the canonical-owner branch with the secrets unset
every third-party action is pinned to a commit sha	test_ci_workflows.py, mechanically
the bench job runs make bench and publishes its report	test_ci_workflows.py, mechanically — the job itself is not executed here
that any of this ran on GitHub	nobody here — see below

The honest limit. No test in this repository may reach the network, so none of this has been observed on GitHub’s runners. What is verified is that the commands the jobs run succeed on this tree, that the job graph gates what it claims to gate, and that the files parse the way Actions parses them. A first push will still be the first time a runner has ever executed them.

The harness runs run: steps for real and provisions uses: steps: being in a checkout satisfies actions/checkout, uv on PATH satisfies setup-uv, a directory stands in for the artifact store, and cosign-installer is recorded as UNPROVISIONED because a local run cannot install it. An action missing from workflow_harness.KNOWN_ACTIONS is an error, so a new dependency cannot enter the pipeline without a reviewer reading it.

The job names are a contract

.github/branch-protection.yaml names four required status check contexts: lint, typecheck, test, check-canon. A check run’s context is the job’s name, so those four names in pr.yml are the same fact as those four names in the policy. Consequences:

• renaming a job removes a required check. The hook then waits forever for a context nobody publishes, and the tempting fix — editing the policy — silently stops requiring something. Change both, deliberately, in one commit.
• no job may use a strategy: matrix. A matrix job’s context is test (3.12), which is not test. The harness refuses to parse a workflow with one.
• pr.yml declares its five jobs directly rather than calling a reusable workflow. A called workflow’s contexts are prefixed (gates / lint), which would not be the required names. main.yml may call it, and does, because branch protection guards pull requests, not pushes.

The eval gate

release.yml’s third stage runs openregs eval --release "$TAG" --path dist --gate. It asks a serving instance built on the artifacts the pipeline just signed every question in fixtures/eval/questions.jsonl that release may be asked, scores the answers per category, writes dist/eval/report.json, and exits 1 when a category falls below the floor config/eval.yaml states. publish needs eval, so a regressed release has no path to a package index — see docs/runbooks/eval.md for how to read a red gate and what the floors mean.

Pending stages

One stage calls work a later spec task still owes: the attaching end of the release pipeline. (The benchmarks, the retrieval eval, package publishing and the end-to-end pipeline test were there too, until they landed — see below for what taking an entry out of that table looks like.) That step runs tooling/ci/pending.py <capability>, which proves the capability is missing — it runs a probe and requires the exact failure that absence produces — and exits 0 with a ::warning:: annotation naming the task that owes it.

The guard is red in both directions of drift:

• the capability landed → the probe now succeeds → fail, with instructions to wire the real command in and delete the entry;
• the capability landed and is broken → the probe fails, but not with the marker absence produces → fail.

So a pending stage can never hide a failing check; it can only be quiet while there is no check to fail. When you implement one of those tasks, expect this guard to go red — that is it asking you to finish the job in .github/workflows/.

The performance budgets

config/perf-budgets.yaml declares five limits — the p95 of POST /v1/answer, openregs serve from start to /readyz, the fixture regime’s release build, the whole make e2e pipeline run, and the canonical as-of query on corpus.sqlite. make bench measures all five, writes dist/bench/report.json and dist/bench/report.md, and exits 1 naming any it exceeded.

• It runs in main.yml, on the merge and on the nightly schedule, never on a pull request. A benchmark measures a machine as much as a change, so it wants the same runner at the same hour rather than whichever runner a contributor’s push happened to land on.
• The report is uploaded as the bench-report artifact, if: always(). The run worth reading is the one that went red, and a failing step would otherwise take the evidence down with it. That is the only if: in these files, and it does not rescue the job — the step failed, so the job failed.
• It is not run by make test. A wall-clock threshold inside the test suite would fail on a busy laptop and teach everybody to re-run until green. What the suite covers instead is the enforcement: tooling/tests/test_perf_budgets.py hands python -m openregs.bench a measurement over budget and asserts the real non-zero exit and the named budget, with no timing of its own anywhere.

A budget that has to move is a reviewed change to that file, with the commit message saying what got slower and why the new number is right. Raising a limit to turn a red build green is the one use of the file that is never legitimate; the seeded regression fixtures/patches/answer-600ms-sleep.patch is there to show what the red build is supposed to look like.

Moving a pin

Every uses: is owner/repo@<40-hex sha> # vX.Y.Z. A tag is a pointer its owner can move; a sha is bytes. To update one:

curl -s "https://api.github.com/repos/<owner>/<repo>/tags?per_page=10" \
  | python3 -c "import sys,json;[print(t['name'], t['commit']['sha']) for t in json.load(sys.stdin)]"

Put the sha in uses: and the tag it came from in the trailing comment — the tests require both — and say in the commit message why the version moved. Runner images are pinned for the same reason: ubuntu-24.04, never ubuntu-latest.

When a job goes red

1. Read which job. The five PR jobs each run one command, and every one of them runs identically on a laptop: uv run --frozen ruff check ., uv run --frozen mypy --strict tooling/, make test, make check-canon.
2. Do not re-run it and take the second answer. There is no continue-on-error and no retry anywhere in these files, on purpose: a suite allowed to be re-run until it is green reports the best of N runs, which is not a result about the tree. If a job is flaky, the flake is a bug — the last one was a race in tooling/tests/http_control.py, where the control server recorded a request after answering it, so a client could observe the response before the recording existed. It was fixed there.
3. A red check-canon is not a CI problem. It is the canon contradicting itself, and fixtures/violations/README.md describes each rule and what trips it.

Releasing

Push a tag named <regime>@<version> (fixreg@2025.04). The five stages run in order, and the order carries the meaning: you sign what you built, you score what you signed, you publish only what the eval gate passed, and the GitHub release carries what was published. Publishing is live only under the openregs owner; in a fork the same command runs as a dry run, decided from the owner rather than from whether a secret happened to be set — and a live run with PYPI_TOKEN or NPM_TOKEN unset fails rather than skipping the upload. What the packages contain, and how to install one from the local registry a dry run writes, is release-publish.md.