Runbook: pushing a release tag

Everything else in the release plane can be undone. This cannot.

.github/workflows/release.yml and .github/workflows/package.yml both fire on refs/tags/*@*, and the repository’s tag ruleset (release tags are immutable) carries deletion and non_fast_forward with no bypass actors. A pushed release tag cannot be deleted and cannot be moved by anybody, including the owner, without first disabling the ruleset. So the tag is not the step where a mistake is discovered; it is the step where a mistake becomes permanent. Everything below is the work of finding the mistake beforehand.

Read this alongside release-build.md (what is being tagged), release-sign.md (what signs it) and release-publish.md (where it goes).

Which commit the tag points at

The tag says which release. The release says which commit. They are two different commits and conflating them is the trap.

A release is a function of the commit it was compiled from, and it records that commit in its own release-meta.yaml. But a release directory is committed after the canon commit it records — a release cannot be inside the commit it was built from — so the recorded commit is always an ancestor of, never equal to, the commit that holds the release.

It is therefore tempting to reason “a correct tag points at the commit the release records”. It does not, and the reason is not about this repository at all:

GitHub runs the workflow file from the ref that triggered it. A tag pushed at commit X runs .github/workflows/release.yml as it stood at X — not the one on main, not the one you reviewed.

Tag a canon commit from a hundred commits ago and you run that era’s pipeline. For this repository that is not a hypothetical inconvenience: at the commit fixreg@2025.04 records, four of the five stages were tooling/ci/pending.py guards, which print a warning and exit 0. The run would be entirely green having signed nothing, scored nothing and published nothing — and package.yml did not exist yet, so no CLI binaries at all.

The build stage therefore does not use the tagged commit. It asks tooling/ci/release_target.py which commit the release records and compiles that, which leaves the tag free to sit on the pipeline that is meant to run:

tag  ──▶  the commit whose workflows run       (the head of `main`)
 │
 └──▶ regimes/<regime>/releases/<version>/release-meta.yaml
                      │
                      └──▶ commit: …  ──▶  the canon the artifacts are built from

So: tag origin/main. Not a canon commit, not a local branch, not HEAD of whatever is checked out. The build derives the rest, and refuses if the recorded commit is not an ancestor of the commit being tagged.

Before you push

Run all of it. Each line is a question a green pipeline cannot answer for you.

1. The release exists, is filed correctly, and its commit is reachable.

TAG=fixreg@2025.04
uv run --frozen python tooling/ci/release_target.py "$TAG" regime    # -> fixreg
uv run --frozen python tooling/ci/release_target.py "$TAG" version   # -> 2025.04
uv run --frozen python tooling/ci/release_target.py "$TAG" commit    # -> 0f466185ac19…

The third refuses if the repository holds no such release, if the release calls itself something else, if the recorded sha is one a rewritten history left dangling, or if it is not an ancestor of HEAD. Run it on the commit you are about to tag, because that is the ancestry it is asking about.

2. The release in the repository rebuilds from that commit, byte for byte.

uv run --frozen pytest tooling/tests/test_release_build.py -k rebuilding_from_the_same_commit

This is the check that makes every digest downstream mean something. If the committed release and its recorded commit have drifted apart, fix that first — never by re-pinning the tag.

3. The signing posture is the one you expect.

uv run --frozen python tooling/ci/signing_posture.py "$TAG"          # keyed | keyless

fixreg@* is keyed and signs with the published fixture key, which vouches for nothing and says so. Every other regime is keyless and needs an OIDC token, which means it can only run in openregs/openregs. A posture that surprises you is a config/trust.yaml problem, not a tag problem.

4. Rehearse the whole pipeline offline. The workflow harness executes every run: step for real, in needs: order:

uv run --frozen pytest tooling/tests/test_ci_workflows.py tooling/tests/test_release_attach.py -q

The stage-by-stage rehearsal that goes with this runbook lives there: the build compiling the recorded commit, the eval gate leaving publish and attach unrun, the live publish branch failing on an empty token — and the release reaching the channel anyway when it does — and the sign stage going green on a runner whose cosign it cannot drive.

The second file is the last stage, run against a stand-in forge because it is the one stage that writes to a real one — including the two answers you want before a tag, not after: an upload that dies part way leaves a draft nobody can download, and a re-run finishes it.

5. The commit you are tagging is the one CI has already gone green on.

git fetch origin
gh run list --branch main --limit 1
git rev-parse origin/main

6. Decide, deliberately, whether publish should upload. publish derives its mode from the repository owner: in openregs/openregs it is live and uploads to PyPI and npm. It fails — loudly, by design — if PYPI_TOKEN or NPM_TOKEN is empty. Today the organisation has no secrets configured at all, so the live branch stops there with OPENREGS_PYPI_TOKEN is empty and the run goes red. attach runs anyway — it needs eval, not publish — so the GitHub release is still created and openregs pull still has something to fetch. Expect that shape and read it correctly: a red run whose only failure is publish has still distributed the release. What is missing is the two packages, and nothing else.

A PyPI upload is the second irreversible step in this pipeline and is easy to overlook behind the first: a version number, once uploaded, cannot be re-uploaded with different bytes. Do not add the secrets in the same change as the first tag.

7. The tag is not already taken. It can never be reused.

git ls-remote --tags origin

The push

git fetch origin
git tag -a fixreg@2025.04 origin/main -m "fixreg@2025.04"
git push origin fixreg@2025.04

An annotated tag records who cut it, which is worth having on something nobody can move. Actions treats it identically to a lightweight one.

The push starts two workflows, and they are independent — release and package have separate concurrency groups, and neither needs the other.

What each stage should emit

release.yml:

stage	what a correct run says
`build`	`building fixreg@2025.04 from <recorded sha>`, then five digests. Compare them against `regimes/fixreg/releases/2025.04/release-meta.yaml`. They must be equal — that is the whole claim of the tag.
`sign`	`signing posture: keyed (config/trust.yaml, for fixreg@2025.04)`, `8 blob signature(s)`, then `verify: OK — … verified entirely offline` with all five checks passing. Under the keyless posture, a `cosign verify-blob` that names the pinned issuer and identity as well.
`eval`	one line per scoring category and `overall`, each `ok` and above its floor in `config/eval.yaml`.
`publish`	`publish mode: live (owner: openregs)`, both packages written, both uploads run. In a fork, `dry-run` and `[not run (dry run)]`.
`attach`	`assets: <tag> -> dist/mirror/<tag>` and the count it wrote, then `attach: the release for <tag> is absent`, then `attach: published <tag> with N asset(s)`. N must equal the asset count `assets` printed. The GitHub release for the tag now exists, published, carrying the release tree flattened into `__`-joined asset names.

publish and attach both hang off eval and neither waits on the other, so they may interleave in the log and either may be the last to finish. A failed publish does not stop attach: the packages and the release are two distributions, and only the eval gate governs both — ci.md, Why attach does not need publish.

package.yml: four platform freezes, then one sign job that writes SHA256SUMS, signs the set with the fixture key and re-verifies it. Its output is a workflow artifact called standalone-signed and nothing else — the binaries are not attached to the GitHub release. release.yml’s attach publishes the content release and only that; the two pipelines are independent and neither waits on the other, so a job that attached both would have to.

Telling success from green-but-wrong

Two shapes of wrong are green, and only two. Check for both.

The wrong bytes. Every stage after build is about the bytes in dist/, so a build that compiled the wrong commit signs, scores and publishes perfectly cleanly. The signature is valid, the digests reconcile, the eval passes — for a release the repository does not hold. The digests build prints are the tell, and the check is against release-meta.yaml in the repository, not against anything in the log. Do this after the run, from the artifacts:

gh run download <run-id> -n "release-fixreg@2025.04" -D /tmp/tagged
gh run download <run-id> -n "attestation-fixreg@2025.04" -D /tmp/tagged/attestation
diff -r --brief /tmp/tagged regimes/fixreg/releases/2025.04 -x attestation
uv run --frozen openregs verify --release fixreg@2025.04 --path /tmp/tagged

diff must report nothing. verify must print verify: OK. A verify: OK on its own proves only that the release is internally consistent and signed by a key the roster accepts — it says nothing about which release it is.

A stage that did not run. No stage in these pipelines is a pending-capability guard any more, and a guard exits 0 — so a ::warning::…: PENDING in the log means a stage regressed to one. Read the log for the word PENDING rather than for red. attach has a second, quieter form of this: attach: already published, carrying exactly these assets. Nothing to do. is correct on a re-run and wrong on a first one, because it says the tag’s release existed before this pipeline ran.

If it goes wrong

The tag cannot be deleted and cannot be moved. Plan around that rather than against it.

what happened	what to do
a stage failed for an environmental reason	`gh run rerun <run-id> --failed`. Re-running is free and changes nothing about the tag.
the build failed because the release, the recorded commit or the ancestry is wrong	fix it on `main` through a pull request and re-run. The tag still points where it pointed; the build no longer depends on that.
the tag points at the wrong commit	you cannot move it, so establish what that commit’s pipeline did rather than what this one would have. A tag on an older `main` runs that era’s `release.yml`, which may predate the rule that the build compiles the recorded commit — in which case it compiled the tagged commit’s canon instead. Check the artifacts against the repository (the `diff -r` above) before trusting anything the run published. If they match, the tag is merely untidy and nothing needs doing. If they do not, treat it as the row below.
`attach` failed	look first at whether it left a draft release for the tag (`gh release view "$TAG" --json isDraft`). If it did, nothing was ever downloadable and `gh run rerun <run-id> --failed` resumes it. If it refused because the tag’s release is already published with other assets, do not hand-edit the assets: that is the row below.
`publish` failed and `attach` went green	the release is on the channel and the two packages are not, which is the whole of what is missing. Adding `PYPI_TOKEN` and `NPM_TOKEN` is a deliberate decision of its own — the upload is irreversible — and once they exist `gh run rerun <run-id> --failed` re-runs `publish` alone, because nothing needs it.
`attach` was skipped because `publish` failed	that is a tag whose pipeline predates the fork in the graph, and a re-run will not fix it: GitHub runs the workflow file as it stood at the tagged commit, so the version on `main` is not the one a re-run of that tag executes. Attach the assets by hand — the section below.
the wrong bytes were published to PyPI or npm	the version number is spent. PyPI refuses a re-upload of an existing version. Yank it (`pip`/`npm` both support this), publish the correction as a new release version — a new date-tagged release, built and committed the ordinary way — and say so in the release notes. Yanking hides it from resolution; it does not remove it. There is no second tag for the same version: `fixreg@2025.04` is taken forever, and a `fixreg@2025.04.1` alongside it would be two tags claiming one release.
the tag genuinely must go	this is an owner decision, not an operator one. It requires setting ruleset `20202372` to `enforcement: disabled`, deleting the tag, and re-enabling it — and for as long as it is disabled every release tag in the repository is deletable and movable. Do it in one sitting, with the re-enable prepared in advance, and never as a way of avoiding a version bump.

Attaching a tag whose pipeline was the old graph

attach used to need publish, so a tag pushed before that changed can have gone green through eval, failed at publish for want of a token, and left the release attached to nothing. fixreg@2025.04 is exactly that tag.

A re-run does not fix it. GitHub runs the workflow file from the ref that triggered the run, so re-running that tag re-runs its release.yml, in which attach still hangs off publish. Adding the two package secrets would make the old graph reach attach — and would spend a PyPI version on the release as a side effect of trying to attach it, which is the coupling this graph removed rather than a way to work around it. So the assets go up by hand, once, with the same bytes and the same checks the stage would have used.

Everything below is the attach stage, run by a person. It needs gh authenticated as somebody with contents: write on the repository, and it pushes no tag and creates nothing that a later re-run would collide with — a re-run that found this release published carrying exactly these assets exits 0 having done nothing.

TAG=fixreg@2025.04
WORK=$(mktemp -d)

# The run that built and signed it — the one whose `build` printed the digests
# you checked against release-meta.yaml. Read its id off the listing.
gh run list --workflow release.yml --limit 10
RUN=<that run's id>

# 1. The bytes that run built and signed — not a local rebuild of them. The
#    attestation was uploaded separately, and goes back beside what it covers.
gh run download "$RUN" -n "release-$TAG"     -D "$WORK/release"
gh run download "$RUN" -n "attestation-$TAG" -D "$WORK/release/attestation"

# 2. The release's five checks, then the flat layout `pull` reads. This refuses
#    the whole release rather than uploading part of one.
uv run --frozen openregs release assets \
    --release "$TAG" --path "$WORK/release" --out "$WORK/mirror"

# 3. The notes the stage writes: the commit, and the command that fetches and
#    verifies the lot. `__`-joined asset names are not self-explanatory.
COMMIT=$(sed -n 's/^commit: *//p' "$WORK/release/release-meta.yaml")
cat > "$WORK/notes.md" <<NOTES
$TAG — the release this repository holds under that tag, built from commit $COMMIT.

Fetch and verify all of it with:

    openregs pull $TAG --from https://github.com/openregs/openregs --into ./vendor
NOTES

# 4. Draft, upload, compare the listing, and publish in one edit. Nothing is
#    downloadable until the last line: a draft's assets are not served.
gh release create "$TAG" --draft --verify-tag --title "$TAG" --notes-file "$WORK/notes.md"
gh release upload "$TAG" "$WORK/mirror/$TAG"/* --clobber
gh release view "$TAG" --json assets --jq '.assets[] | "\(.name) \(.size)"' | sort
gh release edit "$TAG" --draft=false

Step 4’s listing is the check, and it is the operator’s here because no job is comparing it: every name and size the assets command reported must be on the release before the last line runs. If they are not, leave it a draft — a draft serves nothing — and fix the difference first.

Then confirm it from outside, as a consumer with no account would:

openregs pull "$TAG" --from https://github.com/openregs/openregs --into "$(mktemp -d)"

Nothing about this is owed for a tag pushed on the current graph: there, a failed publish leaves attach to do its own work.

release-build.md — what is being tagged, and why a release cannot be inside the commit it was compiled from.
release-sign.md — the two postures, and what the keyless path needs that no offline run can supply.
release-publish.md — what publish uploads.
ci.md — the pipelines as a whole.

Rendered from openregs/openregs@f3a2d10:docs/runbooks/release-tag.md