Runbook: building a content release

A release is the boundary between the corpus and everybody who uses it. Upstream of it the truth lives in git — one file per legal unit, one YAML file per atom, a reviewed edge list, a lock of source digests. Downstream of it a consumer gets a directory of artifacts and never reads the canon at all.

openregs release build --regime fixreg --tag 2025.04 [--commit <rev>] [--out <dir>]
make build TAG=2025.04 [REGIME=fixreg] [COMMIT=<rev>]

writes regimes/<regime>/releases/<tag>/.

The two rules that make a release rebuildable

A release is a function of a commit. The working tree is never read. The regime directory is materialised out of a git commit (git archive), so the same tag built from the same sha is the same bytes, whatever the checkout has been doing since. --commit takes any revision and defaults to HEAD.

The build timestamp is the commit’s committer date. Never the wall clock. built_at in release-meta.yaml is that instant, normalised to UTC and written …Z.

Everything else that could vary is pinned: SQLite gets a fixed page size, UTF-8 encoding, journal_mode = DELETE and a final VACUUM; Parquet gets format version 2.6, no compression, statistics on, page index off and one row group; every table is inserted in sorted order; every rendered file ends in exactly one newline; and no artifact is an archive, so no tar or zip header can smuggle an mtime in.

The bundle does not name the SQLite library that wrote it. That library is the one thing the build cannot hold still, and it leaks into corpus.sqlite twice: the header’s SQLITE_VERSION_NUMBER field at offset 96, and the residue some versions leave in the free space of a vacuumed page where others zero it. Both were seen for real — rebuilding the committed releases under SQLite 3.45.1, having been built under 3.50.4, reproduced every artifact except the bundle, which came out the same length and 20 613 bytes different. So the builder sets PRAGMA secure_delete = ON and, on the closed file, overwrites the header field with a declared constant (HEADER_VERSION_PIN, also written to release_info.sqlite_header_version so it is visibly a pin and not a claim). A rebuild that disagrees now disagrees about the corpus.

What a tag means

A content release is tagged YYYY.MM and speaks for the first day of that month: 2025.02 is read as of 2025-02-01, 2025.04 as of 2025-04-01. The date is derived from the tag, never declared.

The corpus is not truncated to that day. Every unit version the canon holds ships with its validity window, which is what lets one bundle answer “what did this say in January” and “what does it say in June” from the same file. The as-of date is used for one thing: reducing each release to the view a reader has of it, so the diff manifest can compare two views rather than two file lists.

The five artifacts

artifact	what it is	who deepens it
corpus.sqlite	every unit version, atom, edge and window, queryable offline	the SQLite-bundle task (documented queries in docs/sqlite-queries.md)
graph.idx	the same edges as an adjacency index a serving engine loads in one pass	the serving tasks
embeddings.parquet	one row per retrieval chunk — one legal unit version each — with a vector from the configured model	the serving tasks (retrieval closure)
constants-src/	the executable blocks as Python and TypeScript source	the distribution-packaging task
<tag>.diff.json	what changed since the previous tag, classified	the impact and report tasks (openregs impact, the reader-facing rendering of openregs diff)

plus release-meta.yaml.

corpus.sqlite

Tables: units, atoms, edges, expressions, modifications, release_info.

The row grain of units is a unit version — every addressable eId of every Expression, not one row per file. art_5__para_3 is the address an atom cites, a graph edge points at and an as-of query asks for, and it lives inside art-005.akn.xml; one row per file would make an atom’s unit_eid join to nothing. A nested eId takes the window of the file that carries it, which is the rule openregs.bitemporal states and everything else reads.

units.text is the unit’s text including everything nested inside it. units.own_text is its own words only — what the diff compares, so a changed paragraph is reported on the paragraph and not also on every unit containing it.

The row grain of atoms is an atom version, keyed (id, effective). An atom id is permanent and names an obligation, not a wording of it, so a regime’s atoms/ directory may hold several files carrying one id — FIXREG-Art5.3-Ob1 is EUR 5000 until 2025-02-28 and EUR 10000 from 2025-03-01 — and the bundle ships all of them so that one file answers a question about any date. Consequences a build has to respect, both of them handled in the builder:

• constants-src/ is generated from the versions in force on the release’s own as_of, because a generated constant is imported by name with no date beside it. 2025.02 therefore states 5000 and 2025.04 states 10000, each with its window in the metadata mapping.
• the diff reduces each release’s atoms to the versions in force on that release’s as_of before comparing, exactly as it does for provisions.

modifications is one row per passive modification of a consolidated Expression: the unit version an amendment produced (node, with provision beside it), the instruction that ordered it (source_href), Akoma Ntoso’s word for what it did (type), and the amendment parser’s ruling — classification (substantive or editorial) with classification_basis in the parser’s own words. The ruling is decided from the amending act’s recitals, and a recital is not a legal unit, so it never enters the canon on its own; consolidation writes it onto each textualMod in the text it produces and this table carries it into the release. It is the first half of the diff manifest’s classification rule, and without it every amendment reads as substantive.

The header carries application_id = 0x4F524753 (“ORGS”) and user_version = <bundle version> — 3 since modifications was added (2 was the atoms grain becoming a version).

graph.idx

UTF-8, LF, one trailing newline:

openregs-graph-idx<TAB>1
regime<TAB>fixreg
tag<TAB>2025.04
nodes<TAB>44
edges<TAB>99
#nodes
<node>                                  (`nodes` lines, byte-sorted)
#edges
<type><TAB><src><TAB><dst><TAB><attrs>  (`edges` lines)

src and dst are decimal indices into the node table, so an edge is three integers and a short string. attrs is - when the edge carries nothing beyond its endpoints, otherwise compact JSON with sorted keys — the same spelling corpus.sqlite stores in edges.attrs, so the two artifacts cannot disagree.

The edges come from the committed regimes/<regime>/graph/edges.yaml, not from a rebuild: the edge list is reviewed data with its own CI rule (dangling-reference rebuilds it and fails on drift), so a release ships what was reviewed.

embeddings.parquet — chunks, and which model embedded them

One row per legal unit version, which is the same grain as units in the bundle. A chunk is an article or a paragraph carried across whole — never a fixed token window, never an overlap, never a split — so a chunk boundary is always a boundary the legislator drew and a retrieval hit is an address (eu-act-reg-2024-1/eng@2025-03-01/art_5__para_3) rather than “somewhere around here”. The two versions of art_5__para_3 are two rows with two windows, which is what makes “what did this say in January” answerable at all.

19 columns: chunk_id, node, work, work_iri, expression, expression_iri, eid, language, jurisdiction, regime, corpus_tag, entity_types, valid_from, valid_to, known_at, text, model, dim, embedding. valid_to is the only nullable one — an open-ended window says so with a null rather than with a date nobody chose. entity_types is the set of entity types the atoms anchored to that unit apply to, so a retriever can filter to “obligations on a small operator” before it ranks anything.

The model is configuration, not code. config/embeddings.yaml (spec/schemas/embeddings.schema.json) names it:

schema_version: 1
embedder: fixture-hash
plugins: []

openregs release build resolves that name before it reads a commit, creates a directory or writes a byte. A name nothing is registered under fails there, listing what is registered:

release build: config/embeddings.yaml: unknown embedder 'acme-text-3'; available
embedders: fixture-hash (fixture-hash/1.0.0, 64 dimensions). A deployment
registers its own model with openregs.release.embedders.register() from a module
named under `plugins:` in config/embeddings.yaml

There is no fallback. A release that quietly shipped fixture vectors where a deployment expected real ones is indistinguishable from a working one until retrieval quality is measured in production.

fixture-hash/1.0.0 is the model this repository ships and the only one CI ever configures: tokens into 64 signed BLAKE2b buckets, L2-normalised. No weights, no randomness, no clock, no network. It is a fixture — it makes similarity between texts sharing vocabulary measurable and claims nothing about meaning.

A deployment with a real model writes a module that calls openregs.release.embedders.register("<key>", <factory>), lists that module under plugins:, and names <key> as the embedder. The factory receives the options: block (an endpoint, a model id, a batch size — things this repository cannot state), and the embedder it returns must offer three things and no more: a name (<model>/<version>, written into every row’s model column), a dimensions count, and embed_all(texts). A batch call, because a real model is behind a network hop and one call per chunk would make a build a thousand round trips. What comes back is shape-checked: a wrong count or a wrong width fails the build naming the model, rather than writing a plausible-looking artifact.

Determinism, and the pyarrow question. Format version 2.6, no compression, statistics on, page index off, one row group, rows sorted by node. pyarrow does stamp its own version into the footer’s created_by — the parquet analogue of the SQLite header field above — but it is not the same hazard: libsqlite3 is whatever the machine has, while pyarrow is a wheel uv.lock pins, so the field is an input to the build rather than a property of the box. It is left as written rather than overwritten, and a test asserts the committed artifact’s created_by matches the locked version, so moving the pin is a red test and not a silent change of bytes: rebuild every standing release in the commit that moves it. options: must not vary between two builds of one tag either — they are part of what makes a release reproducible.

<tag>.diff.json — the shape

This is the input to the canon bundle’s diff-classification rule and to openregs diff, so the shape is stated here rather than left to be read off an example.

{
  "schema_version": 1,
  "regime": "fixreg",
  "tag": "2025.04",
  "as_of": "2025-04-01",
  "commit": "<40-char sha>",
  "previous_tag": "2025.02",
  "previous_as_of": "2025-02-01",
  "previous_commit": "<40-char sha>",
  "counts": {
    "units": {"added": 6, "changed": 2, "removed": 0, "reissued": 20},
    "atoms": {"added": 1, "backfilled": 1, "changed": 1, "deprecated": 0, "removed": 0},
    "edges": {"added": 70, "removed": 0}
  },
  "changes": [
    {
      "kind": "unit",
      "id": "eu-act-reg-2024-1/eng/art_7",
      "change": "changed",
      "classification": "editorial",
      "classification_basis": "amendment-editorial",
      "citation": {
        "from": "eu-act-reg-2024-1/eng@2024-06-01/art_7",
        "to": "eu-act-reg-2024-1/eng@2025-03-01/art_7",
        "eid": "art_7",
        "work_iri": "/akn/eu/act/reg/2024/1",
        "in_force": ["2025-03-01", null]
      },
      "detail": {
        "before": "…must recieve confirmation…",
        "after": "…must receive confirmation…",
        "amendment": {
          "instruction": "/akn/eu/act/reg/2025/7/eng@2025-03-01/!main#art_1__point_3",
          "operation": "substitution",
          "classification": "editorial",
          "basis": "rec_2 declares a correction to art_7 that does not alter its meaning"
        }
      }
    }
  ],
  "notes": ["…"]
}

• kind — unit or atom.
• id — for a unit, the provision <work>/<language>/<eId>; for an atom, the atom id. A provision is the thing that has versions; a node names one version.
• change — added, changed, removed or deprecated.
• classification — substantive or editorial. Never null: a manifest entry with no classification is an entry nobody can act on.
• classification_basis — the evidence that decided it, one of five words: added, removed, deprecated, amendment-editorial (the amending act’s own recital says the change does not alter meaning), whitespace-equal (the texts are one text under spec/whitespace-policy.md), amendment-substantive (an amendment produced this version and declared nothing) or text-differs (the words changed and no amendment record covers them).
• detail.amendment — present when an amendment record decided or corroborated the classification: the instruction that ordered the change, what it did, and the parser’s ruling in the parser’s own words. “Editorial” is a claim, and a reader is entitled to the evidence for it.
• citation — for a unit, the two canon addresses and the window; for an atom, the source, unit_eid, quoted_text and its offsets.
• Every list is sorted (kind, id, change); keys are sorted; two-space indent; one trailing newline.

What is compared. Not the raw file list: consolidation writes a whole new Expression, so a node-level diff would report every provision of an amended act as added. Each release is first reduced to the view a reader has of it — for every provision, the one version in force on that release’s own as_of — and the two views are compared on the units’ own words. A provision that was reissued in a new Expression without a change in words is counted in counts.units.reissued and is not a change entry.

Where the old side comes from. The previous release’s corpus.sqlite, read from disk through SQL. A release built at commit N cannot be inside commit N, so the previous release is never materialised from git; and reading it rather than re-deriving it means the diff says what a consumer of that release would actually have seen.

How a change is classified. “Editorial only when the amendment parser marked the underlying operation editorial, or the normalized texts are equal under the whitespace policy; everything else is substantive.” Both disjuncts are checked, in that order. The first is answerable because the ruling travels with the text: the amending act’s recital is read at consolidation time, written onto the textualMod in the consolidated Expression, carried into corpus.sqlite as a modifications row, and quoted in the entry’s detail.amendment. So the FIXREG Article 7 typo correction is editorial on basis amendment-editorial, while the threshold change — same amending act, no recital declaring it meaning-preserving — is substantive on basis amendment-substantive. Over-reporting substantive remains the safe direction, which is why a unit two operations landed on is editorial only if both were, and why a change no amendment record covers is text-differs rather than an assumption.

Atoms the older release never recorded. An atom the new release carries whose duty was already in force on the older release’s as_of is a backfill: the corpus learned it, the law did not change. Those are counted in counts.atoms.backfilled and are not change entries — reporting one as added would tell openregs impact that an obligation commenced between the two releases, which is false. An atom whose duty commenced after the older release’s date is a real addition and is reported as one. Removals are never suppressed this way.

Reproducing a manifest. openregs diff <regime>@<old>..<regime>@<new> --json computes the same document from the two built releases’ bundles alone, so running it for a tag and its predecessor writes that release’s <tag>.diff.json byte for byte. That equality is a test (tooling/tests/test_diff_manifest.py), and it is what keeps the command from becoming a second opinion about a shipped artifact.

release-meta.yaml and the digest recipes

Records schema_version (the spec version the release was built under), spec_schema_range, bundle_version, builder, licenses, regime, tag, as_of, commit, commit_subject, regime_tree, built_at, counts, diff and artifacts. Its shape is spec/schemas/release-meta.schema.json, so openregs validate release-meta <path> checks a manifest the way any other document is checked.

licenses is the licensing split — {code: Apache-2.0, data: CC-BY-4.0} — stated in the manifest so a downstream packager reads it programmatically; it is the one field a release published to PyPI and npm turns into registry metadata. The values come from openregs.governance.licensing.RELEASE_LICENSES, the single place this repository’s choice is written down, which is also what make check-licenses enforces per file. A release cut before spec 1.2.0 declares no block, and the field is optional in the schema for exactly that reason.

schema_version is read out of spec/VERSION at the commit being built, not from the checkout doing the building, and spec_schema_range is derived from its major. That is what keeps the two version axes independent: bumping spec/ never restamps a release published under an earlier version, so its digests, its signatures and the combination pinning it all stay valid, and rebuilding an old tag on a machine whose spec/ has moved on still reproduces the shipped bytes. The builder refuses outright to build a commit whose spec major it does not implement. See ../schema-compatibility.md.

regime_tree is the sha of the regimes/<regime> subtree at that commit. It is recorded beside the commit sha because a commit sha dies if history is ever rewritten and a tree sha is content-addressed, so a rebuild can still prove it read the same bytes.

Digests:

• a file artifact’s sha256 is the sha256 of its bytes;
• the directory artifact (constants-src) declares every file inside it with its own size and digest, and its sha256 is the sha256 of the listing: <path>\n<sha256>\n per file, paths release-relative and POSIX-separated, byte-sorted. Nothing about the filesystem — mode, mtime, iteration order — enters a digest.

openregs release verify regimes/fixreg/releases/2025.04

recomputes every one of them.

The three standing fixture releases

fixreg@2025.02, fixreg@2025.04 and fixreg@2025.06 are built and committed under regimes/fixreg/releases/; index/combinations/fixture-set.yaml pins the first two there.

They are built from three different commits, and that is the point:

tag	built from	corpus
2025.02	the last commit before the consolidated Expression landed	17 canon files, 32 unit versions, 31 edges, 1 atom — Article 5(3) reads EUR 5000 with an open window
2025.04	the commit that recorded the amendment parser’s ruling in the consolidated text	30 canon files, 55 unit versions, 101 edges, 4 atom versions, 3 modifications — both Expressions, so 2025-01-15 answers EUR 5000 and 2025-06-01 answers EUR 10000
2025.06	the commit that recorded the French modality divergence on FIXREG-Art5.2-Ob1	the same canon as 2025.04 and one atom record richer — its whole diff is one editorial atom change, which is what gives openregs diff a range spanning more than one release move

A pre-amendment release must not know that the amendment exists. The committed canon holds both Expressions and one capture instant (every unit was learned at 2025-04-01T00:00:00Z, because the fixture ingest was one offline run), so no transaction-time filter can hide the amendment from a release built at HEAD. Building from the earlier commit is the only honest way to get a corpus that predates the change — and it is what makes 2025.04.diff.json a real diff rather than a comparison of a corpus with itself.

To rebuild them after the canon moves:

make build TAG=2025.02 COMMIT=<the pre-amendment sha in its release-meta.yaml>
make build TAG=2025.04 COMMIT=<the consolidation sha in its release-meta.yaml>
make build TAG=2025.06 COMMIT=HEAD

They must be built in that order, oldest first: each one’s diff is computed against the previous tag’s bundle on disk.

COMMIT=HEAD only works when HEAD already holds the canon change being released. A release is a function of a commit and cannot be inside the commit it was compiled from, so a change to regimes/<regime>/canon/, atoms/ or graph/ lands in its own commit first; the rebuild then names that sha, and the rebuilt artifacts are committed on top of it. Rebuilding from a commit that is later amended or squashed away leaves release-meta.yaml pinning a sha nobody can git archive, which fails the rebuild test in a fresh clone rather than here.

Next: signing

A built release is unsigned. openregs release sign adds the signature per artifact, the SLSA provenance statement and the transparency-log entries, and openregs verify --release <regime>@<tag> checks all of it offline — see release-sign.md. Its output lives in attestation/ inside the release directory and is deliberately not committed: everything on this page is about the build being reproducible, and a signature is not.