Runbook: the golden-fixture conformance suite

What to do when make conformance goes red, and what blessing means.

What the suite is

One blessed Akoma Ntoso output per captured document per adapter, committed under fixtures/goldens/. make conformance runs each shipped adapter’s pure side — parse() then normalize() — over every document the offline catalogues resolve, and compares the bytes it gets against the blessed copy. Any difference fails, naming the golden file.

It exists for one failure the rest of the pipeline cannot see: a mapper change that alters the output of every document at once. The validation gates check one document against itself, the purity sweep checks one mapper against itself, and make check-canon reads a canon nobody has re-ingested. None of them notices a corpus-wide rewrite; on fifty thousand canonical units, neither does a reviewer. The blessed files are the approval record, and moving one is a deliberate act.

make conformance      # compare
make bless-goldens    # approve what the mappers produce now

When it goes red

Read which goldens are named. There are four kinds of failure:

kind	means
golden	fresh output differs from the blessed copy
unblessed	a captured document with no blessed output on record
orphan	a blessed file no captured document produces
mapper	a document did not map at all, or mapped differently twice
exclusion	a document declared as having no blessed output no longer fits that declaration

Then decide which of two situations you are in.

You changed a mapper on purpose. The named goldens are the corpus change your edit causes. Read the diff excerpts the suite prints, satisfy yourself that the change is the one you meant, and run make bless-goldens. It rewrites the blessed files and writes fixtures/goldens/BLESS-SUMMARY.md — commit that summary with them. It is what a reviewer reads to see what moved without diffing ten thousand lines of XML.

You did not change a mapper. Then something changed underneath one, and blessing would launder it into the corpus. Find the cause first: a dependency bump that altered XML serialization, a captured fixture edited in place, an adapter that started reading something outside its snapshot. git log -p on tooling/openregs/akn/ is usually the shortest route.

A mapper failure saying two runs produced different bytes is never blessed. It means a transform stopped being a function of its input — a clock read, a random value, a dictionary whose order leaked into the output — and the corpus cannot be pinned to a transform that answers differently each time. Fix the mapper.

What blessing does not do

It does not re-ingest anything. The committed canon under regimes/*/canon/ is unchanged by a blessing, and a mapper change only reaches the corpus through openregs replay (see docs/ARCHITECTURE.md). Blessing records what the mapper would now produce; replay is what makes the canon agree with it.

Adding an adapter

Nothing to add here by hand. The inventory is derived: an adapter is swept because it is in BUILTIN_ADAPTER_MODULES, and its documents are the instruments in the offline catalogue its config/sources.yaml entry names. A shipped adapter with no catalogue fails the suite rather than being skipped. Add the adapter, add its catalogue, run make bless-goldens, commit the goldens and the summary.

A captured document that legitimately has no blessed output — a change notice that carries no legal text, a scan its adapter refuses — is declared in openregs.conformance.NO_GOLDEN with a reason. Every declaration is re-proved on each run, so it cannot quietly become a skip list.

Exercising the suite

fixtures/patches/mapper-whitespace.patch is the mapper change the suite is tested with: the shared serializer’s indent width, which every mapper serializes through and no gate can see. Apply it, watch every blessed output get named, revert it with git apply -R. Nothing in this repository is committed in the patched state.