Runbook — capturing a real act from EUR-Lex

Audience: whoever is bringing a real EU regime online. Trigger: you are about to point the eur-lex adapter at the Publications Office for the first time, or you are debugging a capture that came back wrong. Time: minutes for one act.

Everything else in this repository is exercised against fixtures, and must be: no test, CI job or fixture run may contact a regulator. This page is the exception that proves the rule — it is about production ingestion, the one place a real endpoint is contacted. Read it before you run it, because two of the four steps look like they are working when they are not.

---

0. What a Union act actually is, on the wire

Three facts decide everything below, and none of them is guessable from the adapter’s own vocabulary.

An act is addressed in CELLAR, not on the website. The URL that looks obvious — eur-lex.europa.eu/legal-content/EN/TXT/XML/?uri=CELEX:32024R0575 — answers 200 text/xml with about a megabyte of <NOTICE>: the bibliographic record, not the law. The act lives at publications.europa.eu/resource/celex/<CELEX>, which serves whichever rendition the request asks for.

Formex arrives as a zip. Accept: application/zip;mtype=fmx4 with Accept-Language: eng gets a 303 to a manifestation, and the manifestation is a set of files: the act, one document per annex, the document wrapper and the issue’s table of contents. Ask for anything else and you get something that is not Formex at all.

The archive says which of its files are the act. The <DOC> wrapper’s FMX block names the body (DOC.MAIN.PUB) and each printed section after it (DOC.SUB.PUB TYPE="ANNEX", numbered NO.SEQ="0001.0001"). That is what the mapper assembles from — not filenames and never the order of the zip, which no packer promises. The body and the annexes become one Akoma Ntoso act, the annexes under attachments; the wrapper and the issue’s table of contents are named in the mapping report and are not part of the act.

A published act does not carry its own identity. There is no <NO.CELEX> in the Formex the Journal prints — the CELEX number is assigned by the Publications Office and lives in CELLAR. There is no entry-into-force date either: the act says “shall enter into force on the day following that of its publication” in its own text and the resulting date is CELLAR’s resource_legal_date_entry-into-force. Both have to come from the capture, not from the file.

1. Bound the discovery

A CELLAR query with no bound answers with the Official Journal. The bound is a regime’s declared instruments plus the directory branches its corpus covers, and both are configuration:

discovery:
  regimes: [<your regime>]        # regimes/<name>/SCOPE.yaml declares the CELEX numbers
  directory_codes: ["04.10.30.10"]

A directory code is written the way the EUR-Lex directory prints it. CELLAR binds the same branch as an authority-table URI (.../authority/dir-eu-legal-act/04103010); the adapter matches both, so the configuration does not have to know which.

Find the branch an act sits in before you scope a source to it:

PREFIX cdm: <http://publications.europa.eu/ontology/cdm#>
SELECT ?dc ?inforce WHERE {
  ?work cdm:resource_legal_id_celex "32024R0575"^^<http://www.w3.org/2001/XMLSchema#string> .
  ?work cdm:resource_legal_is_about_concept_directory-code ?dc .
  OPTIONAL { ?work cdm:resource_legal_date_entry-into-force ?inforce }
}

Sent to https://publications.europa.eu/webapi/rdf/sparql with format=application/sparql-results+json. Note the ^^xsd:string: the endpoint does not treat a plain literal and an xsd:string as the same term, which is why the adapter’s own query compares STR(?celex) and never a bare literal.

2. Run the capture

openregs ingest --source eur-lex --state-dir $WORK/state --lock $WORK/SOURCES.lock

Without --offline-fixtures this contacts the Publications Office. It is rate limited to one request per second per host, identifies itself with the contact_url from config/sources.yaml, sends Accept-Encoding: identity so the digest pins the bytes as served, and sends the conditional validators from the last run — a second run of an unchanged act costs a 304 and transfers nothing.

What it writes, in this order and never another: the payload under its own sha256, a WARC record of the exchange, a SOURCES.lock entry pinning it, then the queue entry is dropped.

3. Check what you got

openregs normalize $WORK/state/snapshots/sha256/<..>/<digest> --format formex

Four things are worth reading before the corpus believes any of it:

• the SOURCES.lock entry — uri, retrieved_at, sha256, storage_path. Recompute the digest over the stored bytes; it is the whole provenance chain.
• the WARC request record — it preserves the Accept that decided which rendition you captured. A capture whose request headers you cannot see is a capture you cannot reproduce.
• the coverage report — every element of the act the mapper walked past, by path. Only the act: its body and the sections the archive’s index attaches to it. What that number is not measured over is as much the point — the issue’s table of contents lists other people’s regulations, and an act whose coverage fell because the Journal printed somebody else beside it would be a number nobody could act on.
• the manifestation list — one line per file the capture held and what each one was to the act. Read it against the coverage report: an act that is missing an annex and an act whose capture never carried one look identical everywhere else, and only this tells them apart.
• the gate verdict — all four gates, and a document that fails any of them is quarantined rather than written to canon. That is the pipeline refusing to call an incomplete act complete, and the document waits there until a person decides; see quarantine-triage.md.

4. Footnotes, and where they go

Nearly every published act footnotes the Official Journal reference of every act its preamble names — the first citation, the first recital, sometimes the title — so how NOTE is handled decides whether any real act can be captured. Both halves matter and they pull in opposite directions:

• the note’s elements have to be mapped, or the act never reaches 100% structural coverage and never leaves quarantine;
• the note’s words must not become part of the sentence they interrupt. The Journal prints them at the foot of the page. Splicing them in gives text nobody enacted — “…the common fisheries policyOJ L 343, 22.12.2009, p. 1., and in particular Article 36(2) thereof” — and moves every character after the marker, which silently invalidates every provenance offset into that unit.

What the mapper does is emit an <authorialNote> attached to the point in the text where the marker stood, never inserted into it. The note’s paragraphs live inside that element; unit_text — the string a quoted_text slices byte for byte — reads straight past it; and the round-trip gate sees the note’s text as a block of its own, which is exactly how a walk of the source meets it. Adding a footnote to a sentence therefore moves no offset in the text around it.

Two things follow that are worth knowing before reading the output:

• No marker is written. A published act’s Formex says the notes are numbered in arabic figures (NUMBERING="ARAB") and that the sequence continues from outside the file (NUMBERING.CONTINUED="YES"). It never says what the page printed. The attribute is optional in Akoma Ntoso and is left off rather than filled with a number derived from the note’s position, which would be this project stating something about the page it cannot know.
• A note’s eId is note_N, numbered across the emitted document. It is not a reference target and no atom cites it; it exists because Akoma Ntoso requires an eId on everything in the content of a document, and because a renderer needs something to key the note it prints at the foot of the page to the place it belongs. Addressing one with unit_text raises: a footnote is not a unit and has no provenance span.

A footnote in a heading — an article’s TI.ART or STI.ART, an annex’s TITLE — is still reported unmapped. A heading is a label a publisher attaches to a provision and is a plain string in the model, with nowhere to hang a note; no act this project has captured carries one there.

5. Known limits, as of the first real capture

Read these before scoping a corpus to this adapter. They are not bugs in the sense of “the code does not do what it says”; they are the places where what the code does is not yet enough for law.

• The Expression is dated from the document date. No published Formex carries entry into force, so the mapper falls back to the date the act bears and writes name="docDate" on the FRBRdate to say so. The Expression IRI is therefore a few days early, and closing this needs CELLAR’s in-force date carried from discovery through the snapshot into normalize() — provenance the pure side does not have today.
• A note’s text is not scanned for internal cross-references. “Article 1 thereof” inside a footnote belongs to the act the footnote cites, not to the act carrying it, and telling one from the other means reading the sentence around the number. The preamble is left unscanned for the same reason.
• The cursor advances on work_date_document, which is the day the act bears, not the day the Journal published it. Two acts signed in one order and published in another can leave a gap, so a first real deployment should keep the cursor behind the publication frontier rather than trusting it exactly.