Runbook — the release feed service

Audience: whoever runs the feed service, and whoever operates a system that subscribes to it. Trigger: a release was built and downstream needs to hear about it, a subscriber says it never got a payload, or a subscription database needs backing up or restoring. Time: minutes.

The feed service is how a downstream system learns about a release without watching a git repository. It reads a built release’s diff manifest, keeps its atom changes as feed entries, serves them as a JSON feed and an RSS feed per filter, and POSTs the ones each subscription asked for to that subscriber’s endpoint, signed and retried until they land.

It is outbound. The feeds a watcher adapter polls — a regulator’s own change notifications — are the inbound ones and belong to openregs ingest; nothing here ever contacts a regulator.

---

Running it

openregs feed serve --store /var/lib/openregs/subscriptions.sqlite \
                    --host 0.0.0.0 --port 8787 \
                    --base-url https://feeds.example.com

--base-url is what the links inside the feeds are written against, so set it to whatever a reader sees. Without it the service writes links against the address it bound to, which is right for a laptop and wrong behind a proxy.

The service does no authentication and terminates no TLS. It publishes material that is already public and holds exactly one thing that is not — the subscription secrets, which it never returns in any response. Put it behind whatever does the rest.

Route	What it is
POST /subscriptions	register an endpoint, its secret and its filter
GET /subscriptions	what is registered; secrets are never included
DELETE /subscriptions/<id>	stop sending
POST /publish	announce a built release ({"regime": …, "tag": …})
POST /deliveries/retry	one retry pass over everything now due
GET /deliveries	what is owed, and what state each delivery is in
GET /log	the service log: every attempt at every delivery
GET /feed.json, GET /feed.xml	the feeds, filtered by the query string
GET /releases/<regime>/<tag>/diff.json	the release’s own diff manifest

Filtering

Four dimensions, named the same way in a subscription’s filter and in a feed request’s query string: jurisdiction, regime, topic, entity_type. A dimension left unnamed matches everything; a dimension named matches when the change carries any of the values (?regime=fixreg&entity_type=operator).

Entity types follow the taxonomy: small_operator receives changes to obligations declared on operator, because a small operator is an operator. It does not work the other way round.

topic matches nothing today. No atom in spec/schemas/atom.schema.json carries a subject vocabulary, so no entry declares a topic. The dimension is wired and will start matching the day atoms carry tags:; until then a subscription filtered on a topic is a subscription that hears nothing, which is deliberate — the alternative, treating an unknown dimension as “everything”, would silently widen every such subscription later.

Publishing a release

openregs feed publish --regime fixreg --tag 2025.04 \
                      --store /var/lib/openregs/subscriptions.sqlite

The release must already be built under regimes/<regime>/releases/<tag>/. The entries are the manifest’s atom changes, copied verbatim — including their substantive/editorial classification and its basis — plus the entity types and jurisdiction resolved out of that release’s own corpus.sqlite. Unit changes are not fanned out; they are one click away in the diff each entry links to.

publish exits 1 while a delivery is still owed. That is not a failure of the publication — the release was announced and every payload is durable — it is the signal that a drain is still to come.

Publishing the same tag twice is a no-op: a delivery’s id is derived from the subscription and the payload bytes, so a re-run does not re-send anything.

Signatures — what a subscriber must check

Every POST carries:

Header	Value
X-OpenRegs-Signature	sha256=<hex>, HMAC-SHA256 of the request body bytes under the subscription secret
X-OpenRegs-Delivery	the delivery id — stable across retries, so deduplicate on it
X-OpenRegs-Subscription	which subscription this answers
X-OpenRegs-Event	release.published

Verify over the raw bytes as they arrived, before parsing them. Re-serializing the JSON and signing that will not match, and is not meant to.

Delivery, retries and the log

Delivery is at-least-once. The row is durable before the first POST, so a service that dies mid-flight still owes the payload when it comes back. A failed attempt — a refused connection, a timeout, or any non-2xx — sets a next-attempt time and the delivery stays pending:

Attempt	Sent after
1	immediately, at publication
2	1 minute
3	5 minutes
4	30 minutes
5	2 hours

After the fifth the delivery is exhausted: it stops being retried and stays in the log saying so. A retry re-sends the same bytes with the same signature and the same id — a retry is the same delivery, not a new rendering of it.

Run the retry pass from a cron entry, or by hand:

openregs feed drain --store /var/lib/openregs/subscriptions.sqlite
openregs feed log   --store /var/lib/openregs/subscriptions.sqlite

drain exits 1 while anything is still pending.

“A subscriber says it never got the payload”

1. openregs feed log --store <db> — find the delivery and read its attempts. A failed line names the endpoint and the error verbatim.
2. If it is pending, fix the subscriber and run openregs feed drain.
3. If it is exhausted, the five attempts are spent. Re-register or re-enable the subscriber and re-publish the tag: the delivery id is a function of the subscription and the payload, so a new subscription gets a new delivery while the old one stays in the log as the record of what happened.
4. If nothing was ever created for that subscriber, its filter did not match. The publication response and GET /log both name the subscriptions that were silent; compare the filter against the entry’s facets in GET /feed.json.

Backing it up

Everything else this system produces is a function of a git commit and can be rebuilt. The subscription database cannot: it is somebody’s registration, made at a moment. One SQLite file holds all of it —

state/feed/subscriptions.sqlite      (the default; --store overrides it)

— with the subscriptions, the publications and their entries, the deliveries and the attempt log. state/ is gitignored on purpose: a subscription is nobody’s reviewed data and must never reach the canon. Back the file up whole (it is opened in WAL mode, so copy the -wal and -shm siblings with it, or use sqlite3 <db> ".backup"), and treat it as a credential store: the secret column is the key each subscriber verifies with.

What is deterministic and what is not

Everything a feed publishes is a function of the release: an entry’s date is the release’s built_at, which is its commit’s committer date, so two services announcing one release render identical feeds and identical payload bytes.

A delivery attempt is the exception and is stamped with a real moment, because no commit can say when a subscriber was tried. --now pins it, which is how the backoff schedule is tested without sleeping through it.