Runbook: disaster recovery — backing up and restoring the two stateful stores

Audience: whoever operates an OpenRegs deployment. Trigger: the nightly backup job, a lost disk, a bad restore, or the quarterly drill. Time: minutes for the drill, minutes for a real restore. The targets are below and they are not aspirational — the drill in tooling/tests/test_backup_restore.py runs these exact commands on every make test.

Almost nothing in this system needs backing up. A release, the canon, the graph, every artifact under a tag: all of them are functions of a git commit and rebuild byte for byte from it. Push the repository somewhere and they are safe.

Two stores are not.

Store	Default location	Why it cannot be rebuilt
The raw snapshot object store	state/snapshots/ (or an S3-compatible bucket)	It holds the bytes a regulator actually served. Every canonical text traces to a sha256 in SOURCES.lock that names an object here. A regulator that withdraws or silently edits a document does not owe us a copy
The feed service’s subscription database	state/feed/subscriptions.sqlite	Somebody registered a webhook at a moment: an endpoint, a filter and the secret that subscriber verifies deliveries with. Nothing recomputes it

Lose the first and the corpus becomes unprovable — the texts are still there, but nothing backs their provenance. Lose the second and every downstream system stops being told about releases and nobody finds out until somebody asks.

---

Objectives

	Target	What it means here
RTO (recovery-time objective)	60 minutes from declaring the incident to a verified store, per store	The restore itself is a decrypt, a write and a re-hash — minutes for a store of this size. The hour is the budget for finding the archive, finding the passphrase, and running the integrity scan afterwards
RPO (recovery-point objective)	24 hours — at most one day of captures or registrations	Both jobs run nightly. What is lost in the worst case is the captures and registrations made since the last run. Captures are recoverable by re-running openregs ingest against sources that still serve the document; registrations are not, which is why a deployment taking many subscriptions should run the feed job hourly and buy an RPO of 1 hour

The snapshot store is content-addressed and append-only, so a backup is never inconsistent with itself and re-running ingestion after a restore adds what is missing without duplicating what is not. The subscription database is the one that has to be right, because nothing anywhere else knows what it held.

---

The archive format, in one paragraph

openregs backup create writes two files: an encrypted archive and a plaintext hash inventory beside it (<archive>.inventory.json). The archive is a sequence of framed members, each sealed on its own with AES-256-GCM under a key derived from your passphrase with scrypt; the member’s key is bound into its authentication tag, so ciphertext cannot be moved from one key to another. Per member rather than one blob over the whole file, because a restore has to be able to name the damaged object — the difference between knowing which payload to re-fetch and knowing only that the backup is bad. The inventory carries the member keys and their sha256s and no secrets, so the two questions of a recovery — “what should be there” and “does what came back match it” — can both be answered before the archive key is anywhere near the machine.

The archive is deterministic: the same store and the same passphrase produce the same bytes, so two nights’ archives can be compared directly.

The passphrase is the whole of the secret. scrypt’s salt is fixed and published (that is what makes the output deterministic), so a weak passphrase is a weak archive. Generate one with real entropy, keep it in the same place as the release signing key, and keep it somewhere the disaster cannot reach. An archive whose passphrase is gone is a file nobody can read.

---

Backing up

Nightly, from cron. Both jobs are safe to run while the system is serving: the object store is read through the same interface the ingestion plane writes through, and the database copy is taken with SQLite’s own online backup API rather than by copying a WAL-mode file out from under a live writer.

openregs backup create --kind snapshots \
                       --store /var/lib/openregs/snapshots \
                       --out /backups/openregs/snapshots-$(date +%F).orbk \
                       --key-file /etc/openregs/backup.key

openregs backup create --kind feed \
                       --store /var/lib/openregs/subscriptions.sqlite \
                       --out /backups/openregs/feed-$(date +%F).orbk \
                       --key-file /etc/openregs/backup.key

--key-file is a file rather than a flag on purpose: a passphrase passed as an argument ends up in the shell history and in the process table of everything else on the box.

Each run prints the archive, its size, and the inventory with the number of members and the number of bytes covered. Ship both files off-site. An archive without its inventory cannot be restored by these commands at all — that refusal is deliberate, because a restore nobody can check is not a restore.

---

Restoring the snapshot object store

The drill, and the real thing, are the same three commands.

openregs backup restore --archive /backups/openregs/snapshots-2026-07-28.orbk \
                        --into /var/lib/openregs/snapshots \
                        --key-file /etc/openregs/backup.key

openregs verify --snapshots --store /var/lib/openregs/snapshots

What happens, in order:

1. the inventory is read (<archive>.inventory.json unless --inventory says otherwise) — no inventory, no restore;
2. pass one: every member is decrypted, authenticated, re-hashed and matched against the inventory. Nothing is written to the target yet;
3. pass two: the objects are written into the store;
4. pass three: every object is read back out of the store and re-hashed against the inventory, so what the command reports is what the store now holds and not what it was handed.

Then openregs verify --snapshots runs the store’s own integrity scan, which trusts nothing the restore said: it recomputes the digest of every payload against the key it is stored under and re-verifies every WARC capture record against the payload it names. Zero mismatches is the pass condition. Run it after every restore; it is also worth a weekly cron entry of its own.

Restoring into a store that still holds objects is fine and is additive — keys are content hashes, so a re-written object is the same object.

When the restore fails

A damaged archive fails by name and writes nothing:

backup restore: FAIL — sha256/3f/a1/3fa1…: this member does not decrypt — its
  ciphertext, its frame or the passphrase is wrong, and the archive must not be
  restored
backup restore: /backups/…/snapshots-2026-07-28.orbk was not restored into …

The exit code is 1 (a check failed) rather than 2 (nothing to check), and the target directory is left exactly as it was — not partly filled. Then:

1. try the previous night’s archive; the store is append-only, so an older archive is a smaller store and not a wrong one;
2. if every archive names the same member, that object is lost. Find what cited it: grep <digest> regimes/*/SOURCES.lock names the document, and openregs ingest --source <id> re-fetches it if the regulator still serves it;
3. if it is gone for good, the affected canonical text has no provenance any more. Say so — do not quietly re-derive it from a copy nobody can vouch for.

A member that fails to decrypt while the others succeed is corruption. Every member failing is almost always the wrong passphrase or the wrong key file.

---

Restoring the subscription database

openregs backup restore --archive /backups/openregs/feed-2026-07-28.orbk \
                        --into /var/lib/openregs/subscriptions.sqlite \
                        --key-file /etc/openregs/backup.key

Stop the feed service first. Restoring over a database that is already there is refused unless you pass --force: the file on disk may have taken registrations since the backup, and overwriting it would lose them silently. Move it aside instead, restore beside it, and reconcile if you have to.

The restore deletes any -wal and -shm siblings at the target before it writes. That matters: a leftover write-ahead log from the database that used to be there would be replayed on top of the restored one, producing a third database that never existed on any machine.

Then bring the service back and check that a subscriber that registered before the backup is still being delivered to:

openregs feed list  --store /var/lib/openregs/subscriptions.sqlite
openregs feed drain --store /var/lib/openregs/subscriptions.sqlite
openregs feed log   --store /var/lib/openregs/subscriptions.sqlite

feed list shows the registrations that came back (never the secrets). drain makes every attempt that is now due — deliveries owed before the disaster are still owed, because the delivery row is durable — and feed log shows each attempt with its outcome. A delivery that was already exhausted before the backup stays exhausted; re-publish the tag to a re-registered subscriber if it still needs it. See release-feeds.md.

---

The drill

Run it quarterly, and after any change to either store’s layout. It is the same sequence the acceptance suite runs, against a scratch copy — never against the live store.

1. Take a backup of both stores with the commands above.
2. Confirm both outputs exist for each: the archive and its inventory.
3. Restore into a scratch directory, not over the live store.
4. Run openregs verify --snapshots --store <scratch> and require 0 mismatches.
5. Corrupt a byte in a copy of the archive and confirm the restore fails naming the damaged key and writes nothing. A drill that only proves the happy path proves the wrong half.
6. Restore the subscription database into a scratch path, point a feed service at it and publish a release to a test endpoint; confirm the webhook registered before the backup receives the delivery, and that its signature verifies under the secret that came back.
7. Write down how long steps 1-6 took. If it is longer than the RTO above, the RTO is wrong or the deployment is — fix one of them.

tooling/tests/test_backup_restore.py is step 1-6 as code, run on every make test against fixture data and loopback servers. It is not a substitute for running the drill against a real deployment’s volumes, which is where the surprises are.