Runbook: deploying the hosted stack

Audience: whoever runs an OpenRegs instance — hosted, or self-hosted inside a bank’s own network. Trigger: a first deployment, a version bump, or a release roll-forward. Time: minutes. The image build dominates.

Two files are the whole deployment:

File	What it is
deploy/Dockerfile	The server image. Multi-stage, non-root, base images pinned by digest, labelled with the commit that built it
deploy/docker-compose.yaml	The composition: three services and two named volumes

tooling/tests/test_deploy_compose.py asserts every property claimed below against those two files on every make test, so this page and the deployment cannot drift apart silently.

---

The three services

Service	What it runs	Port
server	openregs serve — REST and MCP over one pinned release	8080
objectstore	MinIO, the S3-compatible bucket the snapshot archive sits on	9000 (console 9001)
feed	openregs feed serve — release feeds and webhook delivery	8787

server and feed are the same image with a different command, because they are the same codebase. A second image would be a second thing to build, sign, scan and keep in step.

What is state and what is not

Almost nothing here is state. The canon, the atoms, the graph and every built release are functions of a git commit, so the checkout is bind-mounted read-only at /srv/openregs: a release the server could write to is a release nobody can verify.

Two things are state, and they are the two docs/runbooks/disaster-recovery.md backs up. Both sit on named volumes, so docker compose down keeps them and only docker compose down -v throws them away:

Volume	Mounted at	Holds
objectstore-data	objectstore:/data	The bytes a regulator actually served
feed-state	feed:/var/lib/openregs/feed	subscriptions.sqlite, including each subscriber’s secret

---

Build the image

make image                     # tags openregs/server:dev
make image IMAGE=ghcr.io/acme/openregs-server:2025.04

The target derives both build arguments from the commit, never from the clock:

docker build --file deploy/Dockerfile --tag openregs/server:dev \
  --build-arg GIT_COMMIT=$(git rev-parse HEAD) \
  --build-arg BUILD_DATE=$(git show -s --format=%cI HEAD) .

The Dockerfile refuses to build without them. An image whose org.opencontainers.image.revision is blank is an image nobody can trace back to a commit, and the point of the label is that somebody can. Because both are functions of the commit, two builds of one commit carry identical metadata.

Confirm what an image says about itself:

docker image inspect openregs/server:dev \
  --format '{{index .Config.Labels "org.opencontainers.image.revision"}} {{.Config.User}}'

That must print the commit sha and openregs — never an empty user, which means root.

What the build needs to reach

Two things, and only two: the registry holding the base image python:3.12-slim-bookworm (pinned by digest in deploy/Dockerfile) and PyPI. uv itself comes from PyPI too — deploy/uv-requirements.txt pins it by version and by the sha256 of every wheel it may resolve to, installed with --require-hashes, so the builder’s own tooling is verified rather than trusted. That is deliberate: one registry and one package index, not two of each.

A builder behind a TLS-terminating egress proxy must trust that proxy’s CA or it cannot reach PyPI at all. Pass the bundle as the optional build_ca_bundle build secret:

make image                                    # picks up $SSL_CERT_FILE if set
OPENREGS_BUILD_CA_BUNDLE=/path/to/ca-bundle.crt \
  docker compose --file deploy/docker-compose.yaml build

It is a secret mount, so no layer of the image holds a CA, and it is optional: where the variable is unset the secret is /dev/null, the build sees an empty file and nothing changes. A builder that cannot reach the base image’s own registry can point the daemon at a pull-through mirror (/etc/docker/daemon.json, "registry-mirrors"); the digest pin still decides which bytes are accepted, so a mirror can serve the image but cannot change it.

Bring the stack up

GIT_COMMIT=$(git rev-parse HEAD) BUILD_DATE=$(git show -s --format=%cI HEAD) \
  docker compose --file deploy/docker-compose.yaml up --build --detach

Both processes are told where the checkout is — OPENREGS_ROOT and OPENREGS_SPEC_DIR point at the read-only mount at /srv/openregs. The image holds the installed package and nothing else, and spec_dir() finds spec/ by walking up from the module, which inside site-packages finds nothing: a container that is not told dies on the first schema it needs.

Then wait for readiness rather than liveness:

curl -fsS http://localhost:8080/readyz     # 200 once a verified release is loaded
curl -fsS http://localhost:8080/healthz    # 200 as soon as the process is up

The two probes answer two different questions and are never interchangeable:

Probe	Question	200 when	503 when
GET /healthz	Is this process alive?	From the moment the socket is bound, always	never — a process that cannot answer this is a process to restart
GET /readyz	May I send it traffic?	A verified release is loaded and its index is warm	during boot, and again from the moment SIGTERM arrives

/healthz is what the container runtime asks (it is the image’s HEALTHCHECK) and it must never require a token or a release: a liveness check that fails while a release is loading gets the container killed for being slow. /readyz is what an orchestrator asks before it routes traffic, and it is what deploy/smoke.py and make bench wait on.

The socket binds before the release is verified, which is what makes that table true. It does not weaken the gate: until the release has passed verification there is no application behind the listener, so /readyz and every other route answer 503 not_ready and no regulation is served at all. A release that fails a check is never loaded — the process closes the listener, prints the failing checks and exits 2.

Neither probe is rate-limited and neither needs a bearer token.

GET /metrics is the third operational route — Prometheus exposition, never rate-limited, no corpus content, but behind the same bearer token as every other route — and the container’s stderr is a stream of JSON log records, one per line, one schema. Point a scraper at the first and a log pipeline at the second; both are documented in observability.md.

Environment variables the composition reads, all with defaults:

Variable	Default	Meaning
OPENREGS_RELEASE	fixreg@2025.04	Which release the server serves
OPENREGS_AS_OF	2025-06-01	The date a request that names none is answered for
OPENREGS_SERVER_PORT	8080	Published server port
OPENREGS_FEED_PORT	8787	Published feed port
OPENREGS_FEED_BASE_URL	http://localhost:8787	What links in the feeds are written against
OPENREGS_OBJECTSTORE_USER / _PASSWORD	openregs / openregs-dev-secret	MinIO root credentials — change both outside development
OPENREGS_IMAGE	openregs/server:dev	The image tag to build and run

---

The serving policy

Who may ask, how often, from which browser origin and how long a shutdown waits is config/serving.yaml, read from the bind-mounted checkout before the socket is bound. openregs serve --config <path> reads another file instead. A policy that does not parse is a server that does not start: exit 2, before a port exists.

schema_version: 1
auth:
  anonymous: false            # no default — a public instance is always a decision
  tokens:
    - name: bank-ci
      sha256: <printf %s "$TOKEN" | sha256sum>
      rate_limit_per_minute: 120
rate_limit:
  default_per_minute: 60
  anonymous_per_minute: 30
cors:
  allowed_origins: ["https://console.example.test"]
  allow_credentials: false
  max_age_seconds: 600
shutdown:
  drain_seconds: 30

• Authentication. Authorization: Bearer <token>. A token is stored as the hex sha256 of its value, so this file carries no secret and is safe to commit and to bake into an image; token: accepts a plaintext for development and starts the server with a warning naming the file. A missing or unrecognised token is 401 with WWW-Authenticate: Bearer. auth.anonymous has no default. A file that omits it is refused, and so is false with no tokens — that instance could never answer anybody. The checkout ships anonymous: true because the only release it holds is the FIXREG fixture; a hosted instance serving real regulation sets it to false.
• Rate limiting. Per token, over a rolling minute rather than a calendar one. Over the limit is 429 with Retry-After in whole seconds. A refused request does not consume budget.
• CORS. Empty by default: no browser origin is allowed. Name origins in full — scheme, host and port. ["*"] allows any and is refused alongside allow_credentials: true, which a browser refuses anyway. A preflight (OPTIONS) needs no token.
• Errors. Every failure path, from any layer, is {"error": {"code", "message", "request_id"}}. The same request_id is on the X-Request-Id response header and in the server’s log line for that request, so a user report of “I got a 500” is answerable. A caller may pin its own id by sending X-Request-Id; anything that is not 64 characters of [A-Za-z0-9.:_-] is discarded and a fresh one minted.
• Graceful shutdown. On SIGTERM /readyz immediately goes 503, the listener stops accepting (new connections are refused at the TCP level), and the requests already in flight are given drain_seconds to finish — to the last byte on the wire, not merely to the last line of handler code. Then the process exits 0.

Watch it: the log, the metrics, the request id

The log is standard error, one JSON object per line. Standard output is the process talking to whoever started it (serve: bound …, serve: listening on …); standard error is the log, and every line of it is a record in one schema — spec/schemas/log-record.schema.json:

{"component": "serve", "fields": {"duration_ms": 4.185, "method": "POST",
 "principal": "anonymous", "route": "/v1/answer", "status": 200},
 "level": "info", "message": "POST /v1/answer -> 200",
 "request_id": "6fbd96b6-000001", "timestamp": "2026-07-29T21:51:13.861Z"}

Six keys, always all six. Anything specific to the event is in fields, never interpolated into message, so a search for a message finds every occurrence of it. request_id is null for an event that belongs to no request — the process starting, a release loading — which is a statement rather than an omission. Validate a shipped line with openregs validate log-record <file.json>.

One id traces one request. The same request_id is on the X-Request-Id response header, inside every error body, and on every log line the request produced — including the ones written by layers below the gateway, which are never handed it: it is bound to the request’s context. It is deliberately not in a successful response body, because that document is a function of the release and the question and two instances of one release must render it byte for byte.

Metrics are GET /metrics, in the Prometheus text exposition format (text/plain; version=0.0.4). It is answered from the moment the socket is bound — boot is exactly when an operator wants numbers — and is never rate limited, but it is behind the same authentication as every other route: a deployment that requires a token to be asked a question requires one to be measured. Point a scraper at it with bearer_token when auth.anonymous is false.

metric	type	labels	what it says
openregs_http_requests_total	counter	route, method, status	requests answered
openregs_http_request_duration_seconds	histogram	route	seconds spent deciding a response
openregs_queries_total	counter	release, route	queries answered against a release
openregs_rate_limit_rejections_total	counter	principal	429s, by the token that earned them
openregs_documents_ingested_total	counter	source	documents whose bytes reached the store
openregs_gate_failures_total	counter	gate	validation gate failures, one per failed gate
openregs_quarantine_depth	gauge	—	documents waiting for a person
openregs_replay_progress	gauge	source	share of a source’s captures re-run, 0 to 1

The route label is the set of routes this deployment serves; a path it does not serve is route="other", because a label value a caller chooses is a label value a caller can make unbounded.

The pipeline has no endpoint to scrape — openregs ingest and openregs replay start, work and exit — so they write the same exposition to <state-dir>/metrics/<command>.prom, which is what a node exporter’s textfile collector reads. <state-dir> is --state-dir, or OPENREGS_STATE_DIR, or state/ under the checkout — so point the collector at whichever one that deployment’s ingestion runs under.

Smoke-test it

make smoke

That builds the image, brings the composition up, waits for /readyz inside the cold-start budget, POSTs the first query of fixtures/queries/replay-50.jsonl to /v1/answer and checks the atom ids the fixture expects come back, then writes an object into objectstore-data, runs docker compose down, brings it back up and reads the object back byte for byte. Exit codes: 0 success, 1 the deployment is wrong, 2 it could not be run, 3 no image registry is reachable.

Roll a new release

The release is not in the image, so rolling one forward is not a rebuild:

OPENREGS_RELEASE=fixreg@2025.06 docker compose --file deploy/docker-compose.yaml up --detach server

Take it down

docker compose --file deploy/docker-compose.yaml down      # keeps both volumes
docker compose --file deploy/docker-compose.yaml down -v   # DELETES the regulator bytes

Never run the second one against an instance whose object store has not been backed up. See docs/runbooks/disaster-recovery.md.