Runbook — observability

What OpenRegs emits about itself: one log schema, one metric format, one request id. This is the operator’s page. Read it before wiring a dashboard, an alert or a log pipeline, because everything below is a contract and everything not below is not.

deploy.md covers bringing the stack up; this page covers watching it once it is up.

---

1. The two streams

openregs serve writes to both of its streams and they are not the same thing.

Stream	Who reads it	Shape
stdout	a person, or a caller that needs the bound port	prose, one line per event
stderr	a log pipeline	JSON, one record per line, one schema

Every line on stderr is a record against spec/schemas/log-record.schema.json, including the startup announcements. The prose on stdout exists because --port 0 lets the kernel choose the port and the announcement is the only way anybody finds out which one:

serve: listening on http://127.0.0.1:53422

Parse that line if you are scripting a start-up. Everything else you want, read off the log or the metrics.

openregs serve --mcp has no stdout to spare — stdout is the MCP protocol — so that door says everything it says as log records on stderr, under the component serve.mcp.

2. The log record

Six keys, always all six. Anything specific to the event goes inside fields, never interpolated into message, so a search for a message finds every occurrence of it.

{"component": "serve", "fields": {"duration_ms": 4.185, "method": "POST", "principal": "anonymous", "route": "/v1/answer", "status": 200}, "level": "info", "message": "POST /v1/answer -> 200", "request_id": "6fbd96b6-000001", "timestamp": "2026-07-29T21:51:13.861Z"}

• timestamp — RFC 3339, UTC, to the millisecond, trailing Z. Always that precision, so records from two hosts sort against each other as plain strings.
• level — debug, info, warning, error. Four, and no others.
• component — dotted from the outside in: serve, serve.api, serve.mcp, ingest, ingest.gates, e2e.
• request_id — the request this event belongs to, or null for an event that belongs to none: a process starting, a release loading, a pipeline run. null is a statement, not an omission — do not filter it out.
• message — one short sentence for a person, stable across occurrences.
• fields — always present, possibly empty.

Check a record by hand:

$ openregs validate log-record one-record.json

3. Request ids

The server returns the id of every request on the X-Request-Id response header — success, error and refusal alike — and puts the same value in every error body, {"error": {"code", "message", "request_id"}}. A caller holding a response can therefore find every line it caused with one grep:

$ grep '"request_id": "6fbd96b6-000001"' openregs.log

The id is bound to the request’s context, not passed down by hand, so log records written by layers below the gateway carry it too without being handed it.

A minted id is this process’s nonce and a counter — 6fbd96b6-000001 — so ids from two instances never collide and the counter says how many requests an instance has answered.

Send X-Request-Id on the way in to propagate an id you already have. It is adopted only if it is at most 64 characters of A-Za-z0-9.:_-; anything else is discarded silently and a fresh id is minted, because a header written into a log line is an injection vector and that alphabet is what closes it.

The id is not in a successful response body. Two instances started from one release answer byte-identically, and an id in the body would end that.

4. Metrics

The server

GET /metrics — Prometheus text exposition format (text/plain; version=0.0.4). It is answered from the moment the socket is bound, because boot is exactly when an operator wants numbers, and it is never rate limited. It is behind the same bearer token as every other route: a deployment that requires a token to be asked a question requires one to be measured. Point a scraper at it with bearer_token when auth.anonymous is false.

Metric	Type	Labels
openregs_http_requests_total	counter	route, method, status
openregs_http_request_duration_seconds	histogram	route
openregs_queries_total	counter	release, route
openregs_rate_limit_rejections_total	counter	principal

The route label is the set of routes this deployment serves; a path it does not serve is route="other", because a label value a caller chooses is a label value a caller can make unbounded. Latency buckets are chosen around config/perf-budgets.yaml, so “did we meet the budget” is a bucket count rather than an interpolation.

The pipeline

A pipeline run is a command, not a process with a port: by the time anybody asks how it went there is nothing to scrape. So openregs ingest and openregs replay write the same exposition format to a file, and a deployment points a textfile collector at the directory.

$ openregs ingest --source eur-lex --offline-fixtures --quarantine work/quarantine
...
ingest: metrics written to state/metrics/ingest.prom

The path is <state-dir>/metrics/<command>.prom, and <state-dir> is --state-dir, or OPENREGS_STATE_DIR, or state/ under the checkout — so point the collector at whichever one that deployment’s ingestion runs under. A run that cannot write its metrics warns and still succeeds: a corpus that was ingested correctly was ingested correctly whether or not the observer could take notes.

Metric	Type	Labels
openregs_documents_ingested_total	counter	source
openregs_gate_failures_total	counter	gate
openregs_quarantine_depth	gauge	—
openregs_replay_captures_total	counter	source
openregs_replay_progress	gauge	source

gate is one of the four validation gates: xsd-validity, round-trip-integrity, structural-coverage, reference-resolution.

openregs_quarantine_depth is the quarantine this process last wrote to, so it reports the backlog including items an earlier run left. openregs_replay_progress is a share from 0 to 1, which is what makes a stalled replay visible as a number that stopped moving rather than as an absence.

5. Rate limiting

Set in config/serving.yaml, not on the command line — the deployment policy is a file somebody chose, not a flag somebody remembered:

rate_limit:
  default_per_minute: 60
  anonymous_per_minute: 30

A rolling minute, not a calendar one, because a fixed window lets a caller send two full allowances back to back across the boundary. Over the limit is 429 with Retry-After and the ordinary error body. A refused request does not consume budget. A named token may override its own allowance with rate_limit_per_minute.

/healthz and /readyz are exempt from both the token and the limit: a readiness probe refused with 429 reads as an instance that is down and gets the process killed. /metrics is exempt from the limit but not from the token — a scrape refused with 429 takes away the only view of why the budget is being spent, at the moment it is being spent.

Every 429 increments openregs_rate_limit_rejections_total labelled by the principal that earned it, so “who is hot” is a query rather than an investigation.

6. What to alert on

Symptom	Query	What it means
The corpus stopped ingesting	increase(openregs_documents_ingested_total[1d]) == 0	a poll is failing, or the source stopped publishing
A mapper regressed	increase(openregs_gate_failures_total[1h]) > 0	a document was refused; the gate label says which check
The backlog is growing	openregs_quarantine_depth	documents are waiting for a person — see quarantine-triage
A replay stalled	openregs_replay_progress below 1 and unchanged over an hour	the replay stopped making progress against a source
The API is slow	histogram_quantile(0.95, rate(openregs_http_request_duration_seconds_bucket[5m]))	compare against config/perf-budgets.yaml
Callers are being throttled	rate(openregs_rate_limit_rejections_total[5m])	the budget is too tight, or one principal is hot
Errors	rate(openregs_http_requests_total{status=~"5.."}[5m])	a 5xx is a bug; the request id in the log names the request
An instance never became ready	/readyz failing while /healthz passes	the release did not verify; the startup log says which check

7. When something is wrong

1. Get the X-Request-Id from the response the caller saw — or, on an error, read request_id out of the error body.
2. grep the log for it. Every record the request produced carries it, including the ones written below the gateway; the gateway’s own record carries the route, the status, the duration and the principal.
3. If the answer looks wrong rather than failed, the release the instance loaded is what to reproduce against: openregs serve --release <tag> and ask again.
4. If a document is missing from the corpus, look at openregs_documents_ingested_total for its source and openregs_gate_failures_total for the gate that refused it. quarantine-triage is the next page.