openregs eval

Score a release against the adversarial retrieval eval.

Ask a serving instance built on one release every question in fixtures/eval/questions.jsonl that it may be asked, and score the answers per category: exemption traps (an answer that omits the derogation in force scores zero), as-of dates (the answer must be the law in force on the date asked about) and cross-regime near-misses (a question about a regime this release does not hold must not be answered out of the one it does). Writes the report BESIDE the release and never inside it — a sibling directory <release-dir>.eval/report.json, so scoring dist/ writes dist.eval/report.json — because a release may hold no file its own signature does not cover, and a report written in would be a release verify refuses and release sign would otherwise seal in. With --gate it exits 1 when a category falls below the floor config/eval.yaml states, which is what stops a regressed release from being published.

openregs eval [-h] --release <regime>@<tag> [--path <release-dir>] [--gate]
                     [--questions <path>] [--config <path>] [--json <path>] [--no-report]
                     [--root <path>]

Option
--release <regime>@<tag>	the release to score (required)
--path <release-dir>	score the release in this directory instead of resolving it in the checkout
--gate	exit 1 when a category scores below its floor, naming the regression; without it the score is reported and the exit status is 0
--questions <path>	the question set to score against (default fixtures/eval/questions.jsonl)
--config <path>	read the thresholds here instead of config/eval.yaml
--json <path>	also write the JSON report here; - writes it to stdout
--no-report	do not write the report beside the release, in <release-dir>.eval/
--root <path>	repo root