openregs eval

Score a release against the adversarial retrieval eval.

Ask a serving instance built on one release every question in fixtures/eval/questions.jsonl that it may be asked, and score the answers per category: exemption traps (an answer that omits the derogation in force scores zero), as-of dates (the answer must be the law in force on the date asked about) and cross-regime near-misses (a question about a regime this release does not hold must not be answered out of the one it does). Writes the report into the release’s own directory and, with —gate, exits 1 when a category falls below the floor config/eval.yaml states — which is what stops a regressed release from being published.

openregs eval [-h] --release <regime>@<tag> [--path <release-dir>] [--gate]
                     [--questions <path>] [--config <path>] [--json <path>] [--no-report]
                     [--root <path>]

Option
--release <regime>@<tag>	the release to score (required)
--path <release-dir>	score the release in this directory instead of resolving it in the checkout
--gate	exit 1 when a category scores below its floor, naming the regression; without it the score is reported and the exit status is 0
--questions <path>	the question set to score against (default fixtures/eval/questions.jsonl)
--config <path>	read the thresholds here instead of config/eval.yaml
--json <path>	also write the JSON report here; - writes it to stdout
--no-report	do not write the report into the release directory
--root <path>	repo root