One golden set, two production brains, scored nightly.

Twelve incidents, four from real meridian-ops runs and eight authored to be hostile, each labelled with bounds rather than answers: a severity floor, permitted categories, and whether the security floor must hold. Both brains that run in production take the same set: Claude inside the fleet, and gpt-4.1-mini behind an Azure Function. rendering…

the run
every case, expected against decided
S C F = the three gate checks per case: severity at or above the floor, category in the permitted set, security floor held where it must. A case can also be marked context-dependent: scored, reported, outside the gate.
what the harness caught before it ever scored a model

Its unit tests found a live production bug pre-run: the security regex ended in a trailing word boundary that never matched "phishing" or "ransomware". It had survived three deployments across two codebases, masked by fixtures that matched other branches. Fixed and regression-tested in all three implementations, then redeployed to both clouds.

The first live run then caught two more: coverage drift between the deployed regexes, and the harness's own token starvation, where the model's thinking block ate an 800-token budget before the verdict. And the "unterminated string" errors were not truncation at all: the model was quoting the ticket inside its reasoning field, breaking strict parsing. The parser now repairs newlines and falls back to field extraction, because malformed prose in a commentary field must not cost the model the case.

the regression gate

CI runs the set nightly and on demand: an absolute floor, plus no brain below its own committed baseline minus one case of tolerance, because run-to-run judgment variance means a baseline from one good run lies. Fork pull requests run the gate logic offline, so they are never silently unverified. nightly scorecard status

Every scored case also emits a Langfuse trace from the run: the incident in, the verdict out, the security-floor decision as its own span, with the share id committed into the row. The instrumentation is evals/tracing.py, env-gated so offline runs never depend on it, and each row of the committed results.json carries its trace URL.

This page renders the newest committed results.json, fetched live from the repository, which CI rewrites after each nightly run; if the fetch fails it falls back to the snapshot baked at build time. Bounds come from evals/golden.jsonl. Repository: meridian-evals.