20x cheaperthe azure brain's run cost against the fleet brain's, $0.0022 versus $0.0426 on the 2026-09-01 run. The price gap is measured, not assumed, and so is what it buys.
1 latent buga security-floor regex that never matched, caught by the harness's own unit tests after surviving three deployments in two codebases.
Meridian's gates decide severity floors and security escalations, so 'the brain seems fine' is not a health check. The harness turns judgment into bounds a machine can score, runs both production implementations against the same cases, and gates deploys on the result. A golden set with exact expected labels would itself lie: severity is often legitimately a range, so the labels are floors and permitted sets, and one case whose answer depends on contract context is scored but held outside the gate rather than given a fake truth.
Who it serves
Anyone running an LLM in a decision seat who needs to know when a model, prompt, or provider change makes it worse.
Who it is for
Teams who want eval pipelines that gate deploys, not dashboards that decorate them.
How it works
Author bounds, not answersEach case carries a severity floor, permitted categories, and whether the security floor must hold. Four cases are real incidents from the fleet's history.
Run both brainsThe same twelve cases go to Claude on the Anthropic API and to gpt-4.1-mini behind the Azure Function, so the comparison is between production paths, not lab conditions.
Score against the gatePass rate, per-case checks, cost, and latency land in results.json, committed so the numbers have a history.
Gate the buildAn absolute floor plus a baseline-with-tolerance rule fails CI on regression. Nightly cron appends the history row and commits the fresh scorecard.
The decision that was not obvious
The baseline has a tolerance band of one case. The first version gated on 'never below the committed baseline', and run-to-run judgment variance broke it immediately: a baseline recorded from one good run lies about the model's real floor. One case of drift is noise; two is a regression. That number is written down in the gate, not in a person's memory.
Built with
Pythongolden set with boundsregression gateGitHub Actions nightlyoffline mode for fork PRs