An audit of seven LLMs across four regulated-knowledge domains — 95–100% audited accuracy, every number reproducible from the run records.

Point a test harness at a body of written rules. Generate a fresh question set from it. Give a model the source evidence and have it answer, then audit each answer against that source — with a different vendor's model reviewing every contested verdict, so nothing confirms itself.

Below is the full matrix: every model, every domain, baseline and audited, low scores included. The run records are downloadable so you can check the work yourself.

Results

Baseline = the model with standard BM25 retrieval, fetching its own evidence. Audited = headline accuracy after the pipeline, with malformed questions excluded symmetrically from both columns. Δ is the gap a team could recover. Nothing is cherry-picked — the full matrix is shown, including a weak 70B model, so the numbers can't be accused of selection.

OSHA — 29 CFR workplace-safety regulations

~992–998 scored questions per model after malformed-question exclusion.

FDA — prescription drug labels

~987–1,000 scored questions per model after malformed-question exclusion.

SEC — 10-K financial filings

~900–992 scored questions per model (a mid-run PC restart's dropped API calls were excluded symmetrically from both arms).

IRS — U.S. tax code

~988–999 scored questions per model after malformed-question exclusion.

How the audit works

For a given corpus of written rules, the harness:

• Generates a fresh question set from the source text — including intentionally unanswerable questions, to catch a model that bluffs rather than declines.
• Gives the model the correct source passage and records its answer, then audits that answer against the same passage and the expected answer.
• Sends only contested verdicts (roughly 4% of answers) to a different vendor's frontier model for an independent second opinion. Vendor rotation is enforced in code, so no model ever reviews its own output. Nothing confirms itself.
• Removes malformed test items (bad questions, broken ground truth) from the denominator — symmetrically, from both the baseline and audited columns — and shows the count rather than hiding it.
• Outputs a full run record: every question, the model's answer, the expected answer, the grading rationale, the failures, the removed items, timestamps, and model names.

Known limitations

• The audited number is bounded by the model under test. The harness measures truth; it does not manufacture it. A model that can't reason to the answer stays wrong, and the score says so.
• Audited accuracy reflects an oracle-grounded check, not a live serving uplift — it is the accuracy a team is leaving on the table, measured, not a claim about a deployed RAG stack.
• The contested-verdict reviewer is itself an LLM. The architecture's job is to make imperfect reviewers produce reliable verdicts; cross-vendor rotation is how it limits any single model's blind spot.

Download a run record

Each is the complete, unedited output of one real run — questions, answers, expected answers, grading rationale, failures, and removed items. Open one and check the work yourself.

(Download links are placeholders in this mockup — the packaged run records aren't wired up yet.)