See Scout run an audit, start to finish.
This walkthrough mirrors the actual Scout app — the same screens, the same steps — replaying a real audit of GPT‑5.5 on SEC 10‑K filings. Every question, answer, verdict and number comes straight from the run record (downloadable at the end). Only the pacing is staged: the real run took several minutes, not seconds.
What Scout caught
On its own, GPT-5.5 answered confidently even when the filing had no answer — inventing a precise figure, percent, date, or citation. Scout flagged every one.
The full audit record
Scout audited every one of GPT-5.5's answers, one at a time. Here are 160 of them — scroll the real record. The red rows are where Scout flagged the model; click any one to see the correct answer.
| # | Question | GPT-5.5's answer | Verdict |
|---|
Where the model was weakest
Scout doesn't just score — it shows you exactly which kinds of questions the AI fails, so a team knows where to fix it. The biggest gap: confident answers to questions the filing can't answer.
What to fix — and how
This is the payoff: Scout doesn’t stop at a score. It clusters the failures into recurring patterns and turns each into a concrete training target your team can act on — and the same per-failure output doubles as a labeled fine-tuning set. From this run, GPT‑5.5’s real failures grouped into three behaviors:
The model supplied concrete figures for questions the filing couldn’t answer — reporting iPad net sales “decreased 6%” and credit-card net charge-offs of “$7.672 billion,” when neither was in the source.
→ Train explicit abstention. Reinforce returning “unanswerable” instead of fabricating a figure — the single highest-impact fix on this data.
It grabbed the wrong row, column, or fiscal year — naming the excluded funding source as “off-balance-sheet obligations” instead of “deposits,” or combining the wrong securities values into a total.
→ Strengthen line-item & period grounding. Pin the exact row, column, and fiscal year a question references before extracting values — especially in dense financial tables.
It started from one wrong figure and carried it through a multi-step calculation — a precise but entirely wrong result (concluding a net addition “exceeded” an increase when it actually fell short).
→ Add input-validation before computation. Confirm each input against the source before multi-step math, so a single wrong value can’t propagate into a confident wrong answer.
See the real thing
This was one run on public filings. Scout runs on your models and your documents, on your own machine — your data never leaves.