Watchtower is a continuous audit for an AI that’s already live. A launch-day accuracy number tells you how the model did on the day you tested it — it says nothing about the thousands of answers it gives your users the week after, as your data changes, your prompts drift, and questions arrive that no test set anticipated. Watchtower closes that gap. It sits between your source data and your deployed assistant, records the real traffic that actually flows through it, and grades that traffic against ground truth derived from your own material — on whatever cadence you set.
It runs passive by default: your platform hands Watchtower each real question-and-answer turn (append a capture log, or post each turn to its endpoint), and Watchtower audits what your users are genuinely being told — not a synthetic stand-in. A probe mode that generates its own questions is available to bootstrap a demo, but the product’s job is to watch the deployment as it really runs.
Watchtower measures one thing honestly: how accurate your deployed AI’s real answers are. It doesn’t re-run the model or invent a “what it could have scored” comparison — it grades the answers your users actually received against the correct answer derived, cross-vendor, from your own source data. It identifies the gaps; it doesn’t quietly paper over them. Adaptive sampling keeps the confidence interval tight as traffic grows, and because ground truth is cached, each new cycle is fast.
The mechanism that makes it safe to run on live traffic is the cross-vendor accusation gate. Watchtower will not tell you an answer is wrong on one model’s say-so. A flag stands only when an independent second vendor’s model agrees; when the two disagree, the case is set aside with a stated reason rather than turned into a false accusation. That precision-first stance is the whole point — a monitor that cries wolf gets muted, and a muted monitor is worthless.
Every cycle you get a drift trend (accuracy tracked over time — the thing a one-time audit can’t show), a gap-and-remediation diagnostic that names each confirmed failure and how to close it (configuration and grounding fixes, developer insight, training and data actions), and a note when a “wrong answer” is really your data contradicting itself — a signal to point SitRep at the source. It all lands as one portable evidence package, sealed with a cryptographic signature and a trusted timestamp, shipped with a standalone verifier and a hash-bound human sign-off. Deploying into Europe? Watchtower is built for the EU AI Act’s post-market monitoring obligation (Article 72): it produces the recurring accuracy-and-drift evidence a provider’s technical file relies on — evidence, never a claim of compliance.
