“In every skill, I have a line saying: do two independent verifications with two different agents, then present it to the main adjudicator. If there’s a disagreement, do a deep dive. If there’s no disagreement, then I take that risk of two agents being wrong.” — Jay Drobez
Session context: 2026-07-16_Mastermind — Jay shared the anti-hallucination architecture he runs on fact-critical projects, after Lou walked through the AAI framework and the conversation turned to keeping agents grounded.
Core Idea
The standard advice on hallucination is a prompt constraint: “only use verified information, cite your sources.” Scott’s story from the same session shows exactly how far that gets you — he had explicitly told Gemini not to rely on training data and to return only real institutional URLs. It fabricated 50–70% of them, invented Google search URLs to match, and justified a nonsense conclusion with “I’m going to say that these three nodes were decommissioned so the number of CPUs remains the same.” The instruction was correct. It didn’t hold.
Jay’s move is to stop asking the model to police itself and make verification structural. Every skill carries a standing instruction:
- Two independent sub-agents verify the fact, separately, against the source.
- A high-reasoning model adjudicates the two results.
- Agreement ends it. Disagreement triggers a deep dive — a third agent, or a re-scrape of the source.
The design detail that makes it viable is model tiering. The verifiers don’t need to be smart; they need to be independent. Run them on Sonnet at ordinary reasoning effort. Spend the expensive model — Fable, Opus — only on adjudication, which is a narrow comparison task, not a research task. Jay’s measured finding, from two weeks of running it during a period of aggressive rate limiting: “It really does not burn as many tokens… I thought I was going to blast through my limits.”
Two things make this more than a cost trick:
The residual risk is named, not hidden. Jay is explicit: “I take that risk of two agents being wrong.” Two independent verifiers agreeing on a fabrication is possible. The architecture doesn’t claim zero — it claims a known, priced-in error rate, which is what you can actually build on.
It belongs on the memory write, not just the read. Lou’s extension in the moment: “If you put a hallucination in memory, then you’re really cooked.” A bad answer in a conversation dies with the conversation. A bad answer written to memory.md becomes ground truth for every future session and gets cited back to you with confidence. The verification gate is cheapest and most valuable at the point where a claim earns permanence.
Jay’s related trick, for source-heavy work: put the verification instruction in a small file alongside the source files themselves, so the model is challenged as it makes its inference rather than after — and ask for a source-accuracy estimate on a 1–100 scale with the reasoning exposed, rather than a flat “don’t fabricate.”
Practical Application
Add one line to your global CLAUDE.md (Jay keeps it among his ground truths):
When a factual claim is consequential, verify it with two independent sub-agents against the source. Present both results to a high-reasoning model for adjudication. On disagreement, re-verify from source before proceeding. Never write an unverified claim to memory.
Then set the tiers deliberately: cheap model for the verifiers, expensive model for the adjudicator. The instinct is backwards — people reach for the expensive model to do the checking. Checking is the easy part. Judging a conflict is the hard part.
Coaching question: “Which of my AI outputs get written down somewhere permanent — and what gate do they pass through on the way in?”
Related Insights
- Insight - The 80-20 Rule of AI Security and Hallucination Defense — listed cross-verification as one defense among five; this turns it into a specific, tiered, automatic architecture and prices it.
- Insight - Lock AI to a Ground-Truth Source to Eliminate Hallucination — grounding constrains where the model looks; this constrains whether you believe what it found. Scott’s story proves grounding instructions alone don’t hold.
- Insight - Model Altitude — Route Model and Effort by Workflow Step, Not by Whole Artifact — the general principle; this is a worked case where the expensive model does only the adjudication step.
- Insight - Forked Skills as Context Isolation — Run Sub-Agents Without Polluting Your Conversation — why the verifiers must be sub-agents: independence requires that they not see each other’s context.
- Insight - The Grounded Query Principle — Context-Locked Answers Reduce Hallucination and Increase Trust — the single-model version of the same intent.
- Insight - Your Growing Trust in AI Is the New Attack Surface
- Insight - Fewer, Better Agents — And the Auditor Never Reports to the Audited
Evolution Across Sessions
This resolves a thread the vault has been circling for a year. Insight - The 80-20 Rule of AI Security and Hallucination Defense (2025-06-26) established the hierarchy of defenses and put cross-verification at the top, but as a manual practice for high-stakes content. What’s new is that Jay made it automatic, tiered, and cheap — a standing instruction rather than a discipline you have to remember, with the expensive model reserved for adjudication. It also picks up the model-tiering/stochasticity thesis flagged at 62 in the 2026-07-02 ingest: that flagged candidate was the observation that model choice is a per-step decision; this is the first concrete architecture in the vault that spends tiers deliberately and reports measured cost. The open question Lou raised: whether the same gate should run on every memory write by default, or only on claims marked consequential.
Source
- 2026-07-16_Mastermind (Jay Drobez — two-agent verification with tiered adjudication; Lou — extension to memory writes)