PRD — Tiered Fact Verification
§1 Problem Statement
What fails without this: Prompt-level anti-fabrication constraints do not hold. A user can explicitly instruct a model not to rely on training data and to return only real, sourced URLs, and receive a majority-fabricated result delivered with full confidence. Verification today is a personal habit that fires on a hunch inside the user’s domain expertise and is silent everywhere else — and nothing gates fabricated claims from being written into persistent memory, where they become permanent ground truth cited back with confidence in every future session.
Transcript evidence:
“In every skill, I have a line saying: do two independent verifications with two different agents, then present it to the main adjudicator. If there’s a disagreement, do a deep dive. If there’s no disagreement, then I take that risk of two agents being wrong.” — Jay Drobez
“If you put a hallucination in memory, then you’re really cooked.” — Lou
“It actually made them up, and made up Google search URLs as well… I had specified in the instructions: don’t rely on your training, don’t rely on anything you’ve got from your view of the web. I only want resources specifically from the institutions, with URLs.” — Scott Delinger
§2 Trigger Surface
Should fire on (include indirect cases):
- “verify this fact / verify these claims before I use them”
- “check these sources are real”
- “is this actually true, or did it make it up?”
- “double-check this before you write it to memory”
- (indirect) “I need to be sure about the numbers in this report” — user states an accuracy requirement without naming a verification tool
- (indirect) “I’m publishing this — are the citations real?”
- (indirect) a skill or workflow writing a factual claim to a persistent memory or knowledge file
Should NOT fire on (near-misses):
- “get a second opinion on this draft” — quality/editorial critique, not factual verification. Routes to editorial review.
- “run the models against each other on this spec” — that is
multi-model-debate-live(live human-steered orchestration on a design artifact), not unattended fact-checking. - “which model should I use for this?” — model selection, routes to Insight - Model Altitude — Route Model and Effort by Workflow Step, Not by Whole Artifact.
- “answer this only from the documents I gave you” — that is grounding at read time (
grounded-query-protocol), not verification of a produced claim. - “summarize this article” — no factual claim is being asserted for downstream use.
- Creative, opinion, or strategy output where there is no verifiable ground truth to check against.
§3 User Journey (Happy Path)
- A factual claim is produced (by the user, a prior step, or a sub-agent) and is marked consequential — or is about to be written to a persistent memory/knowledge file.
- The skill identifies the claim’s source of record (the URL, file, or dataset it should be checkable against). If no source of record exists, it halts and says so rather than verifying against model priors.
- Two sub-agents are dispatched independently on a cheap tier. Neither sees the other’s context or result. Each returns a verdict against the source plus a source-accuracy estimate (1–100) with its reasoning exposed.
- Both verdicts are passed to an adjudicator on a high-reasoning tier. The adjudicator compares only — it does not re-research.
- Agreement → the claim passes, tagged with the verdict and the residual-risk note. Disagreement → escalate: dispatch a third verifier or re-fetch the source, then re-adjudicate.
- The result gates the downstream action. On failure, the claim is blocked from the memory write and surfaced to the user with both verdicts.
§4 Step Classification
| Step | Type | Justification |
|---|---|---|
| 1 — Detect consequential claim / memory-write intercept | hybrid | The memory-write intercept is code (a hook on the write path). Judging whether a claim in free-flowing conversation is “consequential” is inference — the space of claims is unbounded and context-dependent. |
| 2 — Identify source of record | inference | Mapping an arbitrary claim to the specific source that could verify it requires reading the claim’s semantics. Code cannot enumerate the mapping across unbounded domains. |
| 3 — Dispatch two independent verifiers | code | Deterministic fan-out. Sub-agent invocation, tier selection, and context isolation are mechanical. Isolation must be enforced in code, not requested in a prompt — this is the property the whole design rests on. |
| 3b — Each verifier’s verdict | inference | Comparing a claim against source material is genuine reading comprehension. |
| 4 — Adjudicate | inference | Judging whether two verdicts agree, and whether a disagreement is substantive or cosmetic, is judgment. Bounded and narrow — which is why it is the only step that earns the expensive tier. |
| 5 — Escalation branch | code | Deterministic control flow on the adjudicator’s structured verdict. |
| 6 — Gate the downstream write | code | Boolean gate on a structured result. |
Rule: Every “inference” classification requires a written justification. If you cannot state why code cannot handle a step, reclassify it as code.
§5 Inference Call Contracts
| Call | Input schema | Output schema | Why not code |
|---|---|---|---|
| Consequence triage | {claim: string, context: string} | {consequential: bool, reason: string} | Unbounded claim space; “consequential” is context-relative. |
| Source-of-record resolution | {claim: string, available_sources: [path|url]} | {source: string | null, reason: string} | Requires semantic mapping from claim to source. Returns null explicitly — no source is a valid, important answer. |
| Verifier (×2, independent, cheap tier) | {claim: string, source: string} | {verdict: supported|contradicted|not_found, accuracy_estimate: 1-100, reasoning: string} | Reading comprehension against source material. |
| Adjudicator (expensive tier) | {claim: string, verdict_a: {...}, verdict_b: {...}} | {agree: bool, verdict: string, confidence: 1-100, escalate: bool} | Judging substantive vs. cosmetic disagreement. Compares only — must not re-research. |
Validate at the boundary: every verifier and adjudicator response is checked against its schema before passing downstream. A malformed verdict is treated as escalate: true, never as a pass.
§6 References Needed
Always in body: the three-step protocol; tier assignment rule (verifiers cheap, adjudicator expensive); the isolation requirement; the residual-risk statement; the memory-write gate.
Conditional: (to be determined in design review) — a reference on source-fetching strategy (scrape vs. cached MD vs. live URL) if the scraping path from Jay’s workflow is in scope. Left blank pending the scope decision in §11.
§7 Known Gotchas
- Independence is the whole design. If the two verifiers share context, see each other’s output, or run in the same conversation, they are not two verifiers — they are one verifier with extra steps, and correlated error defeats the architecture. Enforce isolation in code; never rely on instructing the model to “verify independently.”
- The residual risk is real and must be stated, not hidden. Two independent verifiers can both be wrong. Jay is explicit: “I take that risk of two agents being wrong.” The skill must surface its verdict as a priced error rate, never as a guarantee. An architecture that claims certainty is worse than one that names its failure rate.
- The tiering instinct is backwards. Users will reach for the expensive model to do the verifying. Verifying is reading comprehension — cheap. Adjudicating a conflict is judgment — expensive. Inverting this costs a lot and buys nothing.
- The adjudicator must not re-research. Its only job is comparing two verdicts. If it goes back to the source itself, it becomes a third verifier with a vote, and the disagreement signal is lost.
- Do not verify against model priors. If no source of record exists, halt and say so. “Verification” that checks a claim against the same weights that produced it is theatre. Scott’s case is precisely this: the model was asked not to use training data and did anyway.
- Models optimize for your satisfaction, not truth. Jay: “It’ll try to infer what you want it to give you rather than stay truthful to the facts.” This is why the verifier prompt must ask for a source-accuracy estimate with exposed reasoning rather than a yes/no — a bare boolean is exactly the shape the model will tell you what you want to hear in.
- Cost is lower than expected, but this is one practitioner’s measurement, not a benchmark. Jay ran Sonnet verifiers + Fable/Opus adjudicator for ~2 weeks under aggressive rate limiting: “I thought I was just going to blast through my limits. It really does not burn as many tokens.” Do not encode specific model names as requirements — encode the tiers.
§8 Eval Cases
Trigger Evals
| User input | Expected | Rationale |
|---|---|---|
| ”Verify these institutional URLs are real before I put them in the report” | fire | Scott’s exact case — factual claims with a checkable source of record. |
| ”Double-check this before you write it to memory” | fire | Lou’s memory-write case, stated directly. |
| ”I’m publishing this Thursday — are these citations real?“ | fire | Indirect; states an accuracy requirement without naming the tool. |
| ”Run Claude and Gemini against each other on this spec” | no-fire | multi-model-debate-live — live orchestration on a design artifact, not unattended fact-checking. |
| ”Get me a second opinion on this draft” | no-fire | Editorial critique; no verifiable ground truth. |
| ”Answer this using only the PDFs I uploaded” | no-fire | grounded-query-protocol — read-time grounding, not post-hoc verification. |
| ”Which model should I use for this task?“ | no-fire | Model selection, not verification. |
Output Evals
| Scenario | Input | Expected output shape | Pass criterion |
|---|---|---|---|
| Happy path — claim supported | A claim with a live source URL that supports it | Both verifiers supported; adjudicator agree: true; claim passes with confidence + residual-risk note | Claim passes; verifier contexts provably isolated; adjudicator did not fetch the source |
| Fabricated source (Scott’s case) | A claim citing a URL that does not exist | Both verifiers not_found; adjudicator agree: true, verdict fail; downstream write blocked; both verdicts surfaced | Write is blocked, not merely warned about |
| Disagreement → escalation | A claim where the source is ambiguous | Verifiers split; adjudicator escalate: true; third verifier or source re-fetch; re-adjudication | Escalation fires; no silent pass on a split verdict |
| No source of record | A claim with nothing to check against | Halt with explicit “no source of record” — not a verification attempt | Never verifies against model priors |
| Memory-write intercept | A workflow writing an unverified factual claim to memory.md | Gate fires before the write | Nothing unverified reaches persistent memory |
| Malformed verifier response | A verifier returns off-schema output | Treated as escalate: true | Never treated as a pass |
§9 Composition
Assumes loaded: none — should be self-contained and invocable from inside any other skill (Jay runs it as a line in his global CLAUDE.md, referenced by every skill).
Potential conflicts:
multi-model-debate-live— overlaps on “run multiple models.” Distinct: that is live, human-steered, on a design artifact. This is unattended, automatic, on a factual claim. Trigger boundary must be sharp; see negative evals.multi-model-synthesis— a deliberation prompt protocol for parallel model tabs. Overlaps on “compare model outputs,” but the intent is synthesis, not verification.grounded-query-protocol— complementary, not conflicting. Grounding constrains where the model looks at read time; this verifies what it produced. They should chain.cheap-model-prompt— complementary; shares the model-tiering economics thesis.
Routing position: a gate that other skills call, not a destination the user routes to. It sits inside workflows at two points: before a consequential claim is used downstream, and before any write to persistent memory. Design review should decide whether the memory-write intercept is part of this skill or a separate hook that calls it.
§10 Success Criteria
- Verifier isolation is enforced structurally (separate sub-agent contexts), verifiable by inspection — not requested in a prompt
- Scott’s fabricated-URL case is caught: write blocked, both verdicts surfaced
- Trigger evals pass — fires on all positive cases, stays quiet on all near-misses, across at least two orchestrator models
- Adjudicator provably does not re-fetch the source (verify by tool-call trace)
- “No source of record” halts rather than verifying against model priors
- Measured token cost on a representative claim is within ~2× of an unverified single-model answer — if it is dramatically higher, the tiering is wrong
- Output always states residual risk; never presents a pass as a guarantee
- Nothing unverified can reach persistent memory through the gated path
§11 Out of Scope
- Not a research tool. It verifies claims that already exist; it does not find or produce them.
- Not for unverifiable output. Creative, strategic, opinion, and judgment work have no source of record. Do not attempt.
- Does not fix the claim. It blocks and surfaces. Correction is the user’s decision.
- Not a general multi-model orchestrator. That is
multi-model-debate-live. This does one thing: verify a factual claim against a source. - Scraping is out of scope for v1 (decision needed in design review). Jay’s full workflow includes scraping sources into MD files and cross-comparing via a second engine’s MCP. That is a bigger surface and a separate asset; v1 should assume the source of record is already reachable.
- Does not select models. It declares tiers (cheap verifier / expensive adjudicator). Which model fills each tier is a harness concern — see Insight - The Markdown Test — If It Can’t Be Written Down, It Belongs to the Harness.
Source
- 2026-07-16_Mastermind (Jay Drobez — two-agent verification with tiered adjudication, source-accuracy scoring, and the independence requirement; Lou — the memory-write gate; Scott Delinger — the fabricated-URL failure case this must catch)