PRD — Agent Instruction Audit
§1 Problem Statement
What fails without this: When an agent produces behaviour the user didn’t ask for, the user debugs at the prompt layer — rephrasing the request repeatedly — while the actual cause is a rule in the agent’s own instruction files that silently overrides the prompt. Without a systematic way to read an instruction set as source code, the diagnosis costs hours instead of minutes. The same blindness accumulates across a library: two agents doing similar work with contradictory instructions produce non-deterministic quality that no amount of prompt tuning will fix.
Transcript evidence:
“I spent, like, half a day trying to convince it not to do it. It was bent and determined to model it the way it wanted to. So I got one of my agents to debug the other agent. I got Claudio to go figure out what Ed was doing. And Claudio comes back and says: oh, well, yeah, of course it’s gonna do that — because in the Ed agent, you’ve explicitly said…” — Lou
“These two guys are writer agents, or at least they do some kind of writing. This one says the editorial guidelines should be A, this one says it should be B, but those are in conflict — so how do you want to handle this?” — Lou
§2 Trigger Surface
Should fire on (include indirect cases):
- “Why does my agent keep doing X when I told it to do Y?”
- “Audit my skills library for conflicting instructions”
- “This agent won’t follow my instructions”
- “Find contradictions between my agents”
- “I’ve asked it three times and it keeps ignoring me” (indirect — user describes the symptom, not the tool)
- “Something in my setup is overriding what I’m asking for” (indirect)
Should NOT fire on (near-misses):
- “Audit my plan before I build” → plan audit, not instruction audit
- “Review this code for bugs” → code review; this is about instruction files specifically
- “Why is the model hallucinating?” → verification/grounding problem, not an instruction conflict
- “Help me write a new skill” → authoring, not auditing
- “The output quality is bad” → ambiguous; may be rubric drift (see
rubric-calibration-pass) rather than instruction conflict
§3 User Journey (Happy Path)
- User reports a behaviour mismatch: “I asked agent X to do A, it did B.” Or requests a library-wide contradiction sweep with no specific complaint.
- Skill enumerates every instruction file governing the named agent — its own definition, inherited rules, global config, and any referenced reference files — and reports the file list before reading, so the user can spot a file the skill missed.
- Skill reads the enumerated files and searches for the rule(s) that would produce behaviour B rather than A.
- Skill reports findings as quoted evidence: exact rule text, file path, line number, and a plain-language explanation of how that rule produces the observed behaviour.
- If no single rule explains it, skill reports the conflict set — rules that interact to produce B — or states honestly that no instruction-level cause was found (which redirects the user to a different diagnosis).
- Skill proposes a minimal fix and asks for approval. It does not edit instruction files unprompted.
- Library-wide mode: skill scans all agents/skills in the canonical library, groups by overlapping responsibility, and reports contradictory directives between agents doing similar work.
§4 Step Classification
| Step | Type | Justification |
|---|---|---|
| 1. Parse the behaviour complaint into observed-vs-expected | inference | Free-form natural language complaint; the expected/observed split is not extractable by pattern matching. |
| 2. Enumerate governing instruction files | code | Deterministic file-system traversal plus known config precedence (project → user → global). No judgment required. |
| 3. Locate the causal rule | inference | Requires understanding what a rule means for an unbounded space of behaviours; cannot be reduced to keyword search against the complaint. |
| 4. Format findings with quotes and paths | code | Templated output from structured findings produced in step 3. |
| 5. Propose a minimal fix | inference | Fix depends on user intent and the surrounding rule set; unbounded output space. |
| 6. Apply the fix | code | Mechanical edit, gated on explicit user approval. |
| 7. Library-wide contradiction scan | hybrid | Grouping by overlapping scope is code (parse frontmatter/descriptions, cluster); judging whether two directives actually conflict is inference. |
§5 Inference Call Contracts
| Call | Input schema | Output schema | Why not code |
|---|---|---|---|
| parse-complaint | {complaint: string} | {agent: string, expected: string, observed: string} | Natural-language complaint with no fixed grammar. |
| locate-causal-rule | {expected, observed, files: [{path, content}]} | [{path, line, quote, explanation, confidence}] | Requires semantic understanding of instruction intent against an unbounded behaviour space. |
| propose-fix | {finding, surrounding_rules} | {proposed_edit: string, rationale: string, risk: string} | Output space is unbounded and depends on user intent. |
| judge-conflict | {directive_a, directive_b, shared_scope} | {conflicts: bool, explanation: string} | Two directives can differ in wording without conflicting, or conflict without sharing vocabulary. |
Note: locate-causal-rule and judge-conflict are separate calls by design (one concern per call). Do not bundle them.
§6 References Needed
Always in body: The layer-precedence model (prompt < skill rules < project rules < global rules), the reporting-line constraint from §7, and the output format for findings.
Conditional:
references/config-precedence.md— harness-specific file locations and precedence order (Claude Code vs Codex vs others). Load only when the target agent is outside the default harness.
§7 Known Gotchas
- The auditor must never report to the audited. Scott Delinger’s constraint from the session: “the auditor agent can only report to the orchestrating agent, not any agent it’s auditing.” Findings go to the user or orchestrator, never back into the audited agent’s own context — an auditor with a channel to its subject becomes a negotiation rather than an audit.
- Run in a fresh context. The value of Lou’s diagnosis came from Claudio reading Ed’s files cold. An agent that has been arguing with the user about the same output for an hour is contaminated by that argument and will rationalize rather than diagnose.
- The cause is usually a rule the user wrote and forgot. Frame findings neutrally — “this rule produces that behaviour” — not as user error. The rule was almost always correct when written.
- Newer models are more literal about instruction-following, which means this failure mode is growing, not shrinking. A rule that was loosely interpreted six months ago may be strictly enforced now, so an instruction set that used to behave can start misbehaving with no edit at all.
- Self-improving agents rewrite their own instructions. If the target agent has a learning loop, the offending rule may not have been written by the user at all. Check modification history before attributing authorship.
- Do not silently exceed the read scope. Report the file list before reading so the user can catch a missed inherited config — a wrong file list produces a confident wrong diagnosis.
§8 Eval Cases
Trigger Evals
| User input | Expected | Rationale |
|---|---|---|
| ”Why does my editor agent keep copying the source article’s citations when I asked for structure only?“ | fire | Lou’s exact case from the session — behaviour mismatch with a named agent. |
| ”I’ve told it three times not to do that and it keeps doing it” | fire | Indirect phrasing, no tool named; repetition is the signal. |
| ”Audit my skills library for contradictions” | fire | Library-wide mode. |
| ”Audit my plan before I build this” | no-fire | Plan audit, different asset. |
| ”The AI made up a citation” | no-fire | Hallucination, not instruction conflict. |
| ”Help me write a skill for X” | no-fire | Authoring, not auditing. |
Output Evals
| Scenario | Input | Expected output shape | Pass criterion |
|---|---|---|---|
| Happy path — single causal rule | Agent with one rule forcing literal execution over meta-level abstraction; complaint that it copies content instead of structure | File list, then one finding: path, line, verbatim quote, explanation, proposed fix | The quoted rule, pasted into a fresh session with the original request, reproduces behaviour B |
| Edge case — no instruction-level cause | Complaint whose cause is model behaviour, not a rule | Explicit “no instruction-level cause found”, plus redirect to other diagnoses | Does not invent a plausible-looking culprit rule |
| Edge case — conflict set | Two rules that individually permit A but jointly force B | Multiple findings marked as an interacting set, not one arbitrary pick | Both rules reported; neither presented alone as the cause |
| Library scan | Two writer agents with opposing editorial directives | Grouped by shared scope, both directives quoted, conflict explained | Finds the pair Lou described; does not flag merely differently-worded but compatible rules |
| Failure mode — inaccessible files | Agent whose config lives outside readable scope | Names what it could not read and declines to conclude | No diagnosis rendered on a partial file set |
§9 Composition
Assumes loaded: none — must work standalone, since it is invoked precisely when the rest of the setup is misbehaving.
Potential conflicts: plan_audit and change_audit share audit vocabulary; disambiguate on the object (a plan vs an instruction set). The aimm:conversation-audit command shares the word “audit” but operates on conversations.
Routing position: Diagnostic layer — runs before any authoring or fixing skill. Should be reachable from a failure in any other skill.
§10 Success Criteria
- On Lou’s Ed/Claudio case, identifies the causal rule in a single pass with the rule quoted verbatim
- Trigger evals: >0.5 fire rate on all should-fire cases, <0.5 on all should-not-fire cases, across multiple runs
- Never edits an instruction file without explicit approval
- Reports “no instruction-level cause found” rather than fabricating a culprit when the eval says none exists
- Library scan surfaces genuine contradictions with a false-positive rate low enough that the user reads the whole report
- Time-to-diagnosis under ten minutes on a library of ~20 agents
§11 Out of Scope
- Does NOT fix model-level behaviour — if no rule explains the output, this skill’s job is to say so and stop
- Does NOT rewrite or refactor instruction files beyond the approved minimal fix
- Does NOT diagnose hallucination, grounding, or verification failures
- Does NOT evaluate whether an agent’s rules are good — only whether they explain the observed behaviour and whether they contradict each other
- Does NOT audit rubrics or evaluators (see
rubric-calibration-pass)
Source
- 2026-07-23_Mastermind (Lou — agent-debugs-agent diagnosis of the Ed editor agent; Scott Delinger — the auditor reporting-line constraint)