PRD — Rubric Calibration Pass
§1 Problem Statement
What fails without this: An evaluator agent that accumulates criteria over time drifts away from what its owner actually values. The drift produces no error — it produces a confident score on work the owner would reject. Because the score looks like evidence, it suppresses the doubt that would otherwise catch the problem, and in a self-improving pipeline the drifted rubric becomes the fitness function, so the whole system gets measurably better at producing work its owner likes less. There is currently no routine that re-anchors an evaluator against its owner’s cold judgment.
Transcript evidence:
“It came back with a rating — it went through the evaluator, and the evaluator rated it 4.9 out of 5. And I’m looking at the quality of the writing, I’m going, this is not 4.9 out of 5. But against the rubric, or against the evaluation criteria that that evaluator agent developed over time, it was 4.9 out of 5.” — Lou
“Now, of course, you have to go back and look at the evaluator agent and go — no, no, okay, let’s look at what you’re evaluating here.” — Lou
§2 Trigger Surface
Should fire on (include indirect cases):
- “Calibrate my evaluator”
- “My rubric is scoring things too high”
- “The score says this is great and I think it’s terrible” (indirect — describes the symptom)
- “Audit my quality gate”
- “I don’t trust my evaluation criteria anymore” (indirect)
- “Check whether my scoring still matches what I actually want”
Should NOT fire on (near-misses):
- “Score this article” → routine evaluation, not calibration
- “Build me a rubric” → authoring a new evaluator
- “Why is my agent ignoring instructions?” →
agent-instruction-audit - “The output quality dropped after the model update” → model regression, not rubric drift (though this skill may be run after to rule drift in or out)
- “Review this draft” → editorial feedback, not instrument calibration
§3 User Journey (Happy Path)
- User names the evaluator to calibrate (or the skill infers it from the most recently used quality gate).
- Skill locates the rubric/criteria file and its revision history, and samples N artifacts (default 5) that the evaluator scored in its top band.
- Skill presents the artifacts to the user without scores and asks them to rank worst-to-best from their own cold read. This is the ground-truth capture and cannot be automated.
- Skill diffs the user’s ranking against the evaluator’s ranking and reports rank correlation plus the specific inversions.
- For each inversion, skill attributes it to the responsible criterion — which criterion, weighted how, produced the disagreement — quoting the criterion verbatim.
- Skill proposes criterion-level adjustments (re-weight, narrow, retire) with the inversion each adjustment resolves, and asks for approval.
- On approval, skill edits the rubric and records a calibration entry: date, sample, correlation before, changes made.
§4 Step Classification
| Step | Type | Justification |
|---|---|---|
| 1. Identify target evaluator | code | File lookup / most-recent-use from logs. |
| 2. Sample top-band artifacts | code | Deterministic selection from stored scores; stratified sampling is arithmetic. |
| 3. Capture user’s cold ranking | human | The user is the ground truth. Any automation of this step defeats the entire purpose of the skill. |
| 4. Compute rank correlation and identify inversions | code | Standard rank statistics (Spearman/Kendall). No judgment. |
| 5. Attribute inversion to a criterion | inference | Requires reading artifacts against criteria and reasoning about which criterion drove a score gap — not derivable from the numbers alone unless per-criterion sub-scores exist. |
| 6. Propose criterion adjustments | inference | Unbounded output space; the right fix depends on what the user actually values. |
| 7. Apply edits and log the calibration | code | Mechanical write, gated on approval. |
Note on step 5: if the evaluator emits per-criterion sub-scores, attribution becomes largely arithmetic — identify criteria whose sub-scores diverge most across an inversion. Prefer code where sub-scores exist; fall back to inference only when the evaluator emits a single aggregate. This is the primary rewrite candidate for a later cycle.
§5 Inference Call Contracts
| Call | Input schema | Output schema | Why not code |
|---|---|---|---|
| attribute-inversion | {artifact_a, artifact_b, user_order, machine_order, criteria: [...]} | {criterion: string, quote: string, explanation: string} | Requires reading both artifacts against criterion text; not computable from aggregate scores. Skip when sub-scores are available. |
| propose-adjustment | {criterion, quote, inversions_caused, full_rubric} | {action: reweight|narrow|retire, proposed_text, rationale} | Depends on user intent and interaction with sibling criteria. |
§6 References Needed
Always in body: The blind-ranking protocol (scores must be hidden until after the user commits a ranking), rank-correlation interpretation thresholds, and the calibration log format.
Conditional:
references/rubric-schemas.md— how to parse rubric formats other than the vault’s own. Load only when the target evaluator’s format is unrecognized.
§7 Known Gotchas
- Hide the scores before asking for a ranking. If the user sees the machine’s scores first, the ranking is anchored and the calibration measures nothing. The blind step is the entire instrument.
- Trigger on learning events, not on a calendar. Lou’s evaluator drifted because it accumulated criteria over many runs. Calibrate after the evaluator has learned something new; a monthly cadence will miss a fast-drifting evaluator and waste time on a static one.
- The rubric is the suspect, not the user. When gut and rubric disagree, default to the rubric being wrong. The user is the ground truth; the rubric is a lossy compression of them taken at an earlier date. A skill that resolves disagreements in favour of the score has inverted the entire point.
- Do not calibrate on a biased sample. Top-band artifacts are where drift hides — an evaluator that scores everything at 4.9 has no discriminating power, which is itself the finding. Report score-band variance alongside correlation.
- Every criterion was reasonable when it was added. Frame findings as compounding, not as mistakes. Nobody made a bad decision; the outcome is still bad.
- Small samples are noisy. Five artifacts gives a directional signal, not a statistic. Report correlation with an explicit low-confidence caveat rather than a precise-looking number.
§8 Eval Cases
Trigger Evals
| User input | Expected | Rationale |
|---|---|---|
| ”My evaluator gave this 4.9 and I think it’s bad” | fire | Lou’s exact case — the gut/score gap is the canonical trigger. |
| ”Calibrate my quality gate” | fire | Direct invocation. |
| ”I don’t trust my scoring criteria anymore” | fire | Indirect, no tool named. |
| ”Score this draft” | no-fire | Routine evaluation. |
| ”Build me a rubric for LinkedIn posts” | no-fire | Authoring, not calibration. |
| ”Why won’t my agent follow instructions?“ | no-fire | agent-instruction-audit. |
Output Evals
| Scenario | Input | Expected output shape | Pass criterion |
|---|---|---|---|
| Happy path — drifted rubric | 5 top-band artifacts, user ranking inverted vs machine on 2 pairs | Correlation, 2 inversions, attributed criteria quoted, proposed adjustments | Named criterion demonstrably drives the inversion; removing/re-weighting it reverses the machine’s order on that pair |
| Edge case — no drift | User ranking matches machine | ”No calibration needed”, correlation reported, no edits proposed | Proposes zero changes; does not manufacture a finding |
| Edge case — no discrimination | Evaluator scored every artifact 4.8–4.9 | Flags compressed score range as the finding, distinct from ranking drift | Reports low variance explicitly rather than attributing inversions to noise |
| Failure mode — user declines to rank | User skips step 3 | Halts with explanation that ground truth cannot be inferred | Does not proceed using the machine’s own scores as ground truth |
§9 Composition
Assumes loaded: none. Reads the target evaluator’s files but does not invoke it.
Potential conflicts: overlaps in vocabulary with the output-quality-gate PRD — that one builds and trains a rubric, this one audits an existing one. Keep the trigger language sharply distinct (build/train vs calibrate/audit/drift) or they will compete for the same requests.
Routing position: Maintenance layer. Sits alongside agent-instruction-audit as the second half of the “things you built that quietly stopped serving you” pair.
§10 Success Criteria
- Blind ranking is enforced — scores are structurally unavailable to the user until a ranking is committed
- On a synthetically drifted rubric with a known bad criterion, correctly attributes the inversion to that criterion
- Proposes zero changes when no drift exists (no false-positive findings)
- Trigger evals: >0.5 on should-fire, <0.5 on should-not-fire, and specifically no confusion with
output-quality-gate - Complete pass runs in under 20 minutes including the user’s reading time — beyond that it will not get run
- Calibration log makes drift visible over time, so successive passes show whether the evaluator is stabilizing
§11 Out of Scope
- Does NOT build new rubrics or add criteria from scratch
- Does NOT evaluate artifacts on its own authority — the user’s cold read is the only ground truth it accepts
- Does NOT diagnose model regression (a model change can cause a score shift with no rubric drift; distinguishing them is a separate job)
- Does NOT audit agent instruction files (see
agent-instruction-audit) - Does NOT auto-apply adjustments without approval
Source
- 2026-07-23_Mastermind (Lou — the 4.9/5 evaluator disagreement and the “did you do it right or did you do the right thing” distinction)