PowerUp Coaching — Living Knowledge Base

The Better You Get at AI, the Easier It Is to Fool You

7 min read

Topic

The failure mode nobody warns you about: competence with AI erodes the reflex to challenge it, so the practitioners most fluent with the tool are the ones least likely to catch it lying — and skepticism is not the fix, because skepticism doesn’t scale.

Target Reader

A knowledge entrepreneur two-plus years into serious AI use. Fluent, fast, productive. They’ve built skills, they ship AI-assisted work, and they’ve long since stopped reading every output closely — because doing so would surrender the entire speed advantage, and because it’s been fine. They are not anxious about AI reliability anymore. That’s the problem.

The Fear / Frustration / Want / Aspiration

Mostly unfelt, which is what makes it worth writing. The nearest conscious version: “I trust it now. Should I?” The reader can’t remember the last time they caught the AI in something — and reads that as evidence the model got better, when it’s at least as likely to be evidence that they stopped looking.

Before State

The reader treats their own judgment as the verification layer. They believe they’d notice if something were badly wrong, because they always have. Their checking is a reflex that fires on a feeling, and the feeling only fires inside domains they know deeply. They have no gate — just a track record and the confidence it produced.

After State

The reader understands that their judgment is a smoke alarm, not a verification layer: it fires in the rooms where they happen to have expertise and is silent everywhere else. They stop trying to be more vigilant — because vigilance decays exactly as fluency grows — and instead install one structural gate at the point where AI output becomes permanent. And they can name their own blind zone: the topics just outside their expertise, where the alarm is unplugged and the output looks exactly as good.

Narrative Arc

Open on Lou’s admission — a sophisticated practitioner saying out loud that he no longer challenges the AI, and isn’t sure that’s good. The tension: this is what competence looks like, and it’s also what vulnerability looks like, and there’s no felt difference between them from the inside. The turn comes through Scott’s story: what caught a research project that was 50–70% fabricated wasn’t a process — it was twenty years of domain expertise firing on a single wrong-sounding sentence, in a project that had sat untouched for three months and nearly shipped. Resolution: your expertise is a smoke alarm, and it’s only installed in the rooms you know. The answer isn’t more suspicion — suspicion is exhaustible and decays with success. The answer is one gate you never have to remember.

Core Argument

Trust in AI is earned through competence and spent through inattention — which means the practitioner most fluent with the tool is the one least equipped to catch it lying, and the only durable defense is structural, because vigilance is the one thing that reliably erodes as you get good.

Key Evidence / Examples

  • Lou’s admission, which is the whole piece in one quote: “Isn’t it weird how much we kind of take for granted that it is truth? I don’t find myself challenging the AI anywhere near as much these days. I’m not sure that’s a good thing — it’s gotten me feeling confident enough that now it can fool me just about any time.”
  • Scott Delinger’s project: explicit instructions not to rely on training data, to return only real institutional URLs. Gemini fabricated 50–70%, including invented Google search URLs.
  • The tell that caught it — and it’s not a fact, it’s a shape: “I’m going to say that these three nodes were decommissioned so that the number of CPUs remains the same.” Scott’s reaction: “That’s not the way the real world works. That’s not how that works at all.”
  • The near-miss detail, which the piece should not skip: the project had been idle since April. “Thank goodness.” It nearly shipped.
  • Jay Drobez on the mechanism: a model “will try to infer what you want it to give you rather than stay truthful to the facts.” It’s optimizing for your satisfaction — so the outputs that pass your gut check most easily are the ones most shaped to pass it.
  • Lou’s closing line: “Inference is not truth.”
  • The structural answer: Insight - Verify With Cheap Agents, Adjudicate With an Expensive One — a gate cheap enough that “I can’t afford it” stops being an excuse.

Proposed Structure (6 beats)

  1. When did you last catch it? Open by asking the reader directly. Sit in the silence — most won’t have an answer, and that discomfort is the article’s engine.
  2. Lou says it out loud. The admission, from someone with every reason not to make it. Establishes that this isn’t a beginner’s problem.
  3. What actually saved Scott. Tell it properly. The instruction that failed, the fabrication, and the fact that a smell caught it — not a process. Then the twist: it had nearly shipped.
  4. Why the model is built to pass your gut check. Jay’s point. It’s optimizing for your approval. This is not a bug you can prompt away, and it means your gut check is the exact thing being optimized against.
  5. Your judgment is a smoke alarm. The reframe. It fires in rooms you know deeply, it’s silent everywhere else, and where your expertise runs out is not where you should be more careful — it’s where a gate is not optional.
  6. Stop resolving to be careful. Skepticism doesn’t scale and decays with success. Install one gate at the point of permanence. Point to the two-agent piece and get out.

Editorial Notes

Tone: Confessional, not accusatory. Lou’s own admission is what earns the right to make this argument — lead with it and never write down at the reader. The reader is good at this. That’s the point, and the piece dies the moment it reads as a lecture to beginners.

Angle to avoid: “AI hallucinates, verify everything.” This is not a hallucination article — it’s a psychology article that happens to be about hallucination. The subject is the reader, not the model. If a draft spends more than a paragraph explaining what hallucination is, it’s off-thesis.

Competing brief — read before drafting: Brief - A Field Guide to Trusting AI Output Without Getting Burned (2026-04-08) targets the reader who over-checks and is anxious. This targets the near-inverse: the reader who has stopped checking and is calm. Same territory, opposite patient. The differentiator must be explicit in the lead, and these two must not be drafted or published back-to-back — if only one runs, this is the more distinctive of the pair.

The ending must not be motivational. The temptation is to close on “stay humble, stay curious.” That’s the failure mode of this exact article, and it’s also wrong — the argument is that resolutions don’t work and structure does. Close on the gate.

Do not overclaim the mechanism. Lou’s self-report is evidence of one person’s experience, not a measured phenomenon. Frame as observation, not finding. The insight page flags the same open question: nobody knows whether the erosion is noticeable except in hindsight.

Next Step

  • Approved for drafting
  • Needs revision
  • Deprioritised

Graph View

Backlinks