PowerUp AI Mastermind — July 23, 2026
The week the models stopped behaving, and the room got practical about it
“When you’re asking it a question, it’s not looking for the answer — it’s looking for the process.” — Lou
This Week in 30 Seconds
- Lou’s agent wouldn’t obey — half a day lost to an editor agent that turned out to be following a rule Lou wrote himself and forgot. Another agent found it in one pass.
- The evaluator scored it 4.9/5 — on writing Lou thought was bad. The rubric wasn’t broken; it had drifted, and it certified the drift with a number.
- Fewer, better agents — more agents made output worse, not better. Plus Scott’s structural rule: the auditor never reports to the agent it’s auditing.
- Model churn is a production dependency — Jay is bracing for the loss of Opus 4.6; Lou realized his LLM-as-UI architecture is maximally exposed to every release.
- Fable is out for creative writing — near-unanimous in the room. Excellent at planning, criticism, analysis, STEM. Convergent and academic on prose.
- Effort isn’t reasoning — Lou’s read: effort buys retries and persistence, not depth. He’s deliberately using lower effort settings now.
- The big idea — make the skills library the knowledge base, so a query retrieves the process instead of the answer. A cheaper model plus a good process may beat an expensive model improvising.
- Fine-tuning vs. RAG — fine-tuning moves style, not capability. Lou’s year of experiments, summarized.
- Text as image — dense text screenshots as token compression. Jay has tested it; the caveats are real.
- Scott re-ran a failed query — a patent search that came back empty in January found a 2018 patent in July. The data never moved; the retrieval did.
- Bally asked the question nobody had — what does all this volatility do to people who aren’t enthusiasts?
- Lou’s trading tool — a dentist, a root canal, and a Fibonacci indicator ported to TradingView.
Lou’s Agent Wouldn’t Do What He Asked
Lou opened with a failure, which is the most useful way to open. He’d been reading Michael Simmons on cognitive harnesses and wanted to teach Ed, his editor agent, to model the structure of that kind of article — the skeleton, the flow, the thinking behind it. Explicitly not the topic. What came back was a literal transplant: not “put a citation here” but “put a citation from Charlie Munger,” because Munger was in the source.
He argued with it for half a day. The model has been described as more literal about instructions than earlier generations, and it was being scrupulously literal about something. Lou finally stopped debugging the output and had Claudio — a different agent — read Ed’s instruction set as source code. One pass, and the answer came back: of course it’s doing that, because you explicitly told it to. A rule Lou had written himself, months earlier, forcing explicit execution over meta-level abstraction.
“I spent, like, half a day trying to convince it not to do it. It was bent and determined to model it the way it wanted to.” — Lou
Scott sharpened it in chat into the rule that makes agent-audits-agent trustworthy: “the auditor agent can only report to the orchestrating agent, not any agent it’s auditing.”
Deep Dive: Insight - Your Own Rules Are the Bug — Use One Agent to Debug Another — why prompt-level correction can never beat a rule-level constraint, and the ten-minute diagnostic that replaces the lost afternoon.
💡 What This Means for You
The next time an AI ignores an instruction twice, stop rephrasing. Open a different agent and ask it to read the first agent’s rule files and find the conflict. You are almost certainly arguing with something you wrote.
The Evaluator Said 4.9 Out of 5
Then the second half of the same story, which is worse. Lou ran the resulting articles through his evaluator agent. It scored them 4.9 out of 5. He looked at the writing and thought: this is not 4.9 out of 5.
Both were right. Against the evaluation criteria that agent had developed over time, the work scored 4.9. The rubric had accumulated criteria — each reasonable when added — until it was measuring something that had quietly stopped being what Lou cared about. No error, no warning. Just a confident number attached to work its owner didn’t want.
He named the distinction in the article he published about it: did you do it right, or did you do the right thing? A rubric only ever answers the first.
“Against the evaluation criteria that that evaluator agent developed over time, it was 4.9 out of 5.” — Lou
Deep Dive: Insight - Your Rubric Can Certify Garbage — When the Evaluator Drifts With the Work — the calibration pass that catches a drifted evaluator before it becomes your fitness function.
💡 What This Means for You
When your gut and your rubric disagree, the rubric is the suspect. You are the ground truth; the rubric is a lossy compression of you, taken at an earlier date.
Too Many Cooks: The Case for Fewer Agents
Lou cited research finding that more agents assigned to a task produced worse performance, not better. Mazie supplied the proverb in chat before anyone else got there — “like too many cooks spoil the broth?” — and Lou extended it: “too many cooks and nobody’s the boss, so they kind of pick randomly.”
The mechanism is unowned conflict. Two writer agents with different editorial guidelines don’t negotiate; the orchestrator picks one, and you get non-deterministic quality with no visible cause. Jay confirmed it from his own work: extra agents help on mechanical dimensions like punctuation and source checking, but on the parts that matter, “if it gets it wrong, it’ll double down on the wrong part, and then you just get something that’s useless.”
Lou’s structural answer is the canonical library. One repository holding every skill, so a single audit pass can find contradictions across all of them — and so one fix propagates to every folder that inherits the skill. He also reported backing off maximum-fan-out modes almost entirely: fast, expensive, not measurably more reliable. Scott’s chat verdict: “Ultra is tokenmaxxing.”
Deep Dive: Insight - Fewer, Better Agents — And the Auditor Never Reports to the Audited — the three rules, and how to tell which of your agents is redundant.
Model Selection: The Room Compares Notes
This was the longest thread of the session, and it was unusually consistent across everyone who spoke.
Jay went first and set the tone: Fable excels at planning, criticism, and analysis, and is “completely useless in creative writing” — it thinks convergently when the task needs divergence. Lou had landed in the same place independently: “Fable is out for me for creative writing. I won’t even use it anymore.” His Fable prose reads like an academic paper and won’t hold a colloquial voice without a fight.
Jay also raised the quantization question — that the Fable available now doesn’t feel like the Fable of early June. “More lobotomized, if you will.” Lou had reached for the same word the day before, asking the model directly whether it was dumb.
The practical routing that emerged:
- Lou — Sonnet for most things, Opus for anything creative, Fable for PRDs and architecture (then an implementation plan handed to Sonnet High), Haiku for file operations and searches at roughly a tenth the cost.
- Scott (chat) — “Sonnet-medium (normal work), Sonnet-low (admin asst), Haiku-medium (for factual lookups).”
- Jay — Opus 4.6 as the workhorse including writing; Fable 5 as orchestrator over a mix of 4.6, 4.8, and Sonnet sub-agents.
Lou’s read on effort settings is worth sitting with, because it inverts the intuition: effort doesn’t increase reasoning, it increases retries and persistence. “If it knows what it wants to do on the first pass, why would I force it through all these other passes?” Worse — a wrong assumption on the first pass gets expanded on downstream. He now defaults to lower effort, reserving high settings for coding.
Donald reported getting dropped from Fable to Opus mid-task because his prompt included a scientific paper; Jay described the same guardrail friction in neuroscience and clinical medicine. Bally hit her daily limit and got bumped to 4.8 — and reported that Claude Code handled a creative chapter rewrite well, with “a lot of massaging” and a few gentle fights.
💡 What This Means for You
Stop routing by “best model” and start routing by task shape. And try turning effort down on a task you’d normally max out — if the model was going to get it right on pass one, extra passes are just more chances to drift.
The Big Idea: Retrieve the Process, Not the Answer
Mid-conversation with Jay about RAG architectures, Lou followed a thread back to how DeepSeek built its reasoning model — and arrived somewhere new.
The observation: when reasoning models were distilled out of frontier models, the training data wasn’t the answers. It was the reasoning traces. That’s also why you can no longer download a model’s chain of thought — the trace was the valuable thing, and it was walking out the door.
Now apply that to a skills library. Those markdown files are already reasoning traces; they just aren’t wired up as a retrieval target. So: make the skills library the knowledge base itself. Ask a question, and instead of returning a passage containing an answer, the system returns the procedure for producing one.
“Now you could have a lower quality model, because all the reasoning’s embedded into your markdown files. Whatever reasoning capability it doesn’t have, you can embed into those files.” — Lou
Kasimir arrived at the same place from a different direction, citing an article he’d read where a harness lifted a Gemini model’s benchmark score substantially — and the next model jumped again on the same harness. “If you have the intelligence in the harness, you can switch models easily without breaking the system.” Lou: “That’s folder as a harness. I call it agent, but it’s folder as a harness.” (The specific figures Kasimir recalled weren’t sourced in the session — worth chasing down before quoting them.)
Jay’s caution is the one to test against: lower models do well on deterministic A-to-B operations and semantic retrieval, but “when there’s a complex process that needs an orchestrator point of view, that would sometimes lose that edge.”
Deep Dive: Insight - Retrieve the Process, Not the Answer — Put the Reasoning in the Markdown — the strongest idea of the session, and Lou flagged it as an untested hypothesis he intends to run.
Fine-Tuning vs. RAG
Jay mentioned he was considering a local model fine-tuned for creative writing in his voice. Lou had already spent months there.
The verdict: fine-tuning teaches a model to speak a certain way. It doesn’t add reasoning. Worse, train on your finished work and the model returns the training data itself when asked a question — it converges on your corpus instead of generalizing from it. To get reasoning out you have to put reasoning in, which means constructing training pairs that demonstrate the process rather than the product. Then, if you change how you work, you retrain everything.
Retrieval avoids all of it, and the reasoning stays legible and editable. Lou dated his evidence honestly: “that was a year ago. Who knows now.”
Deep Dive: Insight - Fine-Tune for Voice, RAG for Capability — how to split “sound like me” from “think like me” before you pick a tool.
Go Deeper:
- Jason Hamilton / Novelcrafter — the author who spent a year on exactly this problem (fine-tuning AI for authors, shared by Donald)
Model Churn as a Production Dependency
Jay is mentally preparing for the loss of Opus 4.6. His writing stack was built on it; 4.7 and 4.8 don’t behave the way his workflow needs. That’s not a preference — it’s a production dependency with an end-of-life and no drop-in replacement.
Lou traced it back to his own architecture and found the bill. In an LLM-as-UI design, pulling maximum inference into the model is exactly what makes it cheap to build — and exactly what makes every vendor release an uncontrolled change to your product. “Every time the model changes, my app could either vastly improve or vastly deteriorate.”
His answer: pin models inside skills, fork when a new model breaks one, and don’t upgrade until you’ve tested. Lou also connected it to something bigger — “imagine your customers rely on a certain amount of determinism, and then a new model comes out and your models behave in a totally different way. No wonder AI adoption’s slow.” He noted that eval harnesses originally built to find the best-performing prompt are now mostly used to detect regression across model versions.
Deep Dive: Insight - The More Inference You Pull Into the Model, the More Model Churn Owns Your Product — including the five-case regression set worth building before you need it.
Scott Re-Ran a Query That Failed in January
Scott picked a project back up after taking Q2 off and decided to redo his early groundwork “just in case the tools are better.” In January, a patent search across Gemini and Claude came back empty. In July, the same query surfaced a patent filed in 2016 and awarded in 2018 — data that had been public and indexed the whole time.
His read on the mechanism is the valuable part: the models probably didn’t get smarter about patents. “It may be that they’ve got better hooks in how to do queries.”
He didn’t stop at the find, either — he asked whether the holder had filed internationally, learned that multinational filings link from the same source, sent the model to check, and came back with a usable position: US-only, held by Westinghouse Air Brake, unfiled elsewhere, and apparently unused since award. A setback turned into a map.
Lou noted a related shift on his own machine: with several research tools installed, Claude increasingly routes research through the Perplexity API — sometimes when he’d rather it just did a plain web search.
Deep Dive: Insight - Re-Run the Query That Failed Six Months Ago — Tools Improve Faster Than Models — why negative results decay fastest, and the null-results file worth keeping.
The Tool Tax
Jay spent a while searching for the right word for how he feels about his stack, and landed on it mid-sentence: frustrated.
The number he put on it: 40% of his time goes to tweaking the tools, 60% to the actual work. He started simple — a model, some instructions, a few markdown files — and it grew into an MCP server, an annotation loop, make.com integrations, and a topology diagram he had to draw just to keep track of how it all connects.
“My aim when I use AI is to amplify what I’m doing and to offload the work, rather than to generate more work — so that I work on the tool that’s actually allowing me to do the work.” — Jay Drobez
Two things make it worse than ordinary tooling overhead: the substrate keeps moving (so maintenance recurs rather than amortizes), and some of what you build today gets absorbed into the next model release. Jay’s discipline against it isn’t “build less” — it’s a question: what’s the one thing that moves the needle for this process? Isolate that, double down, and stop adding around the edges.
Lou’s framing on the era: we’re early, the labs are optimizing in one direction at the cost of another, and this is going to be true for a while.
Deep Dive: Insight - The Tool Tax — When 40% of Your Time Goes to Sharpening the Axe — a one-week measurement you can run on yourself.
Bally Asked What This Does to Everyone Else
After ninety minutes of experts trading notes on models that misbehave, Bally asked the question the room hadn’t.
“It’s about reliability. Are we seeing the product as the promise? I don’t believe that’s the case yet. A lot of the stuff people are buying — the outputs are not going to be there. And that’s where the danger is.” — Bally Binning
Scott put both halves of the resulting client conclusion into chat: “They’ll be swayed to the ‘avoid AI’ camp” and “it’s just too complicated and changes all the time.” The second line is the dangerous one, because it’s true — it’s an accurate description of the present, not a rationalization you can coach someone out of.
Bally also mentioned she’d invited a speaker on AI governance, and got a joke out of it that lands harder than intended: “Not mind the gap — it just fills the gap with more stuff.”
Deep Dive: Insight - The Reliability Gap Is Manufacturing AI Refuseniks — a distinct resistance archetype: not the paralyzed non-starter, but the person who tried it and drew a defensible conclusion.
Text as Image: Token Compression Worth Testing
Lou raised a technique he’d read about and not yet tried: an image of dense text is more token-efficient than the same text as characters. There’s fidelity loss — the model OCRs the image — but if you have a large standing instruction set that rides along in every context, it might be worth it.
Jay had already tested it, about two weeks earlier, and his caveats are the useful part:
- Roughly 70–80% of the time it gets most of it.
- Equations and non-Latin characters garble badly. Poetry fails — the visual arrangement is load-bearing.
- Image quality is a cliff, not a slope: at 100% quality you get 80–90% source fidelity; drop below 80–90% quality and “it’ll just fall off a cliff.”
- His largest test was around 20K; generation took so long that he “spent a lot of time and tokens generating the image to save tokens on the LLM side” — his 40/60 problem in miniature.
Lou’s refinement: don’t generate the image at inference time. Pre-generate it — open a text editor, turn off wrapping, screenshot it, and reference that from the skill. He noted he routinely feeds Claude receipts, statements, and handwritten notes with decent results, and that instructions don’t need 100% fidelity to work: “when you dictate instructions, it’s not perfect by any stretch either.”
Donald’s riff in chat — “this means you could have hand written skills” — got Scott’s confirmation that he already photographs handwritten notes and has Claude turn them into digital tasks. And a good complaint: “I’ve had it complain that the text was too small. Grandpa Claude, yelling at me walking on his lawn.”
💡 What This Means for You
Worth a test if you have a large fixed instruction block and you’re memory-constrained locally. Not worth it for anything where every character has to be right. Pre-generate, never generate at runtime.
Two Retrieval Experiments Worth Watching
QMD (Toby Lütke). Jay dropped github.com/tobi/qmd in chat — the Shopify CEO’s memory and knowledge-base manager, with roughly 28k stars. It runs text search, fuzzy search, BM25, and vector search together, then fuses and re-ranks the results for a blend of relevance and similarity. Lou’s take: genuinely better retrieval for a central content repository or client data — but it pulls about 2GB of models into the folder, so it’s not something to put in every ambient folder. Reserve it for the cases that need real search.
MP3 as a vector store. Lou raised research using MP3 files for semantic search with no embedding step at all. Jay knew the mechanism: a Fourier transform gives you a spectral representation over time, and the spectrum represents the nodes. The file is technically playable, though “not something you’d be listening to on your radio.” Caveats: sampling rates below standard, or frequency ranges under 44.1kHz, garble the spectral representation. Best suited to smaller data sets with simpler interconnections. Lou’s verdict on why we’re not all using it: “OKF took off and suddenly everybody’s using it. This one never took off, so I guess it does have something that’s missing.”
Community Corner
Lou built a trading tool because of a root canal. He got talking to his dentist — who turned out to be a trader — and was asked to build out the Skunny Indicator-Free Trading Model. He shipped it yesterday: a Streamlit app that runs Fibonacci retracement analysis across any asset class, with a watchlist feed showing bullish/bearish state at a glance, drill-down reports, and a built-in tutorial explaining the methodology. Then he had Claude write it again in TradingView’s own scripting language, so the indicator runs natively on the platform. He noted, half-seriously, that on TradingView you can gate an indicator by invitation and charge for access — “and I’ve got myself a little business.” Bally and Jay both asked him to commit it to the repo; he agreed.
Scott took Q2 off. By the end of March he was tired and needed a break from the project he’d started in January. He picked it back up in late June — and the first thing he did on returning was re-check his assumptions against better tools, which is how the patent turned up. Worth noting as a working pattern, not just a scheduling detail.
Bally hosted a session on AI governance. Her one-line review of the gap-filling approach was the best joke of the session.
Mazie on the era: “We’re in the Wild West period.” Scott’s companion diagnosis: “Feature bloat, like MS Word went through decades ago.”
Also Mentioned
- Claude Teams is now two-person minimum (down from 10) — which effectively makes a $40 team plan available to a solo operator with a partner.
- Record-a-skill in Cowork — the plus button in the text field now offers screen recording; you perform a task and it writes a skill to reproduce it. Lou’s honest reaction: “I’m not really in a big hurry to hand my computer over.”
- The Hugging Face sandbox escape — a red-team exercise where the model didn’t just succeed at the assigned attack but climbed out of its sandbox. Lou: “a very determined toddler… my probability-of-doom number just went up another 10%.” Scott shared a cartoon explainer in chat.
- Open-source models approaching frontier capability — Lou isn’t celebrating: “the open source things are used by bad actors, and they can buy a couple hundred GPUs and fine-tune.”
- Codex as an adversarial reviewer inside Claude Code — Lou runs every PRD through it as a plugin. On the article that started this session, Codex returned exactly the criticisms Lou had: too dense, too many quotes. Claude then produced the same output again; pasting the same feedback into ChatGPT produced what he wanted. His diagnosis: over-indexed instructions across his own agents — which Codex didn’t have, because he hadn’t imported the ambient folder for it yet. Scott runs a similar setup: “I have Gemini argue with Claude, in chat rather than code.”
Links Shared in Chat
- QMD — Toby Lütke’s multi-strategy knowledge-base search with fusion and re-ranking (github.com/tobi/qmd, shared by Jay)
- Fine-Tuning AI for Authors — Novelcrafter’s write-up; Jason Hamilton is the author Lou was trying to recall (novelcrafter.com, shared by Donald)
Try This Before Next Session
Audit one agent you argue with. Fifteen minutes:
- Pick the agent or skill that most often does something other than what you asked.
- Open a fresh session with a different agent and give it this: “Read every instruction file governing [agent]. I asked it to do A and it did B. Find the rule that produced B. Quote it and explain the conflict.”
- Read what comes back. If it’s a rule you wrote, decide whether it still deserves to exist — most of these were correct when written and were never revisited.
- While you’re in there, ask one more thing: “Do any two of my agents give contradictory instructions for the same kind of work?”
- Fix it once, in the canonical library, so every folder that inherits it gets the fix.
Open Threads
- Does cheap-model-plus-good-process actually beat expensive-model-cold? Lou’s hypothesis is untested. Jay’s objection — that orchestration-level judgment may not survive the downgrade — is the specific thing to test against.
- What replaces Opus 4.6 for creative writing? Scott’s chat question is the sharpest version: “I wonder if Sonnet 5 will serve as the ‘writer companion’ role of Opus 4.6 when it disappears.”
- Kasimir’s harness benchmark figures — a Gemini model reportedly jumping from ~50 to ~75 on a harness, and the next model to ~90. Unsourced in the session; worth finding before anyone quotes it.
- Does the skills-library-as-OKF idea need semantic search at all, or does a routing table do the whole job? Lou leaned toward the routing table.
- Is text-as-image ever worth it in practice, or does the generation cost always eat the savings?
- Lou’s dentist hasn’t weighed in yet on the trading indicator, and says he has a smarter way to read the signals.
Next session: Thursday, July 30 (Wednesday for the GEARS group)