Series
Multi-part series, in reading order.
- Could a Decision-Only Model Take Over? (4 of 4 in English)
A measured record of whether, instead of calling a full LLM every time an Obsidian plugin asks for a judgment such as whether to merge two notes or whether an edited note now has a broken sentence, that judgment could be handed to models built only for decisions (Laya, TypeSafe Jev, OpenAI Decisions API). Beyond accuracy, it follows the confidence attached to wrong answers, the pull from the order of the options, and how the results change depending on how the question is split up.
- What AI Verification Misses (8 of 9 in English)
Putting the checks that verify AI-generated output under measurement to see what they actually miss: a merge audit that mistook a parent concept and its subtype for the same thing, a subagent-driven full audit that skimmed list-heavy files instead of checking item by item, re-verifying only against other cases while never rechecking the original false positive, an ingest step where sibling notes copied paragraphs verbatim and nothing caught it at first, and even the embedding filter built to catch that duplication flipping its verdicts depending on model size (small/large), once that filter was removed, duplication turning up in groups that had been recorded as clean, a merge rule whose wrong example was deleted while the rule that produced it stayed, a real-note test that passed cleanly while the fixed code never ran once, and sentences left pointing at nothing after duplicates were removed, blocked shape by shape in code until I switched to having an LLM reread each edited note and judge it.
- Putting My Confidence Up Against Measurement (2 of 11 in English)
After the redesign, a record of putting plausible-looking confidence up against actual measurement: the trap of few-shot counterexamples, information-density loss, wikilink false positives, a RAG claim made without evidence, a UTF-8 parsing bug, tag and link experiments, reasoning effort and isolation conditions, shoring up judgment with embedding search, filling in a missing prompt rule only to get the same results, and using control groups to isolate when a counterintuitive claim mentioned only once disappears from a summary.
- Before Fixing It, I Had to Recheck the Instruction (3 of 8 in English)
— I was designing a guard against LLM note over-splitting and didn't see the one that already existed
After the redesign, a record of times when the "fix it" instruction or the document itself was wrong: fixing a dead comment and getting it wrong again, a hypothesis that was only half right, a mistake avoided by asking once more before fixing, "delete it" and "how to delete it" turning out to be different questions, fixing a prompt rule twice when the place that needed fixing was the stage that splits notes, carefully built detection code that, once measured, turned out to have done nothing, and designing a safeguard against over-splitting without recognizing the same safeguard built the day before.
- [Complete] I Almost Mistook Two Variables for One (7 of 7 in English)
Same sources, same plugin, but switching machines made the results differ 3x. Tracking down the cause, what I thought was a single culprit turned out to be two different settings, and separating them took rerunning the experiment eight times; along the way I found and fixed three completely unrelated bugs.
- Reviewing MCP Servers and Skills (1 of 9 in English)
A record of reviewing, one by one, tools worth plugging into or swapping in for Claude Code or Hermes. Beyond MCP servers (such as codebase knowledge graphs), it covers workflow skills (Superpowers), token-saving tools (Headroom, OmniRoute), a memory plugin (claude-mem), a browser automation tool (agent-browser), and an open-source coding agent that can take Claude Code's place (OpenCode), all judged by the same question: is it actually worth adding to, or switching from, the setup I use now?