Splitting the question into three made the Decisions API as accurate as Luna

In the previous post I measured whether one judgment in the Obsidian plugin I’m building (not yet public) could be handed to a decision-only model. The judgment asks whether two notes cover the same concept and should be merged, or different concepts and should stay apart. Today it’s a regular gpt-6-luna call, and the candidates were TypeSafe’s Jev and OpenAI’s Decisions API.

Here’s how it went. With the plugin’s actual judgment rule put in as-is, the Decisions API attached high confidence, like 0.94, even to wrong answers. That rule packed several conditions into one question, and both companies’ docs advise against exactly that. This post is a record of actually following that advice. As I would for Luna, I revised the prompt twice and measured each time.

Let me state the scope up front. The test data is 37 note pairs taken from my own note vault, and I revised twice while looking at that same data. This post isn’t claiming that what worked here will work for other judgments or other data. It only covers what worked and what didn’t in this case.

The starting point: just fixing the rule made things better

Before tuning, I had to fix the rule, because of the bug found in the previous post. The rule “two notes from different sources are the same concept only when they cover the same specific event” was being applied to unique things like products and companies too, so pairs like “Claude Code” and “클로드 코드” (Claude Code written in Korean) weren’t being merged. So I added an exception to the test prompt: “the same unique entity is the same thing even if the sources differ, but a subtopic relationship is still a different concept.”

With that rule, and the question structure left as-is, I measured again. I counted 37 pairs, leaving out three where the judgment is ambiguous enough that the expected answer itself is doubtful.

Just plugging the hole in the rule made both decision-only models better too. But the Decisions API still got one pair (notes on a broad topic and on one part of it, from the same article) wrong with confidence 0.94. I took this as the baseline and tested the two pieces of advice from the docs.

Two things the docs recommend: split the question, trim the input (Decisions API and Jev prompt guidance)

OpenAI’s Decisions API guide recommends separating different concerns into different questions. In the document where TypeSafe lists Jev 1.13’s weak spots, it advises against hiding several judgments inside one question, and notes that accuracy falls when the input contains a lot of content unrelated to the decision.

Applied to my case, that means two things. First, split the single choice question “is this the same concept?” into several true/false questions and let code combine the answers. Second, send the notes without the parts that look unrelated to the judgment (metadata like tags and dates, the list of related-note links, link paths).

Trimming the input made it worse (trimming LLM input context)

Trimming failed first. At first I stripped the note header (frontmatter) down to just the title and classification, and the Decisions API dropped from 34 pairs to 29. The reason was quick to spot. Among what I’d removed was source, the line saying which original article the note came from. My rule centers on “did the two notes come from the same original article?”, so I had removed, by my own hand, the very information that judgment needed.

I trimmed again with source kept, and it still got worse: the Decisions API at 32 pairs and Jev at 33, both below baseline. The remaining suspect is the list of related-note links. Sibling notes from the same article link to each other, so that list may have served as a clue that “these were made separate on purpose.” But that’s an unverified guess. One thing is clear in this case: what looks like clutter to a human may have been evidence for the model, so what to remove had to be decided against the judgment criteria.

Splitting the question, round 1: the model read one question differently (Decisions API predicate)

I split the judgment into three true/false questions.

1. Do both notes cover one and the same unique entity (a specific product, company, person, work, and so on)?
2. Is one note a parent concept, with the other being a part of it or a specific case of it?
3. Do the two notes summarize the same specific event, study, or report?

Code does the combining: if 1 is true, same; otherwise if 2 is true, different; otherwise if 3 is true, same; anything else, different.

Correct answers were similar to baseline: 34 pairs for the Decisions API, 33 for Jev. But opening the wrong pairs showed the same problem repeating. For sibling notes made from one article, each covering a different subtopic, both models answered question 3 “true” (0.89 to 1.0). They read “do they summarize the same specific event?” as “did they come from the same article?” That isn’t wrong: same article, so same event. What I wanted to ask was “do they cover the same subject?”, but the question didn’t say that.

Splitting the question, round 2: Decisions API got 36 of 37 (true/false questions, no option-order bias)

I changed only question 3: “Do the two notes cover the same subject at the same scope? False if they cover different subtopics even when they came from the same article; true if they cover the same specific event and share facts like dates, figures, and people even when the articles differ.”

The Decisions API got 36 of 37 pairs right, the same count as gpt-6-luna through a regular call. It now also got the pair it had missed at confidence 0.94 in the baseline, and the similar subtopic pairs. Even the one remaining wrong answer has a doubtful expected answer. One note is a broad take on corporate AI cost issues pulling together four sources; the other covers the same issue centered on one event from them (Uber exhausting its AI budget). I had labeled that pair “same.” The model gave question 2 (the parent/part relationship) 0.93. By the rule, the model may well be the one that’s right.

As a bonus, the option-order bias seen in the previous post disappeared too. A true/false question has no options whose order could change. Speed stayed at about 0.16 seconds per judgment. All three questions fit in one request, so the number of calls didn’t grow either.

The disappointing part is the confidence value. Computing the combined judgment’s confidence from the least confident of the three answers made the values come out low across the board. Following only judgments with confidence 0.9 or higher gives no errors, but only 12 to 13 of the 37 pairs pass that filter. The rest still have to go to the LLM.

Jev did better without splitting (noul probabilities and confidence gating)

Applying the same approach to Jev gave a different result. Correct answers stayed at 35 pairs, the same as baseline, but Jev’s probabilities for the true/false questions were bunched between 0.2 and 0.6. It got answers right but with weak confidence, which sharply reduced how many pairs could be filtered by confidence.

For Jev, the baseline with a single question was the most usable. It got 35 pairs right and gave the two wrong ones low confidence, 0.2 and 0.32. Following only answers with confidence 0.8 or higher, it handles 26 of the 37 pairs with no errors, and only the remaining 11 need to go to the LLM. It’s the setup that keeps the strength Jev has shown since the previous post: when it’s wrong, it isn’t sure of itself.

In short, in this case the right method split by model. The Decisions API needed the question split to answer well, while Jev did better unsplit and filtered by confidence. The same advice from the docs had a different effect on each model.

Time and cost per 1,000 judgments (gpt-6-luna, Decisions API, and Jev pricing compared)

I also calculated how much switching to a decision-only model would save. For gpt-6-luna, I re-ran 10 pairs with the actual plugin setting (reasoning high) and got the usage records. Input was around 3,000 tokens per judgment, and the front part holding the judgment rule, 1,386 tokens, was cached and billed cheaply. Output, reasoning included, was about 100 to 600 tokens per judgment. Prices are the public prices as of October 2026 (gpt-6-luna $0.10 per million input tokens and $0.50 per million output tokens; the Decisions API input-only at $0.10; Jev input-only at $0.042).

The Decisions API and Jev costs are estimates computed from the token counts gpt-6-luna reported for the same input, and the last row is an estimate using the share of pairs Jev handled alone in these 37 pairs (about 70%).

What surprised me was cost. I expected decision-only models to be much cheaper, but gpt-6-luna didn’t think for long on this judgment even with reasoning on high, and the same rule portion was cached every time, so it came out at almost the same price as the Decisions API. The test note vault has accumulated 1,082 judgments so far, and whichever approach you use, that’s under $0.50. At this scale, what a decision-only model buys you isn’t money but time. The same 1,000 judgments take over 40 minutes of waiting with gpt-6-luna and are done in 3 minutes with the Decisions API.

How far can these results be trusted?

A few things to keep in mind when reading the numbers.

• The data is small. 37 pairs, from one person’s note vault, a mix of Korean and English, with topics concentrated on AI and work.
• I revised against the same data. I looked at the same 37 pairs’ results while revising the question wording twice. The results may be fitted to this data to some degree. Until measured on new pairs, 36/37 is a number for this data.
• The expected answers aren’t perfect either. I set them by reading the notes, and for a few pairs the answer depends on how you read the rule. The one remaining wrong answer is such a pair.
• This doesn’t mean the docs’ advice is wrong. Trimming didn’t work most likely because what I removed from my input was evidence for the judgment. Removing content that’s truly unrelated to the judgment may give a different result.

Final thoughts

The previous post concluded that “the Decisions API doesn’t fit the shape of my question.” This time, once I shaped the question to fit the tool, it matched a regular LLM call, at least on these 37 pairs. Much of what looked like the tool’s limits turned out to be about how the question was asked.

What stays with me most is how much weight one question’s wording carries. “Do they summarize the same specific event?” and “do they cover the same subject at the same scope?” look similar to a person, but the model read the first as “is it the same article?” The more a decision-only model answers in one shot without reasoning, the more literally it takes what the question asks. When turning a judgment criterion into a question, I first had to check whether it could be read to mean something other than what I intended.

I haven’t plugged a decision-only model into the plugin yet. I plan to collect new note pairs, measure this setup again, and plug it in only if the results hold up. And after writing this post, I ended up re-examining the judgment rule itself, starting with whether “notes from different sources aren’t the same concept” was the right starting point. That story is in the next post.