Tag: Empirical
All the articles with the tag "Empirical".
-
Raising LLM request concurrency from 2 to 4 surfaced two other bugs
Raising parallel LLM requests from 2 to 4 kept summary accuracy. The audit found missed Korean-English note merges and Korean headings on English notes instead.
-
I used embedding similarity to find out why Korean and English notes wouldn't merge
Korean and English notes on one concept never merged. OpenAI embeddings put them at 0.77, below same-language duplicates; a 0.70 similarity threshold fixed it.
-
OpenAI reasoning effort wasn't the culprit behind the fidelity problems
Lowering OpenAI reasoning effort from High to Medium didn't add LLM summary errors. Note count and run time changed a lot instead; chunk size governed accuracy.
-
Getting ready to rerun the experiment, git conflict markers got committed as-is
Continuing a chunk-size experiment at home, alternating two laptops on one git repo committed conflict markers into 77 files. Then I suspected reasoning effort.
-
Just lowering the LLM chunk size made the fidelity problems go away
I compared the same 30 sources changing only the chunk size, how many go to the LLM at once. Summary errors that drop people and quotes fell from 40% to 0%.
-
189 notes an LLM had organized became 63 one day
I had an LLM turn the same 30 sources into Obsidian notes, and just switching machines cut them from 189 to 63. Not one cause, chunk size, but two settings.
-
I fixed the LLM prompt rule twice, but the note should never have been split
An LLM-made sibling note kept coming out empty but for links. Two prompt fixes later, the source showed both courses shared one description: merge, don't split.
-
Removing the embedding filter caught more duplicates, not fewer
I swapped an embedding filter plus LLM checks for one LLM review of all sibling notes to catch duplication. One hub-note rule took precision from 55% to 100%.
-
Switching embedding models flipped the ranking
Moving from OpenAI's text-embedding-3-small to large flipped the order of true duplicate paragraphs and look-alike sentences mixed at cosine 0.72-0.80.
-
LLM-generated sibling notes were copying the same paragraph word for word
Testing my Obsidian plugin on real articles exposed an LLM bug: sibling notes copied the same paragraph verbatim, and embedding duplicate detection missed it.