A viral claim about search, tested on 5,376 of my own notes. Keyword beat semantic exactly where predicted. Then both of the obvious next moves turned out to be wrong.
The claim came from a short video by Isaac Flath, who builds retrieval systems for a living: for technical questions, plain keyword search beats semantic search. Ask for an exact thing, a filename, a paper ID, a config parameter, and matching words wins. Ask a fuzzy conceptual question and embeddings win. Treating your whole knowledge base as one corpus with one retriever means picking wrong for half your queries.
This is a checkable claim, and the published benchmarks back the first half: lexical search is stubbornly hard to beat on rare exact terms. But benchmark corpora are not my corpus. My notes are a deliberately messy pile: session logs, project notes, config gotchas, 3,485 podcast summaries, 675 paper summaries. So instead of sharing the video, I spent an afternoon testing it.
The design mirrors the claim. Ten questions contain an exact
identifier I know lives in my notes: a script name, a database table, a
parameter like max_df_ratio, an arXiv number, a version pin.
Ten are conceptual paraphrases with none of the target document's
vocabulary. Every question has a verified answer set, so scoring is
mechanical: Recall@10 asks whether a right document shows up in the top
ten, and MRR rewards it being near the top. Three retrievers ran every
question: keyword search (BM25), a local embedding model, and the
standard blend of both rankings.
| question type | retriever | Recall@10 | MRR@10 |
|---|---|---|---|
| exact identifier | keyword | 0.80 | 0.581 |
| exact identifier | semantic | 0.50 | 0.370 |
| exact identifier | blend | 0.60 | 0.395 |
| conceptual | keyword | 0.90 | 0.783 |
| conceptual | semantic | 0.80 | 0.714 |
| conceptual | blend | 1.00 | 0.824 |
Semantic search lost half the identifier questions outright, and its misses are instructive. Asked about a health database column, it returned sleep podcasts. Asked about a specific arXiv number, it returned papers that felt thematically nearby. Embeddings compress meaning, and an identifier's meaning is not the point of an identifier. Its exactness is.
On conceptual questions the two modes were close, which also matches the claim. So far the video is two for two. The surprises were in what I assumed would follow.
The textbook resolution is hybrid retrieval: run both, merge the rankings, get the best of each. The merge I used is the standard one, reciprocal rank fusion, which rewards documents that rank well on either list.
On identifier questions the blend was worse than keyword alone: 0.60 recall against 0.80. The mechanism is blunt. When one ranking is confidently right and the other is confidently wrong, averaging them drags the right answer down. The arXiv question is the clean example: keyword search had the correct document at rank one, and fusing in the semantic ranking pushed it out of the top ten entirely.
Keyword search still missed two identifier questions, which looked like the ceiling of the approach. It was not. It was a tokenizer bug, and a common one.
My indexer split text on anything that was not a letter or digit, the
usual default. That turns max_df_ratio into three tokens:
max, df, ratio. Each is common. The identifier's entire value, its
rarity, evaporates at the first step of indexing. The top result for that
question was a document containing zero occurrences of the identifier and
plenty of the fragments. Same story for a table name of the
user_something_keys shape: three ordinary words, none rare.
The fix is one line: keep joined identifiers as whole tokens alongside their fragments. That alone moved keyword search to 0.90 recall on identifiers and a clean sweep on conceptual questions.
| question type | tokenizer | Recall@10 | MRR@10 |
|---|---|---|---|
| exact identifier | default split | 0.80 | 0.581 |
| exact identifier | identifiers kept whole | 0.90 | 0.761 |
| conceptual | default split | 0.90 | 0.783 |
| conceptual | identifiers kept whole | 1.00 | 0.794 |
This inverts the diagnosis. What looked like evidence for needing a second retrieval mode was mostly evidence that the first one was quietly destroying its own best signal.
The video's implied architecture is a router: send each query to the retrieval mode that suits it. That turns out to be almost embarrassingly cheap to build. A regular expression that looks for identifier-shaped tokens, things with underscores, hyphens, dots, or camelCase, classified all twenty questions correctly. No model in the loop. The two question populations separate on surface shape alone.
| configuration | Recall@10 | MRR@10 |
|---|---|---|
| semantic everywhere | 0.65 | 0.542 |
| blend everywhere | 0.80 | 0.675 |
| keyword everywhere | 0.95 | 0.778 |
| routed: identifiers to keyword, rest to blend | 0.95 | 0.793 |
Routing is the best configuration, and it recovers every keyword win that blind fusion destroyed. But the honest reading of that table is the gap between the last two rows: plus 0.015 MRR, at twenty questions, is noise. Fixed keyword search alone captures nearly everything.
So the deployment rule I actually took away: fix tokenization first, because it is free and load-bearing. Add routing and a blend only if you already maintain an embedding index for other reasons. Never fuse blindly.
The limits, stated plainly: twenty questions is a probe, not a benchmark. I wrote both the questions and the answer sets, and the identifier stratum is keyword-favorable by construction, since a correct document is one that contains the identifier. The probe measures each mode's failure pattern, not a fair horse race. The direction is clear at this size; the decimals are not.