Every RAG guide recommends the same upgrades once basic vector search works: add keyword search and combine the two ("hybrid"), then re-score the candidates with a reranker. This lesson tries all of them against the golden set, one at a time, so we can see what each contributes. The answer is not what the guides would predict, and getting a fair answer needed fixing the golden set again.
Code: ask_lmsc/retrieve.py holds every strategy behind one function;
lessons/06_hybrid_rerank.py sets up keyword search, evaluates every mode, and runs the
pooled judging described below.
Why keyword search at all
Embeddings match meaning, which is their strength and their blind spot. They are good at
"put my changes aside" ≈ "stash", and weak at exact tokens: an error code, a flag like
--no-cache-dir, an API name like rememberSaveable. Keyword search is the
opposite. Hybrid search runs both and merges the results.
Postgres has full-text search built in, so no second engine is needed. A generated column keeps a searchable form of each chunk, and a GIN index makes it fast. Neither needs re-embedding:
def ensure_fulltext(conn, model: str) -> None:
conn.execute(sql.SQL(
"ALTER TABLE {} ADD COLUMN IF NOT EXISTS tsv tsvector "
"GENERATED ALWAYS AS (to_tsvector('english', content)) STORED").format(table(model)))
conn.execute(sql.SQL("CREATE INDEX IF NOT EXISTS {} ON {} USING gin (tsv)").format(
sql.Identifier(f"chunks_{model}_tsv_idx"), table(model)))
Keyword search scored 3%
The first version used Postgres's own ranking, ts_rank_cd, and turned the question's
words into an OR query (Postgres's websearch_to_tsquery ANDs every word, and a
twelve-word question then matches almost nothing). On the hand-written golden set,
keyword hit@1 was 3%. Before concluding that keyword search is
useless, look at what it does with one question:
How do I put my uncommitted changes aside so I can switch branches?
tsquery: 'put' | 'uncommit' | 'chang' | 'asid' | 'switch' | 'branch'
matching chunks: 2577
3.30 git/git-branch
1.20 git/git-cherry-pick
...
share of chunks containing: chang 14.1% branch 2.2% uncommit 0.1% stash 0.1%
Postgres's ranking counts how often and how close together the words appear, but it has no notion
of how rare a word is. "change" appears in 14% of all chunks, "uncommitted" in 0.1%, and they
count the same. Search engines weight words by inverse document frequency (IDF):
a word that appears everywhere tells you little. That idea is the core of BM25, the ranking almost
every keyword engine uses, and Postgres doesn't have it. (The ParadeDB pg_search
extension adds real BM25 if you want it.)
Adding IDF ourselves takes a few lines. Count once how many chunks contain each word, then score each matching chunk by the summed IDF of the question's words it contains:
return {lex: math.log(1 + (n_docs - _df_cache[(model, lex)] + 0.5) / (_df_cache[(model, lex)] + 0.5))
for lex in lexemes}
def _keyword_ids(conn, model, question, n) -> list[int]:
"""Chunks containing any query word, scored by the summed idf of the words they contain."""
tsq = conn.execute(f"SELECT {TSQUERY}::text", (question,)).fetchone()[0]
lexemes = list(dict.fromkeys(re.findall(r"'([^']+)'", tsq)))
if not lexemes:
return []
idf = _idf(conn, model, lexemes)
# 'simple' because these words are already stemmed; 'english' would stem them a second time.
terms = sql.SQL(" + ").join(
sql.SQL("{} * (tsv @@ plainto_tsquery('simple', {}))::int").format(sql.Literal(idf[lex]), sql.Literal(lex))
for lex in lexemes)
return [r[0] for r in conn.execute(
sql.SQL("SELECT id FROM {t} WHERE tsv @@ to_tsquery('english', %s) "
"ORDER BY {terms} DESC, length(content) LIMIT %s").format(t=table(model), terms=terms),
(tsq, n)).fetchall()]
Keyword search went from 3% hit@1 and 30% hit@5 to 33% and 67%. Scored against the corrected golden set further down, IDF weighting took it to 39% hit@1. Still far behind vector search, as expected for natural-language questions, but now a fair participant.
Hybrid: reciprocal rank fusion
Vector scores (cosine similarity) and keyword scores (summed IDF) are on different scales, so you can't add them. Reciprocal rank fusion ignores the scores and uses positions: each list gives every chunk 1/(60 + rank), and the sums decide the order.
def _rrf(*rankings: list[int]) -> dict[int, float]:
fused: dict[int, float] = {}
for ranking in rankings:
for rank, i in enumerate(ranking, 1):
fused[i] = fused.get(i, 0.0) + 1 / (RRF_K + rank)
return fused
A chunk near the top of both lists wins; a chunk at the top of one list and absent from the other still scores well. The constant 60 is the value from the original paper and works almost everywhere.
Reranking
An embedding model compares the question and the chunk separately: each becomes a
vector without seeing the other. A cross-encoder reads the question and one chunk
together and outputs a relevance score. It is far more accurate and far too slow to run
over eleven thousand chunks, so it is used as a second stage: take the top 30 from the first search,
re-score them, keep the best five. We use BAAI/bge-reranker-base, locally:
if mode.endswith("rerank") and hits:
ids = list(hits)
# The cross-encoder reads question and chunk together, so it can tell "Stash my changes so
# I can switch branches" quoted as an example prompt from a section that explains git stash.
scores = _reranker().predict([(question, hits[i]["text"]) for i in ids])
for i, s in zip(ids, scores):
hits[i]["score"] = float(s)
A cap per post
The cheapest idea is not from the guides at all. The naive pipeline gave three of five slots to one interview post. Allow at most two chunks per post, and the other slots go to other posts:
out, per_url = [], {}
for h in sorted(hits.values(), key=lambda h: h["score"], reverse=True):
if per_post and per_url.get(h["url"], 0) >= per_post:
continue
per_url[h["url"]] = per_url.get(h["url"], 0) + 1
out.append(h)
if len(out) == k:
break
return out
The reranker looked worse. It wasn't.
The first evaluation said the reranker lowered hit@1 on the hand set, at forty times the latency. Reading the questions where it changed the top result showed why:
h20 In TypeScript, should I use interface or type?
reranker's #1: [typescript] TypeScript – Interview Questions > interface or type?
"They overlap almost entirely. Three real differences: ..."
h09 How do I figure out why a MySQL query is slow?
reranker's #1: [sql] MySQL Interview – Fundamentals > How do you find out why a query is slow?
"EXPLAIN it. Read type first ..."
Those are correct answers. Interview-question posts are full of short, direct answers to exactly these questions, and the golden set didn't list them because the person writing it was thinking of the main tutorial for each topic. The reranker was being marked wrong for finding answers the golden set's author hadn't thought of.
This is the standard problem with hand labels, and the standard fix is pooled
judging: take every chunk that any method put in its top five, and judge each one
on whether it answers the question. Posts judged relevant are added to the golden set. We used Claude
as the judge with a strict prompt ("a chunk that only mentions the topic does not count"), asked for
a one-sentence reason with each verdict, and kept every verdict in
eval/judgments_hand.json.
python lessons/06_hybrid_rerank.py judge # 335 chunks, about $1
It judged 335 chunks for $1.07; 32 were relevant, adding 29 posts to 16 questions. Then the judge was checked. It agreed with all five calls made by hand earlier (YES for the two interview posts above; NO for three chunks that only mentioned the topic). One verdict was overruled: the judge said the Claude Code prompt "Stash my current changes so I can switch branches" answers the stash question, while admitting in its own reason that it doesn't explain how. A judge is a tool; read its reasons, and record any override next to the label so the next reader can see it.
The results
python lessons/06_hybrid_rerank.py eval --per-post 2
python lessons/06_hybrid_rerank.py eval --per-post 2 --model base
Hand set (33), after pooled judging. Max 2 chunks per post unless noted.
Model Mode hit@1 hit@5 MRR ms/query
----- ---------------- ----- ----- ---- ---------
small vector, no cap 64% 79% 0.72 22
small vector 64% 85% 0.73 18
small keyword 39% 76% 0.55 46
small hybrid 64% 88% 0.73 43
small rerank 70% 88% 0.76 680
small hybrid + rerank 61% 88% 0.72 1,284
base vector 73% 91% 0.81 66
base hybrid 64% 85% 0.74 86
base rerank 70% 94% 0.79 895
Read it in order of cost:
- The per-post cap is free and real: 79→85% hit@5 for free.
- Hybrid search helped the weaker model and hurt the stronger one:
(small 85→88%, base 91→85% hit@5). With
bge-base, the keyword list mostly adds noise to rankings that were already right. - The reranker gains one or two questions at 13 to 40 times the latency, about 0.7 to 0.9 seconds per query on a laptop.
- The best simple setup is
bge-base, plain vector search, two chunks per post: 73% hit@1, 91% hit@5, 66 ms.
That reverses last lesson's decision to keep bge-small, which is what the "test again
after lesson 7" note was for. The pooled judging only covered bge-small's results, which
if anything understates bge-base. And the honest caveat stands: these differences are
one to three questions out of 33. They are evidence, not proof. The decision is the simplest setup
that is at least as good as everything else we measured.
Three questions are still missed by every method: the git stash question (beaten by a near-identical sentence in another post), NestJS request validation, and JPA projections. They come back in lesson 10.
Take-aways
- Check that a component is implemented fairly before rejecting it. Keyword search wasn't bad; Postgres's ranking lacked IDF.
- When a method "loses", read the questions it lost on. The reranker was right more often than the labels.
- Pool and judge when you compare methods: otherwise you reward whichever method your labels came from.
- Measure the stronger baseline. A better embedding model beat every add-on here.