Home›AI with Lovemesomecoding›RAG – Measuring Retrieval with a Golden Question Set

RAG – Measuring Retrieval with a Golden Question Set

November 6, 20267 min readUpdated 11/14/2026

Every lesson so far has judged retrieval by looking at a few questions. That was enough to spot a TypeScript post winning a Python question, but it cannot answer the question the last lesson ended on: is bge-base better than bge-small, or did it get lucky once? To answer that, and to judge every change from here on, we need a number.

This lesson builds a golden set: questions with known answers, scored automatically. It is the most important lesson in the track. Several later "improvements" turned out to make things worse, and this is how we found out.

Code: ask_lmsc/evaluate.py, lessons/05_eval.py, and the questions in eval/golden_hand.json and eval/golden_synthetic.json.

What a golden question looks like

{
 "id": "h05",
 "question": "How do I put my uncommitted changes aside so I can switch branches?",
 "expected": ["/git/git-stash"]
}

A question, phrased the way a reader would ask it, and every post that answers it. Some questions have two or three: "should I rebase or merge?" is answered by both the rebase and merge posts, and "how do I return only a few columns from a JPA query?" by three Spring Data posts. An answer from any of them counts.

Questions the site does not cover get an empty list:

{"id": "o01", "question": "How do I deploy a Kubernetes operator written in Rust?", "expected": []}

Those don't score retrieval (there is nothing right to find). They exist to test whether we can tell that a question is off-topic, which matters in lesson 8.

Writing them by hand

The hand-written set has 33 answerable questions across 22 categories and 7 the site doesn't cover. Three rules made them useful:

  • Ask like a learner, not like the post. "My Docker image is over 1 GB. How do I make it smaller?", not "Docker image size and base images". If the question repeats the title, you are testing string matching.
  • Ask about a problem ("my API starts before the database is ready and crashes"), because that is how people search.
  • Check that off-topic questions really are off-topic. Search the corpus for the key terms first. More on why below.

Generating them with Claude

Hand-writing is slow. The common shortcut is to let a model write questions from your documents: pick a random section, ask for a question it answers, and the section's post is the expected answer.

GEN_PROMPT = """Here is one section of a programming tutorial.

<section>
{text}
</section>

Write one question that a learner might type into a search box, which this section answers.
Rules:
- Use your own words. Do not copy distinctive phrases, the post title or the section heading.
- Ask about a problem or a task, the way someone who has not read the section would.
- One sentence, under 25 words.
Reply with the question only."""

Thirty questions cost $0.07. They read well: "Why does my LIMIT OFFSET query sometimes show duplicate rows or skip rows across pages when sorting by a column with repeated values?" is exactly what a reader might ask. We will see what they are worth.

Three numbers

def first_rank(hits: list[dict], expected: set[str]) -> int | None:
    for rank, h in enumerate(hits, 1):
        if h["url"].removeprefix(SITE) in expected:
            return rank
    return None
...
    return {
        "n": len(questions),
        "hit@1": statistics.mean(r == 1 for r in ranks),
        "hit@5": statistics.mean(r is not None and r <= 5 for r in ranks),
        "mrr": statistics.mean(1 / r if r else 0 for r in ranks),

For each question, search returns the top ten chunks and we record the rank of the first one that comes from an expected post:

  • hit@5: the right post is in the top five. This is the headline number, because five chunks is what Claude receives. A miss here means the answer cannot be in the context.
  • hit@1: the right post is first. Stricter; it matters when you show one result.
  • MRR (mean reciprocal rank): 1 for rank one, 0.5 for rank two, 0.33 for three, 0 if not in the top ten, averaged. One number that rewards ranking the right post higher.

One defensive detail in load_set: every expected path is checked against the exported posts. A typo in a golden set never errors; it just scores as a miss forever, and you conclude your search is bad.

The first run

python lessons/05_eval.py generate --n 30
python lessons/05_eval.py run --misses

model  set          n  hit@1  hit@5    MRR
small  hand        34    53%    76%   0.64
base   hand        34    56%    82%   0.69

Before trusting that, read every miss. A miss means either search failed, or the golden set is wrong. Both happen.

The golden set was wrong five times

Reading the misses one by one turned up 5 labels were wrong on the first run:

  • "How do I undo my last commit but keep the changes?" expected /git/git-reset. That post turns out to be 389 characters long and covers only reset --hard, which discards changes. The site doesn't answer this question. Search wasn't failing; there was nothing to find. It was relabelled as not covered, and noted as a content gap.
  • "How do I set up GraphQL subscriptions with Apollo?" was in the off-topic list, because a search of the text for "graphql subscription" found nothing. Search returned /hasura/hasura-subscription with a high score, and it was right: that post covers GraphQL subscriptions. A phrase search is not a topic search. Replaced with a question about Arduino, which the site truly doesn't cover.
  • Three generated questions were "missed" because search found a different post that also answers them. "Why shouldn't I use floating-point numbers for currency?" was generated from a backend post, and search returned the MySQL data types post, which has a section literally headed "Money: DECIMAL, never FLOAT". Generated questions list only the post they came from, so any other correct answer scores as a miss.

After fixing those, with 33 hand questions and the reviewed synthetic set:

model      set              hit@1  hit@5   MRR
---------  ---------------  -----  -----  ----
bge-small  hand (33)          55%    79%  0.66
bge-small  synthetic (30)     77%    93%  0.85
bge-base   hand (33)          58%    85%  0.71
bge-base   synthetic (30)     87%    93%  0.90

Same model, same search: 93% vs 79% hit@5, synthetic against hand-written. The prompt told Claude not to copy the section's wording, and it mostly didn't copy phrases. But a question written while reading the answer naturally uses the answer's vocabulary and framing, and embeddings reward exactly that. A real reader, who hasn't read the post, asks differently.

Generated questions are still useful: they are cheap, they cover the corpus broadly, and a drop in their score is a real signal. But report the hand-written number as the headline. A 93% you measured on questions written from the answers is not the 93% your readers will get.

What the misses say

The remaining misses fall into a clear pattern, which is worth more than the score:

miss h05 (rank >10): How do I put my uncommitted changes aside so I can switch branches?
     expected /git/git-stash  got /claude/claude-git-workflow (0.721)
miss h18 (rank 8): How do I validate the request body in a NestJS controller?
     expected /nestjs/nestjs-dtos-and-validation  got /nestjs/nestjs-controllers (0.828)

The git question is lexical bait. The Claude Code git post contains, as an example prompt, the sentence "Stash my current changes so I can switch branches", nearly the question word for word. It wins, though it never explains git stash, and the real stash post isn't even in the top ten. Several other misses rank the right post sixth to tenth, just outside what Claude sees. Those are what the next lesson's techniques target.

A score threshold cannot tell you "off-topic"

An obvious idea for detecting off-topic questions: if the best match scores low, the site probably doesn't cover it. The golden set lets us test that:

bge-small  top-1 score  answerable: min 0.705 median 0.791   off-topic: max 0.806 median 0.697
bge-base   top-1 score  answerable: min 0.646 median 0.738   off-topic: max 0.727 median 0.637

The ranges overlap: small answerable min 0.705 vs not-covered max 0.806. The highest-scoring off-topic question is the "undo my last commit" one, which matches the Claude Code post's near-identical prompt at 0.806. Any threshold that rejects it also rejects real questions. Scores are not even comparable between models; bge-base runs about 0.05 lower across the board. A low score is a useful hint, but deciding "the site doesn't cover this" needs something that reads the text. That is lesson 8.

Small or base?

bge-base leads on the hand set, 85% against 79% hit@5. With 33 questions, that is two questions. Two questions is not enough evidence to pay 2.7 times the embedding time and twice the storage, so the decision at this point was to keep bge-small and test again after the next lesson changes the search. Write decisions like that down with the reason: the next lesson reverses it, and the reason is what makes the reversal easy to accept.

Take-aways

  • Write the golden set by hand, for the headline number. Ask like a reader.
  • List every correct answer, not just the one you had in mind.
  • Read every miss before believing a score. Five of our first findings were labelling errors, not search errors.
  • Generated questions are a cheap supplement, and they overstate quality.
  • Small differences on a small set are not results. Say how many questions a difference is.

Next: hybrid search, reranking and pooled judging, where the golden set turns out to be wrong a sixth time, and catching it changes the outcome.