Lessons 8 and 9 judged answers by their form: full, partial, or "not covered". A fluent, complete-looking answer can still be wrong in two different ways. It can state things the sources never said (it is not grounded). Or it can be grounded and still miss the point the post makes (it is not correct, or not complete). This lesson measures both, for the three prompt setups we have built, on the 33 questions the site covers.
Code: lessons/09_answer_quality.py. It grades answers already saved by lessons 8 and 9,
so the comparison is like for like.
Two checks, two references
The two checks compare an answer against different things, and mixing them up is the most common mistake in RAG evaluation:
- Groundedness compares the answer with the passages the model was given. It asks: did the model stay inside its sources? It catches knowledge from training, embellishment and invention. It does not care whether the sources were the right ones.
- Correctness compares the answer with what a good answer should say, written independently from the full posts. It catches answers that are faithful to the wrong section, or faithful but incomplete.
An answer can pass one and fail the other, and the most interesting finding of this lesson is an example.
Judges that return structured data
Both checks use Claude as a judge. Free-text verdicts need parsing, and a parser that fails quietly
loses data. The API's structured outputs remove the problem: define the result as a Pydantic model,
call client.messages.parse, and the response is validated against the schema.
class Claim(BaseModel):
claim: str
verdict: Literal["SUPPORTED", "NOT_SUPPORTED", "CONTRADICTED"]
evidence: str # short quote from the passages, or "" when there is none
class Grounding(BaseModel):
claims: list[Claim]
class PointCheck(BaseModel):
point: str
verdict: Literal["COVERED", "PARTLY", "MISSING", "CONTRADICTED"]
class Correctness(BaseModel):
points: list[PointCheck]
g = client.messages.parse(model=MODEL, max_tokens=8000, output_config={"effort": "low"},
messages=[{"role": "user", "content": GROUNDING_PROMPT.format(
sources=sources, question=q["question"], answer=a)}],
output_format=Grounding)
g.parsed_output is a Grounding instance. Every verdict is one of the
listed values; nothing to regex, nothing lost.
Groundedness: claim by claim
GROUNDING_PROMPT = """You check whether an answer stays within its sources.
<sources>
{sources}
</sources>
<question>{question}</question>
<answer>
{answer}
</answer>
Split the answer into its factual claims: technical statements, what code or a command does, rules,
recommendations, numbers. Skip statements about coverage ("the tutorials don't cover X") and citation
markers. Merge trivial restatements; at most 15 claims. For each claim:
SUPPORTED the sources state it or directly show it (code in the sources counts)
NOT_SUPPORTED the sources do not say it, even if it is true in general
CONTRADICTED the sources say otherwise
Give a short quote from the sources as evidence for SUPPORTED or CONTRADICTED, else ""."""
Judging a whole answer as "grounded or not" hides where the problem is. Splitting it into claims gives a rate (what share of claims is supported) and a list of the exact unsupported sentences to read. Two details in the prompt matter: "even if it is true in general", because a true fact the sources don't contain is still a grounding failure on a site that should answer from its tutorials; and evidence quotes, so a reviewer can check the judge.
The sources given to the judge are re-retrieved exactly as they were when each answer was generated. Retrieval is deterministic, so the judge sees what the model saw.
Correctness: key points from the full posts
For each of the 33 covered questions, Claude read the full expected posts (not the retrieved chunks; that would grade retrieval against itself) and listed the two to five key points a correct answer must make. That produced 160 key points, for $0.76. For "How do I keep a long-running command alive after I disconnect?":
- Run the command inside a GNU screen session, because a process started directly in an SSH shell
stops when the connection drops or times out.
- Start a session with `screen` or a named one with `screen -S <name>`.
- Detach with Ctrl+a Ctrl+d; the program keeps running after you detach or disconnect.
- After reconnecting, reattach with `screen -r`, or `screen -ls` to list sessions.
Generated references are only as good as their review, so every key point was read by hand
against the posts. Two were edited: one was true to the post but a side note rather than part of the
answer, and one bundled two unrelated points. Both edits are recorded next to the question in
eval/keypoints.json.
One caveat shapes how to read the scores: the key points are everything the posts say on the question. A focused answer of three hundred words won't make all five points. Absolute coverage will look low; compare setups with each other, not with 100%.
The results
python lessons/09_answer_quality.py keypoints
python lessons/09_answer_quality.py grade --config v1 # about $1.30 each
python lessons/09_answer_quality.py report
Setup Claims Supported Answers w/ unsupported Key points covered Partly Missing
--------------------------------- ------ --------- ---------------------- ------------------ ------ -------
v1 naive prompt 306 93% 17 / 33 36% 39% 24%
v2 grounded prompt (lesson 8) 266 96% 7 / 33 32% 37% 30%
w1 grounded + neighbours (lesson 9) 308 99% 4 / 33 51% 29% 19%
The current setup is best on both axes. The grounded prompt alone (v2) traded a little completeness for groundedness: fewer unsupported claims, but more honest partial answers, so fewer key points covered. Adding neighbouring chunks (w1) gave the model enough material to be complete and grounded: 99% of claims supported, and 15 points more key-point coverage than v1 (51% against 36%).
Check the judge
A judge's verdicts are data to inspect, not facts. Every unsupported claim in w1, and a sample from v1:
w1 h04 Make sure you pick the correct hash from the reflog.
w1 h13 Use exactly the address shown in your browser's address bar for the React app.
w1 h15 Shared state lives in a provider component and other components read it through a hook ...
w1 h22 The @JoinColumn points to the parent's foreign key column.
v1 h19 For admins you would write @Roles(UserRole.ADMIN).
v1 h16 The same advice from Vue applies to React.
v1 h02 Syncing often and merging quickly reduces conflicts, whichever method you use.
The judge is right, and the difference is instructive. w1's unsupported claims are harmless glue:
reasonable advice the sources just don't state. v1's include genuine extrapolation. The NestJS post
never defines an admin role (it deliberately uses staff), so
@Roles(UserRole.ADMIN) is invented code. And carrying a Vue post's advice over to React
is exactly the kind of leap a tutorial assistant shouldn't make silently.
Grounded, and contradicted
One key point came back CONTRADICTED for the best setup. The question: "How do I keep
each Python project's packages separate from the others?" The answer recommended:
pip freeze > requirements.txt # record exact versions
pip install -r requirements.txt # reproduce them elsewhere
and the key point says "pip freeze > requirements.txt is not equivalent [to a
lockfile]: it records whatever is installed, including hand-installed extras." Was the answer making
things up? No. It was quoting /python/python-modules-packages, which says "pip freeze pins
exact versions, which is what makes an environment reproducible". The key point came from
/python-advanced/python-advanced-virtual-environments-pip, which says freeze "is not the
same thing" and recommends pip-compile and pip-sync.
The site contradicts itself, and the answer was perfectly grounded in one side of the contradiction. That is why groundedness and correctness are scored separately: a grounded answer is only as right as its sources. As a bonus, the evaluation found a content problem worth fixing on the site itself. RAG evaluation regularly does this; treat contradictions it surfaces as bugs in the corpus.
Take-aways
- Measure groundedness against what the model saw, and correctness against an independent reference built from full documents.
- Split answers into claims; the unsupported sentences are more useful than the rate.
- Use structured outputs for judges, and read a sample of verdicts every time.
- Draft references with a model, then review every one by hand and record the edits.
- When a grounded answer is wrong, look at the sources: the corpus may disagree with itself.