Retrieval is now measured and tuned. This lesson turns to the other half: what Claude does with
what it is given. Two problems have been visible since the naive pipeline. When the site doesn't
cover a question, Claude says so and then pads the answer with loosely related tips. And its
[1], [2] citations are claims nobody checks: nothing guarantees that source
3 says what the sentence citing it says.
Both are fixed here: a prompt with explicit rules for full, partial and missing coverage, and the Claude API's citations feature, which returns the exact source text behind each cited statement. Then we measure the old and new prompts on all 40 hand-written questions.
Code: ask_lmsc/grounded.py and lessons/07_grounded_answers.py. Retrieval
is lesson 7's best setup: bge-base, vector search, two chunks per post, top five.
A prompt with three outcomes
The naive prompt said "If the sources do not contain the answer, say so." That covers one case and leaves the model to improvise everything else. The new prompt names all three outcomes and what to do in each:
NOT_COVERED = "The tutorials on this site don't cover"
SYSTEM = f"""You answer questions for readers of lovemesomecoding.com, a programming tutorial site, \
using only the tutorial excerpts provided as documents.
- If the excerpts answer the question, answer it directly. Use their code when it helps. Do not add \
facts, APIs, flags or steps that the excerpts do not contain, even if you know them.
- If the excerpts answer only part of the question, answer that part, then say plainly which part \
the tutorials don't cover.
- If the excerpts do not answer the question, reply with one sentence that starts with \
"{NOT_COVERED}" and stop. Do not add general advice, related tips or anything from outside the \
excerpts: a reader who wants that can ask a general assistant."""
Three choices in it are deliberate:
- "Even if you know them." The model knows a lot about Docker. On a site whose answers should match its tutorials, its own knowledge is the thing to keep out. Saying so explicitly works better than hoping "only the excerpts" is read strictly.
- Partial coverage is its own case. Most real questions are partly answered. Without a rule, a model either refuses or fills the gap from memory. Asking it to say which part is missing turns out to be the most useful instruction in the prompt, as the results show.
- A fixed opening sentence for "not covered". It gives a clean, short reply, and
it lets code detect the outcome with a
startswith, so the UI can show it differently. The prompt also says why it should stop (the reader can ask a general assistant), because a model follows a rule better when it knows the reason.
Citations the API checks
Instead of pasting sources into the prompt as text, each chunk goes in as a
document content block with citations enabled:
def answer(question: str, hits: list[dict], client: anthropic.Anthropic | None = None) -> dict:
documents = [
{
"type": "document",
"source": {"type": "text", "media_type": "text/plain", "data": h["text"]},
"title": h["title"],
"context": h["url"],
"citations": {"enabled": True},
}
for h in hits
]
response = (client or anthropic.Anthropic()).messages.create(
model=MODEL,
max_tokens=4000,
system=SYSTEM,
messages=[{"role": "user", "content": documents + [{"type": "text", "text": question}]}],
)
The response is then a list of text blocks, and each block that relies on a source carries citations: which document, the character range, and the cited text itself. A short probe shows the shape:
text 'Create a Thread with a task (e.g., `new Thread(() -> ...)`, a Runnable implementation, ...'
cite char_location 0 0 562 '[java] Multithreading > Starting a thread\n\n```java\nclass Demo {...'
text ' — never call `run()` directly, since '
text 'run() just calls the method on the current thread — a mistake that produces code which ...'
cite char_location 0 562 709 'run() just calls the method on the current thread — a mistake ...'
The quoted text is guaranteed to be an exact span of the document. That is the
difference from prompt-requested [3] markers: a quote can't be invented. Whether the
quote actually supports the sentence is still worth checking (lesson 11 does), but you can
now show the reader the exact words each claim rests on.
Turning blocks into an answer
def _render(content) -> tuple[str, list[dict]]:
out, quotes, pending, in_fence = [], [], [], False
for block in content:
if block.type != "text":
continue
out.append(block.text)
in_fence ^= block.text.count("```") % 2 == 1
for c in block.citations or []:
n = c.document_index + 1
quotes.append({"n": n, "quote": c.cited_text.strip()})
if n not in pending:
pending.append(n)
if pending and not in_fence:
out.append("".join(f"[{n}]" for n in pending))
pending = []
if pending:
out.append("".join(f"[{n}]" for n in pending))
return "".join(out).strip(), quotes
It joins the blocks into Markdown, adds a [n] marker after each cited block, and
collects the quotes so the UI can show them under each source. One detail matters on a programming
site: a citation can end inside a code block, and a marker inserted there becomes part of the code a
reader copies. The function tracks whether it is inside a fence and holds markers until the fence
closes. Also note that the API attaches citations to prose, not to the code blocks themselves, so
code in an answer carries no marker even when it came straight from a source.
Trying it
python lessons/07_grounded_answers.py ask "How do I deploy a Kubernetes operator written in Rust?"
status: not_covered $0.0189
The tutorials on this site don't cover deploying a Kubernetes operator written in Rust.
One sentence, where the naive prompt produced five paragraphs of Flask and CodeBuild advice. Answered questions now come with their evidence: for "How do I create a thread in Java?" each claim lists the quote it rests on, such as "There are two ways you can create a thread. You can extend the Thread class or implement the Runnable interface."
Measuring it
One example proves nothing, so eval answers all 40 hand-written questions with both
the old prompt (v1) and the new one (v2), and has Claude classify each answer's form, not
its correctness: answered, partial (answers part and says what is missing), a clean "not covered",
or a padded "not covered".
python lessons/07_grounded_answers.py eval # about $2.30
Prompt Site covers it? answered partial clean "no" padded "no"
------ --------------- -------- ------- ---------- -----------
v1 yes (33) 22 10 0 1
v1 no (7) 0 2 2 3
v2 yes (33) 18 14 1 0
v2 no (7) 0 1 6 0
On questions the site doesn't cover, padding goes from 3 to 0 and clean refusals from 2 to 6. The seventh is the "undo my last commit" question, where the site has only a Claude Code prompt about it; a partial answer pointing at that is fair. Costs were similar: v1 $1.04, v2 $1.25 for 40 answers including judging, about 2.5 cents an answer.
This also settles lesson 6's open question. A similarity threshold could not separate answerable from off-topic questions; the model, reading the actual text, does it in six cases out of seven. Abstention belongs in the generation step, not in a score cutoff.
The surprising part: more partial answers
On covered questions, v2 gives fewer full answers (18 against 22) and more partial ones (14 against 10). That looks like a regression until you read them:
h32 "The excerpts I have stop at the 'Start a screen session' heading. They don't include the
commands or key combinations for starting a session, detaching, or reattaching."
h29 "These excerpts don't show how to write a builder class itself, so you'll need the full
Builder Pattern tutorial for that."
h15 "They don't show the code for creating a context, providing a value, or reading it in a
nested component. For that, see the full 'Passing Data Deeply with Context' tutorial."
Every one is true. Search found the right post, but not the chunk that holds the commands or the
code: the screen post's commands sit in the section after the one that matched.
The new prompt is honestly reporting a retrieval gap that the old one glossed over.
Did the old prompt fill those gaps by inventing things? We checked every code block in every
answer against the chunks the model was given. Only 1/42 (v1) and 1/49
(v2) code blocks weren't found in the retrieved text, the same question in both, and on
inspection a false alarm (the post writes those commands with a $ prompt). So v1 wasn't
fabricating code; it was writing around the gaps. v2 names them, which is what tells us what to fix
next.
Take-aways
- Spell out partial and missing coverage in the prompt, with a fixed form for "not covered" and the reason to stop.
- Use API citations instead of asking for
[n]markers. Quotes you can show beat references you have to trust. - Let the model decide "not covered". It reads the text; a threshold doesn't.
- Read the answers your metric dislikes. Here the "worse" prompt was the honest one, and its complaints pointed at the next fix.
Next: search small, answer big, which gives the model the chunks around each hit.