Some questions are still missed by every search method we have tried. The usual explanation is that a chunk, cut out of its post, loses context: a section that says "use a projection to fetch only certain fields" may never mention JPA or say what a projection is for. Contextual retrieval, a technique published by Anthropic, addresses that directly. Before embedding each chunk, have an LLM read the whole document and write a sentence or two situating the chunk in it, then embed the context and the chunk together.
This lesson implements it for all 11,701 chunks, measures it, and does not adopt it. The result is as useful as a win: it shows when a well-known technique has nothing to add, and how to be sure that a "no improvement" result is real.
Code: lessons/08b_contextual_retrieval.py, with commands
contextualize, ingest and eval.
Writing the contexts
The prompt is the one from Anthropic's write-up. The whole post goes in the system prompt, the chunk in the message:
PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the \
purposes of improving search retrieval of the chunk. Answer only with the succinct context and \
nothing else."""
r = client.messages.create(
model=CTX_MODEL,
max_tokens=1000,
output_config={"effort": "low"},
# The whole post is identical for every chunk of that post, so it is cached: the first
# call writes it, the rest read it at a tenth of the price. (Posts shorter than the
# model's minimum cacheable length simply aren't cached.)
system=[{"type": "text", "text": f"<document>\n{document}\n</document>",
"cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user", "content": PROMPT.format(chunk=c.text)}],
)
Two cost decisions make this affordable:
- Prompt caching. A post with twelve chunks means twelve calls with the same
post in front. Marking the post with
cache_controlmeans the first call writes it to the cache and the other eleven read it at a tenth of the input price. The script sends each post's first chunk alone, then the rest in parallel, so the cache exists before they need it. - A small model. 11,701 short labelling calls is a bulk job, so it runs on Claude Haiku 5.5 rather than Opus. Estimated with Opus, this would have cost $50 to $80; with Haiku, about $2.50. If a reader can't afford to reproduce a tutorial step, it isn't a tutorial step.
A test on five posts first (46 chunks, a cent) confirmed the cache was working and produced contexts like this one:
This chunk is from a post on AI-assisted AWS investigation, specifically the section on how the
assistant reasoned about cost. It explains that Cost Explorer could not price the leftover API
Gateway and CloudFront resources individually, so the assistant used whole-account spend ...
Specific and accurate, which is the point.
Two things that went wrong running it
The full run stopped after 2,757 chunks:
anthropic.RateLimitError: Error code: 429 - Number of concurrent requests across all models has
exceeded your organization's limit.
New API accounts start on a low limit for simultaneous requests, separate from requests
per minute. Eight posts at a time with four calls each was too many. Two changes fixed it: three
posts at a time with two calls each, and anthropic.Anthropic(max_retries=8), so a
brief 429 waits and retries with backoff instead of killing the run. Because every context is
appended to a JSON-lines file as it arrives, the rerun picked up where the first one stopped.
Make long batch jobs resumable; you will need it.
The second problem was quieter. The script ran in the background as
python ... | grep -v noise, and the job reported success. It had crashed: the pipeline's
exit status is grep's, not Python's. set -o pipefail makes the pipeline fail
when any part of it fails. Without it, the only clue was a traceback in a log nobody was reading.
The complete run: $2.05, with 43M cache-read vs 2.3M cache-write tokens. Caching did almost all the work. The median context 57 words.
Embedding with context
vectors = embed.embed_passages("base_ctx", [f"{contexts[key(c)]}\n\n{c.text}" for c in chunks])
with store.connect() as conn:
store.recreate(conn, "base_ctx")
# `content` stays the original chunk: the context is for search only, Claude never sees it.
store.load_rows(conn, "base_ctx", chunks, vectors)
The contextualised vectors go into their own table, chunks_base_ctx, using the same
bge-base model, so the comparison with chunks_base measures the contexts and
nothing else. Only the vectors change: the stored text, and what Claude reads, is the original
chunk.
The result
Table Set hit@1 hit@5 MRR
-------- -------------- ----- ----- ----
base hand (33) 73% 91% 0.81
base_ctx hand (33) 70% 91% 0.78
base synthetic (30) 87% 93% 0.91
base_ctx synthetic (30) 80% 90% 0.85
No gain. The same hit@5, slightly lower hit@1 and MRR. Top-1 changed on 8/33 hand questions: three wins (Terraform remote state and iOS networking moved to first), three losses, two neutral. One loss shows how a context can hurt. For "How do I store JSON in a Postgres column and query fields inside it?", the MySQL JSON post now ranks first. Here are the two contexts Haiku wrote:
MySQL chunk (now #1):
This chunk is the decision guidance near the end of a MySQL JSON columns lesson, ... to explain
when to store data as JSON versus ordinary columns, ...
Postgres chunk (was #1):
This chunk opens a guide on PostgreSQL's JSON and JSONB data types, ... The rest of the document
covers operators for extracting values, containment and existence searches, indexing strategies,
jsonb's effects on stored data, modifying documents, converting between JSON and rows, ...
The MySQL context is about storing JSON in columns, almost the question's own words. The Postgres context dutifully summarises the whole post, and a vector of a list of eight topics is close to none of them. The context that was meant to situate the chunk diluted it instead.
Making sure "no improvement" is real
Lesson 7 showed how a fair method can look worse when the golden set is incomplete. The pooled judgments covered what the earlier setups returned, not this one. So the posts contextual retrieval put in its top five that had never been judged were judged now: 27 (question, post) pairs for $0.09. None was relevant. The losses are real.
And there is a cost we can measure without any labels. bge models read at most 512 tokens. The added context pushes chunks over that limit from 21 to 86, and those chunks lose their tails from search entirely.
Why it didn't help here
Since lesson 4, every chunk has started with
[category] Post title > Section heading. That line is a context, a cheap,
deterministic one, and on this site it carries most of what the LLM-written sentence adds: which
language, which framework, which topic. Contextual retrieval was designed for chunks that lose their
meaning when cut out, such as financial filings where a paragraph says "revenue grew 3%" without
naming the company or quarter. Tutorial sections with descriptive headings are close to the opposite
case.
That makes the decision easy: don't adopt it. The table and the script stay in the repository, because "we tried it, measured it and here is why it didn't help" is worth keeping. On a different corpus, the same script might well show a gain, and now you have the harness to find out for two dollars.
When it is worth trying
Contextual retrieval is likely to help when chunks depend on their document for meaning:
- Chunks full of pronouns and references: "the company", "this quarter", "the function above".
- Documents without useful headings, so there is no cheap title line to add.
- Many near-identical documents that differ by one fact, such as contracts, filings or per-product pages, where the distinguishing fact is far from most chunks.
It is less likely to help, as here, when documents have descriptive titles and headings you can put in every chunk for free. Either way, run the comparison: it takes one script, about two dollars and an afternoon, and it replaces an opinion with a number.
Take-aways
- Try the cheap, deterministic version of an idea first. A title and heading line got most of this technique's value for free.
- Prompt caching and a small model turn a $50 job into a $2 one.
- Make batch jobs resumable, respect concurrency limits, and use
pipefail. - Before accepting "no improvement", check the labels cover what the new method returns.
- Look for costs labels can't see, like chunks pushed past the embedding limit.