The last lesson's grounded prompt kept reporting the same kind of gap: "the excerpts stop at the 'Start a screen session' heading", "these excerpts don't show how to write a builder class itself". Search had found the right post but not the chunk that held the commands or the code. This lesson fixes that without touching search at all.
Two jobs, one chunk size
Chunk size pulls in two directions:
- Search wants small chunks. An embedding is one point for the whole chunk. A focused chunk about one thing matches a question about that thing strongly; a large chunk covering four things matches each of them weakly.
- Answering wants big passages. The section that matches the question is often
the one that introduces the answer, and the commands or code come right after it. The
screenpost's matching chunk explains why you would want screen; the next one hasscreen -S,Ctrl+a dandscreen -r.
Making chunks bigger would weaken search. The usual answer, often called small-to-big or parent-document retrieval, is to stop using one size for both jobs: search on small chunks, then give the model a bigger window of text around each hit.
Expanding hits into passages
Each chunk row has an id, and because ingest inserts each post's chunks in document
order, the chunks before and after a hit are simply id - 1 and id + 1 with
the same URL. expand() in ask_lmsc/retrieve.py grows each hit to a window
of neighbours and merges windows that touch:
def expand(conn, model: str, hits: list[dict], window: int = 1) -> list[dict]:
if window <= 0 or not hits:
return hits
ranges: dict[str, list[list]] = {} # url -> [[lo, hi, best_hit], ...]
for h in hits:
lo, hi = h["id"] - window, h["id"] + window
spans = ranges.setdefault(h["url"], [])
for span in spans:
if lo <= span[1] + 1 and hi >= span[0] - 1: # overlaps or touches: merge
span[0], span[1] = min(span[0], lo), max(span[1], hi)
break
else:
spans.append([lo, hi, h])
Merging matters. Two hits from the same post a section apart would otherwise produce two overlapping passages, repeating text and wasting tokens. Merged, they become one coherent passage. The per-post cap from lesson 7 still applies to the hits, so no post can take over the context.
Then each range is read back in order and stitched together, keeping the
[category] Title > Heading line once and turning later section changes into Markdown
headings:
passages = []
for url, spans in ranges.items():
for lo, hi, best in spans:
rows = conn.execute(
sql.SQL("SELECT id, heading, content FROM {} WHERE url = %s AND id BETWEEN %s AND %s ORDER BY id")
.format(table(model)), (url, lo, hi)).fetchall()
parts, heading = [], None
for _, hd, content in rows:
if heading is None:
parts.append(content) # keeps the "[category] Title > Heading" line once
elif hd != heading:
parts.append(f"## {hd}\n\n{_body(content)}")
else:
parts.append(_body(content))
heading = hd
passages.append({**best, "text": "\n\n".join(parts), "chunks": len(rows)})
rank = {h["id"]: i for i, h in enumerate(hits)}
return sorted(passages, key=lambda p: rank[p["id"]])
The WHERE url = %s keeps a window from crossing into the neighbouring post, whose
first chunk has the next id. Passages keep the rank of their best hit, so the most relevant passage
is still document 1 for Claude's citations.
For the screen question, five hits became four passages, and the context roughly
doubled, from about 5,100 to 10,200 characters. The screen passage now runs through
"How does screen work?" and on to the commands.
The ordering assumption
"Neighbours are id ± 1" is only true while each post's chunks have consecutive ids in document order. A full ingest guarantees it. Updating posts one at a time (lesson 12) has to preserve it: deleting a changed post's chunks and re-inserting them all in one transaction, in order, keeps them consecutive with a single writer. Lesson 12's tests check exactly this. If your ingest ever runs in parallel, store an explicit position column instead of relying on ids.
Measuring it
Search is unchanged, so the retrieval numbers from lesson 7 still hold. What should change is the
answers. lessons/08_small_to_big.py reruns lesson 8's answer evaluation with expansion,
for one and two neighbours each side:
python lessons/08_small_to_big.py eval --window 1 # about $1.80
python lessons/08_small_to_big.py eval --window 2
python lessons/08_small_to_big.py changed 0 1 # which questions changed, and why
Window Covered (33): answered / partial / clean no Not covered (7): partial / clean no / padded Input tokens (median)
----------- -------------------------------------------- --------------------------------------------- ---------------------
0 (lesson 8) 18 / 14 / 1 1 / 6 / 0 ~3,200
1 24 / 8 / 1 1 / 6 / 0 5,740
2 26 / 6 / 1 1 / 6 / 0 7,803
One neighbour each side turns six partial answers into full ones, +6 full answers net (7 up, 1 down), with no new padding on the questions the site doesn't cover. The ones that moved are exactly the gaps lesson 8 reported:
h32: PARTIAL -> ANSWERED How do I keep a long-running command alive after I disconnect ...
The answer directly addresses the question with cited, step-by-step instructions for using screen.
h29: PARTIAL -> ANSWERED My Java constructor has eight parameters. Is there a cleaner way ...
h17: PARTIAL -> ANSWERED How do I stop one broken React component from blanking the whole page?
h18: PARTIAL -> ANSWERED How do I validate the request body in a NestJS controller?
h14: ANSWERED -> PARTIAL How do I send an email after returning the HTTP response in FastAPI ...
The one that went the other way, h14, now answers the question and then notes that the tutorials don't show how the email helper itself is written. That is a fair caveat, not a worse answer. With two neighbours it flipped back, which is a good reminder that a single question moving between "answered" and "partial" is within the noise of a model-judged label.
How big a window?
Two neighbours each side added two more full answers (24 to 26), one of them that noisy h14, for another +36% input tokens. The runs cost $1.81 and $2.23 for the same 40 questions. One neighbour gets most of the gain, so it is the default in the API and the test console, where a "Context" selector lets you compare 0, 1 and 2 on your own questions.
Longer context has costs beyond money. More text means more chances for the model to pick up something irrelevant, and more for the reader to check. Measure where the gains stop, and stop there.
Other ways to do the same thing
A fixed window of neighbouring chunks is the simplest version. Three variants are common, and the right one depends on your documents:
- Whole section. Return every chunk under the same heading as the hit. Natural
for well-structured docs, but a section can be one paragraph or twenty, so the context size swings
widely. Our long sections would blow the budget, and the
screencommands sit in the next section, which this wouldn't include. - Whole document. When documents are short (support articles, FAQ entries), skip the windows and send the entire document. Simple, and nothing is ever cut off, but most posts here run to thousands of words.
- Sentence windows. Embed single sentences for very precise matching, then send a few sentences around each hit. Precise, but it multiplies the number of vectors, and code blocks don't split into sentences.
A chunk window suits this corpus: the chunks already follow the author's structure, and one neighbour each side reliably reaches the "how" after a matching "why".
What it doesn't fix
Expansion only helps when search found the right post. The questions where search never finds it (the git stash question beaten by a near-identical sentence elsewhere, JPA projections) are unchanged: no amount of context around the wrong chunk helps. The next lesson tries a well-known technique aimed at exactly that kind of search failure.
Take-aways
- Search and answering want different chunk sizes. Don't compromise; use small chunks to find and bigger passages to answer.
- Merge overlapping windows, keep them inside one document, and keep the best hit's rank.
- Know what your neighbour lookup assumes about ordering, and test it.
- Measure window sizes; the gains flatten fast while the cost keeps growing.