The naive pipeline's worst problems all came from one place: how it turned posts into chunks. It
stripped tags blindly, so code came out as System . out . println (. It cut every
thousand characters, so chunks started mid-word. And no chunk said which post it came from, so a
TypeScript section about decorators beat the Python post for "What is a Python decorator?".
This lesson changes only the chunking. The embedding model, the NumPy search and the prompt stay
exactly as they were, so any difference in the results comes from chunking alone. The code is in
ask_lmsc/chunking.py, and lessons/03_chunking.py runs it with a
--compare flag that prints the naive and new results side by side.
Five rules
- Split at headings. A section under an
<h2>is usually about one thing, and an author's headings are a better guide to topic boundaries than a character count. - Never split inside a code block, and keep code exact.
- Drop the "Copy" buttons.
- Prefix every chunk with where it came from: category, post title, section heading.
- Skip stub posts with almost no text.
Parse, don't strip
All five rules need the document's structure, so the chunker parses the HTML with BeautifulSoup
instead of deleting tags with a regular expression. One warning first, because it cost time on this
site's original WordPress migration: if your HTML contains code samples with raw,
unescaped markup in them (a <script> tag shown as an example, say), an
HTML parser will treat it as real markup and silently delete it. Extract the <pre>
blocks with a regex before parsing in that case. Our exported pages are the rendered site,
where code is properly escaped, so a parser is safe here. Check which situation you are in.
The parser walks the article and yields a stream of headings, text blocks and code blocks in document order:
def _blocks(node: Tag):
for child in node.children:
if isinstance(child, NavigableString):
if _clean(child):
yield "text", _clean(child)
continue
if not isinstance(child, Tag):
continue
name = child.name
if name == "button": # the "Copy" button on every code block
continue
if name in HEADINGS:
yield "heading", _clean(child.get_text(" "))
elif name == "pre":
lang = next((c[9:] for c in child.get("class", []) if c.startswith("language-")), "")
# get_text() with no separator joins the Prism spans back into the original code.
code = child.get_text().strip("\n")
for piece in _split_code(code):
yield "code", f"```{lang}\n{piece}\n```"
...
else: # div, section, ul, ol, figure, ... : look inside
yield from _blocks(child)
Three details carry most of the weight:
get_text()with no separator on a<pre>joins the syntax-highlighting spans back together exactly, soSystem.out.println(comes out as written, with its original indentation and newlines. Everywhere else we useget_text(" "), because there a missing space would glue two words together.- Code is wrapped in a Markdown fence with its language, so Claude sees clearly where code starts and ends, and in which language.
- The last branch recurses. On a modern post, headings and paragraphs are direct
children of the article. But 142 posts nest content inside WordPress
layout
<div>s, three or four levels deep. A chunker that only looked at top-level elements would see nothing in those posts. The recursion stops at the first block-level element, so a paragraph is never emitted twice.
Tables become one row per line with | between cells, and list items keep a
- prefix. Before writing this, a quick survey of every post's top-level tags (8,033
h2/h3 headings, 492 tables, 7,339 divs) said which shapes
needed handling. Survey first; it is a ten-line script and it stops you guessing.
Grouping blocks into chunks
for kind, text in _blocks(soup):
if kind == "heading":
flush()
heading = text
continue
# Start a new chunk at a block boundary once this one is full. A single block bigger than the
# target (a long code sample) still goes in whole: half a function is worse than a big chunk.
if parts and size + len(text) > TARGET_CHARS:
flush()
parts.append(text)
size += len(text)
flush()
return chunks
A heading always starts a new chunk. Within a section, blocks are added until the next one would take the chunk past about 1,200 characters, and then a new chunk starts, always at a block boundary. Nothing is ever cut mid-sentence or mid-code.
Every chunk says where it came from
@dataclass
class Chunk:
url: str
title: str
category: str
heading: str
body: str
@property
def text(self) -> str:
"""What gets embedded and shown to the model: context line, then the content."""
where = f"{self.title} > {self.heading}" if self.heading else self.title
return f"[{self.category}] {where}\n\n{self.body}"
A chunk from the Python decorators post now starts with
[python] Python – Decorators > Stacking decorators. That one line is what fixes the
wrong-language problem: the embedding of a chunk now carries "Python" even when the section's
own text never says it. It also helps Claude, which sees which post and section each source is.
Here is the multithreading chunk that the naive version mangled:
[java] Multithreading > Starting a thread
```java
class Demo {
void run() throws InterruptedException {
Thread thread = new Thread(() -> System.out.println("on another thread"));
thread.start(); // start(), never run()
thread.join(); // wait for it to finish
...
```
start() creates a thread and runs the task on it. run() just calls the method on the cur...
Stubs, and the 512-token limit
Posts with under 200 characters of text are skipped: the eighteen stubs from the last lesson produce no chunks at all.
The less obvious rule is a size limit. bge-small reads at most 512
tokens and silently ignores everything after that. No error, no warning: the end of
a long chunk simply doesn't exist as far as search is concerned. Prose runs about four characters
per token, so 1,200-character chunks are safe, but rule 2 says a long code block goes in whole, and
code is dense: closer to three characters per token.
Measure it rather than guess. Counting tokens with the model's own tokenizer showed 120 chunks (1%) over the limit: long code samples whose tails could never be found by search. So rule 2 gets one exception:
def _split_code(code: str) -> list[str]:
"""Whole block if it fits; otherwise line-aligned pieces, preferring blank lines as cut points."""
if not code.strip():
return []
if len(code) <= MAX_CODE_CHARS:
return [code]
pieces, current = [], []
for line in code.split("\n"):
size = sum(len(x) + 1 for x in current)
at_blank = not line.strip()
if current and (size + len(line) > MAX_CODE_CHARS or (at_blank and size > MAX_CODE_CHARS * 0.6)):
pieces.append("\n".join(current).strip("\n"))
current = []
current.append(line)
if "\n".join(current).strip():
pieces.append("\n".join(current).strip("\n"))
return pieces
Code blocks over 1,400 characters are split at line breaks, preferring blank lines, which in most
code fall between functions. That left 21 left (0.2%), all dense
tables such as ASCII EXPLAIN output. That was accepted as a known limit rather than
adding more special cases.
The result: 11,701 chunks (vs 8,171), with a median 179 tokens. More, smaller and cleaner chunks.
Did it work?
python lessons/03_chunking.py --compare "What is a Python decorator?"
lesson 2 (fixed 1,000-char chunks)
[1] 0.789 /typescript/typescript-decorators
[2] 0.767 /swedesignpattern/decorator-pattern
[3] 0.754 /python/python-decorators
[4] 0.735 /python-advanced/python-advanced-interview-questions
[5] 0.733 /python/python-decorators
lesson 3 (structure-aware chunks)
[1] 0.872 /python/python-decorators
[2] 0.858 /python-advanced/python-advanced-interview-questions
[3] 0.811 /typescript/typescript-decorators
[4] 0.803 /swedesignpattern/decorator-pattern
[5] 0.801 /python/python-decorators
The Python post goes from third to first, python-decorators 0.872. Across four test questions:
Question Naive top hit Structured top hit
------------------------------- ------------------------------------ ---------------------------------
Python decorator typescript-decorators 0.789 (wrong) python-decorators 0.872
Thread in Java java-advanced-multithreading 0.797 same, 0.823
CORS in Spring Boot spring-boot-web-mvc 0.827 same, 0.851
Rust k8s operator (not covered) flask-deployment 0.726 flask-deployment 0.717
Two things in that table matter beyond the fix:
- Answerable questions score higher; the uncovered one doesn't. The gap between "the site answers this" and "the site doesn't" roughly doubled, from about 0.05 to about 0.10. Cleaner chunks make real matches stronger without making false ones stronger.
- The code now reaches Claude verbatim. Asked how to create a thread, Claude's
answer quoted the post's code with its original comments,
// start(), never run(), instead of reconstructing it from token soup.
One problem is untouched: three of the top five for the thread question are still the same interview post. That is a ranking problem, not a chunking one, and lesson 7 deals with it.
What to take from this
Chunking is the cheapest, highest-leverage part of a RAG system, and the part most tutorials spend the least time on. Split where the author split. Keep code whole. Put the document's title in every chunk. Check your embedding model's input limit against real chunk sizes with the real tokenizer. None of it needs a better model, and it moved the right answer from third to first.
Three questions are still not evidence, though. Before going further, the next lesson moves the vectors into a real database, and the one after builds the measurement every later change is judged by.