This lesson builds the simplest RAG pipeline that works, end to end, in one file of about a
hundred lines: lessons/02_naive_rag.py. It is deliberately naive. Every choice in it is
the obvious first guess, and the point is to see what those guesses get right and wrong on real data
before improving anything.
Four steps, the same four every RAG system has:
- Turn each post into text and cut it into chunks.
- Embed every chunk once and keep the vectors.
- For a question, embed it and find the closest chunks.
- Give those chunks and the question to Claude and ask for an answer with citations.
Step 1: text and chunks
The naive way to get text out of HTML is to delete every tag. The naive way to chunk is to cut every thousand characters:
def html_to_text(raw: str) -> str:
text = re.sub(r"<[^>]+>", " ", raw) # naive: drops every tag
return re.sub(r"\s+", " ", html.unescape(text)).strip()
def build_index(model: SentenceTransformer) -> None:
chunks, urls, titles = [], [], []
for path in sorted(glob.glob(str(ROOT / "data" / "posts" / "*.json"))):
post = json.loads(Path(path).read_text(encoding="utf-8"))
text = html_to_text(post["html"])
for start in range(0, len(text), CHUNK_CHARS): # naive: fixed-size slices
chunks.append(text[start : start + CHUNK_CHARS])
urls.append(post["url"])
titles.append(post["title"])
print(f"Embedding {len(chunks)} chunks (one-time, cached in {INDEX.name})...")
vectors = model.encode(chunks, batch_size=64, normalize_embeddings=True, show_progress_bar=True)
np.savez(INDEX, vectors=vectors, chunks=chunks, urls=urls, titles=titles)
Why chunk at all, rather than embedding each post whole? Two reasons. An embedding is one point summarising a whole text, so a 15,000-word post that covers twenty topics gets a vector that is close to none of them. And the embedding model can only read so much: the one we use stops at 512 tokens, roughly 2,000 characters of prose, and ignores the rest. A thousand characters keeps each chunk well inside that limit.
Step 2: embeddings
An embedding model maps text to a fixed-length list of numbers. It is trained so that texts with similar meaning land close together: "how do I start a thread in Java" and "creating a new Thread with a Runnable" get vectors pointing in nearly the same direction, even though they share few words.
We use BAAI/bge-small-en-v1.5 through the sentence-transformers library. It is small
(384 numbers per vector), free, runs on a laptop CPU or Apple Silicon GPU, and is good enough to
start. Two details matter:
EMBED_MODEL = "BAAI/bge-small-en-v1.5"
QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
CHUNK_CHARS = 1000
TOP_K = 5
- The query prefix. The bge models were trained with that sentence in front of search queries, and not in front of the passages being searched. Leaving it off a query works, but slightly worse. Every embedding model has conventions like this; read its model card.
normalize_embeddings=True. It scales every vector to length 1. Then the dot product of two vectors equals their cosine similarity: 1 means the same direction, 0 unrelated. One matrix multiplication scores every chunk at once.
The first run embedded 8,171 chunks in about 35 seconds on an
Apple Silicon laptop and saved them to data/naive_index.npz. No database yet; a NumPy
array is plenty for eight thousand vectors.
Step 3: retrieval
def retrieve(model: SentenceTransformer, question: str) -> list[dict]:
data = np.load(INDEX)
query = model.encode(QUERY_PREFIX + question, normalize_embeddings=True)
scores = data["vectors"] @ query # vectors are normalized, so dot product = cosine similarity
best = np.argsort(-scores)[:TOP_K]
return [
{"score": float(scores[i]), "url": str(data["urls"][i]),
"title": str(data["titles"][i]), "text": str(data["chunks"][i])}
for i in best
]
That is the entire search engine: score everything, sort, take five.
Step 4: the answer
def answer(question: str, hits: list[dict]) -> str:
import anthropic
context = "\n\n".join(
f'<source id="{n}" title="{h["title"]}" url="{h["url"]}">\n{h["text"]}\n</source>'
for n, h in enumerate(hits, 1)
)
response = anthropic.Anthropic().messages.create(
model="claude-opus-5-5",
max_tokens=4000,
system=(
"You answer programming questions using only the sources provided. "
"Cite sources inline as [1], [2]. If the sources do not contain the answer, say so."
),
messages=[{"role": "user", "content": f"{context}\n\nQuestion: {question}"}],
)
...
if response.stop_reason == "refusal":
return "(The model declined to answer.)"
return "".join(b.text for b in response.content if b.type == "text")
Each chunk goes in an XML-style <source> tag with a number, which gives Claude a
clean boundary between sources and something to cite. The system prompt asks for three things:
answer from the sources only, cite them, and say so when they don't contain the answer. The script
also prints the token usage, so you see what each question costs.
The API key comes from .env. If you have a Claude Max plan, it includes monthly API
credit once you link a Console organisation to the plan under Settings → Billing on claude.ai;
otherwise buy credit in the Console. Keep the key in .env; if you
export it in your shell profile, Claude Code will bill that key instead of your
subscription.
Running it
python lessons/02_naive_rag.py --retrieve-only "How do I create a thread in Java?"
python lessons/02_naive_rag.py "How do I create a thread in Java?"
Always run --retrieve-only first. It is free and shows what Claude will be given:
[1] 0.797 Multithreading /java/java-advanced-multithreading
[2] 0.791 Java interview – Concurrency /java-interview/java-interview-concurrency
[3] 0.784 Java interview – Concurrency /java-interview/java-interview-concurrency
[4] 0.775 Java interview – Concurrency /java-interview/java-interview-concurrency
[5] 0.771 Java 21 Virtual Threads /java/java-21-virtual-threads
--- [1] ---
Threads let a program do several things at once. ... Starting a thread Copy class Demo {
void run ( ) throws InterruptedException { Thread thread = new Thread ( ( ) -> System . out .
println ( ...
--- [5] ---
t i = 0 ; i < 1_000_000 ; i ++ ) { threads . add ( Thread . ofVirtual ( ) . start ( ...
The full answer was good: both ways to create a thread, the warning to call
start() rather than run(), virtual threads, and a recommendation to use an
ExecutorService, each statement cited. It cost about two cents: around 2,000 input
tokens and 700 output tokens, ~$0.02 per question.
For a pipeline this crude, that is a strong result. It is also hiding several problems.
What is wrong with it
Read the retrieved chunks above closely, then try a few more questions. The failures fall into five groups.
The wrong post wins
What is a Python decorator?
[1] 0.789 TypeScript – Decorators /typescript/typescript-decorators
[2] 0.767 Decorator Pattern /swedesignpattern/decorator-pattern
[3] 0.754 Python – Decorators /python/python-decorators
The TypeScript post ranks first, at 0.789, for a question that says "Python". The chunk that won talks about decorators in general, and nothing in its text says which language it belongs to: the post title, "TypeScript – Decorators", was never part of the chunk. The answer survived because Claude ignored the TypeScript source and used the Python ones, but that is luck we shouldn't rely on.
The code is mangled
System . out . println ( ... is what tag-stripping does to highlighted code. The
site wraps every token in a <span>; replacing each tag with a space puts a space
between every token. Claude quietly repaired it in the answer, which reads well, but it means the code
in the answer is not the code in the post. On a programming site that matters.
"Copy" everywhere
Every code block's copy button contributes the word "Copy" to the text. Harmless to a human, noise to an embedding.
Chunks start mid-word
Chunk 5 starts with t i = 0 ;, the tail of for (int i = 0;. Fixed-size
slicing cuts wherever the thousandth character falls: in the middle of a word, a sentence, a code
sample. The chunk loses its beginning and the previous chunk loses its end.
The same post three times
Three of the five slots for the thread question went to one interview post. Each slot spent on a near-duplicate is a slot not spent on a different post that might add something.
A question the site doesn't cover
The last test is the most important habit in this track: ask something the corpus cannot answer.
How do I deploy a Kubernetes operator written in Rust?
[1] 0.726 Flask – Deployment
[2] 0.696 Flask – Deployment
[3] 0.695 Docker – Running Containers in Production
...
The provided sources don't explain how to deploy a Kubernetes operator written in Rust. ...
Some general container practices in the sources would still apply to an operator image:
- Building the image: Running docker build inside AWS CodeBuild requires privileged mode ...
- Secrets: ... - Container security: ... - Infrastructure as code: ... - Rollout strategies: ...
Claude correctly says the sources don't cover it, then pads the answer with five loosely related tips from Flask and Docker posts. The reader asked about Rust operators and got CodeBuild privileged mode. Notice also the top score: 0.726, against roughly 0.77 to 0.83 for questions the site does answer. Lower, but not dramatically lower. Lesson 6 measures whether a score threshold can catch these, and lesson 8 fixes the padding.
The agenda
Problem Fixed in
--------------------------------------- -------------------------------------------
wrong-language post wins lesson 4 (titles in chunks)
mangled code, "Copy" noise lesson 4 (parse the HTML properly)
chunks cut mid-word lesson 4 (split on structure)
one post fills the top five lesson 7 (a per-post cap)
padded "not covered" answers lesson 8 (a grounded prompt)
"is it actually better?" lesson 6 (a golden question set)
Keep this script. It is the baseline, and every later lesson can be compared against it with the same question.
Next: chunking by structure.