Ask the pipeline we have built "What are the three most recent Spring Boot tutorials?" and it answers, correctly and unhelpfully: "The tutorials on this site don't cover when each Spring Boot tutorial was published." It is right. No chunk of any post says "I am one of the three newest Spring Boot posts". The question isn't about what a post says; it is about the catalogue of posts: titles, categories, dates, counts. Similarity search is the wrong tool, and a one-line SQL query is the right one.
Real assistants get both kinds of question. The capstone gives Claude a set of tools, RAG search
among them, and lets it choose. Code: ask_lmsc/agent.py and
lessons/11_agent.py.
Structured data belongs in a table
The post metadata already exists in every exported JSON file. It becomes a posts
table (URL, category, slug, title, excerpt, publish date), maintained by lesson 12's sync, so it
stays current with the chunks. Every sync upserts the metadata of every post; it is cheap.
That small addition shipped with a bug worth describing. The first version also deleted every
posts row whose post was not in the directory being synced. The test suite syncs a
temporary directory of three fake posts, so running the tests wiped the real table.
It was caught before commit, and the next sync restored all 989 rows. The fix has two parts: sync
deletes only posts it has seen removed, and the table name is a parameter, so tests use their own
table. A regression test now inserts a row the sync didn't create and checks that it survives. If a
function deletes data, ask what it deletes when it is run on a smaller input than you expect.
Three tools
Tools are plain Python functions. The Anthropic SDK's tool runner builds each tool's schema from the function signature and docstring, calls Claude, runs whatever tools it asks for, sends the results back, and repeats until Claude answers.
@beta_tool
def list_posts(category: str | None = None, title_contains: str | None = None,
order: Literal["newest", "oldest", "title"] = "newest", limit: int = 10) -> str:
"""List posts from the site's catalogue with their publish dates. Use for which posts exist,
the latest or oldest posts, or counting posts. Call list_categories first if unsure of
the category slug.
Args:
category: category slug, e.g. "spring-boot", "docker", "python". Omit for all posts.
title_contains: only posts whose title contains this text (case-insensitive).
order: "newest" or "oldest" by publish date, or "title" (A-Z).
limit: how many posts to return (1-50).
"""
The docstring is the tool's documentation for the model, so it says when to use the
tool, not only what it does. The function builds a parameterised query (values are bound, and the
sort order comes from a fixed mapping, never from the model's text) and returns JSON including
total_matching, so "how many Docker posts?" is answered by a count, not by counting a
truncated list.
The third tool, list_categories, returns every category with its post count and
latest date. The interesting one is search.
Keeping citations through a tool
Lesson 8's answers quote their sources exactly because the chunks went in as citable documents.
Through a tool, the same thing works with search_result blocks: return a list of them
instead of a string, with citations enabled, and Claude can cite them precisely.
@beta_tool
def search_tutorials(query: str) -> list[dict]:
hits = retrieve(conn, model, query, k=5, mode="vector", per_post=2)
found = expand(conn, model, hits, window=1)
calls.append({"tool": "search_tutorials", "input": {"query": query},
"result": f"{len(found)} passages: " + ", ".join(p["url"][29:] for p in found)})
blocks = []
for p in found:
passages.append(p)
# One text block per paragraph, so a citation can point at the exact paragraphs used.
paragraphs = [x for x in p["text"].split("\n\n") if x.strip()]
blocks.append({"type": "search_result", "source": p["url"], "title": p["title"],
"content": [{"type": "text", "text": x} for x in paragraphs],
"citations": {"enabled": True}})
return blocks or "No passages found."
Inside, it is everything the track built: bge-base vector search, two chunks per post,
one neighbouring chunk each side. Each passage is split into one text block per paragraph, so a
citation can point at the exact paragraphs used. Citations come back as
search_result_location objects whose search_result_index counts every search
result in the conversation, in order. Recording passages in the same order makes the mapping
direct.
Before writing that, two details were checked in the installed SDK rather than assumed: that a tool may return content blocks rather than a string, and the exact citation fields. The SDK's type definitions answer both in seconds, and API surfaces change faster than anyone's memory of them.
Running the loop
runner = (client or anthropic.Anthropic()).beta.messages.tool_runner(
model=MODEL,
max_tokens=8000,
max_iterations=6,
system=SYSTEM,
tools=[search_tutorials, list_posts, list_categories],
messages=[{"role": "user", "content": question}],
)
messages = list(runner)
max_iterations caps the loop, so a confused model can't call tools forever. The system
prompt says which tool fits which kind of question, repeats lesson 8's grounding rules for search
results, and adds one rule for the catalogue: "Report exactly what they return. Never invent a post,
title, date or count."
What are the three most recent Spring Boot tutorials?
list_posts({"category": "spring-boot", "order": "newest", "limit": 3}) -> 36 matching, returned 3
These are the three newest of the 36 posts in the Spring Boot category:
1. Spring Boot – Interview Questions, published 2026-08-18
2. Spring Boot – Cheat Sheet, published 2026-08-16
3. Spring Boot – Building with Gradle, published 2026-08-14
One call, two turns, about two cents. A how-to question goes to search_tutorials and
comes back with quotes, as before.
Testing routing with answers from the database
The evaluation has 15 questions: six about the catalogue, five content questions from the golden
set, two that need both tools, and two off-topic. The interesting design choice is that the expected
answers are not written down. eval/agent_questions.json holds checks, and the
expected answer is computed from the database when the evaluation runs:
{"id": "c1", "type": "catalog", "question": "What are the three most recent Spring Boot tutorials?",
"tools": ["list_posts"], "check": {"kind": "newest", "category": "spring-boot", "n": 3}}
Publish a new Spring Boot post, sync, and the expected answer updates with it. A hard-coded list of three titles would go stale the first time the site changes, and then the test would fail a correct agent.
Each question checks that the agent called the tools it needed, and that the answer is right: catalogue answers must mention the expected posts or the exact count; content answers must cite a post from the golden set; off-topic answers must say the site doesn't cover it.
python lessons/11_agent.py eval
PASS c1 catalog tools: posts PASS t1 content tools: search
PASS c2 catalog tools: categories PASS t2 content tools: search
PASS c3 catalog tools: categories PASS t3 content tools: search search
PASS c4 catalog tools: categories posts PASS t4 content tools: search
PASS c5 catalog tools: posts PASS t5 content tools: search
PASS c6 catalog tools: posts search PASS m1 mixed tools: posts search
PASS o1 off-topic tools: search PASS m2 mixed tools: categories search
PASS o2 off-topic tools: search
15/15 passed, cost $0.69
Catalogue questions cost about two cents, content questions five to eleven. For "Which topic has
the most tutorials?", the agent called list_categories, answered "fundamental-problem,
123 posts", and added that the Java family of categories together totals 137. The arithmetic checks
out.
Check the grader can fail
A perfect score is a reason for suspicion. baseline sends the six catalogue questions
through the plain pipeline, with no tools, and grades them with the same checks:
FAIL c1 answer: "The tutorials on this site don't cover when each Spring Boot tutorial was published ..."
PASS c2 answer: "The Docker track on the site has **22 lessons**, grouped into five sections: ..."
FAIL c3 answer: "The tutorials don't give a full per-topic breakdown, so they can't say which topic ..."
FAIL c4 answer: "The excerpts identify only one of the first three posts by title. ..."
FAIL c5 answer: "The tutorials on this site don't cover which post is the newest on the site."
PASS c6 answer: "Based on the excerpts, two lessons are dedicated to v3 (DDN), ..."
Four of six fail, so the checks discriminate. The two passes are luck: a Docker post happens to say "22 lessons", and "DDN" is in two post titles. Notice also how plain RAG fails: honestly, saying it can't tell, rather than inventing dates. That is lesson 8's prompt still doing its job.
What the track built
Component Choice (and the lesson that measured it)
--------------------- ----------------------------------------------------------------
Chunking headings, whole code blocks, title line, 512-token aware (4)
Embeddings bge-base, local (5, 7)
Search pgvector HNSW, vector only, max 2 chunks per post (7)
Context ±1 neighbouring chunk (9)
Answers grounded prompt, API citations, one-sentence "not covered" (8)
Not adopted hybrid search, reranking (optional), contextual retrieval (7, 10)
Quality hit@5 91%, 99% of claims supported (7, 11)
Updates incremental sync by content hash (12)
Routing agent: search, list_posts, list_categories (13)
And the habits behind it: a golden set written by hand, every miss read, labels corrected by pooled
judging, judges checked, benchmarks checked with EXPLAIN, tests made to fail once, and
every claim in these posts measured. The repository's test console (uvicorn api.main:app
plus npm run dev in web/) puts all of it in a browser, with an
"Agent with tools" switch and every retrieved passage visible.
Take it to your own documents. The code that changes is the export and the chunker; the golden set has to be written fresh, by hand, for your readers' questions. Everything else carries over, including the most useful lesson: measure, then decide.
Back to the start: RAG – what it is and when you need it.