Before any retrieval or any model, a RAG project needs two boring things: somewhere to keep vectors, and a copy of the documents it will answer from. This lesson sets up both. It is also where the corpus first shows its quirks, which is why it is worth reading even if you have done a Docker setup a hundred times.
All the code is in the
ai_repo_lovemesomecoding.com
repository. One note on numbering: the scripts in lessons/ are numbered by the lesson
that introduces them, starting at 02_naive_rag.py for the next post, so each post names
the exact file it runs.
The project layout
ai_repo_lovemesomecoding.com/
docker-compose.yml Postgres + pgvector
db/init.sql enables the vector extension
scripts/export_posts.py
ask_lmsc/ the reusable pieces: chunking, embedding, storage, retrieval, answers
lessons/ one runnable script per lesson
eval/ golden questions, judgments and saved results
api/ web/ FastAPI + React test console
tests/
data/ exported posts and caches (gitignored: everyone generates their own)
.env ANTHROPIC_API_KEY (gitignored)
Python first:
git clone https://github.com/folaulau/ai_repo_lovemesomecoding.com.git
cd ai_repo_lovemesomecoding.com
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
requirements.txt grows with the track; by the end it holds the HTTP client, NumPy,
sentence-transformers for local embeddings, the Anthropic SDK, BeautifulSoup for parsing, psycopg and
pgvector for the database, FastAPI for the console and pytest. The first install pulls PyTorch, which
is the slow part.
Postgres with pgvector
A vector database stores vectors and finds the ones nearest to a query vector. You could use a
dedicated one, but Postgres with the pgvector extension does the job for millions of
vectors, and it means the chunk text, the metadata, keyword search and the vectors all live in one
place you already know how to query. One container:
services:
db:
image: pgvector/pgvector:pg17
container_name: ask-lmsc-db
environment:
POSTGRES_USER: rag
POSTGRES_PASSWORD: rag
POSTGRES_DB: rag
ports:
- "${PG_PORT:-5434}:5432"
volumes:
- pgdata:/var/lib/postgresql/data
- ./db/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
healthcheck:
test: ["CMD-SHELL", "pg_isready -U rag -d rag"]
interval: 2s
timeout: 3s
retries: 15
The init script runs once, on the first start of an empty volume, and does one thing:
CREATE EXTENSION IF NOT EXISTS vector;
Note the port. The default Postgres port, 5432, is very often taken by a local Postgres
install, and on the machine this track was built on 5433 was taken too, by another container. Rather
than fight it, the compose file maps the container to 5434 by default, overridable
with PG_PORT. .env.example has the matching connection string:
ANTHROPIC_API_KEY=
PG_PORT=5434
DATABASE_URL=postgresql://rag:rag@localhost:5434/rag
Start it and check pgvector works:
docker compose up -d --wait
docker exec ask-lmsc-db psql -U rag -d rag -c "select '[1,2,3]'::vector <=> '[1,2,4]'::vector;"
<=> is pgvector's cosine distance operator. The answer, about 0.0085, says
those two vectors point in almost the same direction. Lesson 5 is about using that operator on real
embeddings.
Getting the posts
The corpus is every post on this site. The site is a static build, so there is no API to call,
but it publishes a search-index.json listing every post's URL, title, category and
excerpt for its own search box. That gives us the list; the posts themselves are ordinary pages.
The script is scripts/export_posts.py. The heart of it fetches one page and keeps
the article:
ARTICLE_RE = re.compile(r'<article class="prose">(.*?)</article>', re.S)
META_RE = re.compile(r'<meta property="article:(published_time|modified_time)" content="([^"]*)"')
...
article = ARTICLE_RE.search(page)
if not article:
return "error: no <article class=\"prose\"> on page", path
meta = dict(META_RE.findall(page))
record = {
"url": SITE + path,
"category": category,
"slug": slug,
"title": entry["t"],
"excerpt": entry.get("x", ""),
"published": meta.get("published_time"),
"modified": meta.get("modified_time"),
"html": article.group(1).strip(),
"etag": resp.headers.get("ETag"),
}
out.write_text(json.dumps(record, ensure_ascii=False, indent=1), encoding="utf-8")
return "saved", path
Each post becomes one JSON file in data/posts/. Run it:
python scripts/export_posts.py --limit 5 # a quick sample first
python scripts/export_posts.py # all of it, about a minute
The full run listed and saved all 989 posts exported, 0 errors,
about 24 MB. The script downloads four pages at a time (be polite to sites you scrape), retries
transient failures with backoff, and skips files it already has, so you can stop and resume it. The
etag field and the --refresh flag are for keeping the copy current, which is
lesson 12.
Why keep HTML instead of plain text
It is tempting to strip the HTML at export time and save plain text. Don't. The structure is the
most useful thing in the page: headings say where one topic ends and the next starts, and
<pre> blocks say "this is code, never cut it in half". Lesson 4 chunks on exactly
those boundaries. Once you flatten a page to text, that information is gone and no amount of clever
splitting gets it back.
First problem: CERTIFICATE_VERIFY_FAILED
The first version of the script used Python's built-in urllib and failed on the very
first request:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
unable to get local issuer certificate
The site's certificate is fine. The python.org installer for macOS ships without root
certificates; you are supposed to run "Install Certificates.command" after installing, and almost
nobody does. Anyone following this tutorial on a Mac would hit the same wall, so the script uses
requests instead, which brings its own certificate bundle through certifi.
If you write HTTP code in Python for other people to run, that is the safer default.
Looking at the data before using it
Before building anything on a corpus, look at it. A few lines of Python over
data/posts answered the questions that matter for the next lessons:
posts 989 categories 48 <pre> code blocks 8088
html chars: min 0 median 18130 max 338876
near-empty posts (under 200 characters of text): 18
Three things stand out, and each changes a later decision:
- Eighteen posts are nearly empty on the live site. They are stubs left from the
WordPress days:
/css/css-gridis just the words "Source code on Github", and several interview pages have no text at all. The export read them correctly; there is simply nothing there. They will need filtering, or they will turn up as useless search results. - There are over eight thousand code blocks. This is a programming site, and a good share of what readers want is code. Any approach that mangles code is a non-starter.
- The saved HTML is the rendered page. Code blocks already carry the
site's syntax highlighting: every token is wrapped in a
<span>, and each block has a "Copy" button. The next lesson shows what that does to naive text extraction.
Lengths vary by three orders of magnitude too, from a one-line stub to a 338,000-character reference page. A chunking strategy has to handle both.
What you have now
ls data/posts | wc -l # 989
docker ps --filter name=ask-lmsc-db
A database that can store and compare vectors, and every post on the site as structured JSON with its HTML intact. Nothing has been embedded yet and no model has been called. That changes in the next lesson, which builds the simplest RAG pipeline that works and lets its failures tell us what to fix.