Every index so far was built from scratch: chunk all 989 posts, embed 11,701 chunks, load them.
With bge-base that takes about three and a half minutes, which is fine for a lesson and
wasteful for a site that changes one post at a time. A real RAG system has to follow its documents:
pick up a new post, replace an edited one, drop a deleted one, and touch nothing else.
This lesson builds that: ask_lmsc/sync.py, run by
lessons/10_reindex.py, and an incremental mode for the export script. It ends with a
test that is shown to catch the bug it guards against.
How do you know a post changed?
Three candidates, and the choice matters more than it looks:
- Timestamps ("modified after the last sync"). A timestamp says when something was written, not whether it changed, and timestamps from different systems disagree. This site's own content sync once shipped stale data for exactly that reason: a local file's modification time was stamped at download, so a newer object in S3 looked older.
- HTTP ETags. The server's fingerprint of the page. Excellent for deciding what to download, but it covers the whole page: a site rebuild that changes the surrounding HTML changes every ETag, though no article changed.
- A hash of the content you index. SHA-256 of the article HTML. It changes if and only if what you embed would change.
So both get used, for different jobs: ETags decide what to download, content hashes decide what to re-index.
def content_hash(post: dict) -> str:
return hashlib.sha256(post["html"].encode("utf-8")).hexdigest()
The hashes live in a small table, one row per indexed post and embedding model:
conn.execute("""
CREATE TABLE IF NOT EXISTS indexed_posts (
model text NOT NULL,
url text NOT NULL,
content_hash text NOT NULL,
chunks int NOT NULL,
indexed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (model, url)
)""")
Planning a sync
plan = {"new": [], "changed": [], "removed": [], "adopted": [], "unchanged": 0}
for url, post in files.items():
digest = content_hash(post)
if url in recorded:
if recorded[url] == digest:
plan["unchanged"] += 1
else:
plan["changed"].append(url)
elif url in in_table:
stored = [r[0] for r in conn.execute(
sql.SQL("SELECT content FROM {} WHERE url = %s ORDER BY id").format(t), (url,)).fetchall()]
if stored == [c.text for c in chunk_post(post)]:
plan["adopted"].append(url)
else:
plan["changed"].append(url)
else:
plan["new"].append(url)
plan["removed"] = sorted((set(recorded) | in_table) - set(files))
Four outcomes per post, plus one for posts that have disappeared:
- unchanged: the recorded hash matches. Nothing to do.
- changed: the hash differs. Replace that post's chunks.
- new: no record and no rows. Chunk, embed, insert.
- removed: recorded, or present in the table, with no file any more. Delete.
- adopted: the interesting one. The table built in lesson 5 has rows but no recorded hashes. Rather than re-embed everything once to get started, sync re-chunks the post and compares the result with the stored rows. If they are identical, the existing vectors are correct for this content, and only the hash is recorded.
Adoption doubles as a check. If the chunker had changed since the table was built, the stored rows wouldn't match, the post would be classed as changed, and it would be re-embedded. Silently mixing old and new chunking in one index is a classic source of results nobody can explain.
Applying it, one post per transaction
todo = plan["new"] + plan["changed"]
chunks = {url: chunk_post(files[url]) for url in todo}
flat = [c for url in todo for c in chunks[url]]
vectors = iter(embed.embed_passages(model, [c.text for c in flat])) if flat else iter(())
for url in todo:
post_chunks = chunks[url]
post_vectors = [next(vectors) for _ in post_chunks]
with conn.transaction():
conn.execute(sql.SQL("DELETE FROM {} WHERE url = %s").format(t), (url,))
if post_chunks:
store.load_rows(conn, model, post_chunks, post_vectors)
_record(conn, model, url, content_hash(files[url]), len(post_chunks))
Embedding happens once for everything that needs it, because the model is far faster on a batch than one post at a time. Writing happens per post, each inside its own transaction: delete the old chunks, insert the new ones, record the hash. A search running at the same moment sees either the old version of a post or the new one, never half of each. A crash midway leaves every post consistent, and the next sync redoes the unfinished ones because their hashes were never recorded.
The HNSW index needs no special handling: pgvector updates the graph on every insert and delete.
Deleted rows leave dead entries behind until Postgres's autovacuum cleans them up, which it does on
its own. After a very large batch of replacements, a manual VACUUM on the table reclaims
the space sooner.
The transaction has one more job. Lesson 9's neighbour expansion assumes each post's chunks have consecutive ids in document order. Inserting a post's chunks in order, in one transaction, with a single writer, keeps that true after a replacement.
Running it
python lessons/10_reindex.py sync --dry-run
python lessons/10_reindex.py sync
python lessons/10_reindex.py sync # again: should do nothing
chunks_base: would add 18, replace 0, remove 0, adopt 971; 0 unchanged (9.5s)
chunks_base: did add 18, replace 0, remove 0, adopt 971; 0 unchanged (17.0s)
chunks_base: did add 0, replace 0, remove 0, adopt 0; 989 unchanged (0.3s)
All 971 indexed posts were adopted without re-embedding anything: re-chunking reproduced the stored rows exactly. The 18 "new" ones are the stub posts the chunker skips; they are recorded with zero chunks so they aren't rechecked. The second sync took 0.3 s.
Keeping the downloaded copy current
The export script now saves each page's ETag and sends it back with --refresh.
Unchanged pages answer 304 Not Modified with no body:
def fetch(url: str, retries: int = 3, etag: str | None = None) -> requests.Response:
"""GET with retries. With `etag`, the server answers 304 (no body) if the page is unchanged."""
headers = {"If-None-Match": etag} if etag else {}
for attempt in range(1, retries + 1):
try:
resp = session().get(url, headers=headers, timeout=30)
if resp.status_code != 304:
resp.raise_for_status()
Adding the etag field rewrote every saved JSON file, and the next sync still reported
zero changes, which is the content hash doing its job: only the article counts.
The first no-change refresh got 989 responses of 304 and still took 31 seconds, 25 of them CPU. The
downloads weren't the cost: each request opened a new TLS connection. One
requests.Session per worker thread reuses connections:
_local = threading.local()
def session() -> requests.Session:
if not hasattr(_local, "s"):
_local.s = requests.Session()
_local.s.headers["User-Agent"] = USER_AGENT
return _local.s
The same refresh then took 7.9 s (1.7 s CPU). 304s save bandwidth;
connection reuse saves time. --prune deletes local copies of posts that are no longer
listed on the site, so the next sync removes them from the index.
python scripts/export_posts.py --refresh --prune
python lessons/10_reindex.py sync
A test that has been seen to fail
tests/test_sync.py runs against a throwaway table with a few fake posts written to a
temporary directory, so it never touches the real index:
plan = sync(conn, MODEL, tmp_path, catalog=CATALOG)
assert plan["changed"] == [a["url"]]
assert plan["removed"] == [b["url"]]
assert plan["new"] == [d["url"]]
assert plan["unchanged"] == 1
assert any("git stash pop" in t for t in rows(conn, a["url"]))
assert not any("set work aside" in t for t in rows(conn, a["url"])), "old chunks must be gone"
It covers new, changed, removed and unchanged posts, adoption, and consecutive ids after a replacement (checked by running neighbour expansion on the edited post). A passing test proves less than it seems, so the test was checked too: deleting the line that removes a changed post's old chunks makes it fail with "old chunks must be gone". A test you have never seen fail might not be testing anything.
python -m pytest tests/ -q
Take-aways
- Detect change with a hash of exactly what you index. Use ETags only to skip downloads.
- Adopt an existing index by re-deriving and comparing; it also catches a changed chunker.
- Batch the embedding, but write each document in its own transaction.
- Reuse HTTP connections when you make many small requests.
- Make a test fail on purpose once before trusting it.
Next, the capstone: an agent that chooses between search and SQL.