Two problems live in this lesson. Getting a lot of documents into Elasticsearch quickly, which is the easy one. And keeping them in step with a database that keeps changing, which is the one that decides whether your search results are trustworthy.
Why bulk
Indexing 200 documents into a fresh index, same documents, same machine, same mapping:
one call per document 526 ms
one bulk request 47 msRoughly 11x, and the ratio grows with the number of documents because what you are removing is per-request overhead: an HTTP round trip, a request parse, a routing decision and a response, 200 times over.
The measurement is on a local single-node cluster, which is the best case for the one-at-a-time version — there is no real network. Across a network the gap is wider.
The bulk API
The format is unusual and it is worth understanding rather than copying: newline-delimited JSON, two lines per operation. An action line, then the document.
POST /stayhub-properties/_bulk
{"index": {"_id": "8f1e-..."}}
{"title": "Cedar Cabin", "city": "Big Bear Lake"}
{"index": {"_id": "3a20-..."}}
{"title": "Sunlit Loft", "city": "San Francisco"}
{"delete": {"_id": "77c4-..."}}
Two rules that cause most of the failures. The content type is
application/x-ndjson, not application/json. And the body must end
with a newline — without it the last action is silently dropped, which is a
maddening bug to chase.
The format exists so the coordinating node can split the body on newlines and route each action without parsing the whole request. That is also why a bulk request should be sized in megabytes, not in documents: 5–15MB per request is the usual guidance, and a 100MB bulk request will be rejected or will cause memory pressure.
Four actions are available: index (create or replace), create (fail if
it exists), update (partial), and delete (no document line).
The trap: a 200 that failed
Send a bulk request where one document is invalid:
{"index": {"_id": "1"}}
{"n": 1}
{"index": {"_id": "2"}}
{"n": "not a number"}
{"index": {"_id": "3"}}
{"n": 3}
HTTP 200
errors: True
index 1 201
index 2 400 document_parsing_exception
index 3 201HTTP 200. Two of the three documents were indexed and one was not, and the status code says everything is fine.
This is correct behaviour — a bulk request is not a transaction, and there is no single
status that describes a partial result. But it means that checking the HTTP status of a bulk
request tells you nothing. You have to read errors in the body, and when it is
true, walk items to find out which ones failed and why.
Code that does not do this loses documents silently. The index simply has fewer things in it than the database, nobody notices, and eventually a user reports that a listing does not come up in search.
helpers.bulk
The Python client's helper handles the newline format, chunking and the error walk for you:
actions = [
{
"_index": settings.elasticsearch_index,
"_id": str(p.public_id),
"_source": to_document(p),
}
for p in properties
]
success, _ = bulk(client, actions, refresh="wait_for")
logger.info("Reindexed %s properties", success)
return successBy default it raises BulkIndexError if any document fails, carrying the per-document
errors:
raised BulkIndexError: 1 document(s) failed to index.
raise_on_error=False -> ok: 1 errors: 1Which of the two you want depends on the job. For a rebuild, raising is right — a partial
index is worse than an obvious failure. For an ingest pipeline where a few malformed records are
expected, raise_on_error=False and log what came back.
Two more helpers are worth knowing. streaming_bulk yields a result per document, so
you can process a generator of millions without building the list in memory.
parallel_bulk runs several threads, which helps when serialising the documents is the
bottleneck rather than the cluster.
Settings that make a big load faster
Three, applied before and reverted after:
PUT /stayhub-properties/_settings
{ "index": { "refresh_interval": "-1", "number_of_replicas": 0 } }
# ... load ...
PUT /stayhub-properties/_settings
{ "index": { "refresh_interval": "1s", "number_of_replicas": 1 } }
POST /stayhub-properties/_forcemerge?max_num_segments=1Disabling refresh stops a segment being created every second during the load. Dropping replicas
to zero halves the write work, and building them afterwards is a bulk copy rather than a
document-by-document one. The _forcemerge at the end is only appropriate for an index
that will not be written to again — it is expensive and it defeats the normal merge policy.
Backpressure, and what a rejection looks like
Push harder than the cluster can absorb and you do not get a slowdown — you get rejections. Elasticsearch has a bounded write queue per node, and when it is full new bulk requests are refused with a distinctive error:
429 es_rejected_execution_exception
rejected execution of coordinating operation ... queue capacity 10000A 429 is the cluster telling you to slow down, and the correct response is to slow down: retry
that batch after a delay, with the delay growing each time. The client does some of this for you
— retry_on_timeout and max_retries — and
helpers.bulk accepts max_retries and
initial_backoff to retry only the rejected items rather than the whole batch.
What does not work is more threads. That is the instinct when a load feels slow, and it makes the
rejections worse while producing no more throughput. Watch
_cat/thread_pool/write?v during a load: if queue is climbing and
rejected is non-zero, the answer is smaller batches or fewer of them, not more
concurrency.
Sizing a batch
The guidance to size by bytes rather than documents is easy to say and easy to get wrong, because the documents you are testing with are rarely the biggest ones you will ever index. Two habits help.
Cap both: chunk_size in documents and
max_chunk_bytes in bytes, so one unusually large document cannot produce a request ten
times the intended size. And start at a batch you are confident is too small — 500 documents,
or 5MB — then increase it while watching throughput. It flattens out well before it starts
failing, and the flat part is where you want to be.
The harder half: staying in step
A bulk load gets the index correct once. The interesting question is what happens on the next write to the database.
There are three families of answer.
Change data capture. Something reads the database's replication log — Debezium reading the Postgres WAL — and turns every row change into an index update. It catches every write, including ones made by a migration or by hand in a psql session, which nothing else does. The cost is real infrastructure: a connector, a message broker, and a second system to operate.
Periodic reindex. A job runs every N minutes and rewrites what changed. Simple, robust, and the index is up to N minutes stale. Perfectly acceptable for a catalogue that changes daily; not acceptable for a listing a host just published and is now looking for.
From application code. Every write path that changes a listing indexes it. Fast, no extra infrastructure, and it misses anything that does not go through your application.
StayHub uses the third, because every write already goes through one FastAPI service, which is a design decision that makes the simple option viable. If writes come from three services and a batch job, it is not.
Three rules for indexing from application code
1. Index after the commit, never before
"""
⚠️ **Index AFTER the commit, never before.** Indexing first means a rolled-back transaction leaves
a listing in search results that does not exist in the database — and the guest who clicks it gets
a 404 from a page that just told them it was available.
"""Indexing inside the transaction is the instinct, because it feels more atomic. It is strictly worse. A transaction that rolls back after indexing leaves a phantom in search results, and nothing will ever remove it — there is no row left to trigger a correction.
Indexing after the commit has a real gap too: the process can die between the two. But that gap leaves a document missing from the index, which the next write or a rebuild repairs. Missing is a recoverable state; phantom is not.
2. A failed index must not fail the write
except Exception: # noqa: BLE001 — see the module docstring: never fail a write over this
logger.exception("Failed to index property %s", prop.public_id)A host publishing a listing must not get a 500 because a search cluster is restarting. The listing is correct in Postgres, which is what matters; search is a derived view that is briefly behind.
This is only defensible because of lesson 1's rule. If the index were the source of truth, swallowing the error would be data loss.
3. Do not merely log the failure — retry it
Logging alone means a thirty-second Elasticsearch outage silently drops every change made during it, and nothing repairs them until somebody notices and runs a rebuild. StayHub turns a failure into an outbox message:
"""
**And a failure is now retried, not merely logged** (added 2026-08-22). `index_property` returns
False when the write did not land, and `property_service._sync` turns that False into an outbox
message — so a listing indexed during an Elasticsearch outage is re-indexed by the worker minutes
later instead of waiting for somebody to notice and run `rebuild_index`.
"""The shape is worth naming: try inline, fall back to the queue. Doing only one is worse in both directions. Queue everything and the common case — Elasticsearch is fine — pays a worker round trip for no reason, and search is seconds stale after every edit. Queue nothing and an outage loses changes.
The retry that re-reads
One detail in the retry handler is worth the whole section, because it inverts the usual advice about message payloads:
"""Re-index one property, reading it fresh from Postgres.
⚠️ It RE-READS rather than indexing a snapshot from the payload, and that is the opposite of
what `models/outbox.py` says payloads are usually for. The reasoning inverts here because the
index is *derived data whose only job is to match the database right now*. If the listing was
edited three more times while Elasticsearch was down, indexing the first snapshot would write a
stale document and then be marked DONE — leaving search confidently wrong. Re-reading
collapses all four changes into one correct write.
"""The usual guidance is that an event carries what happened, so a consumer is not coupled to the current state. That is right for a booking confirmation email, which describes a moment.
It is wrong here. The index's only job is to match the database now. Four edits during an outage produce four queued messages; indexing four stale snapshots in order happens to end correctly, and indexing the first one and marking it done does not. Re-reading collapses all four into one correct write and is immune to ordering.
Ask what the consumer needs: the event as it happened, or the world as it is.
Why any of this is safe
At-least-once delivery means a handler can run twice. That is only acceptable because of lesson 7's property:
"""
**Idempotent by construction.** The document id is the property's public id, so running this
handler once or five times produces exactly the same index — which is the property that makes
at-least-once delivery safe here rather than merely tolerable.
"""Note also that failure handling is inverted between the two callers. On the request path an exception is swallowed, because a host should not see a 500. In the worker an exception is how you say "not done yet" — swallowing it would mark the message done and discard the retry:
# ⚠️ `raise_on_error` is for the WORKER, and only for the worker. On the request path a
# False keeps a host's save working; in the worker a False would be read as success and
# the message marked DONE, quietly discarding the retry this function exists to enable.
# Same code, two callers, opposite correct behaviours — so the caller chooses.Choosing between the three strategies
Worth being concrete, because the decision is usually made by accident.
Use application-code indexing when every write already goes through one service you control, and when a few seconds of staleness is acceptable. It is by far the least infrastructure, and the code is small enough to read in one sitting. Its failure mode is writes that bypass the service — a migration, a support script, a second service added later — and you will not get an alert when that happens.
Use change data capture when writes come from more than one place, or when
missing one is genuinely unacceptable. It catches the manual UPDATE somebody ran at
2am, which is exactly the change nobody remembers to re-index. Budget for the connector as a
production system with its own monitoring, because that is what it is.
Use a periodic reindex when the data changes on a schedule anyway — a nightly catalogue import, a daily price feed. It is the most robust of the three precisely because it makes no attempt to be clever: it just rewrites the world, so there is no accumulated drift to detect.
And note they combine well. Application-code indexing for freshness, plus a periodic rebuild as the backstop, gets you most of CDC's correctness for none of its operational cost. That is the combination StayHub ends up with, and it is a reasonable default for a system where the index is derived.
What each one does when Elasticsearch is down
This is the question that separates them, and it is worth answering before you need it.
CDC keeps its position in the log, so it resumes where it stopped and nothing is lost — that is its main advantage and the reason it is worth the operational weight.
A periodic reindex simply catches up on the next run, with no special handling. It is the least exciting answer and the least likely to be wrong.
Application-code indexing loses every change made during the outage unless the failure is queued, which is the whole argument for the outbox above. Without it, the recovery procedure is a human noticing and running a rebuild — and "a human notices" is not a recovery procedure.
Keep the rebuild
Whatever sync strategy you choose, keep a way to rebuild the whole index from the source of truth, and run it occasionally rather than only in emergencies.
python -m scripts.reindex --rebuild
# rebuilt from Postgres: 12 documentsIt is the repair path for every bug in the sync layer you have not found yet, it is the seed path for a new environment, and running it periodically against a copy and diffing the result is the only honest way to know whether your sync is actually working.
The rebuild being cheap is also what makes the rest of this lesson's advice affordable. Every "log it and move on" decision above rests on there being a way to put the index right without anybody reconstructing what was lost. Take that away — by letting a field exist only in the index — and swallowing an indexing error stops being pragmatic and becomes data loss.
A drift check is cheap and worth having: count documents in the index, count eligible rows in the database, alert when they differ by more than a threshold. It will not catch a wrong field value, but it catches the whole class of "the sync silently stopped" problems, which is the class that tends to go unnoticed for weeks.
A stronger version, when it is worth the effort: sample a few hundred documents, compare each against the database row it came from, and alert on any field that disagrees. That catches the subtler failure — a sync that runs, succeeds, and writes the wrong value because a mapping changed or a field was renamed. Both checks are a few dozen lines, and they are the only evidence you will ever have that the index is telling the truth rather than merely responding.