Message Queues and Asynchronous Work

The most dangerous line in most codebases is the one immediately after a commit.

self.db.commit()                       # the booking is now real
send_confirmation_email(booking)       # ...and this line never runs

Between those two statements the process can be deployed over, killed by the OOM reaper, or lose its network. The booking exists, the email does not, and nothing anywhere records that it was ever owed. Nobody finds out until a guest complains.

Reversing the order does not help — it makes it worse. Send first and a transaction that subsequently rolls back has emailed somebody about a booking that does not exist.

This post is about the machinery that closes that gap, and about what asynchronous work costs once you accept it.

Why not just do it in the request?

Three reasons, and the third is the one people underrate.

Latency. An email provider takes 400ms. The user does not need to wait for it to see “your booking is confirmed”.

Failure isolation. If the provider is down for ten minutes and sending is on the request path, your booking endpoint is down for ten minutes. With a queue, the work waits and the site keeps taking bookings. A queue converts somebody else’s outage into your backlog.

Back-pressure. A burst of traffic becomes a longer queue rather than a burst of concurrent provider calls. The system degrades by getting slower, which is survivable, instead of by falling over, which is not.

What in-process background tasks are not

Every web framework ships something that runs work after the response. It is not a queue, and the difference costs people real incidents. StayHub’s notification module states it plainly:

It is NOT a job queue, and the difference is not academic:

  * no retry           — a provider blip loses the email
  * no persistence     — a deploy or a crash mid-task loses it too
  * no back-pressure   — a burst of requests is a burst of concurrent tasks
  * no visibility      — nothing anywhere records that it was meant to happen

Those tasks run in the same process, in the same event loop, after the response is sent. That buys latency and nothing else. So the rule is: in-process for work that is genuinely nice to have, a real queue for work that must happen.

The transactional outbox

The fix for the opening problem, and it is simpler than it sounds. If two systems cannot be updated atomically, make the intent to update the second one part of the first one’s transaction.

   BEGIN
     INSERT INTO bookings ...        the business change
     INSERT INTO outbox   ...        "and an email needs sending"
   COMMIT                            both, or neither

           │
           v  a separate worker polls the table
     [ worker ] ──> send the email ──> mark the row DONE

There is now no gap. If the commit succeeded, the row saying “send this” is durable. If it rolled back, the intent vanished with the booking.

In StayHub, the position of one line is the entire pattern:

        try:
            self.bookings.add(booking)

            outbox_service.enqueue(
                self.db,
                notification_service.TOPIC_BOOKING_CREATED,
                {
                    "bookingId": str(booking.public_id),
                    "guestEmail": guest.email,
                    "propertyTitle": prop.title,
                    "checkIn": req.check_in,
                    "checkOut": req.check_out,
                    "total": breakdown.total,
                    "idempotencyKey": f"booking-created:{booking.public_id}",
                },
            )

            self.db.commit()

The enqueue is before the commit. Move it after and the gap is back.

It also inherits something for free: if the database rejects the booking — StayHub has an exclusion constraint that prevents double bookings — the rollback takes the outbox row with it. Nobody is emailed about a booking that lost the race. That is not extra code; it is a consequence of being in the same transaction.

Which makes one rule non-negotiable: enqueue must never commit.

    message = OutboxMessage(
        topic=topic,
        payload=json.loads(json.dumps(payload, default=str)),
        status=OutboxStatus.PENDING,
    )
    db.add(message)
    db.flush()
    return message

flush assigns the id without ending the transaction; the caller’s commit remains the single boundary. A commit hidden inside enqueue would reopen the gap invisibly — the code would look correct and work perfectly until the first crash.

What goes in the message

Two decisions worth making deliberately.

Name events, not commands. booking.created, not send_confirmation_email. A command has exactly one handler by definition, so the producer must know every consumer and gets edited whenever one is added. An event has none of that: today booking.created sends an email, next month it also notifies the host, and the booking service is untouched both times.

Carry a snapshot, not just a key. A foreign key alone is smaller and it is the thing that goes wrong: by the time the worker runs, the row may have changed — so the email describes the current state rather than the state that triggered it — or it may be gone, in which case the message can only be discarded.

The payload above does both: the id, so the handler can re-read anything that must be fresh, and a snapshot of what the email actually says.

The table

The schema is short, and every column is there for a reason worth naming:

CREATE TABLE outbox (
    id           BIGSERIAL PRIMARY KEY,
    public_id    UUID        NOT NULL UNIQUE,   -- the idempotency key sent downstream
    topic        VARCHAR(64) NOT NULL,          -- 'booking.created' — an EVENT, past tense
    payload      JSONB       NOT NULL,          -- a snapshot, not just a foreign key
    status       VARCHAR(16) NOT NULL,          -- PENDING | DONE | DEAD
    attempts     INTEGER     NOT NULL DEFAULT 0,
    available_at TIMESTAMPTZ NOT NULL DEFAULT now(),  -- created_at, then the backoff deadline
    processed_at TIMESTAMPTZ,
    last_error   TEXT,                          -- so a DEAD row is diagnosable without the logs
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX ix_outbox_pending ON outbox (status, available_at);

available_at does two jobs in one column: it is the creation time for a new message and the backoff deadline after a failure. That keeps the worker’s query to a single available_at <= now() rather than a null check plus a comparison.

The index earns its own mention. Equality column first, then the range-and-sort column, so Postgres seeks straight to the pending rows and walks them already in order — no sort step, and the cost does not grow as millions of DONE rows accumulate behind them. A production version makes it partial (WHERE status = 'PENDING'), because the done rows come to outnumber the live ones by orders of magnitude.

Which raises the housekeeping question people forget: DONE rows accumulate forever unless something removes them. Delete or archive them on a schedule — and if the table is partitioned by date, dropping a partition is instant where a DELETE of millions of rows is not.

Claiming work without doing it twice

Once there is a table of pending work, several workers will read it, and the obvious query is wrong.

SELECT * FROM outbox
 WHERE status = 'PENDING' AND available_at <= now()
 ORDER BY available_at
 LIMIT 20
   FOR UPDATE SKIP LOCKED;

Both halves of that last line are load-bearing, and the second is the one people omit.

FOR UPDATE locks the selected rows for the transaction’s duration. Without it, two workers polling at the same moment read the same rows and both send the same email. At-least-once is a guarantee to design around; twice on every message because two workers are running is simply a bug.

SKIP LOCKED tells Postgres to step over rows another transaction already holds and take the next free ones. Leaving it off is worse than it looks: plain FOR UPDATE makes worker B wait for worker A’s rows, so a second worker adds no throughput at all — it just queues behind the first, and one slow handler blocks everybody.

   FOR UPDATE            worker A: takes rows 1-20
                         worker B: BLOCKS until A commits    -> 1x throughput

   FOR UPDATE SKIP LOCKED
                         worker A: takes rows 1-20
                         worker B: takes rows 21-40          -> 2x, no coordination

The difference is only visible under concurrency, which is why the test for it has to be concurrent and has to assert on timing:

                started = time.perf_counter()
                claimed_b = outbox_service.claim(worker_b, limit=10)
                elapsed = time.perf_counter() - started

                assert claimed_b == []
                assert elapsed < 1.0, f"worker B blocked for {elapsed:.2f}s — it is waiting, not skipping"

Empty and fast. Without SKIP LOCKED the result is also eventually empty — it just takes as long as worker A does, and only the clock tells the two apart.

Retries and backoff

The point of a queue is that failure is temporary. A failed message goes back with a delay.

def backoff_for(attempts: int) -> timedelta:
    return timedelta(seconds=min(BACKOFF_BASE_SECONDS * (2 ** max(0, attempts - 1)), BACKOFF_MAX_SECONDS))

StayHub’s schedule is 2, 4, 8, 16, 32, 64, 128, 256 seconds — eight attempts spanning about four and a quarter minutes, then the message is dead.

Three properties of that sequence, each of which is a decision:

It grows. Retrying a struggling provider every second is how a limited outage becomes a total one, because the retries are the load. Exponential backoff means a recovering service sees exponentially less traffic from you.

It is capped. Uncapped doubling reaches days by attempt twenty, so a message that would have succeeded on the next retry waits a week. The cap matters as much as the growth.

It gives up. A cap on attempts is the difference between a queue and a spin loop. A message with a malformed payload fails identically forever; without a limit the worker retries it on every poll and it sits at the front of the queue delaying everything behind it.

One thing production adds that StayHub deliberately leaves out: jitter, a random fraction of the delay. Without it, every message that failed during the same outage retries at exactly the same instant and the recovering service is hit by a thundering herd of your own making. It is omitted here only so the tests can assert exact numbers.

Dead letters

After the last attempt, the message stops being retried and starts being evidence:

    if message.attempts >= MAX_ATTEMPTS:
        message.status = OutboxStatus.DEAD
        message.processed_at = datetime.now(UTC)

A dead-letter queue is not a failure of the design, it is the design working — the alternative is losing the message silently or retrying it forever. What matters is that somebody looks. A DLQ nobody monitors is a folder where data goes to be forgotten, so the count of dead messages belongs on a dashboard with an alert on it.

Keeping the last error on the row is worth the column, because by the time anyone investigates, the logs have very possibly rotated.

At-least-once, and the duplicate you will get

This is the cost of the whole pattern and it cannot be engineered away. The worker can hand a message to the provider and die before marking it done; the next worker sends it again.

“Exactly-once delivery” does not exist across a network — it is the two generals problem from the consistency post. What exists is at-least-once delivery plus idempotent handling, which produces exactly-once effects, and that is the thing you actually want.

Building this, the duplicate showed up immediately and visibly. Both delivery paths were briefly live at once — the old in-process task and the new outbox worker — and one booking produced two files:

$ ls notifications/
20260822T190807315928-guest_at_stayhub.test.json
20260822T190807858799-guest_at_stayhub.test.json    # same email, twice

That was a bug with an obvious fix (the routes stopped calling the in-process task). But it is also exactly what a redelivery looks like, and it is worth seeing, because the same shape arrives legitimately whenever a worker dies at the wrong moment.

How handlers survive it:

• Make the operation naturally idempotent. The index handler writes a document whose id is the property’s id, so running it five times produces the same index as running it once. This is the best answer when it is available.
• Pass an idempotency key downstream. The payload above carries "idempotencyKey": f"booking-created:{booking.public_id}". Real providers accept one and deduplicate on it.
• Record what you did. A table of processed message ids, written in the same transaction as the effect.

Unknown topics stay pending

A small decision with a large consequence. What should the worker do with a message whose topic it has no handler for?

        if handler is None:
            counts["unhandled"] += 1
            logger.warning(
                "No handler for outbox topic %r (known: %s) — leaving it pending",
                message.topic,
                ", ".join(registered_topics()) or "none",
            )
            db.commit()  # releases the row lock so the next poll can see it again
            continue

Not a failure to retry — no amount of waiting grows a handler — but deliberately not dead-lettered either. This is normally a deployment problem: a producer shipped ahead of its consumer. Marking those messages dead turns a five-minute deploy skew into permanent data loss.

There is a trap adjacent to this that cost real time here. Handlers register themselves at import time, so a module nobody imports has no topics — and the symptom is not a crash, it is “No handler for topic booking.created” on a perfectly valid message, forever. The worker therefore imports its handler modules explicitly, and ships a --list-topics flag so the registry can be inspected when this happens.

The worker is a separate process

python -m scripts.drain_outbox              # run until interrupted
python -m scripts.drain_outbox --once       # one batch, then exit
python -m scripts.drain_outbox --list-topics

Separate, and that separation is the point. Inside the API it would compete with request handling and die with every deploy mid-message. As its own process it can be restarted, scaled out or stopped without touching the API — and stopping it is safe, which is the property that distinguishes a queue from a background task. Messages simply accumulate as pending until it comes back.

Two details from the worker loop that generalise. It commits per message, not per batch, so one poisoned message cannot roll back the successful work beside it and cause all of it to be re-sent. And it opens a session per batch, not per lifetime — a long-lived session holds a connection open for days and accumulates every object it ever loaded.

Shutdown deserves a moment too. On SIGTERM the worker sets a flag and finishes the batch in flight rather than exiting immediately. Killed mid-handler, a message is left pending and redelivered — correct, but a duplicate nobody needed. Draining first makes the common case clean.

Ordering, and why you mostly should not need it

A question that comes up immediately: do messages arrive in the order they were sent?

StayHub’s claim clause is ORDER BY available_at, which processes oldest-ready-first — roughly FIFO. It is emphatically not an ordering guarantee, and relying on it as one is a bug waiting to happen:

   worker A claims message 1  (a slow handler, 3 seconds)
   worker B claims message 2  (fast, 20ms)

   message 2 completes first. Always, under any concurrency.

Ordering and parallelism are in direct conflict. Guaranteeing global order means one consumer, one at a time — which caps your throughput at whatever one worker can do.

Systems that need both compromise with partial ordering: messages sharing a key go to the same partition and are processed in order, while different keys proceed in parallel. All events for booking 1234 are ordered relative to each other; booking 5678 is independent. That is almost always what “ordering” actually meant.

The simplest answer, though, is to design so ordering does not matter. If two messages must be applied in sequence, that is usually a sign they should be one message. And if handlers are idempotent and carry the state they need, arrival order stops being interesting — which is another dividend of the design decisions above.

Watching it

An asynchronous system fails silently by construction. The request succeeded; the work did not happen; nobody is holding an error. So the queue needs to be observable or it will fail undetected.

def pending_count(db: Session) -> int:
    """For the health check and the admin page. A number that only goes up means the worker is
    dead, and that is worth an alert — a queue nobody is draining fails completely silently."""

Four numbers cover it:

The second row is the one worth adding if you only add one. Depth alone is ambiguous — a thousand messages is fine if they are draining in seconds — whereas “the oldest message has been waiting eleven minutes” is unambiguous, and it catches a single poisoned message sitting at the head of an otherwise healthy queue.

Queue or log?

The other question in this space, and the distinction is about what happens after a message is read.

Replay is the deciding feature. If you need to add a consumer next year that processes the last six months of events, you need a log. If you need work done once, a queue is simpler and cheaper.

And a database table is a perfectly respectable queue at moderate volume. It gives you transactions with your business data — which is the entire outbox pattern — and it is one less system to operate. Move to a dedicated broker when the volume or the fan-out demands it, not before.

Where the outbox does not reach

Honesty about a pattern’s limits is worth more than the pattern. StayHub has two async paths and only one of them is fully transactional — the code says so:

            outbox_service.enqueue(
                self.db,
                indexer.TOPIC_PROPERTY_CHANGED,
                {"propertyId": str(prop.public_id), "reason": "index-write-failed"},
            )
            self.db.commit()

That commit is the tell. This enqueue happens in the search-index path, which runs after the business transaction has already committed — deliberately, because indexing inside the transaction would let a slow search cluster fail a host’s save. So the message is genuinely a second write, and a crash between the commit and this line loses it.

   BOOKING PATH                       INDEX PATH
     BEGIN                              COMMIT (business change)
       INSERT booking                   ...crash here loses the index update
       INSERT outbox        <- atomic   try to index
     COMMIT                             failed? enqueue + commit
                                        ^ a SECOND transaction, not the same one

Two different guarantees in one application, and both are right for their path. The booking email must never be lost, so it is transactional. The index update can be lost, because the index is derived data with a rebuild button — and buying atomicity there would cost the write path its independence from Elasticsearch.

The general lesson: the outbox is for work that must not be lost, and it costs you an extra table, an extra process and a few seconds of latency. Applying it everywhere is as much a mistake as applying it nowhere. Ask what losing the message actually costs, and pay accordingly.

The summary

• The line after the commit is where messages are lost. The outbox makes the intent part of the transaction.
• enqueue never commits — the caller’s commit is what makes it atomic.
• Name events, carry snapshots, and include an idempotency key.
• FOR UPDATE SKIP LOCKED — both halves. Without the second, more workers add nothing.
• Exponential backoff, capped, with jitter, and a limit on attempts.
• At-least-once is the guarantee. Every handler must be idempotent, and you will see the duplicate eventually.
• Alert on queue depth and dead letters. A queue nobody drains fails silently.

The mental model worth keeping: a queue does not make work reliable, it makes the record of the work reliable. Everything after that — retries, backoff, dead letters, idempotency — exists because delivery itself never becomes certain. You are not building a system where nothing fails; you are building one where a failure is recorded, retried, and eventually visible to a human.

Next: concurrency and locking — what happens when two requests want the same row at the same instant, which the outbox’s row locking has already hinted at.