FastAPI – Background Tasks and Their Limits

BackgroundTasks runs work after the response has been sent. That is the entire feature, and it buys exactly one thing: the caller is not kept waiting. It is also the FastAPI feature most often mistaken for a job queue, and the gap between what it does and what people assume it does is where the outages live.

Using it

Ask for it by type, then hand it a callable and its arguments:

@router.post("", response_model=BookingResponse, status_code=status.HTTP_201_CREATED)
def create_booking(
    payload: BookingCreateRequest, user: CurrentUser, db: DbSession, background: BackgroundTasks
) -> BookingResponse:
    """Hold the dates. The booking is PENDING until payment succeeds.

    ⚠️ The body carries no price. Every figure is computed server-side from the listing.
    """
    booking = BookingService(db).create(user, payload)

    background.add_task(notification_service.send_booking_confirmation, booking.public_id)
    return _to_response(booking)

add_task does not call the function. It appends it to a list Starlette walks after the response is written. The guest sees their booking immediately; the email is sent a moment later, and its latency is invisible to them.

Note when the task is queued — after create() has returned:

    # ⚠️ Queued AFTER create() returned, never before. A task added earlier still runs even if
    # create() then raises — Starlette runs whatever is on the response's task list, and a failed
    # request that emails "your dates are held" is worse than no email at all.

Queue after the work succeeds, never before. The ordering looks arbitrary and is not.

The trap: it half-works

The obvious way to write that call is to pass the object you already have:

background.add_task(send_booking_confirmation, booking)   # ⚠️ do not

It will appear to work. Here is why it does not.

A dependency with yield runs its teardown before background tasks. Measured on FastAPI 0.115.5:

1. dep: open
2. route body
3. dep: CLOSED      <- get_db's `finally: db.close()`
4. background task ran

So by the time the task runs, the request's session is closed and every object loaded from it is detached. What makes this genuinely dangerous is that it fails selectively:

booking.total          -> OK, Decimal('797.40')
booking.property.title -> DetachedInstanceError

A column that was already loaded survives — StayHub sets expire_on_commit=False, so attributes are not invalidated on commit. The first relationship that was not loaded raises, because reading it needs a query and there is no session to issue one.

Put that together and the failure mode is brutal: the version that passes the object passes a test that checks the total, ships, and breaks the day somebody adds the property name to the email — after the response has already gone out with a 200, where nobody is looking.

The fix is in the signature

def send_booking_confirmation(booking_public_id: UUID) -> None:

Take a plain value. Then open a session that belongs to the task:

    with SessionLocal() as db:
        booking = db.execute(
            select(Booking)
            .where(Booking.public_id == booking_public_id)
            .options(selectinload(Booking.property), selectinload(Booking.guest))
        ).scalar_one_or_none()

The relationships are eager-loaded in that one query, precisely because there is no session to lazy-load from later — and because the alternative is three more round trips per email.

Rule: a background task takes ids and strings, never ORM objects, and opens its own session. That removes the entire category rather than working around one instance of it.

Nobody is left to return an error to

        if booking is None:
            # Not an error worth raising. The booking can legitimately be gone by now, and there
            # is nobody left to return a 404 to — the response was sent long ago.
            logger.warning("Booking %s vanished before its email was sent", booking_public_id)
            return

This is the mental shift a background task requires. There is no caller. Raising accomplishes nothing a log line does not, and an unhandled exception in a task is invisible — the response already went out with a 200, so no monitoring based on status codes will ever see it.

Two consequences. Every task should catch its own failures and log them with enough context to act on. And a task must tolerate the world having moved: the row it was given may be gone, already processed, or changed. Write them to be re-runnable and to no-op when there is nothing to do.

Sync or async task

Both work, and the difference matters:

background.add_task(sync_function, booking_id)    # runs in the threadpool
background.add_task(async_function, booking_id)   # runs on the event loop

A plain def task is dispatched to the threadpool, so blocking in it is safe. An async def task runs on the event loop, so a blocking call inside it stalls every request in the process — and it does so after the response, where it looks like unrelated slowness elsewhere.

StayHub's notification functions are plain def, because they do blocking database and file work. That is the right default: if the task blocks, make it def. Reserve async def for tasks that genuinely await.

What it is not

It is NOT a job queue, and the difference is not academic:

  * no retry           — a provider blip loses the email
  * no persistence     — a deploy or a crash mid-task loses it too
  * no back-pressure   — a burst of requests is a burst of concurrent tasks
  * no visibility      — nothing anywhere records that it was meant to happen

Each of those is worth sitting with.

No retry. The email provider returns a 503; the task raises; the email is gone. Nothing will ever try again.

No persistence. Tasks live in the process's memory. A deploy that restarts the container mid-task loses it silently, and deploys happen at exactly the times traffic is highest.

No back-pressure. A thousand requests queue a thousand tasks. There is no worker pool bounding how many run, no rate limit, and nothing to shed load — the process simply takes on more work while still serving traffic.

No visibility. No queue depth, no dead-letter, no record that a task existed. When somebody asks whether the email was sent, the only answer is a log search.

So when is it the right tool?

The test is a single question: if this silently never happens, is that acceptable?

StayHub's confirmation email is on the borderline and lives here because it is a demo. The money-side equivalent deliberately does not: payment confirmation arrives as a Stripe webhook, a real HTTP callback with the provider's own retries behind it. That contrast is the lesson in one application — the thing that must happen does not use BackgroundTasks.

What a queue gives you

The shape barely changes at the call site:

background.add_task(send_booking_confirmation, booking.public_id)   # in-process
send_booking_confirmation.delay(str(booking.public_id))             # Celery
queue.enqueue(send_booking_confirmation, str(booking.public_id))    # RQ

What changes is everything behind it. The task is serialised into a broker — Redis, RabbitMQ, SQS — where it survives a restart. Separate worker processes consume it, so heavy work does not compete with request handling and can be scaled independently. Failures retry with backoff and eventually land in a dead-letter queue you can inspect.

Note the argument in both queue versions is a string, not a UUID. Anything crossing a broker must serialise, which enforces the discipline the in-process version only recommends: you could not pass an ORM object if you tried.

The cost is real — a broker to run, workers to deploy, and a second place code can fail. For a small application that is a lot of machinery for one email. The honest progression is to start with BackgroundTasks, and move the moment something crosses the "must happen" line.

The middle option

Between the two sits a pattern worth knowing, because it needs no new infrastructure: write the intent to your own database in the same transaction as the work, then process it separately.

class OutboxEntry(Base, TimestampMixin):
    __tablename__ = "outbox"

    id: Mapped[int] = mapped_column(primary_key=True)
    kind: Mapped[str] = mapped_column(String(60), nullable=False)
    payload: Mapped[dict] = mapped_column(JSONB, nullable=False)
    processed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
    attempts: Mapped[int] = mapped_column(Integer, default=0, nullable=False)

Because the row is written in the booking's own transaction, "the booking exists" and "we owe an email" become atomic — which is something neither BackgroundTasks nor an external queue can offer. A BackgroundTasks job then tries it immediately for latency, and a periodic sweep catches whatever failed. Persistence and retry, at the cost of one table.

Testing tasks

The awkwardness is that TestClient runs background tasks, which is usually what you want — but it means a test hitting a route also sends email.

Three approaches, in order of preference. Test the task directly, as a plain function, which is most of the value for none of the ceremony. Override the dependency that produces the side effect if it has one. Or assert on the queued tasks without running them:

def test_creating_a_booking_queues_its_confirmation(db, guest, listing):
    background = BackgroundTasks()
    create_booking(payload, guest, db, background)

    assert len(background.tasks) == 1
    assert background.tasks[0].func is notification_service.send_booking_confirmation

Calling the route function directly and inspecting background.tasks asserts the wiring without the side effect. It is also the test that would have caught the "queued before the work" ordering mistake.

StayHub's own tasks write to a directory rather than sending mail, which makes the end-to-end version cheap — the test asserts a file appeared and reads it. A stand-in like that is often the least effort for the most confidence.

Work that is not triggered by a request

BackgroundTasks is attached to a response, so it can only ever run because somebody called an endpoint. A whole category of work is not like that: expiring stale bookings, nightly reports, sweeping temporary files, retrying an outbox.

Three options, in increasing order of seriousness.

A loop in the lifespan, for something small in a single-instance deployment:

@asynccontextmanager
async def lifespan(app: FastAPI):
    async def sweep():
        while True:
            await asyncio.sleep(3600)
            try:
                expire_stale_bookings()
            except Exception:
                logger.exception("sweep failed")

    task = asyncio.create_task(sweep())
    yield
    task.cancel()

Note the try inside the loop. Without it, one failure ends the loop permanently and nothing says so — the sweep just silently stops happening. Note also the reference is held and cancelled on shutdown; a forgotten task can be garbage-collected mid-flight.

The catch is that this runs in every worker and every replica. Four workers across three containers is twelve concurrent sweeps.

An external scheduler — cron, a Kubernetes CronJob, your platform's scheduled tasks — invoking a management command. Runs once regardless of how many replicas exist, which is usually the deciding factor.

Celery beat or APScheduler with a shared lock, when the schedule itself needs to be dynamic or per-tenant.

Whichever you choose, make the job idempotent and make it safe to run twice at once. At some point it will be.

A second example: keeping the index fresh

Email is the obvious case. A more instructive one is StayHub's search index, which is updated from application code on every write:

    """Rebuild the Elasticsearch index from Postgres.

    The repair path for the sync's one weakness: it is not transactional, so a crash between a
    commit and an index call leaves the two out of step. Postgres is the source of truth, so a
    rebuild is always safe.
    """

That docstring contains the whole design. Indexing is a good candidate for a background task — the write should not wait for it, and a search result being a second stale is fine. But it is not transactional: the database commit and the index update can disagree if the process dies between them.

The answer is not to make it transactional, which is not possible across two systems. It is to name one of them the source of truth and provide a repair path. Postgres is authoritative, so a full reindex always fixes any drift, and it is exposed as an endpoint staff can call.

That pattern generalises. Any background task that touches a second system has this problem. Rather than trying to make two writes atomic, decide which one is the truth and make reconciliation cheap and repeatable.

What runs, and in what order

Worth being precise, because the ordering explains most surprises.

1. dependencies resolve
2. route body runs                    -> tasks queued here
3. response is serialised
4. yield-dependency teardown          -> the session closes
5. response is SENT to the client
6. background tasks run, IN ORDER, one after another

Three consequences fall out of that list.

Tasks are sequential, not concurrent. Three tasks of two seconds each is six seconds of work after the response. That is usually fine — nobody is waiting — but it does mean a slow task delays the ones queued behind it, and under load they accumulate.

A task that raises stops the ones after it. Starlette runs them in order and does not catch for you, so the second failing means the third never runs. That alone is a reason to wrap each task's body in its own try.

Every task on the list runs, even if the route raised after queuing it. Which is the ordering rule from the top of this lesson, and the reason to queue after the work rather than before.

Dependencies inside tasks

A background task is a plain function; nothing injects into it. So anything it needs, it acquires itself:

    with SessionLocal() as db:

That is the shape for everything, not only sessions. Settings come from the module-level settings object, which is cached and safe. A shared client held on app.state is not reachable, so either pass what it needs or reach for the module-level singleton instead.

If a task genuinely needs several injected things, that is a signal it should be a service method the route calls with what it already has — and the task becomes a thin wrapper that acquires a session and calls it. Which is exactly what send_booking_confirmation is.

Observing them

The hardest thing about background tasks is that they are invisible by default: no status code, no response, nothing in an access log. Two cheap habits fix most of that.

Log the start and the outcome, with the id the task was given, so the work can be found from the same request id as the endpoint that queued it:

    logger.info("Email queued for %s: %s", to, subject, extra={"recipient": to})

⚠️ One caveat specific to this: the ContextVar holding the request id has usually been reset by the time a task runs, so the correlation is not automatic. If you need it, capture the id in the route and pass it as an argument — the same discipline as everything else the task needs.

Count them. A counter of tasks started and tasks failed, from lesson 15's metrics, turns "did the emails go out?" into a graph rather than a log search. It is also the only way to notice the failure mode where a provider quietly starts rejecting everything.

Choosing a queue, when you need one

Three options dominate, and the differences are practical rather than philosophical.

RQ is the honest first step for a Python service that already has Redis: a few lines, a worker process, and persistence plus retries. Celery is the right answer once you need scheduling, chained tasks, priorities or routing, and it is genuinely more to operate. A managed queue removes the broker from your responsibilities and adds a network hop and a bill.

Whichever it is, the boundary you should design for is the same: tasks take serialisable arguments and are idempotent. Get that right with BackgroundTasks and moving to a queue is changing one line at each call site. Get it wrong — passing objects, assuming exactly-once — and the move is a rewrite.

The decision, in one paragraph

Use BackgroundTasks when the work is genuinely optional, fast, and cheap to lose: warming a cache, updating a search index that has a rebuild path, writing an audit line, tidying a temporary file. Use a queue when losing the work costs something a user or an accountant would notice. And whichever you use, take ids rather than objects, open your own session, catch your own exceptions, and write the task so running it twice is harmless — because at some point it will.

Two habits worth forming

Make tasks idempotent. With retries, "exactly once" is not on offer; "at least once" is. A task that sends an email should record that it did, so a retry after a partial failure does not send a second one.

Keep them small and specific. One task that sends an email, another that updates the index — not one that does both. A combined task that fails halfway has no good retry, because retrying repeats the half that worked.

Next: async def or def — the same "which one blocks" question that decided how these tasks are written, with measurements.