AWS – SQS: Queues, Visibility Timeout and DLQs

A queue is the simplest way to stop a slow or broken dependency from becoming a 500. The request handler writes a message and returns; something else does the work later. If the worker is down, messages wait instead of being lost.

Standard or FIFO

Standard unless you can name why not. FIFO's ordering is per message group id, not across the queue — which is the useful part: order per customer, per booking, per whatever, while different groups still process in parallel.

Your consumer must be idempotent

Standard queues are at-least-once. Not "rarely twice" — at least once, by design. A duplicate can arrive because the visibility timeout expired, because a delete failed, or because of ordinary distributed-systems reality.

So "send the welcome email" must not send two. Give each message a business key and record what you have processed:

def handle(message):
    body = json.loads(message["Body"])
    key = body["booking_id"]

    # A conditional write is the whole idempotency mechanism: the second
    # attempt fails the condition instead of doing the work again.
    try:
        table.put_item(
            Item={"pk": f"PROCESSED#{key}"},
            ConditionExpression="attribute_not_exists(pk)",
        )
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return          # already done
        raise

    send_confirmation(body)

FIFO's exactly-once applies only within a five-minute deduplication window, so it does not remove this requirement either — it narrows it.

Visibility timeout: the setting that duplicates your work

When a consumer receives a message it is not deleted — it becomes invisible for the visibility timeout, default 30 seconds. The consumer must delete it before that expires, or it returns to the queue and another consumer picks it up.

So if your handler takes 45 seconds and the timeout is 30, every message is processed twice, forever. The work completes, the delete arrives too late, and the queue looks like it is mysteriously growing.

Set the timeout to comfortably more than your slowest run — a common rule is six times the expected duration. For genuinely variable work, extend it while running instead:

aws sqs change-message-visibility \
  --queue-url https://sqs.us-west-2.amazonaws.com/111122223333/bookings \
  --receipt-handle "AQEB..." \
  --visibility-timeout 300

Long polling

Short polling returns immediately, often with nothing, and you pay for the request. Long polling waits up to 20 seconds for a message to arrive.

aws sqs set-queue-attributes \
  --queue-url https://sqs.us-west-2.amazonaws.com/111122223333/bookings \
  --attributes ReceiveMessageWaitTimeSeconds=20

This is close to free money: fewer empty receives, a much smaller bill on an idle queue, and lower latency, because the message is delivered the moment it lands rather than at your next poll. Set it to 20 on every queue you create.

Dead-letter queues

Without one, a message that always fails is retried until it expires — up to 14 days of a consumer crashing on the same input, and then the evidence is deleted.

A redrive policy moves a message aside after N failures:

aws sqs set-queue-attributes \
  --queue-url https://sqs.us-west-2.amazonaws.com/111122223333/bookings \
  --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:us-west-2:111122223333:bookings-dlq\",\"maxReceiveCount\":\"5\"}"}'

Note RedrivePolicy is a JSON string inside JSON, which is why the escaping looks like that. It is not a mistake.

Alarm on the DLQ's depth. A dead-letter queue nobody looks at is a folder of silent failures — the alarm is the point, not the queue.

Lambda as the consumer

An event source mapping polls for you and invokes your function in batches, scaling automatically. Two settings matter.

The visibility timeout must exceed the function timeout, and by a margin. AWS recommends at least six times. Get this backwards and messages are redelivered while still being processed.

Enable ReportBatchItemFailures. By default, one failure in a batch of ten returns all ten to the queue — including the nine that succeeded, which are then processed again. Reporting per-item failures returns only the ones that actually failed.

Delay, and scheduling work

Two ways to make a message arrive later. A delay queue holds every message for a fixed period via DelaySeconds on the queue. A per-message delay sets it on publish, which is more useful — "retry this in five minutes" without a scheduler.

The cap on both is 15 minutes. For anything longer, use EventBridge Scheduler or Step Functions; people routinely try to build a delayed job system on SQS and hit this wall after the design is finished.

What it costs, and why the queue is usually not the expensive part

SQS is billed per request, with the first million a month free, and a request covers a batch of up to ten messages. Batching therefore cuts your SQS bill by up to a factor of ten, and long polling removes the empty receives that make an idle queue cost anything at all.

In practice the queue is rarely the line item worth optimising — the consumers are. A queue is one of the cheapest pieces of infrastructure AWS sells, which is part of why reaching for one before reaching for a stream is usually right.

Things that are easy to get wrong

• Maximum message size is 256 KB. For anything larger, put the payload in S3 and send the key — the extended client library does this for you
• Retention defaults to 4 days, maximum 14. A consumer down over a long weekend can lose messages
• Delete after processing, not before. Deleting on receipt turns a crash into lost work
• A queue is not a database. You cannot query it, and reading a message to check something is not free