Designing a Reliable API Request Queue

1. Give the queue one clear purpose

An API request queue accepts a durable description of work now and dispatches it when a worker and the provider have capacity. It can keep a user request short, smooth a temporary burst, isolate background traffic, and survive a worker restart without losing accepted work.

A queue does not create provider capacity. If work arrives faster than it can be completed for a sustained period, backlog and age keep growing. The design must eventually add capacity, reduce arrivals, coalesce work, expire low-value tasks, or reject new work. Unlimited backlog merely converts an immediate failure into a delayed one.

producer -> durable queue -> bounded workers -> provider
              ^                  |
              |                  v
        status record      retry or complete

2. Make the enqueue boundary durable

Decide exactly when the application may tell a caller that work was accepted. That point should come only after the task is durably stored. If the application changes local state and publishes a message as two independent operations, a crash between them can leave state without a task or a task without matching state.

A transactional outbox is one common solution when the database and broker cannot share a transaction. Write the business change and an outbox record in one database transaction, then let a separate publisher deliver the outbox record to the queue. The publisher can repeat delivery, so the queue task and worker still need deduplication and idempotency.

begin transaction
  update local business state
  insert outbox event with stable job_id
commit

outbox publisher -> queue
worker -> provider -> durable result -> acknowledge

3. Put the contract in the task envelope

A worker should not have to reconstruct the task's intent from mutable application state alone. Give each task a stable identifier, operation version, provider and operation, priority, creation and expiry times, idempotency key, correlation identifier, and either a minimal payload or a protected reference to one.

Version the envelope so workers can reject or migrate old formats deliberately. Avoid secrets in messages, logs, and dead-letter records. Encrypt queue data when required, restrict producer and consumer permissions, and define retention that matches the sensitivity and useful lifetime of the work.

{
  "job_id": "job_01...",
  "version": 1,
  "provider": "example-api",
  "operation": "refresh-account",
  "priority": "normal",
  "created_at": "2026-07-13T14:00:00Z",
  "expires_at": "2026-07-13T14:15:00Z",
  "idempotency_key": "refresh:account_42:v7",
  "payload_ref": "account_42",
  "correlation_id": "req_01..."
}

4. Acknowledge after the durable outcome

Many queue systems use an acknowledgment deadline, lease, or visibility timeout. A worker claims a message, the broker temporarily hides it from other consumers, and the worker acknowledges or deletes it after successful processing. If the worker crashes or the claim expires first, the message becomes eligible for another delivery.

Set the claim long enough for normal processing, but not so long that a crashed worker delays recovery excessively. Extend it with a heartbeat only while useful work is progressing. Record the provider result or local state change durably before acknowledging the task. Acknowledging first can lose work when the process fails before the result is saved.

claim task
if expired: record expiration and acknowledge
if already completed: acknowledge
perform bounded provider request
commit durable result
acknowledge task

5. Design every worker for repeated delivery

Amazon SQS standard queues and Google Cloud Tasks both document at-least-once delivery behavior. A timeout, worker crash, acknowledgment failure, or producer retry can therefore cause the same logical work to appear again. Broker-side deduplication can reduce duplicates, but it does not replace an idempotent business operation.

Use the stable job identifier or business idempotency key to find a prior durable result. For provider writes, use a provider-supported idempotency key or conditional request when available. If neither exists, store a local operation record and make the state transition conditional. Keep deduplication records at least as long as the task can be delivered or redriven.

• Create: use a stable provider idempotency key or a locally reserved resource identifier.
• Update: use a version, ETag, or compare-and-set condition where supported.
• Delete: treat already absent as the intended final state when the API contract permits it.
• Notification: record the recipient and logical event so a duplicate does not send twice.
• Read or refresh: coalesce newer work for the same resource when older results have no value.

6. Bound dispatch rate and concurrency

A time-based dispatch rate controls how quickly workers start requests. A concurrency limit controls how many requests can remain in flight. Both matter. A slow provider can exceed connection, memory, or provider concurrency limits even while requests per second remain below the published rate.

Start below documented provider capacity and preserve headroom for interactive traffic, retries, and other clients sharing the same quota. Separate independent provider buckets when their scopes differ. Google Cloud Tasks exposes both maximum dispatches per second and maximum concurrent dispatches, illustrating why one limit does not replace the other.

estimated_in_flight = dispatch_rate x response_time

Example:
80 dispatches/second x 0.20 seconds = 16 in flight

Choose the actual limit from provider quotas, worker capacity,
connections, memory, measured latency, and reserved headroom.

7. Use a few priority classes and protect fairness

Priority should represent business criticality, not which caller shouts loudest. A practical design might separate interactive work, ordinary background work, and deferrable backfills. Google SRE describes a small set of criticality values and treats batch work as more sheddable than user-visible production traffic.

A numeric priority field does not guarantee that high-priority work starts first. Consumers may already hold prefetched low-priority messages, and a continuous high-priority stream can starve everything else. RabbitMQ recommends a small number of priorities and notes that consumer prefetch affects priority behavior. Separate queues with reserved worker shares are often easier to reason about.

• Reserve some capacity for interactive work instead of letting batch work consume every slot.
• Give normal and low-priority queues a minimum service share so they eventually progress.
• Limit per-tenant in-flight work when several tenants share one provider credential.
• Measure oldest-task age separately for each priority and tenant class.
• Do not let a retry silently move into a higher priority class.

8. Delay work without sleeping a worker

A delayed task is unavailable until a scheduled time. Use that mechanism for a future job, a provider reset time, or a transient retry. Do not keep a worker process asleep while it owns a task because that consumes concurrency and can allow its claim to expire.

A delivery delay and a visibility timeout serve different stages. Amazon SQS documents a delay queue as hiding a newly enqueued message, while a visibility timeout hides a message after a consumer receives it. Preserve the original creation time and attempt history when scheduling retries so queue age and retry budgets remain honest.

next_attempt_at = max(
  valid provider Retry-After or reset time,
  current_time + jittered local backoff
)

if next_attempt_at >= expires_at:
  expire instead of retrying

9. Expire and cancel work that lost its value

Not every accepted task remains useful. A refresh may be superseded, a user may cancel an export, or a response may no longer fit the product deadline. Put an explicit expiry time in the task and check it before every attempt. Expiration is an expected outcome, not a processing failure that belongs in an endless retry loop.

Cancellation in a distributed queue is usually cooperative. Mark the job canceled in durable state and have workers check that state before side effects. If a provider request is already in flight, cancellation may stop later steps without undoing the completed remote operation. Design compensation separately when the business process requires it.

10. Treat the dead-letter queue as an investigation queue

A message that repeatedly fails can block ordered work, waste provider capacity, and hide a permanent data or code problem inside normal retries. After a bounded number of eligible attempts, move it to a dead-letter queue or terminal failure store with its job identifier, failure classification, attempt count, and safe diagnostic context.

A dead-letter queue is not a backup or success archive. Alert when messages arrive, assign ownership, define retention, and provide a review workflow. Amazon SQS warns that moving a FIFO message to a dead-letter queue can break exact ordering, so ordering requirements must be considered before enabling that path.

• Transient failure: delay and retry within the task's attempt and time budgets.
• Permanent task failure: dead-letter with a specific reason and no automatic retry.
• Expired or canceled work: record a terminal outcome without calling the provider.
• Malformed or unknown version: quarantine safely and alert the owning team.
• Systemic provider outage: pause dispatch or open a circuit instead of dead-lettering the entire backlog.

11. Redrive slowly and only after the cause is understood

Redrive returns reviewed dead-letter work to processing. First identify and fix the cause, validate that the work is still wanted, confirm idempotency, and choose whether the original task or a corrected replacement should run. Preserve an audit link to the dead-letter record.

Do not release an entire dead-letter backlog at maximum speed. It can recreate the incident and compete with new work. Amazon SQS supports a custom redrive velocity and recommends beginning at a small rate, checking the destination queue, and increasing gradually. The same controlled approach applies to any queue product.

review -> repair or replace -> sample redrive -> verify outcomes
       -> bounded batch -> monitor provider and queue -> continue

Never redrive blindly into an unchanged failure path.

12. Monitor age and outcomes, not backlog alone

Backlog count shows stored quantity, but oldest-task age shows how long useful work is waiting. Amazon SQS publishes separate metrics for visible backlog, delayed messages, in-flight messages, and the approximate age of the oldest message. Use the equivalent signals provided by the chosen broker, and add application outcomes that the broker cannot know.

Track enqueue failures, accepted tasks, dispatches, completions, expirations, cancellations, retry attempts, acknowledgment or lease expirations, dead-letter arrivals, and redrive outcomes. Break them down by provider, operation, priority, and tenant where cardinality and privacy permit. Alert when age approaches the product deadline, not only when the queue contains a large number.

• Ready, delayed, and in-flight task counts
• Oldest ready-task age by priority
• Enqueue-to-start and enqueue-to-completion latency
• Provider response status, latency, throttling, and timeouts
• Attempts per completed logical job and duplicate suppression
• Dead-letter arrival rate, oldest dead-letter age, and redrive success
• Worker saturation, dispatch rate, concurrency, and unused capacity

13. Walk through one bounded design

Assume an illustrative provider policy allows 100 requests per second and no more than 20 concurrent requests. The application chooses 80 dispatches per second and 16 concurrent requests to reserve headroom. Interactive refreshes, ordinary synchronization, and backfills use three separate priority queues with explicit worker shares.

Each task has a stable job identifier, 15-minute expiry, provider idempotency key, and payload reference. A worker claims one task, checks cancellation and prior completion, calls the provider with a bounded timeout, commits the result, then acknowledges. A transient response is rescheduled with provider timing or jittered backoff. Permanent failures move to review after the configured attempt limit.

These numbers are examples, not universal defaults. Production values must come from the provider's real quota scopes, measured response times, worker resources, acceptable queue age, and the other applications sharing the same credentials.

dispatch rate:       80 requests/second
concurrency:         16 in flight
priority classes:     3
task lifetime:       15 minutes
maximum attempts:     5 total attempts
retry policy:         documented pushback, then bounded jitter
dead-letter redrive:  reviewed, sampled, and rate limited

Request queue checklist

• Define when work is durably accepted and how enqueue failures reach the caller.
• Use a stable job identifier, versioned envelope, expiry, priority, and correlation identifier.
• Keep secrets and unnecessary personal data out of task payloads and logs.
• Acknowledge only after the intended outcome is durably recorded.
• Make provider operations and local state transitions idempotent.
• Bound dispatch rate, concurrency, worker prefetch, attempts, and task lifetime.
• Use a few priority classes with reserved capacity and starvation protection.
• Schedule delayed retries in the broker instead of sleeping workers.
• Classify success, transient failure, permanent failure, expiration, and cancellation separately.
• Alert on dead-letter arrivals and redrive only after review at a controlled rate.
• Monitor oldest-task age, completion latency, retry amplification, and provider behavior.
• Test worker crashes, duplicate deliveries, late acknowledgments, provider outages, and redrive procedures.