Choosing Reasonable API Timeouts

1. Start with the end-to-end deadline

A timeout limits how long one operation may take. A deadline identifies the point when the complete result is no longer useful. Starting with the deadline keeps every queue, connection, dependency call, retry, and response step inside one user-facing budget.

Reserve part of that budget for work after the last dependency returns. The application might still need to combine results, serialize a response, write it to the network, and record essential state. A dependency that consumes the entire deadline has not left enough time to deliver a useful result.

total_deadline = 5000 ms
safety_reserve = 1000 ms
usable_dependency_budget = 4000 ms
remaining_budget = deadline - current_time

2. Know what each timeout actually measures

There is no universal connection timeout, read timeout, or request timeout. Libraries start and stop their timers at different stages. Before choosing a value, verify the exact version of the client library and learn what the setting includes.

For example, Go documents Client.Timeout as a total time limit that includes connection time, redirects, and reading the response body. Node.js documents request.setTimeout as a socket inactivity timer installed after the socket connects. Its timeout event does not abort the request by itself. Those two settings have different boundaries and failure behavior even though both are commonly called timeouts.

• Queue or pool-acquisition timeout: waiting for local capacity before a connection is used.
• Connection timeout: establishing a network connection, sometimes with DNS and TLS handled separately.
• Response-header timeout: waiting until the server begins a response.
• Idle or read timeout: the maximum permitted gap while response data is transferred.
• Total request timeout or deadline: the maximum elapsed time for the complete operation.
• Cancellation behavior: whether expiration only reports an event or also stops the request and downstream work.

3. Build the critical path

Only work on the critical path determines the earliest possible completion time. Required sequential calls add together because each waits for the previous call. Truly independent concurrent calls can overlap, so the slowest required branch determines their dependency time.

Concurrency does not make work free. A client still needs enough connection-pool capacity, server workers, quota, memory, and CPU. Excess concurrency can create a queue that consumes the deadline before work starts.

Sequential: 3 calls x 600 ms = 1800 ms
Concurrent: max(600 ms, 600 ms, 600 ms) = 600 ms

Also budget for pool wait, local work, and response delivery.

4. Allocate from the time that remains

Equal allocation is a useful first estimate, not a final production rule. Real dependencies have different latency distributions and importance. Start with the remaining end-to-end budget, reserve application time, and allocate more of what remains to the calls that need it.

A child operation should never receive a deadline later than its parent. Recalculate the remaining time immediately before each call. That prevents queue delay or earlier work from silently extending the complete operation.

usable = total_timeout x (1 - safety_margin)
dependency_budget = usable / sequential_calls
attempt_budget = dependency_budget / planned_attempts

child_timeout = min(local_limit, parent_deadline - current_time)

5. Make retries earn their place

A retry is another full attempt, not a free extension. It needs time to wait, connect, receive a response, and leave enough budget for the caller to finish. Retry only failures that are plausibly transient, and only when repeating the operation is safe or protected by an idempotency mechanism.

Keep retries at one deliberate layer when possible. Retries at several layers multiply attempts and can intensify an overloaded dependency. Cap the attempt count, backoff, and total retry time. A Retry-After instruction can also make a retry impossible within the current deadline.

minimum_attempt = connection_budget + processing_budget
minimum_finish = minimum_attempt + response_reserve

if remaining_time < minimum_finish:
    do_not_retry

6. Propagate deadlines and cancellation

Pass the remaining deadline to downstream work instead of giving every dependency a fresh duration. gRPC supports deadline propagation and converts the deadline to a timeout while deducting elapsed time. Explicit propagation keeps a dependency from working past the point when its result can be used.

A client timeout does not guarantee that the server stopped. Cancellation support depends on the protocol, client, server, and application. When cancellation is available, propagate it to child work, check it during long operations, release resources, and avoid committing side effects after the caller has abandoned the result.

• Send the remaining deadline or timeout to each child operation.
• Cancel child requests when the parent request ends or expires.
• Stop queued and spawned work that no longer has a caller.
• Release sockets, pool slots, locks, memory, and temporary resources.
• Record whether the deadline expired before queueing, connection, response, or body completion.

7. Choose values from latency evidence

A reasonable timeout is based on observed end-to-end latency and an explicit tolerance for false timeouts. Examine distributions such as the median, 95th, 99th, and 99.9th percentiles instead of using only an average. A narrow latency distribution can make a small change near the tail produce many unexpected timeouts.

Measure from the caller's point of view. Segment results by endpoint, payload size, region, connection reuse, deployment state, and other factors that materially change latency. Validate the candidate value with realistic load rather than copying one number to every dependency.

• Pick a false-timeout objective that matches the operation's reliability needs.
• Measure warm and cold paths, including DNS, connection, and TLS setup when applicable.
• Test the timeout when the system is busy, not only during quiet periods.
• Use separate policies for materially different endpoints and workloads.
• Revisit the value after architecture, traffic, provider, or regional changes.

8. Account for cold paths and deployment effects

New connections can include DNS lookup, TCP establishment, TLS negotiation, proxy work, and authentication. Connection pools, DNS caches, TLS sessions, application caches, and newly deployed instances can all be cold at the same time. A timeout tuned only from warm steady-state traffic may fail during startup or deployment.

Measure these phases separately when the client permits it. Preconnect or warm pools only when the operational cost is justified, and retain a bounded total deadline. Continuously increasing the timeout can hide queue growth and make failures slower without improving success.

9. Observe and revise

Timeout telemetry should explain where the budget went. Record the configured deadline, remaining budget at each dependency, elapsed time, attempt number, queue or pool wait, connection time, response time, cancellation outcome, and final result. Avoid logging credentials, request bodies, or sensitive response content.

Compare client and server observations. A client timeout followed by successful server completion is evidence of abandoned work. A rising pool wait can indicate local concurrency pressure, while rising service time can indicate a downstream problem. These are different causes and need different fixes.

10. Work through a concrete budget

Suppose a user-facing operation has a 5,000 ms deadline and a 20% safety reserve. That leaves 4,000 ms for dependencies. The critical path contains three sequential calls. Each attempt needs a 100 ms connection budget and a 500 ms processing budget, and the plan permits one retry after the initial attempt.

The three dependencies need 3,600 ms if every call uses both attempts. The plan retains 400 ms of dependency headroom plus the separate 1,000 ms application reserve. An equal allocation gives each dependency about 1,333 ms and each of its two attempts about 667 ms. Production evidence may justify shifting those allocations between dependencies.

Total deadline                 5000 ms
20% safety reserve            1000 ms
Usable dependency budget      4000 ms

One attempt: 100 + 500         600 ms
Two attempts per dependency   1200 ms
Three sequential dependencies 3600 ms
Dependency headroom            400 ms

Implementation checklist

• Define when the complete result stops being useful.
• Reserve time for local work and response delivery.
• Document the exact semantics of every client timeout setting.
• Map sequential and concurrent work on the critical path.
• Allocate child deadlines from the parent's remaining time.
• Fund only retries that can complete before the deadline.
• Propagate cancellation and stop abandoned work.
• Set candidate values from caller-observed latency distributions.
• Exercise cold connections, deployments, regional paths, and realistic load.
• Record timeout phase and budget consumption, then revise from evidence.