Limit the resource that is actually scarce
Requests per second are easy to count but may not reflect cost. A small invoice lookup and a large export can use very different CPU, memory, and database capacity. Consider separate controls for request rate, concurrent work, response volume, and expensive operations.
Choose a quota scope deliberately: client identity, tenant, operation, or a combination. An IP address alone may group unrelated enterprise users behind one proxy. State the scope in documentation so consumers can coordinate workers rather than discovering that each process independently consumes the same shared allowance.
Communicate throttling through clear HTTP behavior
HTTP 429 indicates too many requests within the server's chosen rate limiting policy. The response can include Retry-After, which HTTP defines as either a delay in seconds or an HTTP date. Clients need robust parsing and should not assume every throttled response includes the header.
Separate quota exhaustion from service outages. Return the appropriate status and a clear error identifier without exposing other tenants' demand or internal capacity details. Document the enforced quota and its scope.
HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 12
{
"type": "https://example.com/problems/rate-limit",
"title": "Request allowance exceeded",
"status": 429,
"detail": "Wait before retrying this operation."
}Coordinate retries across workers
Bound the number of in flight requests and the size of local queues. When throttled, respect a valid server retry instruction and add suitable jitter to prevent synchronized clients from returning together. Without a server delay, use a bounded backoff policy with a maximum elapsed retry budget.
Retry only operations whose repeat behavior is understood. A timeout during a write can mean the server committed the change before the response was lost. Use an idempotency strategy or query the operation result instead of assuming no work occurred. Avoid nesting retry loops at several layers, which can multiply traffic unexpectedly.
Measure fairness and recovery behavior
Observe rejected requests, queue age, active concurrency, and latency by meaningful bounded dimensions. An acceptable average can conceal one tenant experiencing sustained starvation. Ensure batch jobs cannot consume all capacity required by interactive or critical operational work.
Test coordinated bursts, slow dependencies, retrying clients, and recovery after a prolonged outage. Verify that queued work drains without causing another incident. A rate limit is one control in a broader capacity model; admission control, bounded queues, deadlines, and graceful degradation still need their own design and operating evidence.