Assign delivery and processing ownership
The sender owns delivery until the receiver acknowledges the agreed acceptance boundary. The receiver owns processing after that boundary. If acceptance means a durable queue write, an HTTP success response tells the sender it may stop retrying; it does not claim that the business action is already complete.
Document the provider's actual retry window and status handling. Stripe, for example, documents automatic retries for live events for up to three days with exponential backoff. That behavior is provider specific. A different sender may use another schedule or stop after a different set of responses.
Bound retry attempts and elapsed time
When designing a sender, use an explicit retry policy with a maximum elapsed time, delay cap and randomized jitter. Randomization reduces the chance that many clients retry together after a shared outage. Respect Retry-After where the protocol and response semantics make it applicable.
Classify responses rather than retrying every failure identically. A timeout or temporary service error may be recoverable. A permanently rejected payload needs correction. Authentication failures may require configuration repair. Preserve enough status context to explain why delivery stopped without recording sensitive request contents.
export function retryDelayMs(
attempt: number,
random: () => number = Math.random,
): number {
if (!Number.isInteger(attempt) || attempt < 0) {
throw new RangeError('attempt must be a nonnegative integer');
}
const cap = Math.min(60_000, 1_000 * 2 ** Math.min(attempt, 16));
return Math.floor(random() * cap);
}Make receiver acceptance durable
Verify authentication and validate the envelope before handing the event to processing. Persist accepted work before returning success. If that persistence fails, return a failure consistent with the sender's protocol so the event remains eligible for retry.
Deduplicate using the sender's documented event identity, such as source plus id in CloudEvents. Do not use the signature as the duplicate key: providers may produce a different signature for each attempt. Keep request attempt metadata separately from the durable event record.
Recover deliveries after retries are exhausted
For each item in a dead letter queue, record the failure category, first and latest attempt time, event reference and responsible owner. Separate invalid contracts from temporary downstream failures so operators can choose the appropriate next step.
Define how operators inspect, correct, replay or discard work. Access to payloads and replay controls should follow the sensitivity of the underlying business action. Retain an audit record for manual decisions, especially when replay can create financial or customer facing effects.
Test uncertainty and concurrent redelivery
Simulate a receiver that commits the event but loses the response. Test two identical attempts arriving together and a manual redelivery overlapping an automatic retry. The final business outcome should remain consistent with the idempotency contract.
Monitor accepted event age, attempt count, retry exhaustion and processing completion separately. During recovery, pace replays so they do not overwhelm the same downstream system that caused the outage. Confirm completion through durable state rather than the number of requests sent.