Define what each acknowledgement confirms
A producer acknowledgement might mean a broker accepted a record under its configured durability policy. It does not mean a downstream service updated its database or a customer received a notification. Write down the producer, broker, consumer and external side effect boundaries separately.
For every boundary, specify durable storage, acknowledgement timing, timeout behavior and recovery responsibility. Identify what can be lost or repeated under each failure condition, including the systems and configuration covered by an exactly once guarantee.
Understand the common delivery models
At most once delivery permits loss while avoiding redelivery at the stated boundary. At least once delivery retries uncertain work and therefore permits duplicates. Neither phrase alone explains ordering, latency, retention or the correctness of an application side effect.
Kafka documents transactional processing capabilities for specific Kafka operations. Extending a guarantee to an external database or service requires cooperation from that destination. A transactional stream processor cannot make an arbitrary payment request or email send part of its local commit.
| Model | Primary risk | Application response |
|---|---|---|
| At most once | A failure can lose work | Accept loss or reconcile |
| At least once | A retry can repeat work | Deduplicate and make effects idempotent |
| Transactional processing | Guarantee stops outside its transaction | Define the external commit boundary |
Acknowledge after the agreed durable step
Acknowledging before durable processing can lose an event if the consumer crashes. Acknowledging after a committed state change can cause a duplicate when the acknowledgement is lost. The latter is often manageable with an atomic inbox and an idempotent effect.
For a webhook receiver, a durable queue handoff may be the correct acceptance boundary. The sender can stop retrying while the receiver owns further processing. Make that distinction visible in operational status so accepted, processing, completed and failed are not collapsed into one success value.
Route failures for repair or replay
An invalid event should not occupy a retry loop forever. Classify failures into transient, permanent and uncertain outcomes. Bound retries, retain diagnostic context and route unresolved messages to an operator owned quarantine or dead letter process.
Replay requires authorization and an auditable reason. Preserve original event identity when redelivering the same event so consumers can apply their duplicate policy. If operators intentionally request a new business action, represent it as a new command with a causal reference instead of disguising it as transport recovery.
Verify outcomes with failure injection
Stop a producer after sending but before observing an acknowledgement. Stop a consumer before and after its state commit. Delay the broker, reject a destination write and exhaust a retry budget. Each exercise should produce an expected business result and an expected operational record.
Measure unresolved message age, duplicate rate, processing lag and replay completion. Assign an owner to stranded work and reconcile source and destination state after recovery.