Duplicate messages, delays and failures affect every event driven system. Plan how consumers detect them and recover.
Delayed and repeated order events
Order 1042 · 24 September 2026 · UTC
Dispatched v8The later version arrives first.
Picked v7An earlier event arrives late.
Dispatched v8The same event arrives again.
Define what an event asserts
An event should describe a fact that happened, with a meaning consumers can understand without inspecting the producer’s database. Order accepted is different from order row updated. The first identifies a business transition; the second reports a storage change that may need interpretation. Both can be useful, but mixing them under one event name transfers ambiguity into every downstream workflow.
Document the event type, business subject, occurrence time and authority of the producer. State whether an event reports a completed transition or a request for work. CloudEvents standardizes common event metadata such as source, identifier and type. It supports interoperability of the envelope, while the business meaning and data contract remain responsibilities of the producing domain.
Protect the transaction boundary
A common failure occurs when an application commits a business record and then fails before publishing its event. Reversing the order creates the opposite risk: consumers may receive a notification about a transaction that never committed. Treat the database update and the intention to publish as one local transaction where the application’s storage model allows it.
The transactional outbox pattern records that intention beside the business update, then publishes it through a separate process. Debezium documents an outbox event router that maps outbox records into event messages. The pattern reduces a particular consistency gap; it does not remove the need to monitor publication lag, manage retained records or make consumers resilient to repeated deliveries.
Receipt time changes on redelivery. Source plus event ID does not. Version order is this producer’s explicit contract; timestamps alone do not establish it. Reconcile an uncertain external action before retrying.Event identity and source version explain how the same business occurrence is interpreted across repeated deliveries.
Design for repeated delivery
A network timeout does not establish whether a receiver processed a message. Retrying is often necessary, so consumers must define how they identify work already completed. Use a stable event identity scoped by its producer and a durable record of processing where the action warrants it. A short lived in memory cache is insufficient if duplicates can arrive after a restart.
Make the business action itself idempotent where possible. Applying a desired order status is different from incrementing a quantity blindly. For an external side effect, preserve the provider’s request identity or an application operation key and reconcile uncertain outcomes. Avoid describing transport guarantees as a guarantee that an invoice, payment or notification can never be applied twice.
Choose the ordering that matters
Most business workflows require ordering for a particular entity rather than one total order across the enterprise. Events about a single order may need a consistent sequence, while unrelated orders can proceed independently. Select partitioning keys based on that business requirement, and document how key changes, multiple producers or migration might disrupt the assumption.
Track event time separately from processing time. A delayed warehouse scan can arrive after a later operational update and still be valid evidence. Define whether consumers reject older versions, revise derived state or open an exception. Include a version or sequence when the producer can provide one, and avoid treating timestamps from unrelated clocks as a complete ordering mechanism.
Make replay an explicit operating mode
Replaying a retained stream can rebuild a projection, repair a consumer or support an investigation. It can also resend customer messages or reopen closed tasks if operational side effects are coupled directly to event processing. Separate projection updates from action execution, and require a clear policy for whether replay is allowed to create external effects.
Record the code and contract version used for a replay, the offset range and the expected output. Validate a representative segment before rebuilding a large history. Provide a way to stop or isolate malformed records without losing the processing position. A failed event should remain discoverable with its cause and owner, rather than disappearing into an unattended error queue.
Measure the business processing path
Broker availability alone does not indicate that the business is current. Observe source commit time, publication time, consumer receipt time and the point at which a derived view becomes usable. Monitor lag by business stream and consumer, because an aggregate can hide a stalled workflow that affects a small but important set of orders.
Test duplicate events, unavailable dependencies, poison messages, schema changes and delayed delivery. Verify that recovery preserves correctness and audit evidence. Operators need to locate an event, identify the work already completed and understand what a retry or replay will do next.
