Order the smallest useful business scope
An invoice may require its own updates to be processed in sequence while updates for unrelated invoices proceed independently. Define that scope before choosing a partition key. A global sequence reduces concurrency and can make an unrelated producer delay every consumer.
Kafka records have an order within a partition. This does not create an order across partitions, nor does it ensure that application side effects finish in the same order when processing is concurrent. The producer key, partition assignment and consumer execution model must support the same business requirement.
Give sequence numbers a precise meaning
A per entity sequence can help reject stale updates, but the producer must define how it is assigned. Multiple writers need a shared serialization mechanism or an explicit conflict policy. A timestamp from each writer is not an adequate substitute when clocks differ or events occur together.
Specify whether the sequence is contiguous for the public stream. If filtered events are omitted, a jump may be expected rather than evidence of data loss. Treat source record versions and event sequence numbers as separate concepts unless the producer guarantees that they advance together.
| Observed sequence | Possible action | Required contract |
|---|---|---|
| Equal to applied sequence | Inspect as repeated or conflicting delivery | Stable event identity |
| Below applied sequence | Ignore stale projection update or investigate | Monotonic entity state |
| Above expected sequence | Buffer or reconcile | Defined gap semantics |
Invoice projection · contiguous per-invoice sequence · applied sequence 40
| Incoming | Decision against sequence 40 |
|---|---|
39Older | Keep the current projection Retain the event without overwriting sequence 40. Investigate or rebuild separately if the history needs correction. |
40Same | Check for a duplicate or conflict Compare the stable event identity and payload. If either differs, resolve the conflict under the receiving contract. |
41Next expected | Apply the next update Commit the projection change and sequence together, with a concurrency check. |
42Gap | Wait for sequence 41 Buffer sequence 42 within the agreed limits. Reconcile or quarantine it if the gap remains, and keep applied sequence 40 until the policy permits progress. |
Bound the waiting period
A reordering buffer trades latency and memory for the chance to receive missing predecessors. Bound it by event count, bytes and waiting time. An unbounded buffer can turn a missing event into a resource outage. Define a visible terminal action when the window closes.
Depending on the business operation, that action might retrieve authoritative state, quarantine the entity or request an explicit replay. Financial posting may require different handling from a dashboard projection. Make the choice a documented domain rule rather than an implicit behavior hidden in a queue client.
Keep replay separate from live progress
Replayed events can arrive behind a live stream. A projection that blindly overwrites state by arrival order can therefore restore an obsolete value. Store the applied sequence or another validated concurrency token with the projection, and update both atomically.
A rebuilt projection may instead consume a bounded history into a separate destination before switching readers. Document the reconciliation point and how live changes are included. Event time watermarks help stream computations reason about late data, but do not prove that every business event has arrived.
Test reordered and missing events
Test reversed pairs, duplicates, missing predecessors, producer restarts and two updates processed concurrently. Include a change in partition count when the selected platform supports it, because key placement may change. Observe the final business state as well as message processing success.
Measure gap age, buffer occupancy, stale update rejection and reconciliation outcomes. For each ordering policy, keep tests that verify the final state after delayed, missing and repeated events.