Map connectivity and trust boundaries
Identify where source systems reside, which connections they permit, and whether communication must be initiated from inside a restricted network. Record DNS, proxy, certificate, firewall, and outbound access requirements. An architecture that ignores network ownership often fails before its first useful data exchange.
Separate public request handling from privileged source access where that reduces exposure. Authenticate service calls even within private networks, and define how credentials are delivered and rotated. Confirm the permitted data locations and operational access model with the relevant owners rather than assuming a particular cloud or region is acceptable.
Source, worker and destination
Separate connectivity, processing and stored recovery state.
Restricted source
Approved connectivity and scoped credentials.
Replaceable worker
Persist progress outside the worker process.
Durable destination
Verify the write before advancing the checkpoint.
Persist recovery state. Retain the source position, operation IDs and transformation versions.
Recovering from a worker failure
A replacement worker needs a valid resume position and a way to verify writes whose responses were lost. Preserve offsets, deduplication records and source history for the recovery period.
Separate environments and classify state
Use distinct identities, configuration, queues, and storage for development, test, and production. Avoid copying production secrets or unrestricted records into lower environments. Use approved test datasets that preserve important integration edge cases without exposing sensitive production records.
Classify components as stateless, durable, or reconstructible. A projection may be rebuildable only if its source history and transformation versions remain available. A queue offset and a deduplication store may be essential recovery state even when the worker process itself is disposable. Back up the state required to recover correctly, not only the most visible database.
| Component | Operational concern | Evidence to request |
|---|---|---|
| API service | Capacity and graceful shutdown | Load and deployment test |
| Event worker | Offsets and in flight work | Restart and replay test |
| Projection store | Durability and rebuild time | Restore or rebuild exercise |
| Identity dependency | Credential renewal and outages | Rotation and failure procedure |
Use health signals according to their purpose
A readiness signal determines whether a component should receive work or traffic. A liveness signal helps detect a process that needs restart. A startup signal can allow a slow initialization period without triggering premature restarts. Kubernetes documents these as separate probe types with distinct effects.
Avoid tying liveness directly to every remote dependency. A database outage that restarts all application containers can make recovery harder without fixing the database. Check whether the process can make progress, then use readiness, deadlines, and controlled degradation for dependency failures. Exercise shutdown while requests and event handlers are active.
- Readiness
- Should this instance receive traffic?
- Liveness
- Does this process need a restart?
- Startup
- Has initialization completed?
Prove the selected failure domains
Decide which failures must be contained: a worker crash, a host failure, an unavailable zone, or a regional outage. More distributed deployment can improve some resilience properties while increasing coordination, consistency, and operating complexity. Align the design with agreed recovery objectives and actual support capacity.
Test resource exhaustion, dependency latency, certificate expiry, rolling updates and restoration. Measure the time to restore business processing, including backlog drain and reconciliation. Record which data was delayed, duplicated, lost or repaired.