Assign roles before the first alert
Name the service owner, escalation path and responders for source systems, transport and downstream processing. Clarify who can authorize a replay, pause ingestion or change a retention setting. During an incident, unclear authority can be as disruptive as an incomplete runbook.
Google SRE separates incident command, operations and communications responsibilities. The incident commander maintains the overall picture while designated operators make changes and a communications role keeps stakeholders informed. Small teams may combine roles, but the responsibilities should remain explicit and handoffs should be recorded.
- Command
- Coordinate the response and authorize recovery.
- Operations
- Apply controlled changes and record the evidence.
- Communications
- Keep stakeholders informed of impact and uncertainty.
Describe impact in business terms
A broker alert is a signal, not the complete impact statement. Identify which event types, tenants and business workflows are affected using authorized operational evidence. Distinguish delayed processing from lost data and from uncertain external effects. Avoid claiming loss merely because a dashboard has stopped updating.
Maintain a timeline with detection, first known impact, mitigations and observed changes. Record confidence and unknowns. When a dependency is outside the team's control, name the dependency and the owner responsible for communication rather than leaving its status as an unexplained waiting state.
Choose and verify a mitigation
Runbooks should state prerequisites, expected effects, rollback conditions and verification steps for each action. Pausing a consumer may protect a destination while increasing queue age. Dropping a PostgreSQL replication slot can release its WAL retention requirement, but destroys that slot's recovery position and may require a new snapshot. Explain these tradeoffs before the emergency.
Capture safe diagnostic metadata before destructive recovery. Avoid repeatedly restarting services without a hypothesis, because that can erase useful state and create retry storms. Track configuration changes in the incident record and designate the team responsible for applying them.
| Observation | Immediate question | Recovery evidence |
|---|---|---|
| Capture lag rising | Can source logs support the delay? | Checkpoint and retained log window |
| Retries increasing | Is the destination unavailable or rejecting work? | Failure class and acceptance state |
| Replay requested | Could the effect repeat? | Deduplication and reconciliation result |
Reconcile work before declaring recovery
When a service returns, compare capture progress, durable queue position and applied destination state. A falling backlog does not prove that every event produced the expected effect. Sample and reconcile the affected interval using stable event and business identifiers.
Pace replays according to destination capacity, preserve event identity for redelivery and monitor errors separately from normal traffic. Ambiguous external actions may require provider queries or human review. Record remaining exceptions with owners instead of treating an available endpoint as proof of full business recovery.
Rehearse, review and improve
Exercise a failed destination, an expired checkpoint, a full quarantine queue and an unavailable telemetry backend. Include role handoff and stakeholder updates in the exercise. Measure whether responders can identify affected work and apply the documented recovery steps without undocumented access or specialist memory.
After an incident, distinguish triggering events from contributing conditions. Assign concrete improvements such as a safer replay control, a missing alert or a tested checkpoint procedure. Each action needs an owner and verification evidence. Review recurrence and recovery quality rather than assigning blame for normal human error.