Model business states and transitions
Begin with the states visible to the business: received, validated, awaiting review, approved, rejected and completed. Define which transitions are allowed, which role or event can initiate each transition and which evidence is required. A diagram of service calls alone does not describe the business process.
Use a stable workflow identity and connect it to the relevant resource, such as invoice inv_1042. Keep business state distinct from execution state. An approved invoice may still be waiting for an external posting, and a temporary worker failure does not necessarily revoke the approval.
Invoice approval and posting
Example: inv_1042
Invoice approved. Retain the authorization and reviewed version while verifying the posting.
- Response timed out
Posting unresolved
A timeout may occur after the destination posts the invoice.
Reconcile before retrying. Look up the operation ID at the destination to avoid posting the same invoice twice.
- Destination record verified
Posting confirmed
The destination record matches the operation ID.
Record completion. Confirm all required updates before completing the workflow. Link the approval to its execution records.
Persist progress before advancing
An orchestrator records durable progress and schedules the next step from that record. Activities must tolerate retry because a worker can complete a remote action and lose its response. Use stable operation identifiers and an explicit reconciliation strategy for ambiguous outcomes.
Define timeouts separately for starting a step, executing it and waiting for an external response. A timeout means the orchestrator stopped waiting; it does not prove that remote work stopped. Avoid launching a conflicting replacement until the effect of the original attempt is understood.
| State | Exit condition | Failure handling |
|---|---|---|
| Awaiting review | Authorized approval or rejection | Escalate after a defined deadline |
| Posting requested | Verified destination outcome | Retry or reconcile by operation identity |
| Completed | Required effects confirmed | Retain evidence and monitor later corrections |
Treat compensation as business work
A saga coordinates local transactions with compensating actions when a distributed operation cannot complete as intended. Compensation is not necessarily an exact technical rollback. A dispatched notification cannot be unsent, and a posted financial entry may require a corrective entry rather than deletion.
Document each reversible step, its compensation and any point beyond which human resolution is required. Compensation can fail too, so it needs its own retry policy, evidence and owner. AWS guidance also highlights saga limitations, including the lack of transaction isolation across participating operations.
Make human decisions explicit and current
A review task should present the information needed for a decision, the acting identity and the version of the resource being reviewed. Recheck authorization at decision time. Access held when a task was created may no longer be valid when someone responds days later.
Prevent a stale approval from applying to a materially changed invoice. Compare the reviewed version with current state, then require renewed review when the policy calls for it. Record the decision, reason and relevant evidence while minimizing unnecessary personal information in the audit trail.
Test recovery as a sequence of decisions
Test worker restarts, duplicate callbacks, late approvals, expired deadlines and compensation failure. Verify the durable state after each interruption, including which actions are still permitted. Workflow version changes need migration rules for executions already in progress.
Monitor the age of waiting states, repeated activity failures and unresolved compensation. Provide recovery controls that let operators retry, reconcile or escalate work with a recorded reason and outcome.