Start with the invariants that matter
State the properties that must hold under normal operation and failure. An unauthorized tenant must not read an invoice; a repeated delivery must not repeat a financial side effect; an older source event must not overwrite a newer accepted state. These are more useful test goals than maximizing a coverage percentage without regard to risk.
Map each invariant to the component and boundary responsible for enforcing it. Some rules belong in a database constraint, others in authorization middleware or a workflow transition. Test the enforcement through its actual interface so the test does not merely reproduce the same assumption made by the implementation.
Layer contract tests with realistic behavior
Use schema and contract tests to check payload shape, required fields, and compatibility. Add semantic fixtures for currencies, time boundaries, partial updates, unknown enum values, and historical versions. A schema passing test does not establish that an amount has the correct unit or a relationship the correct grain.
Run integration tests with the real database and protocol behavior where it matters. Mocks can isolate logic but may hide transaction conflicts, row security bypass, cursor behavior, and acknowledgment timing. Keep a smaller set of complete workflow tests that prove a representative business path across the integrated environment.
Test coverage and results
Record the revision, configuration, schema versions and dataset beside the result.
Test tenant isolation, duplicate delivery, partial failures, schema changes and out-of-order events.
| Risk | Test condition | Expected evidence |
|---|---|---|
| Duplicate side effect | Deliver the same event concurrently | One durable business effect |
| Tenant leakage | Use another tenant's resource key | Consistent denial across interfaces |
| Schema drift | Replay retained older payloads | Compatible result or controlled quarantine |
| Partial failure | Interrupt after a committed write | Correct recovery without duplication |
| Stale state | Reorder updates for one entity | Documented ordering policy holds |
Control time and inject realistic failures
Use controlled clocks for token expiry, replay windows, backoff, and retention tests. Exercise network timeout, connection reset, unavailable dependencies, and partial responses. Distinguish a failure before a commit from a failure after a commit but before acknowledgment; the caller sees uncertainty in both cases, while recovery requirements differ.
Make randomized and property based tests reproducible with recorded seeds and clear invariants. Avoid indiscriminate failure injection in shared production systems. Choose isolated environments and bounded experiments with known recovery procedures, then record which assumptions were actually tested and which remain unverified.
Timeouts before and after commit
- Before the write commits
- The write has not committed. Use a stable operation ID when retrying.
- After commit, before acknowledgment
- The write may have succeeded. Check the destination before retrying.
Record release validation
Tie a test run to the application revision, configuration, schema versions, dependencies, and dataset used. Keep meaningful failure artifacts and record unresolved limitations. A green pipeline from a different configuration does not prove that the intended deployment is ready.
Include operational and interface behavior: empty states, denied access, slow responses, progress feedback, cancellation, and recovery after an interrupted workflow. Review test failures for product impact rather than automatically retrying until green. The release decision should be supported by a concise evidence set that explains what was verified and why it covers the important risks.