Start with operational questions
Define what the operator must answer before choosing charts. Has the latest source change been captured? Which consumer owns a delayed event? Did an approved invoice reach the downstream system? Those questions connect infrastructure signals to a business path with clear owners.
Track availability, latency, error and saturation signals at service boundaries, then add progress measures such as the age of the oldest unresolved event. A consumer can return successful health checks while its queue grows indefinitely. Process health and successful business processing should remain separate measures.
Use each signal for its strengths
Metrics reveal aggregate behavior and support alerts. Traces describe the path and timing of individual operations. Logs record selected details that explain decisions and failures. OpenTelemetry provides a common model for collecting and exporting these signals, while the storage and analysis backend remains a separate choice.
Do not duplicate every payload into every signal. Record a safe event reference and a failure category in a log, then link it to the relevant trace where available. Keep an authorized route to business records instead of turning the telemetry system into another unrestricted operational database.
- Metrics
- Is unresolved work accumulating?
- Traces
- Where did this operation spend its time?
- Logs
- What caused this outcome?
Keep sensitive business data out of diagnostic records.
Propagate context across asynchronous boundaries
A request may publish an event that is processed long after its original span ends. Carry validated trace context where the transport and instrumentation support it. Use span relationships or links appropriate to batching, fan out and delayed work rather than inventing a single continuous synchronous call.
W3C trace context includes identifiers that enable correlation; it does not confer permission to access an event. Treat received context as untrusted input, apply propagation limits and avoid placing secrets in baggage. Correlation identifiers and business identifiers need separate retention and access decisions.
{
"eventSource": "https://example.com/erp",
"eventId": "evt_2048",
"stage": "projection_commit",
"outcome": "retryable_failure",
"errorCategory": "destination_unavailable"
}Keep metrics bounded and alerts actionable
Use a small, controlled label set for service, environment, stage and failure class. An event ID or arbitrary customer value can create excessive metric cardinality. Put detailed identifiers in appropriately protected logs or traces instead, with sampling and retention chosen for the investigation need.
Alert on meaningful user impact or an approaching operational limit. Combine a symptom, a severity, an owner and a runbook. A dashboard is useful for exploration; an alert should require a decision. Review noisy alerts after incidents rather than accepting persistent notification fatigue.
Test the diagnostic path
Inject a known failure and verify that an operator can find its event, processing stage and recovery action. Confirm that dashboards distinguish capture lag from consumer lag. Check that trace sampling retains enough detail to investigate high impact errors.
Measure telemetry volume, export failures and collector resource use. Instrumentation must degrade safely when its own backend is unavailable. Review example logs for personal data and credentials, and test that redaction remains effective when a downstream service returns an unexpected error body.