Define health by the work the data supports
A data product serves a particular use, such as showing current order exposure or supporting a period close. Its health should be defined in relation to that use. A successful pipeline run does not establish that the expected records arrived, that an important field retained its meaning or that users can access the result before their decision deadline.
Write a small set of observable expectations with the product owner. These might describe maximum age, required source coverage, reconciliation tolerance or the availability of critical fields. Specify the population and observation window for each measure. Avoid an aggregate health badge whose calculation is unclear, because it makes both escalation and meaningful improvement difficult.
Connect runtime signals to the data path
OpenTelemetry describes signals including traces, metrics and logs. Each contributes a different view of system behavior. Traces connect execution steps, metrics describe measured behavior over time and logs record events. Use them to follow source extraction, transformation, storage and serving, with identifiers that let an operator connect a product symptom to its relevant processing activity.
Keep business context in telemetry carefully scoped. A product or pipeline identifier can support investigation without placing customer names or full payloads in every log. Avoid high cardinality metric labels such as individual transaction identifiers when they would make aggregation costly or impractical. Preserve detailed references in appropriately controlled records that can be consulted after a signal identifies the affected area.
Measure freshness at the right boundary
The time since the last successful job is not always the age of the data it produced. A job can run successfully against an unchanged or incomplete extract. Track a source observation watermark where possible, along with ingestion and publication times. Distinguish source inactivity from a stalled integration using the expected behavior of the underlying business process.
Define freshness at the level consumers need. An overall maximum timestamp can hide a stale region or missing business unit if another source continues to update. Break important indicators down by relevant partitions and record the expected arrival pattern. Show users the age of the information they are viewing, particularly when an operational action depends on its currency.
Observe completeness and meaning separately
Completeness checks ask whether the expected population arrived. Structural checks ask whether records satisfy the contract. Reconciliation checks compare data with an independent or authoritative source. These are distinct forms of evidence. A record count within its normal range cannot prove that customer identifiers are correct, and a valid schema cannot prove that a financial amount has the intended meaning.
Choose checks that correspond to plausible failure modes in the product. Compare source and destination counts within a defined scope, detect missing entity references and inspect distributions where unexpected shifts would matter. Explain tolerances and expected seasonal behavior. Anomaly detection can identify useful candidates for review, but an unusual pattern should not automatically be classified as a data error.
Design alerts around an owner’s next action
An alert should identify the affected product, violated expectation, observed impact and responsible team. Include enough evidence to begin investigation without requiring access to every infrastructure dashboard. Route warnings differently from failures that should pause a dependent action. Repeated alerts with no clear owner or resolution path usually become background noise rather than an effective control.
Write recovery guidance alongside the alert definition. State whether a retry is safe, whether a backfill may create side effects and how completeness is verified after recovery. OpenLineage’s execution and dataset model can help connect a failed run to affected outputs. Give operators a way to distinguish a repaired pipeline from a repaired product whose consumers have received corrected information.
Use incidents to refine the observability model
Test the monitoring with controlled failures: a stopped source, a partial extract, a schema change and a delayed partition. Verify that the expected team receives a useful signal and can identify the impacted consumers. Record detection and recovery evidence rather than assuming that the presence of a dashboard proves the system is observable.
After an incident, ask which promise was violated and whether a signal existed before a user reported the problem. Add a targeted check where it would provide earlier or clearer evidence, and remove alerts that do not support action.