Define the investigation you need to support
Lineage is often presented as a map of datasets and transformations. That view is useful for identifying dependencies and the impact of a schema change. An operational investigation asks a narrower question: why did this order receive an escalation at this time? Answering it requires the actual input versions, applicable rules and execution record, not only a diagram of systems that could have participated.
Choose representative investigations before designing the evidence model. A financial adjustment may require reconciliation inputs and an approval record. A supply warning may require the event sequence and customer commitment in effect. Determine the precision needed to explain each decision and the period for which that explanation must remain reconstructable.
Distinguish datasets, jobs and runs
OpenLineage separates concepts such as datasets, jobs and runs. This distinction matters because a transformation definition is not the same thing as one execution of that transformation. A job may run repeatedly with different inputs, code versions or operating conditions. The lineage associated with a specific result should identify the relevant execution rather than pointing only to the job’s current definition.
Use stable names for logical datasets and unique identities for executions. Capture input and output versions when the source supports them. Record execution state and the relevant transformation version. Where only a timestamp or source watermark is available, preserve that limitation explicitly so an investigator does not assume a level of historical precision the system cannot provide.
Capture the interpretation between input and output
An operational result often depends on more than a data pipeline. It may use a business calendar, an entity mapping, a risk threshold or a model configuration. These dependencies can change without modifying the source table. Include the versions of important interpretation rules in the decision record, particularly when they can alter the recommended action for the same underlying facts.
W3C PROV offers a model for describing entities, activities and agents involved in producing information. Distinguish evidence from the process that interpreted it and the person or service responsible. Retain the structured inputs needed to verify a decision alongside its written explanation.
Connect the event path without confusing causality
Trace identifiers and correlation identifiers help connect activity across services. They are useful for investigation, but a shared identifier does not by itself prove that one event caused another. A workflow can combine several inputs, retry an operation or receive a late correction. Preserve explicit relationships between the decision and the evidence it actually used.
OpenTelemetry defines signals that include traces, metrics and logs. These signals complement lineage by explaining execution timing and runtime behavior. Connect a decision record to its relevant traces while retaining a distinct business evidence record. Trace sampling and telemetry retention may be appropriate for operations but insufficient as the only durable explanation for a consequential business action.
Keep evidence useful without collecting everything
Lineage records can contain sensitive field names, document references and business relationships. Apply access controls to the evidence itself, and avoid copying entire payloads into unrestricted logs. Identify what must be retained to reproduce a decision, what can be referenced securely and what should be removed under the organisation’s retention rules.
Decide how deletion, source corrections and restricted evidence affect historical explanations. A record may need to preserve that a decision used a now unavailable source without exposing the removed content. Document these cases with governance owners.
Test reconstruction as an operational capability
Select a completed decision and ask a reviewer who did not implement the workflow to reconstruct it. They should find the source observations, identity mapping, rule version, execution status and resulting action. Include cases with a failed dependency, a retried event and a subsequent correction. Record where the evidence requires undocumented knowledge or access that the reviewer cannot obtain.
Use those gaps to improve capture and presentation. Establish an owner for missing lineage and an alert for broken evidence links where appropriate. Repeat the exercise after significant workflow changes and verify that investigators can still recover the evidence needed to explain a decision.