Choose the granularity required by the question
Dataset lineage identifies which tables, files, or streams depend on each other. Column lineage explains how individual fields are derived. Record or decision provenance can identify the specific input versions used in an outcome. These levels have different costs and should be selected around an actual investigation or control requirement.
An impact analysis for a renamed column may need field level dependencies, while explaining an invoice approval may require the contract version, source event, and policy decision. Do not assume a catalog edge proves that a specific record was used in a particular execution. Static design lineage and observed run lineage answer different questions.
Capture execution context alongside dependencies
Record a stable job identity, a unique run identifier, input and output dataset identities, and relevant schema or transformation versions. OpenLineage distinguishes jobs, runs, and datasets, with facets that carry additional metadata. W3C PROV provides a broader model of entities, activities, and agents involved in producing information.
For invoice inv_1042, a decision record could refer to the observed invoice version, contract version, policy version, and workflow run. Keep the reference stable even if the underlying source later changes. Where exact source snapshots cannot be retained, state the resulting reproducibility limits rather than implying a complete historical reconstruction.
| Evidence | Purpose | Example |
|---|---|---|
| Dataset identity | Locate the source or projection | ERP invoice collection |
| Run identity | Distinguish executions | One processing attempt |
| Transformation version | Explain derivation | Approved mapping revision |
| Input version | Identify observed state | Invoice revision used for review |
Make gaps and sensitive metadata visible
Lineage collected through query parsing can miss dynamic SQL, stored procedures, external scripts, or manual data movement. Runtime instrumentation can also be incomplete when an integration lacks a supported emitter. Record coverage and collection methods so users can distinguish an absent dependency from an unobserved one.
Metadata itself can be sensitive. Dataset names, query text, filters, and actor identifiers may reveal business activity or personal information. Apply access and retention controls to the lineage store, redact unsafe payloads, and avoid placing credentials or full records in freely searchable annotations.
Use provenance in change and incident workflows
Before a schema change, inspect downstream dependencies and confirm their owners. During an incident, trace an error to its processing run, input versions and authoritative source.
Test the evidence path with an actual sample result. Confirm that referenced runs and datasets exist, that timestamps are interpretable, and that the access policy permits the intended investigator to follow the path. Measure missing or stale lineage so the quality of the explanation can improve alongside the data pipeline itself.