Privacy across the data lifecycle

Privacy engineering starts with understanding why information is processed and where it travels. Apply that understanding to source access, projections, logs, exports, and the lifecycle of derived records.

Selecting fields for an invoice report

A report of invoice totals by supplier and currency needs three fields.

Review the source fields

supplier_id
Keep to group records by supplier.
invoice_amount
Keep to calculate the total.
currency
Keep to avoid combining different currencies.
contact_email
Exclude. The report does not need contact details.
free_text_notes
Exclude. Notes are not needed to calculate invoice totals.

Report fields

supplier_id
supplier_42
invoice_amount
6200.00
currency
USD

Supplier references and amounts can still be sensitive. Restrict access even when personal contact details have been removed.

Prevent deleted data from returning during replay

Replaying old events or restoring backups can reintroduce deleted data. Reapply deletion and suppression rules, then check the resulting stores. Removing a field from the current report does not erase earlier copies or exports.

Review access, retention and deletion requirements for every store that contains these fields.

Map purpose before collecting fields

For each use case, identify the information required, the purpose it serves, the people or organizations affected, and the owner accountable for the decision. An invoice analytics view may need a supplier identifier and amount without needing personal contact details or free text notes.

Map how information moves from sources to caches, event streams, projections, search indexes, logs, and exports. Derived information can remain sensitive even when direct identifiers are removed. The technical inventory should support review by the relevant privacy and legal stakeholders rather than assuming that an integration purpose automatically authorizes every downstream use.

Minimize information at each boundary

Select only needed fields at extraction and avoid sending complete source records through every component for convenience. Apply field access controls and consider tokenization or pseudonymous references where the workflow permits them. These techniques can reduce exposure but do not automatically make information anonymous.

Review free text, error messages, tracing attributes, and filenames. They often carry sensitive values outside the carefully designed record schema. Keep diagnostic content proportional to the problem being investigated, use restricted access where detailed inspection is necessary, and ensure temporary debugging changes have an owner and an end condition.

LocationDesign questionPossible control
Source extractionWhich fields are necessary?Explicit field selection
Event streamDoes every consumer need this detail?Minimal payload and authorized lookup
Operational logsCould context expose a person or secret?Redaction and structured safe metadata
ExportsWho can retrieve the result and for how long?Authorized access and controlled lifecycle

Plan retention across derived stores

Define retention by purpose and applicable organizational requirements with accountable stakeholders. A database deletion may leave copies in search indexes, materialized views, caches, object storage, retained events, and backups. Record those paths and determine how each participates in deletion or restricted retention.

A restored backup or replayed event can reintroduce information removed from a live projection. Maintain a suppression or deletion reconciliation process where required, and test it during recovery exercises. Document when deletion reaches each store, including backups and retained events.

Verify controls through realistic workflows

Test ordinary access, privileged investigation, export creation, membership revocation, and deletion requests through the complete information path. Confirm that tenant and field restrictions remain effective in aggregates, search, caches, and downloaded files. Record exceptions with their owner and review date.

NIST's Privacy Framework helps teams organize privacy risks and controls. Record the controls adopted, their owners, validation results and unresolved issues. Legal and regulatory requirements need a separate review.

References