Schema evolution and compatibility

Schema evolution is a coordination problem across deployed software and retained data. Evaluate both the shape of a change and the meaning consumers attach to it before choosing a rollout sequence.

Test compatibility against the reader and writer versions in use.

Adding an optional field

Example invoice messages with a new note field.

Writer v1

{
  "id": "inv_1042",
  "currency": "GBP",
  "amountMinor": 125000
}

Writer v2

{
  "id": "inv_1042",
  "currency": "GBP",
  "amountMinor": 125000,
  "note": "Review requested"
}

Old reader permits unknown fields

It accepts this addition under its configured schema and ignores note. The existing identifier, currency and amount keep their meanings.

Old reader rejects unknown fields

It rejects note because its closed object schema does not list that property. Deploy a compatible reader or an agreed migration before relying on this message.

A v2 reader can also read a v1 message if note is optional and the original fields remain compatible.

Define compatibility from the reader's perspective

Test compatibility in both directions: a new reader consuming old messages, and an old reader consuming new messages. Record the reader and writer versions in each test.

Adding an optional field can be safe for a tolerant reader and unsafe for a strict one. Adding an enum member can break exhaustive branching. Renaming a field or changing a unit usually requires coordination even when the physical type remains unchanged. Test with the actual serialization format and configured validators.

Separate semantic changes from structural changes

Changing amount from minor units to major units while retaining a numeric type is a severe semantic change. Similarly, broadening approved to include conditional approval changes business interpretation even if the schema is identical. Version and review these changes with the same care as a removed required field.

Document examples showing how the old and new definitions differ. Identify reports, workflows, and external consumers that depend on the prior meaning. The schema version, model version, and deployment version may move independently; a traceable mapping helps operations understand which combination produced a particular result.

ChangePotential hazardReview evidence
Optional field addedStrict readers reject itOld reader with new payload
Enum value addedUnhandled branchUnknown value behavior
Unit changedPlausible incorrect calculationSemantic reconciliation
Required field removedReader assumptions failMigration and rollback plan
Changing the unit of an amount
"amount": 125000
Old agreement · GBP minor units
£1,250.00
Proposed agreement · GBP major units
£125,000.00

Changing the unit changes the amount represented, even though the JSON type stays the same. Producers and consumers need a coordinated migration.

Use an expand, migrate, contract sequence

First expand the interface so old and new representations can coexist. Then migrate consumers and backfill data where appropriate. Only remove the old representation after usage evidence shows the migration is complete and the agreed compatibility period has ended. Avoid forcing every consumer to deploy at the same instant.

For database changes, examine locks, backfill load, index creation behavior and application compatibility. Dual writes require reconciliation because one write can succeed while the other fails. Decide which store is authoritative and how a failed write will be repaired.

Test historical replay and rollback

A consumer that processes today's events may still fail when replaying months of retained data. Keep historical fixtures and run them through the new implementation. Preserve transformations or readers needed for old versions, and define how invalid historical records are quarantined without stopping the entire recovery process.

A code rollback does not automatically reverse a data migration. State which changes are reversible, which require a forward repair, and how affected records can be identified. Confirm that telemetry reports version distribution and compatibility failures so the rollout can stop on evidence before a subtle semantic mismatch spreads downstream.

References