Expire raw tracking payloads while retaining minimal lineage
Carrier payloads contain recipient notes unnecessary after the exercise's 30-day debugging window. A cleanup script removes artifacts still used by an active replay.
- Focused work estimate
- 3h + prerequisites
- Priority in the scenario
- Medium
- Engineering practice
- Retention · Lease lifecycle · Data minimization
Estimated field mix
- Privacy engineering50%
- Data engineering30%
- Storage systems20%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional delivery marketplace receives JSON batches from three carriers. One uses local timestamps, another retries whole batches, and a third corrects delivery scans. Customer support needs a stable timeline rather than the last payload received.
Setup prerequisites
- JSON schema validation
- SQL queries
- Event-time concepts
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- PIPE-101 · Validate carrier events without discarding a whole batch
- PIPE-102 · Attach source lineage to every accepted scan
- PIPE-103 · Deduplicate carrier retries and surface changed duplicates
- PIPE-104 · Stop interpreting offset-free scans as server-local time
- PIPE-105 · Keep a late pickup scan from undoing delivered status
- PIPE-106 · Apply an explicit carrier correction without erasing the scan
- PIPE-107 · Replay a parser fix into a new projection generation
Acceptance criteria
- Delete eligible raw artifacts after configured retention.
- Protect artifacts referenced by an active authorized replay lease.
- Retain minimal hashes, deletion receipts, and normalized non-sensitive facts.
Implementation constraints
- Thirty days is a fictional exercise policy; expired leases cannot pin data forever.
Verification to include
- Expire an eligible artifact and verify its deletion record.
- Protect an active replay artifact, then expire the lease and delete it.
Deliverables
- Retention job and replay-lease cleanup cases
Rollout and recovery
Review a dry-run deletion manifest before enabling deletion on synthetic artifacts.
Value of the work
For the engineer: Practice ingestion contracts, deduplication, event-time reconciliation, and replayable data repair.
For the team: Examine how an engineer preserves source lineage and handles bad partner data without losing valid business events.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.