Prevent a backfill from replacing a newer nightly partition
A noon backfill finishes after the nightly job. Its older snapshot overwrites May's newer partition and removes refunds received at 18:00.
- Focused work estimate
- 5h + prerequisites
- Priority in the scenario
- High
- Engineering practice
- Snapshot isolation · Batch concurrency · Atomic publication · Recovery
Estimated field mix
- Data engineering60%
- Distributed systems40%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional marketplace reports daily sales to merchants. Refund joins inflate revenue, merchants close books in different time zones, and backfills compete with nightly loads. All source orders and merchants are synthetic.
Setup prerequisites
- SQL joins and aggregates
- Batch pipelines
- Metric definitions
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- LAKE-102 · Profile source keys before trusting the order feed
- LAKE-101 · Write the daily net-sales contract with worked rows
- LAKE-104 · Assign sales to the merchant's reporting day
- LAKE-105 · Preserve historical merchant plans in revenue breakdowns
- LAKE-103 · Remove the order-line and refund join fan-out
- LAKE-106 · Pick up late refunds in an incremental load
Acceptance criteria
- Associate each generation with a source snapshot boundary.
- Reject cutover when another generation superseded the intended partition revision.
- Resume backfills without publishing incomplete partition sets.
Implementation constraints
- Document locking or compare-and-swap policy; finish time is not freshness.
Verification to include
- Overlap backfill and nightly load; prove the fresher state wins.
- Interrupt before publication and verify readers see a complete prior generation.
Deliverables
- Partition publication protocol and overlap reproduction
Rollout and recovery
Publish shadow generations first; retain the previous complete manifest for rollback.
Value of the work
For the engineer: Practice metric contracts, historical dimensions, incremental processing, and reproducible warehouse releases.
For the team: Inspect whether an engineer can reconcile business totals and explain metric changes in a reviewable data product.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.