Pick up late refunds in an incremental load
The nightly job filters on order creation date. A refund posted today against a six-week-old order never reaches the mart.
- Focused work estimate
- 3h 30m + prerequisites
- Priority in the scenario
- High
- Engineering practice
- Incremental processing · Watermarks · Idempotent loads
Estimated field mix
- Data engineering70%
- Database engineering30%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional marketplace reports daily sales to merchants. Refund joins inflate revenue, merchants close books in different time zones, and backfills compete with nightly loads. All source orders and merchants are synthetic.
Setup prerequisites
- SQL joins and aggregates
- Batch pipelines
- Metric definitions
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
Acceptance criteria
- Select changed keys using source change positions or update times.
- Recompute affected reporting partitions beyond today's orders.
- Advance checkpoints only after selected changes commit.
Implementation constraints
- Document watermark precision and ties; overlapping reads must be idempotent.
Verification to include
- Apply a late refund and reconcile its affected date.
- Fail mid-load and replay equal-timestamp updates without loss or duplication.
Deliverables
- Incremental model and checkpoint recovery tests
Rollout and recovery
Compare incremental and full builds on identical synthetic changes before switching schedules.
Value of the work
For the engineer: Practice metric contracts, historical dimensions, incremental processing, and reproducible warehouse releases.
For the team: Inspect whether an engineer can reconcile business totals and explain metric changes in a reviewable data product.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.