Profile source keys before trusting the order feed
The warehouse assumes order_id is globally unique, but two merchants both supplied order 7001. Daily deduplication keeps only one.
- Focused work estimate
- 1h 30m + prerequisites
- Priority in the scenario
- Medium
- Engineering practice
- Data profiling · Composite keys · SQL
Estimated field mix
- Data engineering70%
- Database engineering30%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional marketplace reports daily sales to merchants. Refund joins inflate revenue, merchants close books in different time zones, and backfills compete with nightly loads. All source orders and merchants are synthetic.
Setup prerequisites
- SQL joins and aggregates
- Batch pipelines
- Metric definitions
Preceding work
No earlier ticket is required. Complete the project setup above.
Acceptance criteria
- Report duplicates at global and merchant-scoped order grain.
- Count missing merchant IDs, order IDs, currencies, and update timestamps.
- Produce bounded sample references without customer details.
Implementation constraints
- Profiling reports anomalies and never deletes source rows.
Verification to include
- Detect intentionally duplicated merchant/order pairs.
- Keep same-ID orders from different merchants distinct and report missing keys.
Deliverables
- Read-only profiling SQL and annotated output
Rollout and recovery
Profile before changing ingestion; retain only minimized aggregate reports.
Value of the work
For the engineer: Practice metric contracts, historical dimensions, incremental processing, and reproducible warehouse releases.
For the team: Inspect whether an engineer can reconcile business totals and explain metric changes in a reviewable data product.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.