Throttle ingestion without losing accepted batches
After an outage, one carrier sends a day's events in 30 seconds. Memory pressure restarts workers and the API cannot tell which batches were accepted.
- Focused work estimate
- 3h 30m + prerequisites
- Priority in the scenario
- High
- Engineering practice
- Backpressure · Durable acceptance · Capacity measurement
Estimated field mix
- Data engineering50%
- Performance engineering30%
- Distributed systems20%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional delivery marketplace receives JSON batches from three carriers. One uses local timestamps, another retries whole batches, and a third corrects delivery scans. Customer support needs a stable timeline rather than the last payload received.
Setup prerequisites
- JSON schema validation
- SQL queries
- Event-time concepts
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
Acceptance criteria
- Bound request size and active ingest concurrency.
- Acknowledge only after durable source and dispatch intent exist.
- Return retryable overload before accepting work above the backlog limit.
Implementation constraints
- Distinguish rejected-before-acceptance from delayed-after-acceptance responses.
Verification to include
- Burst synthetic batches and account for every acknowledged batch.
- Fail storage and saturate the queue; assert no false acceptance.
Deliverables
- Backpressure limits and accepted-batch accounting test
Rollout and recovery
Start with conservative limits; reduce intake while draining accepted work if lag grows.
Value of the work
For the engineer: Practice ingestion contracts, deduplication, event-time reconciliation, and replayable data repair.
For the team: Examine how an engineer preserves source lineage and handles bad partner data without losing valid business events.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.