Keep stalled stock calls from consuming every refund execution slot
Stock reservations and refunds share a 12-slot provider pool. Twelve hanging stock requests prevent an otherwise healthy refund provider from receiving work, and the pending promise list grows without a limit.
- Focused work estimate
- 3h + prerequisites
- Priority in the scenario
- High
- Engineering practice
- Concurrency limits · Backpressure · Cancellation
Estimated field mix
- Site reliability40%
- Distributed systems40%
- Performance engineering20%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Pattern topics
- BulkheadApply
Separate provider execution capacity so a stalled stock dependency cannot consume refund slots, with bounded queues and explicit permit cleanup.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional equipment retailer lets customers choose a refund or replacement after inspection. Its payment and stock providers can time out after accepting an operation, so retrying the whole return is unsafe. Create a local TypeScript application, PostgreSQL state/outbox tables, and synthetic provider doubles with controllable outcomes; no starter code or fixtures are supplied. Use invented orders and integer minor-unit amounts only, with no real payments or external provider calls.
Setup prerequisites
- SQL transactions
- Idempotent commands
- Async failure handling
- State modeling
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- PRECOVER-101 · Reject return decisions that skip inspection or reverse a completed refund
- PRECOVER-102 · Bind a repeated refund request to its original merchant and intent
- PRECOVER-103 · Commit approved return work and its dispatch record together
- PRECOVER-104 · Persist replacement progress across stock reservation and shipment creation
- PRECOVER-105 · Open the refund circuit without declaring timed-out refunds failed
Acceptance criteria
- Give stock and refund operations independent concurrency limits and bounded waiting queues with explicit admission outcomes.
- Timeout, cancellation, and provider rejection each release a slot exactly once while preserving durable work for the declared retry policy.
- When stock capacity is saturated, refund work continues within its configured limit and neither queue exceeds its cap.
Implementation constraints
- Document what is isolated by the pools and what remains shared, including database capacity; do not claim full fault isolation from separate counters alone.
Verification to include
- Hold all stock operations behind a barrier and complete a refund batch without exceeding either provider's concurrency limit.
- Cancel queued work, time out running calls, and overfill both queues; assert no leaked permits or unbounded pending promises.
Deliverables
- Provider capacity limits, admission contract, and saturation reproduction
Rollout and recovery
Start with conservative synthetic pool limits and expose queue depth/rejection counts; pause intake before lowering limits below active work.
Value of the work
For the engineer: Practice durable workflow state, ambiguous provider outcomes, compensation, concurrency isolation, and recovery that survives process restarts.
For the team: Inspect whether a proposed workflow preserves refund and inventory invariants, contains provider failures, and gives operators a bounded recovery path.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.