noCV
QUEUE-105 · Handle failures deliberately

Keep one team's bulk import from occupying every worker

Practice briefStoryAdvanced

A synthetic tenant uploads 10,000 documents. A second tenant's single preview waits behind the entire batch although the renderer has spare concurrency slots between completions.

Focused work estimate
4h + prerequisites
Priority in the scenario
High
Engineering practice
Fair scheduling · Concurrency limits · Capacity accounting

Estimated field mix

  • Platform engineering50%
  • Performance engineering30%
  • Distributed systems20%

Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.

Your next step

Review it, then add it to your workspace.

The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.

Project context

A fictional internal document portal creates previews through a trusted mock renderer. Large batches crowd out small teams, failed documents retry forever, and operators lack a safe replay command. This exercise never executes uploaded code or real document macros.

Setup prerequisites

  • Queue semantics
  • Database transactions
  • Operational metrics

Preceding work

Complete these dependencies, or supply their agreed outputs before taking this ticket.

Acceptance criteria

  • Enforce configured global and per-tenant active-job limits.
  • Allow eligible tenants to make progress while a bulk tenant has backlog.
  • Release capacity after success, terminal failure, cancellation, or expired execution lease.

Implementation constraints

  • Do not create an unbounded queue or metric family per tenant; explain the fairness policy.

Verification to include

  • Run one bulk tenant and two small tenants and measure their wait times.
  • Crash a worker while holding capacity and prove another eligible tenant eventually progresses.

Deliverables

  • Fair dispatch policy and controlled-load report

Rollout and recovery

Start with conservative synthetic limits; disable new intake and drain leases if accounting diverges.

Value of the work

For the engineer: Practice asynchronous lifecycle control, fair scheduling, bounded retries, and artifact recovery with a deterministic provider.

For the team: See how an engineer accounts for accepted work and reduces operational toil without granting operators broad data access.

Evidence boundaries

Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.

Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.