Measure oldest runnable work instead of queue size alone
The queue contains only ten jobs, but the oldest eligible preview has waited 40 minutes. A queue-depth alert never fires, while scheduled retries inflate another dashboard.
- Focused work estimate
- 2h + prerequisites
- Priority in the scenario
- Medium
- Engineering practice
- Queue observability · Metric semantics · Alert design
Estimated field mix
- Site reliability70%
- Platform engineering30%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional internal document portal creates previews through a trusted mock renderer. Large batches crowd out small teams, failed documents retry forever, and operators lack a safe replay command. This exercise never executes uploaded code or real document macros.
Setup prerequisites
- Queue semantics
- Database transactions
- Operational metrics
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- QUEUE-101 · Expose a document job's current phase and terminal reason
- QUEUE-102 · Reject oversized document batches before accepting work
- QUEUE-103 · Commit job acceptance and dispatch intent together
- QUEUE-104 · Stop retrying unsupported document formats
- QUEUE-105 · Keep one team's bulk import from occupying every worker
Acceptance criteria
- Measure oldest eligible-job age separately from delayed retries and running age.
- Expose attempt exhaustion and stale-lease counts with bounded labels.
- Alert when eligible wait exceeds the fixture's ten-minute budget for two observations.
Implementation constraints
- Exclude document IDs and tenant IDs from metric label sets.
Verification to include
- Advance a controlled clock and trigger runnable-age paging.
- Keep a future retry delayed and verify it does not inflate eligible wait.
Deliverables
- Queue age metrics and diagnosis-oriented alert
Rollout and recovery
Compare alerts with the synthetic workload trace in report-only mode before paging.
Value of the work
For the engineer: Practice asynchronous lifecycle control, fair scheduling, bounded retries, and artifact recovery with a deterministic provider.
For the team: See how an engineer accounts for accepted work and reduces operational toil without granting operators broad data access.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.