Rehearse recovery of a queue with mixed expired and terminal jobs
After an outage, operators need to resume assignments without redoing completed work or accepting stale reviewer results.
- Focused work estimate
- 5h + prerequisites
- Priority in the scenario
- Medium
- Engineering practice
- Recovery · Data integrity
Estimated field mix
- Database engineering60%
- Site reliability40%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional inspection service assigns review jobs to staff. Polling clients sometimes claim the same job and a disconnected client can finish work after reassignment.
Setup prerequisites
- Create synthetic jobs and two independent database clients.
- Use explicit state methods and an injected clock.
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- ACLAIM-101 · Define assignment states and their required database fields
- ACLAIM-102 · Order eligible jobs with a stable tie break
- ACLAIM-103 · Scope work assignment queries to the operator's organization
- ACLAIM-104 · Claim one pending job atomically under competing pollers
- ACLAIM-105 · Issue a monotonically increasing assignment generation on every reclaim
- ACLAIM-106 · Renew assignment leases without extending a completed job
- ACLAIM-107 · Finish a job and publish its result reference in one transaction
- ACLAIM-108 · Reclaim expired assignments using current row authority
- ACLAIM-109 · Measure assignment lock behavior without claiming a throughput guarantee
Acceptance criteria
- Classify terminal, live and expired assignments from durable fields.
- Reclaim only expired authority and preserve terminal results.
- Report inconsistent rows instead of silently normalizing them.
Implementation constraints
- Use a fabricated outage dataset with one contradictory row.
Verification to include
- Recover valid mixed states and finish one reclaimed job.
- Submit a stale completion and inspect the contradictory row report.
Deliverables
- Queue recovery runbook and executable drill
Rollout and recovery
Run the drill before increasing concurrency; keep uncertain jobs quarantined for review.
Value of the work
For the engineer: Practice database concurrency, leases and transactional invariants.
For the team: Review whether assignment authority remains reliable during contention and recovery.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.