Drain long requests before retiring an API instance
During release, an instance exits halfway through a report download. The load balancer keeps sending new requests during its shutdown grace period.
- Focused work estimate
- 2h 30m + prerequisites
- Priority in the scenario
- High
- Engineering practice
- Graceful shutdown · HTTP lifecycle · Timeouts
Estimated field mix
- Platform engineering50%
- Networking30%
- Site reliability20%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional scheduling service has an API, a worker, and PostgreSQL. Releases use immutable images on two application instances. The team needs compatibility checks, staged traffic, and a rehearsed rollback without introducing a new orchestration platform.
Setup prerequisites
- HTTP health checks
- CI pipelines
- Database migrations
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
Acceptance criteria
- Mark the instance unready before starting its bounded drain period.
- Allow existing requests to complete until the documented deadline.
- Close remaining connections and report forced terminations at deadline without hanging shutdown.
Implementation constraints
- Do not count idle keep-alive connections as unfinished business requests indefinitely.
Verification to include
- Start a long synthetic request, initiate shutdown, and observe completion.
- Keep a request stuck beyond the deadline and confirm bounded exit and termination count.
Deliverables
- Graceful shutdown behavior and request-drain reproduction
Rollout and recovery
Exercise draining on one test instance; restore the previous grace settings if traffic does not stop arriving.
Value of the work
For the engineer: Practice release contracts, compatibility windows, measurable canaries, and incident decisions on a modest application topology.
For the team: Review whether an engineer can make deployments diagnosable and reversible while identifying when rollback is unsafe.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.