Fall back predictably when the shared cache is unavailable
A Redis outage makes every request retry repeatedly, overwhelming the authoritative database while users wait.
- Focused work estimate
- 3h 30m + prerequisites
- Priority in the scenario
- Medium
- Engineering practice
- Failure containment · Admission control
Estimated field mix
- Distributed systems40%
- Site reliability30%
- Performance engineering30%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
A fictional wholesale portal caches product availability descriptions across several API instances. Delayed invalidations and slow refreshes bring back old content after edits.
Setup prerequisites
- Create two local cache clients and a synthetic authoritative product store.
- Use a fake clock and controllable read/write barriers.
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- ACACHE-101 · Define cache keys from every product-view authority input
- ACACHE-102 · Specify what stale product data the portal may display
- ACACHE-103 · Record authoritative product revision with each cached projection
- ACACHE-104 · Commit product changes with a durable invalidation fact
- ACACHE-105 · Prevent a slow cache fill from overwriting a newer product revision
- ACACHE-106 · Handle delayed invalidation without evicting a newer cache revision unnecessarily
- ACACHE-107 · Bound duplicate refresh work across API instances
Acceptance criteria
- Bound cache attempts and stop retry amplification.
- Apply an explicit source-read admission limit.
- Return the declared degraded or unavailable response when both paths lack capacity.
Implementation constraints
- Use local adapter failures and synthetic requests.
Verification to include
- Disconnect the cache and serve an admitted authoritative read.
- Exceed fallback admission and return a bounded failure without unbounded source calls.
Deliverables
- Cache-outage fallback and overload probe
Rollout and recovery
Canary with conservative fallback limits; shed excess reads until cache recovery is verified.
Value of the work
For the engineer: Practice cache consistency, version guards and recoverable invalidation.
For the team: Review fast derived reads without losing authoritative state or tenant isolation.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.