Return insufficient information when the corpus cannot answer
Someone asks whether a bicycle repair is reimbursable, but the corpus contains only airfare and laptop rules. The prototype invents a limit. Add an explicit abstention path.
- Focused work estimate
- 2h + prerequisites
- Priority in the scenario
- High
- Engineering practice
- Abstention · Product judgment · Evaluation design
Estimated field mix
- Applied AI100%
Field percentages are editorial estimates of the ticket's engineering focus. They total 100%; they are not measured time, proficiency scores, or ownership evidence.
Review it, then add it to your workspace.
The board opens an editable draft; nothing is saved until you confirm it. Sign-in and workspace permissions apply, and Demo boards remain ephemeral.
Project context
Employees search fictional travel and equipment policies. The current prototype blends draft and approved text and sometimes answers using a policy the employee cannot open. Use a small synthetic corpus and a deterministic answer-provider double.
Setup prerequisites
- Author synthetic policy documents with revisions, effective dates, and access groups.
- Use a local deterministic provider double; paid model access is optional and not required.
Preceding work
Complete these dependencies, or supply their agreed outputs before taking this ticket.
- SEARCH-101 · Import policy documents with stable revision identities
- SEARCH-102 · Keep chunk citations anchored to the original policy text
- SEARCH-103 · Apply group access before ranking or sending context to a provider
- SEARCH-104 · Exclude drafts and future policies from current-policy answers
- SEARCH-105 · Reject answers with invented or mismatched citations
Acceptance criteria
- Empty retrieval returns INSUFFICIENT_INFORMATION without calling the answer provider.
- A provider abstention is shown as missing support, without a fabricated policy rule.
- The response may include authorized source links but never presents retrieval rank as certainty.
Implementation constraints
- Use a documented conservative context-selection rule with inspectable thresholds.
Verification to include
- Ask an unsupported bicycle-repair question and confirm no rule or amount is invented.
- Ask an exact covered policy question and confirm authorized sources are retained in the supported fixture response.
Deliverables
- Abstention contract and covered/uncovered question fixtures
Rollout and recovery
Default uncertain cases to source search; tune thresholds only against a versioned evaluation set.
Value of the work
For the engineer: Practice access-aware retrieval, source provenance, structured model boundaries, and reproducible evaluation.
For the team: Inspect how an engineer prevents unsupported or unauthorized answers and handles changing policy content.
Evidence boundaries
Outcome Evidence: Tests, patches, and runbooks are requested deliverables. They become Outcome Evidence only through a qualified Mission and immutable Evidence IDs.
Ownership Evidence: Independent adaptation must be observed under a declared verification policy and cite immutable Evidence IDs. Completing a planning ticket establishes no Ownership Evidence.