← Back to prototypes
Case study

AI Knowledge Answer Reviewer

A trustworthy question-and-answer workflow that makes source material, uncertainty, feedback, and escalation visible.

The problem

Operations teams need faster access to procedure knowledge, but unexplained AI answers create trust and compliance risks. A plain chatbot response gives no way to verify the source, report a gap, or escalate an edge case.

Target user

Operations staff and customer-service staff who need a reliable answer quickly. Secondary user: the knowledge owner who maintains the source material and resolves gaps.

Product goal

Give users a trustworthy question-and-answer workflow that makes source material, uncertainty, feedback, and escalation visible.

User flow

  1. 1Select an operational area (Claims, Policy Servicing, Customer Onboarding).
  2. 2Ask a question, or choose a suggested question.
  3. 3Read a structured answer: answer, steps, exceptions, and sources.
  4. 4Open the source citation cards to verify the guidance.
  5. 5Rate the answer as Useful, Incomplete, or Inaccurate.
  6. 6If incomplete, create an escalation request with the question and context prefilled.
  7. 7A knowledge owner reviews feedback trends and resolves unresolved questions.

Design decisions

  • Structure beats prose — splitting the answer into answer/steps/exceptions makes it scannable and reduces the risk of an overconfident wall of text.
  • Sources are first-class — every answer shows its supporting sources with review dates, so trust is earned rather than asserted.
  • Uncertainty is explicit — the "Review Recommended" label appears when the question is not clearly covered, instead of inventing an answer.
  • Feedback is a product feature, not a footnote — Useful/Incomplete/Inaccurate feeds the owner dashboard and unlocks escalation.

AI interaction design

Input is KnowledgeAnswerService.answerQuestion(question, category). Output is a structured AiAnswer with steps, exceptions, sources, and a confidence label. The human review loop is: verify sources → rate → escalate if incomplete. Escalations are routed to a knowledge owner, closing the feedback loop.

Technical approach

Next.js App Router with a React client component for the ask flow and a separate owner dashboard route. TypeScript types for KnowledgeSource, AiAnswer, Feedback, and EscalationRequest. Atomic component structure with reusable atoms and feature-scoped components. Mock adapter with simulated latency and an explicit unavailable state. Feedback and escalations are held in lightweight in-memory stores so the owner dashboard reflects real interactions during a session.

Tradeoffs

  • No real retrieval layer: the mock service matches on keywords against fictional documents.
  • Feedback and escalations are session-scoped rather than persisted.
  • The owner dashboard uses a small seed dataset to make trends legible in a demo.

Success measures

  • Answer usefulness rating
  • Escalation rate and time to resolution
  • Time to find guidance
  • Unresolved-question volume
  • Source coverage

Next steps

Connect a real retrieval system with citations and evaluation, add authentication and role-based access, persist feedback and escalations, and measure answer quality against a labelled question set.