A useful first knowledge assistant answers one bounded class of questions. Start with a policy desk, a product manual or a team handbook. Decide who can ask, which documents are approved, and which answers require an owner. A convincing paragraph is not your acceptance criterion.
1. Make the evidence set manageable
Choose a small set of current documents you are authorized to use. Keep a document identifier, title, owner, revision date and access rule with each one. Remove superseded copies from the active collection without destroying the audit history. If two policies disagree, resolve the conflict with the policy owner before expecting the assistant to choose.
Retrieval-augmented generation combines generation with retrieved material. The original RAG paper is a useful primary reference for that architecture. Retrieval supplies context; your application still needs to check whether the answer is supported by that context.
2. Specify the answer contract
Write down the expected output before choosing a model: a short answer, the supporting passage and document reference, the applicable version, and an uncertainty or escalation reason when the evidence is incomplete. Ask the assistant to distinguish a quotation from an interpretation. A link that merely mentions the subject does not support every claim in an answer.
Worked example: the remote-work handbook
This is an illustrative exercise, not a customer result. Your approved handbook says employees must request remote work through their manager. A user asks, “Can I work abroad for three months?” The passage supports how to request remote work; it does not establish international eligibility, duration or immigration rules. The useful answer explains the supported request process and routes the unresolved eligibility question to the policy owner.
3. Build a question set before the demo
Create questions with known evidence, ambiguous wording, outdated evidence and no authorized evidence. Add a question that asks for a document the user cannot access. Record the expected supporting source and whether an answer or escalation is appropriate. Keep these cases out of demonstrations used to tune the assistant, or record that tuning explicitly.
4. Score the claim, not the writing style
- Was the relevant authorized passage retrieved?
- Does it support the answer’s actual claims?
- Are the source and version shown correctly?
- Did the assistant escalate when the evidence was missing?
- Could a user inspect the source without gaining extra access?
Review failures separately. Missing retrieval, unsupported interpretation and wrong permissions require different fixes. Keep a plain search or a human support desk available while you test. The decision to widen the scope should come from the observed errors and the cost of review, not from a polished demonstration.
A small next step
Pick one handbook, write ten questions and manually mark the source passages before building anything. That exercise reveals whether the documents can answer the proposed questions at all. Use synthetic or approved test material; do not upload confidential documents to a provider without authorization.