A RAG assistant that can say "I don't know"

An assistant that answers questions from your company's documents is easy to build. The usual recipe is retrieval-augmented generation (RAG): connect an embedding model to a chat model, load the handbook, ask "how many days of parental leave do we offer", and a clean paragraph comes back. The demo always works.

The trouble starts later. Imagine a support agent asking the same assistant about the refund window. It answers "60 days" with full confidence; the written policy says 30. The refund goes out, and from then on nobody fully trusts anything the tool says. An assistant that is right 92% of the time and says "I don't know" the rest of the time is more useful than one that is right 92% of the time and confidently wrong the rest, because in the second case nobody can tell which answers are the wrong ones.

This article is about the second half of the work: making the assistant show its sources, refuse when it has none, respect who may see what, and stay up to date.

Split documents by their structure, not by token count

The usual first version cuts every document into fixed windows of a few hundred tokens because a tutorial did. That slices tables in half, separates a heading from the paragraph it governs, and glues the end of one section to the start of the next. The retriever then finds half a rule, and the model fills in the other half.

Split on the structure the author already gave the document: headings in Markdown or HTML, sections in a PDF outline, numbered clauses in a contract. Each chunk should hold one idea and carry its document title and section path as a prefix, so that "30 days" arrives together with "Refund policy › Standard orders". Keep tables whole, and store a short plain-language summary next to each one, because embedding models handle prose much better than grids.

Make retrieval visible

If you cannot see what the retriever found, you cannot fix a wrong answer; you can only reword the prompt and hope. So every answer should be traceable to the chunks behind it: the source document, the section, the similarity score and the exact text the model received.

With that trace, a wrong answer falls into one of three groups, and each needs a different fix:

the right passage was never retrieved: a chunking or search problem;
it was retrieved and the model ignored it: a prompting problem;
it was retrieved and it is itself wrong or out of date: a content problem, and no model will fix that.

Pure vector search is not enough on its own. It misses exact identifiers, error codes and product names that people type word for word. Hybrid search, which combines keyword and vector results, catches those, and a reranker then picks the best five or six passages out of the twenty or so candidates before anything goes to the model.

Citations, and "I don't know" as a normal answer

Ask the model to attach, to each statement, the passage that supports it, and have your application check those citations before showing anything. With structured output the answer might look like this:

{
  "answer": "Refunds are available within 30 days of purchase.",
  "citations": ["policy-refunds#standard-orders"]
}

If citations is empty, or points to a passage that was not retrieved for this question, the application discards the answer and shows an honest fallback instead: "I couldn't find this in the documents." The same happens when retrieval finds nothing above a relevance threshold. That fallback has to be a normal, expected result, not a failure the model tries to avoid.

If you use Claude, its Citations feature does part of this bookkeeping for you. You pass the documents in, and each citation in the answer comes back with the exact quoted text, guaranteed to point into the documents you sent. As of September 2026 it cannot be combined with structured outputs in the same request, so you choose one or the other.

A valid citation proves that the passage exists. It does not prove that the passage supports the sentence next to it. Checking that is the job of the evaluation further down.

Respect who can see what

An internal assistant reads everything, so it can leak everything. If salary bands, unreleased plans and legal memos sit in the same index, a well-phrased question can pull out a document the person asking was never allowed to open.

Filter at retrieval time. Every chunk carries the access rules of its source document, and each search runs only over what the current user is allowed to read. Never retrieve a restricted passage and then ask the model not to use it: the model must never see what the user cannot. That means your ingestion has to copy document permissions and keep them in sync, which is real work, and it is the main difference between an internal assistant and a public FAQ bot. The broader rules are in A security checklist for AI features.

Measure before you trust it

"It looked good in the demo" is not a measurement. Before real users get it, build a test set: 100 to 150 real questions that people actually ask, each paired with the document that should answer it. Add questions the documents cannot answer, because the right response to those is "I don't know", and you want to know how often the assistant bluffs instead.

The set gives you three numbers:

Retrieval recall: how often the right passage appears among the retrieved ones. If it is missing, the model never had a chance.
Faithfulness: how often the answer is fully supported by the passages it cites, with nothing added. A second model can grade this, as long as you spot-check its grades by hand.
Abstention: how often the assistant says "I don't know" to the questions it should not answer.

Rerun the set after every meaningful change to chunking, search, prompts or model, the same way you run tests. It is the same idea as the fifty examples in Choosing a model: price, speed and quality, applied to retrieval.

Keep the index current

Documents change, and an assistant answering from last quarter's policy is worse than none. Re-embedding everything every night gets slow and wasteful once you pass a few thousand documents. Index incrementally instead: keep a content hash per document, and when a document changes, re-chunk and re-embed only that one. When a document is deleted, delete its chunks too. Leftover chunks from removed documents are a quiet source of confident wrong answers.

Store a last-updated date with every chunk and show it next to the citation. "Source updated three days ago" and "source updated in 2023" lead people to trust the answer differently, and they should.

What it costs

The model bill is usually the smallest part. Take a team that asks 2,500 questions a week. Each question sends about 3,000 input tokens (six passages plus instructions, as Claude Sonnet 5 counts them) and gets about 300 tokens back. At Anthropic's prices on 25 September 2026, $2 per million input tokens and $10 per million output tokens for Sonnet 5, that is about $0.009 a question: roughly $22 a week, or just under $100 a month. Embedding the questions costs next to nothing; embedding the documents is a one-off cost that depends on how many you have. Choosing the model itself is covered in the article linked above.

The real costs are elsewhere: building and rerunning the test set, the ingestion and permission plumbing, and keeping the documents themselves in order. The assistant is only as good as what it reads. Point it at a wiki full of contradictions and out-of-date pages, and you get fluent, well-cited wrong answers.

In short: split by structure, make retrieval visible, require citations and accept "I don't know", filter by permissions before the model sees anything, and measure with your own questions before and after every change.