An AI assistant can produce a confident answer from the wrong document, an outdated policy, or information the user should never have seen. A successful model response is not enough to prove that the retrieval workflow is ready for real work.
Retrieval-augmented generation (RAG) systems have at least two separate jobs: find useful context and turn that context into an answer. A practical QA plan tests both jobs, plus the permissions and freshness rules around them. Microsoft’s RAG evaluation guidance separates retrieval quality from response quality, including groundedness, relevance, and completeness. NIST’s generative AI profile also emphasizes documenting data provenance, data quality, limitations, and evaluation data as part of responsible system design.
This article presents a focused pre-launch test plan for an AI retrieval workflow. The examples are intentionally platform-neutral. The same principles apply whether the source is a document library, a knowledge base, a CRM, a reporting database, or an internal API.
Start with the answer contract
Before testing an assistant, write down what a good answer is allowed to do. Otherwise, reviewers tend to grade tone and fluency instead of operational correctness.
An answer contract should define:
- Scope: Which questions the assistant should answer and which it should decline.
- Sources: Which systems or collections are authoritative for each question type.
- Evidence: Whether the response must cite a document, record, date, or source link.
- Freshness: How old information may be before the answer must show a stale warning or refuse to answer.
- Access: Which user, team, or role may retrieve each source.
- Escalation: When a human must review an answer or take over the request.
For example, an operations assistant might be allowed to explain the current intake process from an approved procedure library. It might be required to show the procedure’s revision date. It might be prohibited from exposing individual customer records, even if those records appear in the same search index.
This contract gives QA reviewers something concrete to verify. It also prevents a common mistake: treating “the model answered” as the acceptance criterion.
Test retrieval before testing prose
The first failure point is often the search step. If the workflow retrieves irrelevant, incomplete, or unauthorized context, a polished answer can hide the problem.
Create a small test set of representative questions. Include common requests, ambiguous wording, synonyms, misspellings, recently changed topics, and questions that should return no answer. For each question, record the documents or records that should be retrieved and the ones that must be excluded.
Then evaluate the retrieval result separately from the final response:
- Coverage: Did the result include the source that contains the needed fact?
- Relevance: Are the top results actually about the question?
- Ranking: Is the most useful source early enough to influence the answer?
- Version: Did the search prefer the current approved version over an archived copy?
- Authorization: Did every returned item pass the user’s access rules?
Keep the evidence for each test. Store the question, user role, retrieved source IDs, source versions, filter decisions, and retrieval settings. This makes a later regression test possible when documents, chunking rules, indexes, or ranking parameters change.
Check groundedness, not just confidence
Groundedness asks whether the answer is supported by the context that was retrieved. It is different from confidence, grammar, or general plausibility.
Use test questions that make unsupported claims easy to spot. Ask about a policy exception, a date, a threshold, or a multi-step procedure. Review each material statement in the response and mark it as supported, contradicted, unsupported, or unclear. If the system shows citations, verify that each citation actually supports the nearby claim.
A grounded answer may still be incomplete. A response can accurately quote one paragraph while omitting an important exception in another section. That is why groundedness should be tested alongside completeness rather than used as a single pass/fail score.
Microsoft’s evaluation guidance describes groundedness as the precision side of the problem: the response should not introduce information that is absent from the grounding context. In practice, this means the assistant needs a safe behavior for missing evidence. “I could not verify that from the approved sources” is often a better result than a plausible guess.
Test completeness with known answers
Completeness asks whether the response covers the information needed to answer the question. It is a recall problem, and it requires a slightly different test design.
For each high-value question, define the minimum facts a reviewer expects. A customer-support answer might need the eligibility rule, the required documents, the time limit, and the escalation path. A reporting answer might need the metric definition, date range, source system, and known caveat.
Compare the generated answer with that expected-facts list. Do not require identical wording. Require the answer to contain the necessary meaning and to distinguish facts from recommendations or uncertainty.
Low completeness can point to several different causes:
- The correct source was never retrieved.
- The retrieved chunk omitted surrounding context.
- The search returned too many loosely related documents.
- The prompt did not tell the model to address every part of the question.
- The source itself is missing a required definition or exception.
Fix the cause at the right layer. Adding more prompt text will not repair a missing source document or an access filter that removes the needed record.
Make access tests adversarial
Access control should be tested as a retrieval property, not only as a user-interface feature. A hidden button does not protect data if the underlying search endpoint still returns restricted content.
Build paired tests with the same question asked by different roles. Include users who should see all relevant sources, users who should see only a subset, and users who should receive no answer. Also test direct requests for restricted information, indirect requests that try to infer it, and follow-up questions that refer to a document the user was not allowed to retrieve.
Record both the returned sources and the final answer. A system fails the test if it leaks a restricted fact even when it does not reveal the document title. Redact or tokenize sensitive test data, and keep test accounts separate from production identities.
Access rules also need an ownership plan. Decide who reviews permission changes, how revoked access is reflected in the index, and how long cached or embedded content may remain available after a source becomes restricted.
Put freshness into the acceptance criteria
A retrieval system can be technically correct and operationally wrong if it answers from stale material. “Last indexed” is not the same as “still authoritative.”
For each source collection, define the freshness signal that matters. It might be a document revision date, a record update timestamp, a successful synchronization time, or an explicit approval state. Display that context to the user when the answer depends on it.
Test the boundary deliberately. Change or retire a source record, run the indexing or synchronization process, and ask the same question before and after the update. Verify that the old answer is no longer returned, that the new source is preferred, and that a failed refresh produces a visible stale state instead of silent confidence.
For high-risk workflows, add a freshness refusal rule. If the source is older than the allowed threshold or the update status is unknown, the assistant should explain that it cannot verify the current answer and route the request to a human or a deterministic system.
Use a layered QA record
Keep one QA record per test case. A useful record includes:
- Question and expected intent
- User role and access context
- Source IDs, versions, and timestamps
- Retrieval configuration and returned ranking
- Generated answer and citations
- Groundedness, completeness, relevance, and access results
- Reviewer notes and severity
- Owner, fix, and retest date
Classify failures by layer: source content, synchronization, indexing, retrieval, authorization, prompt behavior, answer formatting, or human handoff. This prevents teams from repeatedly tuning the model when the real defect is a stale export or missing identifier.
Run the test set again after material changes. Changes to chunk size, embedding or search configuration, source permissions, document templates, model versions, and prompt instructions can all change the answer path. A small, durable regression set is more useful than a one-time demo.
Launch only the workflow you can explain
An AI retrieval system is ready for production when the team can explain where an answer came from, why the user was allowed to see it, how current the source was, what the answer omitted, and what happens when evidence is missing.
That standard does not require perfect automation. It requires visible boundaries, testable evidence, and a human path for uncertainty. DigitalWerks helps organizations connect websites, data stores, reporting systems, and operational workflows while keeping identifiers, permissions, freshness, and validation in view. Ask us to review an AI retrieval workflow before launch, including its source inventory, access model, test set, synchronization path, and escalation process.