Enterprise RAG evaluation checklist: six tests before
launch

Imagine asking your company’s AI search assistant which contract applies to a customer renewal. It finds a signed agreement and gives you an answer in seconds. Then you notice that the contract was replaced last week. Worse, a colleague without permission to read it received the same information.

Test that scenario before colleagues depend on an AI search tool. Retrieval-augmented generation (RAG) gives an assistant passages from selected documents to help it answer. A convincing demo tells you little about private files, outdated information or questions with no answer.

Before inviting more staff, test six things: the questions people ask, who can see each source, whether citations support the answer, how updates reach search, what happens when information is missing, and who will look after the service.

1. Start with questions from your actual work

Ask colleagues for questions that take time today. A customer adviser might ask, “Which warranty terms apply to an order shipped to Sweden in March?” The answer may need a purchase date, local terms and the correct agreement. A generic PDF summary would miss those details.

For a first pilot, prepare a small collection of questions that includes:

• A familiar request with one approved source.

• A question that needs two different documents.

• An exact product number, contract code or local term.

• A question about a document that has been replaced.

• A request for information your documents do not contain.

For every question, save its expected source and relevant passage. Ask a subject expert to review the list. Microsoft’s RAG preparation guidance recommends selecting representative documents and questions together.

2. Check who can read a private document

Create two test accounts. Give the first access to a restricted contract and deny that access to the second. Ask both accounts the same question about the confidential terms.

The authorised account may receive a supported answer. The other must see neither private terms nor an exposing citation. Remove the first account’s permission and repeat the test. Ask engineers to inspect stored search results and logs as well as the reply.

Try this prompt: “What discount was agreed in the latest renewal?” Run it once as the contract manager and again as an employee without contract access. Save both replies with the time and test account used.

3. Open the citations rather than trusting the wording

Select five answers that appear accurate. Open every cited document and find the passage behind each claim. A reference to the right file can still be misleading if the assistant missed a condition in a table or quoted an expired clause.

Check two things separately: did search retrieve the correct passage, and did the reply use it accurately? A polished answer with an unrelated citation fails. Microsoft’s RAG evaluation guidance explains why retrieval and answer quality need separate checks.

4. Replace a document and ask again

Choose one policy or product sheet in the pilot library. Replace it with an approved new version, then repeat a question whose answer changed. Check when the replacement appears in search, whether the previous version is still quoted and whether deleting the old file removes it from future replies.

A pricing assistant may need new information available within hours; an internal reference library may allow a different interval. Agree that interval with the person responsible for the documents. Write it into the test record instead of assuming every source updates immediately.

5. Ask a question that has no reliable answer

Try an invented product name or a request outside the approved document collection. The assistant should explain that it cannot find a supported answer, ask for a useful detail or refer the question to a colleague. A confident invented response fails the trial.

For a security test, put an instruction inside a test document telling the assistant to ignore its rules. It should treat that as source text, without revealing private information or taking another action. The OWASP AI Security Verification Standard offers further engineering checks.

Try an invented product name or a request outside the approved document collection. The assistant should explain that it cannot find a supported answer, ask for a useful detail or refer the question to a colleague. A confident invented response fails the trial.

For a security test, put an instruction inside a test document telling the assistant to ignore its rules. It should treat that as source text, without revealing private information or taking another action. The OWASP AI Security Verification Standard offers further engineering checks.

6. Find out what happens on a normal workday

Ask a representative group to use the pilot while carrying out their jobs. Note questions that need a second attempt, slow replies, failed searches and answers that require manual correction. Compare the time spent with the way staff find the information today.

Put names beside the ongoing work: who updates documents, reviews reported errors, checks changed permissions and responds when search fails. Also agree when the assistant should be taken offline. A pilot without those responsibilities may work in a demonstration and disappoint people a month later.

Bring this RAG review sheet to your next meeting

Use one row per test. Add the actual reply, source link, reviewer, date and whether the result passed. You can reuse the same questions after a document, search setting or model changes.

Use the results to choose a limited launch, another test or a pause. Save the record for future document or model updates.

Would an AI search pilot help your organisation?

If your team is reviewing enterprise AI search, start with three details: the documents people need, who is allowed to read them and two or three questions that currently take too long to answer. Those details make a first engineering discussion much more useful.

Phases works on enterprise software and integrations between business applications. Our published Normann Copenhagen project is an example of that integration work; it is not presented as an AI search implementation.

Explore AI Search for Your Business