All insights

Would you trust this AI answer on a busy Monday?

Build a small evaluation set from real questions, score answers against evidence, and decide what your assistant can handle.

Precision measuring jaws hold a small violet glass sphere.

The practical takeawayGive the assistant a written test it can fail before giving people a tool they will trust.

  1. Choose real questions
  2. Write the scoring guide
  3. Repeat the test

A convincing demo is easy to arrange. You choose the question, know the answer, and can explain away an odd response. The real test comes the following Monday, when someone asks about an old event, phrases a question differently, or needs an exception the organizer has never approved. Before putting an assistant in front of that person, make those situations part of its rehearsal.

Start with the questions people actually ask

Ask the people who answer the phone to bring examples from an ordinary week. Remove names and personal details. For a nonprofit event assistant, examples might include whether two guests can share a ticket, how a table host changes a guest, and whether someone can register after the deadline. Preserve the awkward phrasing. A tidy question written by the project lead tests a different experience from a hurried question sent from a phone.

Sort the examples by consequence. Giving the wrong parking entrance is inconvenient. Inventing a refund policy makes a commitment on somebody else's behalf. Include at least one question whose answer is missing. In that case the useful response identifies the gap and points to the right person. Deciding what counts as a good handoff is part of designing the service.

Write the answer key before running the test

Use a current, approved document for every question that has a factual answer. Record the expected fact, where it came from, and any condition that changes it. Allow several good phrasings. A response can be shorter than your reference answer and still be completely correct. Conversely, a cheerful paragraph can contain the right date and the wrong instructions for taking action.

For the guest-change question, the reviewer might require three things. The answer must identify who can request the change, describe the published route, and avoid promising that a late request will be accepted. Have two people score the same handful of examples. Wherever they disagree, adjust the scoring guide or clarify the underlying policy. Sometimes the pilot exposes an ambiguous document the team has been interpreting differently for years.

Test what happens at the boundary

Keep some examples out of the initial development work. When you change the instructions or replace a source document, run those examples again. Include different phrasings of the same request, a question with too little information, and a question about last year's event. Record the assistant's exact output along with the source version and the reviewer's decision.

Separate mistakes that visitors can easily reverse from mistakes that need a hard stop. An outdated room description might prompt an editorial revision. A promise to issue a refund should keep that task with staff until the problem is resolved. An average score hides this distinction. Nine correct parking answers cannot make an invented policy acceptable.

Turn the findings into a release decision

A useful test report fits on a page. List the tasks the assistant can handle, the ones that need a human reviewer, and the examples it still gets wrong. Name someone who can withdraw an answer or update its source. If the answer depends on a calendar, include a check when the next event opens. A successful pilot in April does not establish that the same information will be current in September.

Give staff a simple way to flag a bad answer. Ask for the question, the response, and what they expected. Sensitive customer details can stay in the system where the case belongs. Add a sanitized version to the evaluation set. The team can now recognize that particular failure if it returns after another update.

A useful first exercise

Take one policy page and write three questions about it: one the page clearly answers, one that depends on a condition, and one the page leaves open. Give the same questions to a colleague and the assistant. Compare where each found the answer and what each did when the document ran out. The third question often tells you more about readiness than the first two. Keep the answers and the date. You have the beginning of a repeatable release test.

Sources & further reading