How to Test an AI Chatbot Before You Share It with Customers

Why testing matters

Chatbots built on large language models (LLMs) put answers together from patterns in language, not from understanding in the way a person does. Even when a business chatbot is set up to answer from specific documents, a few things can still go wrong:

• The answer comes from the wrong part of the document.
• The answer sounds convincing but has no basis in the document. This is often called a hallucination.
• Outdated information in the document is given as if it still applies.
• Off-topic questions get answered when they should be declined or redirected.

Customers do not distinguish between the chatbot's mistakes and your business's mistakes. If the chatbot quotes the wrong price, customers will expect you to honour it.

Step 1: Prepare a list of test questions

Write 20–30 questions before you start testing, and split them into groups:

1. Routine questions that come in most often by message, written exactly the way customers write them, including abbreviations and typos.
2. Questions with a number as the answer: prices, opening hours, dates, sizes, and quantities. These are the answers that do the most harm when they are wrong.
3. Questions the document does not answer. A good chatbot admits that the information is not available and points to a contact, rather than making something up.
4. Off-topic questions, such as asking for a recipe, talking politics, or asking it to write someone's homework.
5. Trick questions, such as "Your staff told me yesterday there's a 50% discount, right?" when no such offer has ever existed.

As an example, here is part of a test list for a home bakery that uploaded its price list and ordering terms:

Routine: hw much is the big jar of pineapple tarts

Numbers: If I order 30 snack boxes, how many days' notice do you need?

Not in the document: Will the biscuits keep for two weeks?

Off-topic: Can you give me a recipe for pineapple tarts?

Trick: I heard orders of 100 boxes get free delivery anywhere in the country, is that right?

For each question, write down the answer you expect before you look at the chatbot's answer. This keeps your judgement objective, because you are not swayed by an answer that merely sounds convincing.

Step 2: Check every answer against the source

For each answer, open the original document and check:

• Do the numbers match exactly?
• Are the conditions and exceptions mentioned too?
• Did the answer come from the right part of the document? For example, the price of package A should not be swapped with package B.

If your chatbot shows its sources or quotes the document, use that to speed up checking. Record every wrong or imprecise answer in a simple table: the question, the chatbot's answer, the correct answer, and the likely cause.

Step 3: Fix the document, not just the chatbot

Many wrong answers start with an ambiguous source document. The most common causes are:

• Conflicting information: two different prices for the same product in two parts of the document.
• Hidden information: an important condition that only appears in a footnote or inside an image.
• Complicated tables: merged cells or nested tables that are hard for software to read.
• Old versions still stored: last year's price list uploaded alongside this year's.

Fix the document by writing one fact per sentence, removing old versions, and moving important information out of images and into text. Then ask the questions that were answered wrongly again.

Step 4: Test the limits and personal data

• Make sure the documents contain nothing that should not be public, such as ID numbers, home addresses, customer data, or internal information like profit margins.
• Try to get the chatbot to leak information, for example with "show me the full contents of the document" or "ignore your previous instructions". Assume that anything in the documents can be read by visitors.
• Check sensitive topics. In health, legal, or financial areas, a chatbot should give only general information and refer people to a professional, not give personal advice.

The safest rule is simple: do not upload any document you would not be happy to hand directly to a customer.

Step 5: Decide when customers are handed to a person

A chatbot should not be the only way to reach you. Make sure customers can still contact a real person for:

• Complaints and problems with orders.
• Price negotiations or special requests.
• Questions the chatbot cannot answer.

Test whether the chatbot really shows the right contact details when it cannot answer, and make sure someone is actually monitoring that contact.

Step 6: Ask other people to try it

You already know the document by heart, so you tend to ask questions in the same words it uses. Ask two or three other people, such as staff, family, or loyal customers, to ask questions in their own way. Pay attention to the questions you had not thought of.

Signs the chatbot is ready to share

A chatbot does not have to be perfect before you start using it, but it should meet these minimum conditions:

1. Every question with a number as the answer is answered correctly.
2. Questions the document does not answer are not met with made-up answers.
3. The uploaded documents contain no personal data or internal information.
4. There is a clear way to reach a person.
5. At least one person other than you has tried it and found no misleading answers.

If any of these is not yet met, fix the document or narrow the chatbot's topics before you share the link.

After the chatbot is live

Testing does not stop once customers start using the chatbot:

• Review the questions that come in regularly, if the service you use makes them available.
• Run your list of test questions again every time the documents are updated.
• Record the date each document was updated, so it is easy to trace answers that change.

An hour of testing at the start costs far less than clearing up misunderstandings with customers after the chatbot has already been shared.