The chatbot demo always works. The chatbot in production, facing real customers with real edge cases, often does not. Here is what separates the two.
Every chatbot demo is impressive. The questions are the ones the builder expected, the knowledge base is tidy, and nobody is in a hurry. Production is different. Customers ask half-formed questions, mention three problems at once, and are rarely patient. After building assistants for retail, education and logistics clients, we have seen the same failure modes repeat — and the same habits in the ones that work.
If you cannot say what a successful conversation looks like, you cannot improve it. The teams that succeed agree on a measurable definition before building: the customer got a correct answer, completed the task, or was handed to a person with full context. Everything else — deflection rate, containment, “bot satisfaction” — is a proxy that can look healthy while customers quietly give up.
A language model is a poor database. When a chatbot answers pricing or policy from memory, it will sometimes be confidently wrong. The reliable pattern is retrieval: find the relevant passages in your approved documentation, give them to the model, and require the answer to stay within them. We cite the source in the reply so a customer — or a reviewer — can check it in one click.
The fastest way to anger a customer is to trap them in a loop. A good assistant knows its limits. When confidence is low, when the customer asks twice for a person, or when the topic is sensitive, it hands over immediately and passes the whole transcript along, so the customer never repeats themselves.
Export three months of tickets, cluster them by topic, and look at the top ten. If those are answerable from documentation you already have, an assistant will pay for itself quickly. If the answers live in people’s heads, the first project is writing them down — and that work is valuable with or without AI.
