Why Most Chatbots Suck - And How We Fixed Them

Have you ever contacted the support of a company and, you started talking to it and the answers were completely off the track, or it didn’t even help you?

At Quivr, we do things differently!

What is different about our automation?

Our automation is based on statistics. What does it mean?! Allow me to explain.

Most chatbots and support tools automate based on business rules. For example,

These automations are often based on the subject and description of the title for the automation. However, there is way more information in a Zendesk ticket than meets the eye. And these systems tend to miss a lot of information that can hardly be transcribed.

Understanding what’s in a Ticket

Tickets have a lot of useful information that helps a customer agent answer it efficiently. Efficient customer support teams have often enriched their Zendesk tickets with loads of Metadata. This metadata is automatically enriched when the user asks a question, such as:

• their account ID

• if they are paying customers

• if they are recurring customers

• the actions that they have done on the platform

All this information gives great insight to CS team in how they should respond.

For the same question, based on multiple internal information or just even where the user asks from, there could be hundreds of different answers.

Let me give you an example.

You have a metadata field that tells you on which page of your website the user opened the ticket. The user complains about not being able to login.

• The Homepage → this probably means his email is erroneous.

• Inside the app, on a specific page → this probably means he is having issues connecting to a tool or accessing a restricted section of the website.

Ok! So now you start to see the issue with “dumb” business rules. This information gives context to a human that is really challenging to explain or describe to an AI. And the number of combinations can be huge!

Let’s automate these business rules

Rather than having you write down hundred of business rules to allow us to build the best customer support agent tool, we built our own tool that does that automatically – Let me introduce to you: Le Juge!

What is Le Juge?!

It is a complex system made simple that, among many features, helps us find patterns in your tickets where we can automate the resolution.

The process is fairly simple

• Connect Quivr to your Zendesk. Once we are connected to your Zendesk, we ingest up to a year of your previous tickets.

• Generate an answer for all your historical data. If you have 4000 tickets, we generate 4000 answers.

• Le Juge evaluates these 4000 tickets and tells us whether we answered correctly or not by providing a score between 1 and 4. We evaluate, among many things, the tone and the substance of the answer. A good answer that doesn’t respect your brand voice is a bad answer.

• Identify that we can answer correctly to 50% of them for example.

• Pause & compare – generating an answer for a historical ticket vs. a live ticket are two very different things. The historical ticket has an answer to compare; the live one doesn’t.

• Work together – we know that 50% of your tickets can be automated. Now let’s build the model that decides which ones.

Let’s build your own prediction Model

Now we need to build a model that can predict if an answer that we generated can be sent or not. Imagine it as a well-experienced customer agent at your company that, by briefly looking at a ticket and its answer, can tell you whether you can send it or not. That is basically what we are building.

The first thing is to improve the quality of our dataset.

In the background, we are building an AI that predicts if an answer can be sent or not. To improve that, we need you to give us feedback on whether an answer was good (score 1–5).

By giving us feedback, we can align our pre-trained AI more closely to what you would have done. This gives us comparison points that are invaluable, allowing us to improve our AI with what we call labeled data.

Meanwhile, Le Juge is running in the background to find clusters (I’ll explain soon) where our prediction model is very precise at evaluating whether this answer can be sent. We are, by default, aiming at a 99.9% accuracy. It means that when Le Juge says, “Send It,” it means we are sure we can send it.

A Cluster?

Remember what I said about those business rules? A cluster is exactly that: a set of rules and metadata that Le Juge found relevant to group together.

A cluster can be something like:

Then, based on these clusters, Le Juge builds a prediction model that tries to be as accurate as possible. We can have a cluster where we are 99.9% accurate, others at 90%, 74%, 50%. The great thing is these clusters are generated automatically by Le Juge, you don’t lift a finger. Le Juge will understand autonomously which rules need to be applied to be accurate.

Now, we send it!

Once we have identified the clusters where we can accurately predict if we can answer accurately, we simply deploy our prediction models and, when a message arrives, it is evaluated on whether it can be auto-sent. And that’s it!

The great thing is we can, depending on the cluster, decide to deploy prediction models that are only 97% accurate—perfect for:

• tickets about an FAQ on a very specific use case,

• bursts such as Black Friday, a sale, a software change or policy update,

• any scenario where you need to answer quickly to an enormous number of tickets.

The best thing is… it keeps improving.

For tickets where we said we could answer, by looking at the answer you gave, our models learn and improve at each interaction.

If you want to give it a try, please let me know at founders [at] quivr [dot] app.