How to measure success of a conversational AI chatbot

Last year when one of our healthcare partners (we refer to our clients as partners) was looking to build a conversational AI chatbot, I was apprehensive about guiding them. I had only worked on the level 2 (out of the 5 levels of conversational AI) type of bots. But this time our partner wanted to build a contextual/consultative AI-powered chatbot assistant.
To Build a Successful Chatbot…. Start with these 5 questions. | by Yogesh Moorjani | Chatbots Magazine

I was concerned about how the bot would understand end users’ problems. What features can we build to make it more humanistic? Would it be successful in replacing human care and compassion? Would it replicate the same emotions of empathy, compassion, and care?

And even if we managed to do everything, how would we know if the conversational AI chatbot is working the way we designed it? How would we define the ‘success’ of our initiative?

My apprehensions became real when I read a Forbes article about chatbots killing customer service with their clumsy conversations and limited learning capabilities. After reading the below paragraph, I realized the problem-

“The AI didn’t always get it, which was frustrating. Even more irritating — the company using the chatbot seemed to shrug the problem off. I detailed my own experience using Skyscanner’s chatbot, which often misunderstood my requests. Some of the companies I mentioned in the column appeared to shrug off my concerns.”

The problem is with organizations/management who choose to look away and see the importance of data analytics in chatbots for healthcare. They think that understanding the users’ behavior, what disappoints them, what makes them happy, is beyond the scope of their work. Because of this mindset, chatbots are still a lost cause.

Is there a solution in sight?

Yes, indeed there is. We’re at a very interesting place where we hold the future of chatbots in our hands. To make chatbots more welcoming and user-friendly, we not only need to make its software side–engineering, UX design, security– more robust. Rather, we should strive to make data analysis a part of the development process– i.e. we must constantly monitor chatbot’s effectiveness and improve features as per users’ needs.

How can we measure a chatbot’s efficiency?

Building a good conversational AI healthcare chatbot is a daunting task. Even after launching it as a service, one can’t be sure of its success. That’s why it’s crucial to measure every interaction with the end users.

There are certain indicators that one can track to see if the chatbot is serving its purpose.

8 Ways to Improve Chatbots and Boost Customer Satisfaction


Botanalytics - Event / Speaker Platform

If you’re looking for a tool that gives you an overview of the user’s lifecycle, then Botanalytics is for you. It’s a great tool for identifying bottlenecks in the user’s journey. You can deep dive into every conversation (transcripts are available for each conversation) and see where your bot failed to respond.

You can set various goals and categorize chats into conversation paths. This is a great feature as it helps you examine which conversation attained its goal and which didn’t.

For example- if your goal is to make users download your mobile app through the link provided in the chat, then this tool will show you how many conversations ended up in accomplishing that goal.

You can also set conversation paths and check how many conversations were successfully handled by the chatbot.


GitHub - grafana/grafana: The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

Grafana is not a bot-analytics tool. Rather, it’s an open source platform that can be used to monitor applications, websites, and even custom data sources. We integrated it with our platform to use it as a chatbot analytics tool.

One of the advantages of using Grafana is that it’s very easy to customize and you can tweak its dashboard to suit your needs.

If you have a chatbot where there is a lot of data to understand, analyze, and dissect, then you must explore Grafana. Icing on the cake? It’s free. And like I mentioned earlier, highly customizable. You can create dashboards, add panels, change visualizations as per the need of evaluators and stakeholders.


Chatbase : New ChatBot Analytic Tool by Google | by Aydın Fevzi Özçekiç | Chatbots Magazine

Chatbase is a free cloud-based tool that allows you to integrate your bot into the analytics platform. One of the best features of Chatbase is that it helps you both analyze and optimize your bots.

In the analytics part, Chatbase has every possible feature that you can imagine- session flows, creating funnels, the grouping of not-handled messages, chat transcripts, and so on. The UI of the dashboard is quite similar to Google Analytics. So if you’re a GA user, you’ll find it easier to use it.

In the optimization part, Chatbase provides insights to understand your users by tracking how they behave and what works (or not) for them. This is especially helpful when you want to target a specific audience and you want to improve your messaging and promotions according to specific inputs from the analytics tool.

When it comes to building chatbots, including analytics in the strategy is often sidelined. It’s considered an additional responsibility, something that can be easily avoided. However, measuring the performance metrics of a chatbot must be included in the development strategy because it’s the only way to define if your chatbot is working as you imagined it to be.

I hope you learn to integrate these tools and use analytics to enhance the experience of the chatbot for your end-user. In case you begin to feel that it’s a sponsored post, let me tell you it’s not. All these recommendations are personal and I have learned through trial and error. I hope you find the best tool suited to your needs.

Incredible Chatbot Design Strategies to Turn Your Bot into an Interaction Ninja

Why the World Needs Chatbots. When Facebook announced the Messenger… | by Akshay Kulkarni | Chatbots Magazine

48% of global consumers accepted that they would rather connect with a company over chat than any other source of contact. 35% of the users would like to see more companies incorporating chatbots. As for the business side of the story, chatbots save around £6 billion per year and a simple chat option in the mobile app brings up the revenue by over 30%.

These are only a snippet of thousands of statistics that are all around the internet and lying on a million office reports hinting towards how chatbot app development has turned out to be the best business decision.

Done right, chatbots can single-handedly make your company approachable and open to business even when you sleep. Done wrong, it leaves users unsatisfied and complaining about how your brand is slow in the logic department.

What helps in making a chatbot effective, besides of course passing all your chatbot success worries to a chatbot app development company, is having a right strategy in place.

1. Have a Purpose

Chatbot Error: 404 (Witty responses not found) - OnePlus Community

Knowing what purpose your chatbot would solve is the first step in designing a working, revenue-generating chatbot design strategy in 2018.

The last thing that you would want is to join the chatbot world just because every business is on it. While it is true that it benefits a range of different industries, it is also true that not all businesses need it, especially the ones that are working only around a particular geographical location i.e. in the same time zone.

But if yours is a business that would benefit immensely with Chatbots, think of the process that you would like to give your bot the autonomy of. The scale of the process that you are looking forward to leaving in the hands of chatbot will help decide which bot type you have to invest in (more on this later).

2. Conversation Flow

6 Steps for Creating a Smooth Chatbot Conversation Flow

The flow in which your chatbot would converse is one of the most important things for your chatbot design guide to cover.

The ideal situation when it comes to deciding how the conversation should flow is that the bot should start with a general salutation followed by asking questions and then pitching the product when the user put in the respective action words like ‘Tell me More’ or ‘Give more information’.

The biggest mistake that businesses make is pitching in the product as soon as the user becomes active on the website or page when they should go with a more human-like conversation flow.

3. Bot Personality

The Chat Bot Personality. Choosing the perfect personality for… | by Pulse Chat | Chatbots Life

The only way to make your bot more human-like is to give it a personality. Now when I say personality, I am not implying you to create an avatar and anime man. No, when I say personality, I mean giving it a tone, a difference in the US and UK English, slang, and age-appropriate responses.

Now, in the normal scenario, a business doesn’t have a specific exact type of customer base, so would you keep making bots aiming for each of them to meet specific criteria? The answer is obviously No. What you can do is ask your users’ background and change the initial conversation level of the bot.

4. Give Context From the Beginning

How to give your chatbot context and improve CX without coding

Make the purpose of your chatbot clear from the first two messages itself. Aim for a scenario where your user knows exactly what the bot would be helping them with and have no misinterpreted expectations and then a bad customer service instance.

The first two dialogue to and fro should be enough for your user to know what is to be expected from the chatbot. If you miss this step, the chances are that your bot dashboard will get filled with repeated questions and frustrations of not getting answers.

5. Share before Publicizing

So you have developed an amazing bot that converses with masses effortlessly, specializes in giving the users the exact information that they need, which even varies from context to context. What’s next?

And the answer does not make it live on your website or integrate into your mobile app right away. The answer is to first share it with your in-house teams to see if it’s actually able to answer different questions and different contexts.

By sharing it with the team and taking their feedback, you will be able to track exactly how people are reacting to it and what changes would make the conversation more meaningful.

6. Choose your Words Wisely

10 Chatbot designs for inspiration – Customer Service Blog from HappyFox

The most important element of strategic chatbot development according to chatbot design trend 2018 is choosing the words right. Your copywriting technique should be on point to not just entice the users to start the conversation but keep them hooked enough to convert into your site’s loyal customer or subscriber.

Know that your chatbot copy has the power to make your chatbot a conversion magnet and it should be treated just like that – as a mix of emotions and conversion statements.

So, here were the pointers that would help you draft a killer, high revenue generating chatbot design strategy in 2018. The one that would result in a chatbot that people love to converse with and pay to.

Now that you know what an ideal chatbot should operate like, it is time to look into the three types of chatbots that you should invest in developing once you have your chatbot design guide planned.

The Three Types of Chatbots

1. Menu Based Chatbots

Types of Chatbots: An Overview for Business People

These are one of the most common types of chatbot available in the market today. Using this chatbot, brands give multiple options to the users to select from. These are used mainly when the users are looking for a direct answer.

One of the biggest challenges that businesses face when it comes to menu based chatbots is that the process that users will have to follow to come to the point when they get the desired answer is quite long. They have to click through several buttons and options to come on one point.

2. Keyword Recognition Based Chatbots

3 chatbot types: Which is best for your business? | Chatimize

These bots utilize the blend of AI and keywords to determine the appropriate response. They lack in performance when they are asked a lot of similar questions with similar sounding keywords, even when the context is different.

A new set of bots are now present in the market that is hybrid. They automatically switch to the menu based type when they are unable to answer users’ queries through AI.

3. Contextual Chatbots

Conversational AI: Design & Build a Contextual AI Assistant | by Mady Mantha | Towards Data Science

The most advanced of the chatbot family, these bots remember the interaction and its outcome and keeps growing on it to give better result to the users over time. They work strictly on Machine Learning and Artificial Intelligence to help users come to a decision.

As they are based on ML and AI technologies, they come with a self-learning ability, which helps in not just customer relationship management but also a positive sales conversion rate.

The ultimate decision of which chatbot type to invest in depends entirely on your business need. If yours is a simple Q&A type app where you aim at giving a quick in and out experience to your users, answering their questions to the point, menu-based chatbot app development is what you should go for. But, in case you have an assistant like mobile app idea, then you will lose your pockets a little and invest in Contextual chatbot development.

error: Content is protected !!