Determining Metrics

You’ve created your assistant, you’ve set up intents, launched it to production and your customers are interacting with it. How do you determine it’s a success? How do you even measure success? How do you quantify customer satisfaction or determine whether the assistant is working as designed? How do you determine what needs to be optimized if it’s under performing?

“Metrics are like a course in a meal. They should satisfy the need or want for something.”

Without metrics and analytics, there is no real quantifiable determination that the work you and your development team have done is actually helping your customers or streamlining your processes. I’m sure you’re thinking, “I’ve already got metrics on number of conversations being had, number of sessions and users. What else do I need to think about?”

Great question, from a business perspective, we think about performance from the customers viewpoint and what a successful assistant looks like within their experience. You want to think about whether your customers are using it, will they use it again, whether their questions are being answered, whether they are escalating to human agents and at what percentage are these instances occurring.

Collecting Customer Feedback: Customer Satisfaction and NPS scores

A great way to measure customer satisfaction is through feedback. A commonly used method to collect customer feedback that we see is CSAT score or Customer Satisfaction Score, this measures how satisfied customers were with their experience or with using your bot and it can be a simple survey at the end of the conversation asking “How satisfied were you with your experience?” You can absolutely tailor this question to your business but the general thought behind this question is to understand customer sentiment and how your customers feel about their experience with your assistant. It’s also a great way to allow customers to follow-up on their sentiment with additional notes or comments that can help you iterate on the customer experience.

Image: CSAT example

Another alternative to the CSAT is NPS or Net Promoter Score, again this would be a question for customers/users at the end of the conversation like the CSAT survey and can be altered to fit your business strategy but oftentimes, the questions asked here is “How likely are you to recommend this to a friend or colleague?” as seen in the example below.

Image: NPS score example

Whether it be CSAT, NPS, Star Charts or any other methods of collecting customer feedback that you choose, having a customer facing satisfaction survey is a fundamental step that should be taken when building successful metrics.

I’ve got some customer feedback, what else should I think about?

Another metric that we’ve found useful when reporting business performance has been Containment Rate. Containment Rate is purely the percentage of conversations handled exclusively by your bot/assistant and not handed over to a human. This is an important metric simply because in an ideal situation, you would want to ensure that your customers aren’t frustrated by the assistants' responses and solely relying on human interaction. Imagine spending several months building an assistant that’s supposed to cut down on the need for human interaction and finding that only a small percentage of your customers are able to use it successfully. Not exactly a great return on investment.

Nonetheless, Containment Rate is only one measure of success, we also want to think about Abandonment Rates and Escalation Rates in parallel. Abandonment Rates are often thought of as a percentage of conversations customers are ending prematurely and Escalation Rates are the percentage of conversations that are escalated to a human agent to handle. Both Abandonment Rate and Escalation Rates are important because they both provide insight into how well your assistant is doing with answering customer questions and whether customers are spending more time essentially spinning their wheels or forcing human interaction because their questions were not answered successfully by the assistant.

Image: Abandonment rate over time

That’s where Rasa Pro Analytics comes in.

Analytics helps visualize and process Rasa assistant metrics in the tooling (BI tools, data warehouses) of your choice. Visualizations and analysis of the production assistant and its conversations allow you to assess ROI and improve the performance of the assistant over time.

With Rasa Pro Analytics, your chatbot history and conversation data is stored in the data warehouse of your choice (PostgreSQL, Redshift, BigQuery or Snowflake). From the warehouse your BI solution can be used to report on key performance indicators such as the containment, abandonment and escalation rates.

Example graphs and dashboards are provided for Metabase and Tableau. This gives you the flexibility to extract and visualize custom KPI's that aren't available from out of the box solutions. By analyzing these metrics over time, you can identify trends and patterns that can help you improve the overall user experience.

Rasa Pro Analytics enables you to gauge contact-related KPIs across various platforms, including call centers, web and mobile applications, by integrating with the warehouse and BI tool of your preference. You can utilize this tool to track customer touchpoints over time and minimize friction, among other metrics.

Image: Number of sessions per channel

By identifying areas of your chatbot that are not performing well, you can take steps to improve performance. For example, if you notice that users are escalating or abandoning conversations, you can make improvements to the NLU and dialog to keep users engaged and prevent them from dropping out.

Image: Escalation Rate over time

It's also important to understand how specific conversation flows of the bot are performing so you can improve the bot in key areas. For example, in a banking bot, you may have a flow to allow users to transfer funds between two accounts. This requires the user to provide the from and to account along with the amount to transfer. With Rasa Pro Analytics, we can measure the completion rate for key flows by calculating the percentage of users who entered and completed a flow.

Finally, tracking key performance indicators can help you measure the return on investment of your chatbot. By understanding how your chatbot is performing in terms of user engagement and satisfaction, you can determine the value that it is providing. If you are investing time and resources into developing and maintaining your bot, it is important to be able to measure its impact and demonstrate its value to stakeholders.