QA has always been under-rated and thus it is important to consider this equally important as the Dev. If we look at the Chatbot QA, it had been considered as a highly challenging work specially when you do not know where your bot may break while you sequentially will be only running your flow (stories). Most of the companies / tools only check the flow which are coded in a fixed format which often breaks while testing. There may be cases where bot are migrated to new version and it breaks. The presentation will discuss the possibilities to test the bots by helping folks to create their coverage matrix for your stories, efficiently looking at the logs and mine information and most importantly what to test and which components to test. Presented by Director QA, DevOps & AIML at APTY.IO, Soumya Mukherjee at the 2021 Rasa Summit https://rasa.com/summit/

1. How to
”Effectively” ”Test”
your Chatbot
Soumya Mukherjee
Director QA, DevOps & AIML
Apty.IO

2. How are we doing our QA today
• Testing is Blackbox for testers
• Mostly manual testing done in organization
• Conversational flow testing
• Small Talk
• Fallback checks
• Integrations
• Automation done on UI and API layer
• Testing is mostly done on same training data
• Models are trained by engineers and are not being
monitored by QA
• There are analytics tools available to monitor but it
needs technical expertise for the QA
• Result : More than 90% times bot breaks (no one
understands when it will break), most of them fallback
and get stuck - once bot is stuck it is stuck
Q ?
A

3. What are the issues in QA ?
• Bots are evolving and continuous story creation is a problem
• No tool manage story coverage
• Your training data may not correspond to new stories or vice versa (it’s a
mismatch) – most org keep training on the same data
• Most automation tools offers record and playback (My stories are
already written how to port is the question)

4. What are the issues in QA ?
• No (unified) centralized dashboard present where QA can check (everything is quite scattered)
• Intent Matching
• Entity Testing – Slot identification
• Entity Testing – Entity Validation
• Confidence score
• Confusion Matrix along with Precision/Recall/F1-Score
• No easy way to reset the failed bot !
• Bot versioning is a mess and A/B testing becomes difficult
• Multilingual bot QA is a challenge (have to make 2 separate bots)
• High confidence score is also a problem as your bot will only predict same thing (if the data is same
for multiple intents then it will predict the one with highest confidence score – may be incorrect)
How to make sure your bot never breaks ?

5. How to make your test effective ?
• Create scenarios for happy path, contextual questions, digressions, domain
specific questions, stateless conversations
• Map proper entities for common scenarios (example bus fee, tuition fee) –
flow should change with entities in the stories
• Automated tests should consume all stories and run them each time as part
of regression testing
• Story coverage visualization
• For Manual Testing use Bot emulation product (like RasaX, Botfront) to test

6. How to make your test effective ?
• Central dashboarding including :
• Confusion matrix, Precision, Recall and F1-Score
• Cumulative accuracy profile
• Cross validation results
• Perform Exhaustive testing (bot resiliency), Integration checks across
platforms, Webhooks
• Perform fault tolerance testing by performing performance testing (bot
response, session management) & security testing (api interaction,
typing speed check, punctuations, typo errors)

7. Other KPIs to track
• Activity Volume
• Bounce rate
• Retention rate
• Open sessions count
• Session times (conversation length)
• Goal completion rate
• User feedback (sentiments)
• Fallback rate (Confusion rate, reset rate & Human takeover rate)

8. Thanks
@QASoumya
Linkedin.com/in/mukherjeesoumya