Presented by Jielei Li, Cognitive Software Engineer at Orange at the 2021 Rasa Summit. See summit recordings here: https://rasa.com/summit/

1. Continuous Improvement of
Conversational AI
In Production
Jielei LI
Cognitive software engineer for Djingo
▪ Why important
▪ How the process works
▪ Lessons learned

2. Interne Orange
Challenge of handling infinite
spontaneous responses.
Continuous learning cycle based on
real-user conversations and feedback
can be extremely helpful.
Difficult to plan for every eventuality
Chatbot projects learn by doing.
Continuous Improvement

3. Interne Orange
Continuous Improvement Process
User feedback
Analyze
Improve
Test
Deploy
Measure
Effectiveness
Create / Update
Training Data
Analyze Training
Test Data
Pre-Deployment
Testing

4. Collect
feedback
Macro KPI Micro vision
Task completion,
NPS,
Human takeover rate…
Specific needs of every
project
NLU :
Analyze the Quality of
Intent and Entity
Detection
Conversation :
Investigate user
experiences, when
users abandon the
dialog
Measure and Improve Effectiveness
Identify
weak spots
Improve
Fix Existing Training
Adapt to the conversation
flow

5. Interne Orange
2
1
Reading real conversations is critical
to improving the bot.
3 Good tools boost productivity.
Define KPI to better
prioritize future tasks.
4
We will need people with
different skill sets.
Developers
Linguists
Domain experts
Ux designers
Product managers
Devops
Lessons learned
Conversations
K
P
I
S
k
i
l
l
s
e
t
s
Tooling

6. Thanks for your attention
Jielei LI