1.
Future is here or how to
test NLP in chatbots
Iryna Yaroslavtseva

2.
● 5 years of testing;
● 3+ years of testing chatbots;
Cherkasy, Ukraine
IRYNA
YAROSLAVTSEVA

3.
of businesses want chatbots by 2020

4.
Rule based AI based

5.
Rule based AI based

6.
Current project tech stack
NLPStructure
Messenger

7.
“Show me yesterday’s financial news”
Utterance

8.
“Show me yesterday’s financial news”
Utterances
● Delivery
● Thanks!
● Book me a flight to Rio next week.
● I already have a phone 9. What the plan going to
cost me ulimited switching over from crickety eyes
you have anything without 1st month.

9.
“Show me yesterday’s financial news”
Intent: showNews
Verb Noun

10.
“Show me yesterday’s financial news”
● checkCoverage
● buyIphone
● findBus
Intents
● bookFlight
● changeLine
● bookAppointment

11.
“Show me yesterday’s financial news”
Entity Entity

12.
Intent
● bookFlight
● checkCoverage
● buyIphone
Entities
● City, Date
● Location, Country
● Model, Color, Capacity

13.
DISCOVERY MODEL
TRAINING
TESTING LEARNING
ON
PRODUCTIO
N

14.
DISCOVERY

15.
Key requirements
● Bot language
● Feature scope
● Target audience (user’s profile)

16.
Key requirements
● Bot language
● Feature scope
● Target audience (user’s profile)

17.
Key requirements
● Bot language
● Feature scope
● Target audience (user’s profile)

18.
MODEL
TRAINING

19.
TESTING

20.
Challenges
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
Solutions
● Risk analysis
● Data analysis

21.
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Data analysis

22.
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Data analysis

23.
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Data analysis

24.
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Data analysis

25.
Structured response
Turning on / Turning
off NLP
Lost in the flow
Business restrictions
Profanity words
Similar intents
Default answer
Acronyms
Informal shortening
Shortening in writing
Emojis
Slang words
Intent + other words
Intent + small talk
Intent + other sentences
Misspellings
Synonyms
Similar phrases
NLP testing checklist

26.
● Order pizza = Order piza
● Change day to Monday = Change day to Mobday
● Send picture = Senf picture
Misspellings

27.
● New offers = New deals
● Some help = Some support = Some assistance
● Order iPhone = Purchase iPhone
Synonyms

28.
● What benefits do you have? = Show me the advantages
● Where is my parcel? = Track my order
● Make a reservation on Friday = Book appointment,
Friday
Similar phrases

29.
Can I order bundle of
two headphones, one
for me and one for
my dad?
Intent + other words
I saw your promo,
how to order the
headphones
orderHeadphones

30.
Hello, find my bus
station please.
Intent + small talk
findStation
Morning! Nearest
bus station. Thanks.

31.
My inbox is a dumpster! I'm like flooded with all the
messages. How can I unsubscribe ?
Intent + other sentences
unsubNews

32.
● Lol - laugh out loud
● Asap - as soon as possible
● Np (No Problem), But how can I send my
booking confirmation
Acronyms
confirmBooking

33.
Informal shortening
● Gonna - going to
● Lil’ - little
● Wanna - want to

34.
Formal shortening
● I’m - I am
● Smth. - something
● I’ve - I have

35.
Top:
👍 😀
😂 ❤️
Emoji
Banned:
🔪 🖕 🍑 🍆 💩
🖕🖕🌈 + 🚫 ⛪️ +

36.
● Kudos! = praise and honor received for an
achievement
● You rock! = You're awesome (at something)
● On fleek = smooth, nice, sweet
Slang words

37.
● I’d like to order dress
● I’d like to know where
is my dress?
Similar intents
trackOrder
makeOrder

38.
● Sorry, I didn’t get that. Try again
● If you didn’t find what you wanted feel
free to see information in the Main Menu
or type ‘show me main menu’
Default (None) intent

39.
Score for None intent
● If utterance score < 0.3
● Avoid repetitions
none

40.
Dogs suffer = stop animal testing
Business restrictions
getInfo

41.
● Bullshit
● Dammit
● Go to hell
Profanity

42.
● Friday, 29 May 2015 05:50:06
● Test@test.com
● $ 1,234.60 = 1 234,6
Structured response

43.
● Feedback forms
● Free form surveys
Turning off/ turning on NLP

44.
Lost in the flow

45.
Lost in the flow

46.
● Intents - 3
● Utterances - 30
● Checklist based testing ≈ 8 hours
Checklist execution time

47.
LEARNING ON
PRODUCTION

48.
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● Monitoring the results
Learning on production

49.
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● Monitoring the results
Learning on production

50.
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● Monitoring the results
Learning on production

51.
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● Monitoring the results
Learning on production

52.
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● Monitoring the results
Learning on production

53.
New project roles
NLP model training
Statistics
Bot flow
Copy
NLP model training
Statistics
CONVERSATIONAL
DESIGNER
CONTENT
MANAGER

54.
● Chatbot project = regular project
● NLP is the only unique component
● Analyze risks and statistics to enforce
effective and efficient testing
QA takeaways

55.
● Chatbot project = regular project
● NLP is the only unique component
● Analyze risks and statistics to enforce
effective and efficient testing
QA takeaways

56.
● Chatbot project = regular project
● NLP is the only unique component
● Analyze risks and statistics to enforce
effective and efficient testing
QA takeaways

57.
NLP testing checklist

58.
Thank you!

59.
Questions

60.
About me
Facebook LinkedIn