Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

1
Live Note/QA: http://tinyurl.com/Evorus
Evorus:
A Crowd-Powered
Conversational Assistant
Built to Automate Itself Over Time
Ting-Hao (Kenneth) Huang*
Joseph Chee Chang
Jeffrey P. Bigham
[ Question / Feedback: http://tinyurl.com/Evorus ]
*Kenneth will join in Fall 2018

2

3

4

5

6

7
What just
happened?
• Open Conversation
• Multi-turn interaction
• Multiple domains
• Personalized
• Coherent dialog
• Mix of task-oriented
and social conversation

8
Today’s Conversational Assistants…
“What’s new with Alexa?”“Talking to Siri”

9
Existing Approaches to
Open Conversation
• Combining multiple task-oriented dialog systems
• DialPort (Zhao, et al., 2016)
• End-to-end framework for dialogue systems
• Serban, et al. 2016; Li, et al. 2017
• Adapting a model to many other domains
• Walker, et al., 2007; Sun, et al., 2016
• Chit-chat systems (social bot)
• Hold social conversations (Banchs, et al., 2012)
• Still a very hard problem…

10
Existing Approaches to
Open Conversation
• Combining multiple task-oriented dialog systems
• DialPort (Zhao, et al., 2016)
• End-to-end framework for dialogue systems
• Serban, et al. 2016; Li, et al. 2017
• Adapting a model to many other domains
• Walker, et al., 2007; Sun, et al., 2016
• Chit-chat systems (social bot)
• Hold social conversations (Banchs, et al., 2012)
• Still a very hard problem…
MIT Technology Review
Feb 27, 2018

11
A Top-Down Approach
Fully-Automated System
Human-
Powered
System

12
A Top-Down Approach
Crowd-AI System
Human-
Powered
System

13
A Top-Down Approach
Crowd-AI System
Human-
Powered
System

14
In Today’s Talk…
Evorus
Chorus

15
Chorus: A Crowd-Powered
Conversation Assistant
Evorus

16
•Crowd workers collectively hold a
conversation by:
1. Propose Responses
2. Vote Responses
3. Take Notes
Reward points for
each action
"Is there anything else I can help you with?": Challenges in Deploying an On-Demand
Crowd-Powered Conversational Agent. Huang, et al. HCOMP 2016.
Chorus: A Crowd-Powered
Conversation Assistant

17
User Interface
17 / 31

18
User & Worker Interface
18 / 31

19

20
We Deployed Chorus
• Launched on May 20th, 2016
• On Google Hangouts
• 2200+ conversations, 420+ users
• TalkingToTheCrowd.org

21
Evorus
Chorus
Evorus: A Crowd-Powered Conversational Assistant
Built to Automate Itself Over Time

22
Automating Chorus

23
Automating Chorus

24
Automating Chorus

25
Automating Chorus

26
Chatbots
How to select
chatbots
automatically?

27
Ranking Chatbots: Performance & Topic
Chatbot’s
Performance
Topic Similarity
Likelihood
of a
Chatbot

28
Chatbot’s
Performance
Topic Similarity
Likelihood
of a
Chatbot
~= Overall Message
Acceptance Rate

29
Topic Similarity
User Message
Domain of
the Chatbot
Hey what should
I eat in Montreal?

30
Topic Similarity
User Message
Domain of
the Chatbot
Hey what should
I eat in Montreal?
Find me some
good restaurants !
Where can I get
Chinese food?
Example
Triggering
Message

31
Topic Similarity
User Message
Domain of
the Chatbot
Hey what should
I eat in Montreal?
Example
Triggering
Message
Find me some
good restaurants !
Where can I get
Chinese food?
Topic
Similarity

32
Chatbot’s
Performance
Topic Similarity
Likelihood
of a
Chatbot

33
Chatbot’s
Performance
Topic Similarity
Likelihood
of a
Chatbot
 Add more chatbots over time !

34

35
Automatic Upvote
How to estimate
the impact of an
automation?

36
Find the Best Confidence Threshold
• High Threshold
• Only vote when pretty sure
• High precision, but little benefit
• Low Threshold
• Nearly always vote
• Grant agreement bonus by mistake
• Damage conversation quality

37
Find the Best Confidence Threshold
Expected Reward Points Saved

38
Automating Chorus Over Time

39
Automating Open Conversations
• Setup
• A 5-month-long deployment, 80 Users
• 4 chatbots + 1 voting bot
• Result
• Automated responses were chosen 12.44% of the time.
• Human upvotes were reduced by 13.81%.
• The cost of each message is reduced by 32.76%.
• Conversation quality and user
satisfaction level remains.
• Conversation Quality: Satisfaction,
Clarity, Responsiveness, Comfort
(Liu, et al., 2010)

40
Conclusion & Future Work
• Automating some part of an open conversation is
possible.
• Future Work
• 1000+ Chatbots!
• Deploy Chorus as an Open Research Platform
• Chorus on Smart Devices

41
Conclusion & Future Work
• Automating some part of an open conversation is
possible.
• Future Work
• 1000+ Chatbots!
• Deploy Chorus as an Open Research Platform
• Chorus on Smart Devices
@windx0303
KennethHuang.cc
Ting-Hao (Kenneth) Huang
QA: http://tinyurl.com/Evorus

42
Backup Slides

43

44
Features of
Automatic Voting

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

Recomendados

Recomendados

Más contenido relacionado

Similar a Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

Similar a Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time (20)

Más de Ting-Hao Huang

Más de Ting-Hao Huang (7)

Último

Último (20)

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time