Virtual agents are back, and they're everywhere! Their user interfaces tend to be simply those of instant messaging... or none at all. Thus the user experience resides more in the sequencing of bits of natural language than in that of menus or screens. Although everyone knows how to engage in human conversation, creating an app that behaves like one requires a technical knowledge of the mechanics of human conversation. While Conversational UX Design is still a nascent discipline, formal models from Conversation Analysis offer a scientific foundation for design. This session from SXSW 2017 provided design principles and models for creating conversational UX.

1. Applying Science to Conversational UX Design
Bob Moore, Raphael Arar
IBM Research

2. Interfaces have come
a long way.

3. Applying Science to Conversational UX Design
© 2017 IBM Research

7. Conversational Agents

9. “
Jean Baudrillard
Sociologist
The sad thing about
artificial intelligence is
that it lacks artifice and
therefore intelligence.

10. Human
Conversation
APIs
Conversational Systems
Speech to Text Text to SpeechNatural Language
Understanding
Dialog Management

11. Human
Conversation
APIs
Conversational Systems
Text to SpeechSpeech to Text Natural Language
Understanding
Dialog Management

12. USER EXPERIENCE
DESIGN
ARCHITECTURE
INFORMATION
ARCHITECTURE
HUMAN-COMPUTER
INTERACTION
INTERACTION
DESIGN
VISUAL DESIGN
MECHANICAL
ENGINEERING
ELECTRICAL
ENGINEERING
CONTENT CREATION
(Text, Data, Graphics)
Signage
Info Viz
Navigation INTERFACE DESIGN
Ubicomp
Controls
Interactive
Environments
USABILITY
ENGINEERING
INDUSTRIAL
DESIGN
Dan Saffer, UX Designer

13. USER EXPERIENCE
DESIGN
ARCHITECTURE
INFORMATION
ARCHITECTURE
HUMAN-COMPUTER
INTERACTION
INTERACTION
DESIGN
CONVERSATION DESIGN
MECHANICAL
ENGINEERING
ELECTRICAL
ENGINEERING
CONTENT CREATION
(Text, Data, Graphics)
Signage
Ontology Management
Navigation CONVERSATIONAL UI
Ubicomp
Controls
Interactive
Environments
USABILITY
ENGINEERING
INDUSTRIAL
DESIGN

14. What’s the state-of-the-
art for conversational UI?

15. Web. 2017.

16. Applying Science to Conversational UX Design
© 2017 IBM Research

17. Web. 1996.

19. Conversational UI. 2017.

21. Conversational UI. ????

22. Human
Conversation
APIs
Conversational Systems
Text to SpeechSpeech to Text Natural Language
Understanding
Dialog Management

23. Conversational Systems
Dialog Management
Conversation
Analysis
Human
Conversation

24. What does natural
conversation sound like?

25. 20 Des: What is the name?
21 Guy: Detweiler. D-e-t,
22 (1.2)
23 Guy: w-e,
24 (0.4)
25 Guy: i-l-e-r-.
26 (2.0)
27 Des: Foursome?
28 Guy: Yah.
29 (0.4)
30 Des: Electric carts?
31 (0.6)
32 Guy: Uh:::, n:no? I don’t
33 think so.
34 Des: Okay. We'll see yuh then,
35 Guy: Righto,
36 Des: Mm hm, Bye?
01 Des: G'morning. San Juan Hills
02 Country Club?
03 Guy: Guh morning. What’s-w-what
04 kind of a starting time
05 ken:: we get fer::hh
06 sometime this afternoon.
07 (0.7)
08 Guy: Any[time-
09 Des: [Oh:::, [let's see.
10 Guy: [Any time
11 tuhday.
12 Des: Two fordy. One, thirdy.
13 Guy: One thirty?
14 Des: Mm hm::?
15 Guy: One thirty.
16 (0.7)
17 Guy: .hh W'l at sounds like a
18 good time?
19 (0.4)

26. Not every voice or
text interaction is a
conversation.

27. Speaker-change recurs, or at least occurs.1
Overwhelmingly, one party talks at a time.2
Occurrences of more than one speaker at a time are common, but brief.3
Transitions (from one turn to a next) with no gap and no overlap are common. Together with transitions characterized by slight gap
or slight overlap, they make up the vast majority of transitions
4
Turn order is not fixed, but varies.5
Turn size is not fixed, but varies.6
Length of conversation is not specified in advance.7
Relative distribution of turns is not specified in advance.9
Number of parties can vary.10
Talk can be continuous or discontinuous.11
Turn-allocation techniques are obviously used. A current speaker may select a next speaker (as when he addresses a question to
another party); or parties may self-select in starting to talk
12
Various 'turn-constructional units' are employed; e.g., turns can be projectedly 'one word long', or they can be sentential in length13
Repair mechanisms exist for dealing with turn-taking errors and violations; e.g., if two parties find themselves talking
at the same time, one of them will stop prematurely, thus repairing the trouble
14
What parties say is not specified in advance8
— Harvey Sacks, Emanuel A. Schegloff, Gail Jefferson

28. Meet Alma
Natural Conversation Framework

29. Natural Language !=
Natural Conversation

30. intent
distance
cuisine place
Natural Language

31. Natural Conversation
action pair
granting
request

32. dependency
sequence closing
hearing trouble
dependency
dependency
understanding trouble
base second part
base first part

35. Conversational UX
a working set of principles

36. Saying is doing

37. J: T's- tsuh beautiful day out
isn't it?
L: Yeh it's jus' gorgeous...
CA
A: God izn it dreary.
(0.6)
B: [Y'know I don't think-
A: [.hh- It's warm though,

38. UX
“A "signifier" is some sort
of indicator, some signal
in the physical or social
world that can be
interpreted meaningfully.
Don Norman
Cognitive Scientist

39. Recipient design

40. CA
“By 'recipient design' we refer to a
multitude of respects in which the
talk by a party in a conversation is
constructed or designed in ways
which display an orientation and
sensitivity to the particular other(s)
who are the co-participants.
Harvey Sacks, Emanuel A. Schegloff, Gail Jefferson
Sociologists

41. B: Who's doing your remodel?
A: Dave
CA
C: Who's doing your remodel?
A: My neighbor across the street.
He's a contractor.

42. UX
“[Human-centered design is] an
approach that puts human needs,
capabilities, and behavior first, then
designs to accommodate those
needs, capabilities, and ways of
behaving.
Don Norman
Cognitive Scientist

43. Minimization

44. D: Who's doing your remodel?
CA
A: Dave
D: Who?
A: You know, my neighbor across the street.
D: Oh!
A: You had a beer with him?
D: Right.

45. CA
B: uh, yeah, I guess I'd like Mexican food
A: Mañana's is on Fourth and Winchester.
It's a great Mexican restaurant within
walking distance. It gets five out of
five stars. Would you like me to make
a reservation for you at Mañana's?
A: What kind of food would you like?
Mexican
Voice inputs are cheap, but voice
outputs are expensive

46. UX
Occam’s Razor1
Minimize Cognitive Load2
Eliminate Excise3

47. Understanding is
interactional

48. CA

49. UX
Mental Models1
Feedback2

50. Emotions describe
actions

51. CA

52. CA

53. UX
Visceral Behavioral Reflexive
Norman’s 3 Levels of Emotional Design

54. The best input method
is situational

56. UX
Context matters!

57. ibm.biz/conversational-ux

58. Bob Moore
rjmoore@us.ibm.com
Raphael Arar
rarar@us.ibm.com
Thank you.