2. why a personal assistant?
cortana introduction
key scenarios and the cloud
lessons learnt
crazy scenarios we can build
3. increasing focus on information (& knowledge), personalization, context, wearables, …
anticipatory computing
importance of near-real time processing/reactive computing
transition from web to apps to personal assistants
4.
5. PERSONAL
Cortana…
…is your truly personal assistant
…gets to know you
…is transparent
Examples:
Learning
Notebook
Personal suggestions
Transparency & control
LOOKS OUT FOR YOU
Cortana…
…looks out for you
…filters out the noise
…reminds you of what’s important
Examples:
Useful and relevant alerts
Planners
Event scheduling
Quiet hours and inner circle
Reminders
DELIGHTFUL & EASY TO USE
Cortana…
…“just works”
…lets you interact on your terms
…has a fun & engaging personality
Examples:
Voice & natural language
Text input
Personality (visual, spoken voice,
and behavior)
6.
7. bing platform as the foundation for
personal assistant experiences
user centric
cloud-driven
service-oriented
asynchronous, reactive, functional, stateless
data-driven
configuration-driven
secure and available
8. ingredients – how to build a personal assistant
privacy/security
design
knowledge platform
machine learning platform
feedback loop infrastructure
notifications infrastructure
stream/complex event processing
metrics/data-driven engineering
speech recognition
natural language understanding
user understanding
conversational & intent understanding
personality
global datacenter footprint
legal
business development
…
11. flight BA 49 is delayed
notification
information streams: sports, flights,
weather, traffic, news, packages,
user location, …
notification
information streams: sports, flights,
weather, traffic, news, packages,
user location, …
var subscription = streamPlatform
.GetObservable<FlightInfo>(Constants.FlightsStream)
.Where(f => f.StatusCode == FlightStatus.Landed)
.Select(f => string.Format("{0} {1}: {2}", f.Airline.Name, f.FlightNumber, f.StatusCode))
.Subscribe(...);
stream
processing
platform
12. is this your home?
commute to work around
8.30am on Mondays
inferences
user understanding/insight
13. “remind me to wish Paul
happy birthday”
speech recognition
natural language
understanding
conversation
management
representation
of intent
speech stream
Using Deep Neural Networks-trained models
and other machine-learning techniques, we
convert the voice stream to a representation
the computers can understand
We reason over the machine representation of the user’s
intent. We use what we know about the user (e.g. “home”
is transformed to an actual location) in order to fill in any
gaps in our understanding. We maintain a dialog with the
user (e.g. if the user had just said “remind me”, we would
have initiated a follow up question).
“when I get home, remind
me to take out the
garbage”
language
generation
user
profile/context
web socket
Using Deep Neural Networks-trained models
and other machine-learning techniques, we
convert the voice stream to a representation
the computers can understand
We reason over the machine representation of the user’s
intent. We use what we know about the user (e.g. “home”
is transformed to an actual location) in order to fill in any
gaps in our understanding. We maintain a dialog with the
user (e.g. if the user had just said “remind me”, we would
have initiated a follow up question).
“when I get home, remind
me to take out the
garbage”
language
generation
user
profile/context
web socket
mini reactor
14. “will I need a scarf
tomorrow?”
speech recognition
natural language
understanding
conversation
management
user
profile/context
html
speech stream
Using Deep Neural Networks-trained models
and other machine-learning techniques, we
convert the voice stream to a representation
the computers can understand
We reason over the machine representation of the user’s
intent. We consult Bing Knowledge and the user’s profile in
order to construct a response in the conversation with the
user. Since we are keeping the context of the conversation,
the user can follow up with a question such as “How about
next weekend?” or “What about Seattle?”.
language
generation
bing knowledge
rank
17. while at Sydney 2011, you should talk to john s.
both of you were at paris 2010, rio 2009, and
istanbul 2002 conferences
you also seem to be reading the same journals on
knowledge representation
you both found the book “on intelligence”
interesting
finally, you both like sushi…
i suggest “blue sushi”, just two blocks from the
conference center
18. “what was the title of the song
that got me dancing during the
radiohead concert few days ago?”
19. emergence of personal assistant as ux
metaphore, consolidation of user experiences
anticipatory computing, task completion,
conversational
sensors, wearables
reactive, near-realtime
20.
21. Coldplay at the Gorge
2009 – 07 – 10, 9.12pm
Weather: Warm and mostly sunny
Attending: 20,000
Song playing at the time of photograph: Viva La Vida
Did you know that it was Coldplay’s 1st appearance at the
Gorge?
Ad-hoc social streams: photos, tweets, emotions
i
Kinect avatars that are connected to a knowledge base
Kids interact with the avatars and can ask questions
“a virtual tutor, companion”
Social experience… a virtual class