We are a long way from systems that “understand” people, but technologies to “recognize” emotions from word usage, voice characteristics, facial expressions, and gestures are here today and improving rapidly. Not far behind are technologies that allow systems to respond expressively with more appropriate tone and content - written and vocal - and relevant facial expressions on avatars and robots driven by brain-inspired models (not to be confused with neural networks). This webinar will present an overview of emerging technologies and applications for the next generation of Perceptive Input and Expressive Output interfaces. Participants will learn about the state of the art today, and how to identify apps in their own environment that would be good candidates for enhanced interfaces.

1. See Me, Feel Me, Touch Me, Heal Me
The Rise of the Cognitive Interface
Adrian Bowles, PhD
Founder, STORM Insights, Inc.
Lead Analyst, AI, Aragon Research
info@storminsights.com
NOVEMBER 9, 2017

2. Context: Advanced Human Computer Interfaces
Perception & Perceptive Input Management
Emotive Output
When to Use Advanced IO
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
AGENDA

3. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
NATURAL HCI HAS BEEN AN AI GOAL SINCE THE BEGINNING
PERCEPTION
UNDERSTANDING
LEARNING
PROBLEM-SOLVING
NATURAL LANGUAGE
PROCESSING
Big
Data
Classic
AI
Deep
Learning
Context

4. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
HOW DO WE COMMUNICATE?

5. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
NLP
Natural Language Understanding
NLU
Natural Language Generation
NLG
?
COMMUNICATIONS GOES BEYOND NATURAL LANGUAGE PROCESSING (NLP)

6. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
People are good - but not perfect -
when communicating in natural languages. We…
understand in context (environment & our own frame of reference)
attempt to resolve ambiguity
have to deal with competing signals, noise
fill in words and meaning and may not hear/understand
use our knowledge about the speaker and the situation to filter the input
use visual and auditory cues to distinguish between what was said vs what was meant…
X
X’
Y
HOW DO PEOPLE COPE WITH IMPRECISION?

7. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
WHAT DO WE COMMUNICATE?
Facts
I send you a text
at 2AM something bad happened
Concepts
I am angry
I am inconsiderate
I exaggerate

8. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
“You’re Not Just Responsible For What You Say,
You Are Responsible For What People Hear”
August 9, 2016
General Michael Hayden (Retired), former Director,
Central Intelligence Agency and National Security Agency
on advice to his senior staff.
NATURAL LANGUAGE PROCESSING IS HARD - FOR PEOPLE
Watch out for slang, context…

9. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
PICTURES VS WORDS

10. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
GESTURES ADD RICHNESS TO LANGUAGE

11. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
“THAT’S A BIG HOUSE” - MEANING MAY BE DIFFERENT FOR DIFFERENT SPEAKERS
Bob Mary
Al (Capone)
Wikipedia contributors. "Alcatraz Federal Penitentiary." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 30 Oct. 2017. Web. 8 Nov. 2017.

12. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
DRAW A QUARTER, TO SCALE - RESULTS DIFFER ACCORDING TO HIDDEN CONTEXT

13. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
Hearing (audioception)
~12,000 outer hair cells/ear
~3,500 inner hair cells
Vision (ophthalmoception)
Photoreceptors - Per Eye
~120,000,000 rod cells
(triggered by single photon)
~6,000,000 cone cells
(require more photons to trigger)
~ 60,000 photosensitive
ganglion cells
Touch (tactioception)
Thermoreceptors, mechanoreceptors,
chemoreceptors and nociceptors for touch, pressure, pain,
temperature, vibration
Smell (olfacoception)
Chemoreception
Taste (gustaoception)
Chemoreception
Human Cognition
~100,000,000,000 (100B) Neurons
~100-500,000,000,000,000 (100-500T) Synapses
THE HUMAN ADVANTAGE FOR PERCEPTION & COGNITION
LearnReason
Understand

14. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
Listen Look
Touch
Smell
Taste
Your System
Learn
Model
Reason
Understand
PERCEPTIVE INPUT: IT’S A NEW SENSATION Perception

15. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
HOW DO WE COMMUNICATE?
A Mathematical Theory of Communication,
C. E. Shannon,
The Bell System Technical Journal, Vol. 27,
pp. 379–423, 623–656, July, October, 1948.

16. Systems
Controls
LearnReason
Understand
Model
Data Mgmt
Human
Machine
Input Output
Gestures
Emotions
Language
Narrative Generation
Visualization
Reports
Haptics
Sensors
(IOT)
Systems
Controls
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
COGNITIVE SYSTEMS: COMMUNICATIONS & CONTROL
Perception

17. Human Input
Human-Oriented Output
Gestures
Emotions
Language
Narrative Generation
Visualization
Reports
Haptics
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
COGNITIVE SYSTEMS: COMMUNICATIONS & CONTROL
Perception
Meaning
Concepts
Intent
Context
Animation
LearnReason
Understand
Model
Data Mgmt

18. Human Input
Gestures
Language
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
PERCEPTIVE INPUT CLASSIFICATION
Context
LearnReason
Understand
Model
Data Mgmt
Detected by
Human Senses
Derived
ImagesSee
Hear
Touch
Smell
Taste
Sounds
Objects
Emotions
Meaning
Concepts
Intent

19. MULTI-DIMENSIONAL INPUT ANALYSIS
Source: IBM Bluemix

20. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
TONE ANALYSIS OF TEXT: IBM WATSON
Emotion tones:
Anger,
Disgust,
Fear,
Joy, and
Sadness.
Writing tones:
Analytical,
Confident, and
Tentative.
Social tones:
Openness,
Conscientiousness,
Extraversion,
Agreeableness, and
Neuroticism.
Source: IBM

21. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
UNDERLYING ALGORITHMS ARE COMPLEX
Source: Affectiva
Depend on detection and
classification of word use for
text and physiological state
changes for audio and video.

22. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
DEEP LEARNING FOR FEATURE, ENTITY, CONCEPT EXTRACTION/RESOLUTION (SIMPLIFIED!)
Visible Layer
Hidden Layers
Output Layer
Input: Observable Variables
HIGHABSTRACTIONLOW
Output
Pixels
Depth
of the
Model
Edges
Object
Shapes/Parts
Object Class
Brightness/
Contrast
Geometry
Rules
Features
to
Extract
Methods
1. Feed Forward
2. Back-Propagate
(adjust weights)

23. Model
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
CONCEPTS IN SPACE: IMPORTANT ASSUMPTIONS
Principles that control the
development and representation
of natural intelligence in the
neocortex provide a guide to the
implementation of machine
intelligence.(Numenta
Hierarchical Temporal Memory)
A function applied to a string
representing data or a concept
results in a value or vector
meaningful for comparison.
e.g. Using Kolmogorov complexity to
measure the strength of relationships
in Memory-Based Reasoning.
Distance
A metric for the similarity of two items
based on their relative locations in n-
space according to an algorithm.

24. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
Proximity Mapping
Boy
Bay
Map
Mop
Man
Nay May
Mope
Buy
Hop
Mapped with vectors,
proximity algorithm
based on purpose.
Mapping for autocorrect/complete vs Mapping for meaning
Boy
Bay
Map
Mop
Man
Nay
May
Mope
BuyHop
Hope
Hope
Similar structure ->
similar meaning in vision,
not always in language.
CONCEPTS IN SPACE

25. Model
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
COGNITIVE COMPUTING FUNDAMENTALS: SAMPLE ASSUMPTIONS
Model
The Corpus, Assumptions, Algorithms
Used to
Generate & Score Hypotheses
or
Calculate The Strength of a Relationship
Relationships can be discovered
without understanding the
meaning of any individual string,
fact, or
(NL-free concept modeling)

26. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
RECOGNIZING CONCEPTS VS UNDERSTANDING
Courtesy of LoopAI Labs.

27. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
EMOTION DETECTION IN VIDEO
7 Emotions
Anger
Sadness
Disgust
Joy
Surprise
Fear
Contempt
Source: Affectiva

28. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
KEY TREND: EMOTION DETECTION AS A FEATURE OR SERVICE
Source: Affectiva

29. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
HCI ADVANCES AUGMENT UNDERSTANDING
GestureRecognition/Analytics

30. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
NEXT TREND? USING SMARTPHONE SENSORS FOR RICHER CONTEXT
Type Typical Current Use
Accelerometer/motion
Rotate screen, Switch screen to landscape/
portrait
Ambient Light Adjust screen brightness
Barometer Measure altitude
Geo-Location (wifi/cellular) Location/Alerts
3-Axis Gyroscope Rotation rate for games, VR…
Proximity Turn off screen when phone is by your head
Touch ID fingerprint Security

31. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
EXPRESSIVE OUTPUT: EMOTIONS, URGENCY
Model
Emotions Meaning
Concepts Intent
Context
Narrative Generation
Images
Haptics
Animation
Using Language, Avatars, Emotive Text to Speech
Emotive Output

32. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
EMOTIVE TEXT TO SPEECH IS NOT NEW

33. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
DATA Model/
Article
Structure
Template
Generated
Narrative
NLG TECHNOLOGY CAN USE EMOTIONS IF THE MODEL SUPPORTS IT
Needs Context-Relevant
Tone Assignment Data
Model

34. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
Copyright (c)
Narrative Science.
COMMERCIAL NLG TECHNOLOGY WITH TONE ATTRIBUTES

35. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
NEUROBEHAVIORAL ANIMATION

36. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
Sentiment/Emotion/Theme/Concept Analysis
Don’t let the search for perfection interfere with the path to progress.
WHEN SHOULD YOU USE PERCEPTIVE INPUT/EMOTIVE OUTPUT? Getting Started

37. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
WHEN SHOULD YOU USE PERCEPTIVE INPUT/EMOTIVE OUTPUT?
When It Improves the User Experience
The System Tone Should Complement,
Not Mirror, the User Tone
When You Can Select or Generate
Appropriate Responses/Actions

38. adrian@storminsights.com
Twitter @ajbowles
Skype ajbowles
If you would like to connect on LinkedIn,
please let me know that you that you
registered for the Smart Data webinar series.
KEEP IN TOUCH
Upcoming SmartData Webinar Dates & Topics
Dec. 14 The Road to Autonomous Applications
Jan. 11 AI At The Edge:
Pushing Intelligence to Fog Computing Nodes
Feb. 8 A Pragmatic AI Maturity Model:
Choosing the Right AI Technologies Based on
Application Requirements and User Attributes