Emotion Recognition in Artificial Intelligence by Valentina Franzoni, Ph.D. in Engineering for Computer Science, Research Fellow in Artificial Intelligence, adjunct professor in Operative Systems

Emotion
recognition
in artificial
intelligence
VALENTINA
FRANZONI
DEPARTMENT OF MATHEMATICS
AND COMPUTER SCIENCE
UNIVERSITY OF PERUGIA, ITALY
Milan & Paris
WiMLDS
online meetup
June, the 4th
2020
WiMLDS online meeting - June 2020 Valentina Franzoni – Emotion Recognition in Artificial Intelligence

- Emotions and Artificial Intelligence
- Machine Learning and Data Science
approaches
- Errors, bias, and overconfidence
in artificial emotional models
- Open Challenges
Seminar overview

Emotions and Artificial Intelligence

Research questions
 What are emotions?
 How can they be measured in
the brain?
 How can they be measured in
their expressions?
 How can they be represented
in computer science?
 Can they be recognized using
Artificial / Web Intelligence?

Cognitive system input
+
Emotional (limbic) system input
=
Decision output
What is an «emotion»
Cognitive system
Emotional system
Cognitive system
Emotional system
situations

If there is a lion…
Cognitive system: “baby lion, maybe mother around.”
Emotional system: “AAAWW! CUTE!! I would like to touch it!”
Decision system: “better to watch from far.”

And what if…
Cognitive system: “lions running at me!”
Emotional system: “FEAR of death! RUN AWAY!”
Decision system: “EMERGENCY!”

Emotions are something that get us to
act in a particular way.
The “qualia” (feeling) is not needed to
run, but it is needed to teach and plan
for the future. It’s the internal subjective
component of sense of perception,
arising from stimulation of the senses by
phenomena
What are emotions

- Primitives (modular)
- Fast (approximate)
- Executed automatically (reactive)
- Based on environment (adaptive)
- Temporary with learning (evolutionary)
- Can be collective and contagious
- NOT usual feelings or thoughts
- NOT sentiments
- NOT moods
Emotions in AI perspective

Machine Learning and Data
Science Approaches

21 June-27 July 2017, Perugia
Emotions in Artificial Intelligence
Seminar series on Cyber-emotions
at the 6th Summer School
organized within Hong Kong
Baptist University and University of
Perugia, Italy
Organized by
Valentina Franzoni and Alfredo Milani

Web-based proximity models
General idea:
Using search engines as a source of information about the
frequency of terms (such as Google page counts)
Observation:
• approximate the probability P(w1)=f(w1)/N where
f(w1) is the frequency, N =total of indexed objects
Ex. P(gatto)=0.000679
Advantage:
• Applicable to all proximity measures based on
probability or frequency
=cat

Web-based proximity measures
to terms similarity P(x)=
f(x)
𝐍
and P(xy)=
f(xy)
𝐍
Based on probability P(x) of terms. Easy to compute by the
frequency of terms:
f(x), f(y) and f(xy) are the frequencies returned from a search
engine (e.g. Bing, Google, Yahoo, Wikipedia, Youtube) and N is
the total number of indexed documents.

Vector-space Emotion Recognition
Let E be a model of emotions EEkman={anger, disgust, fear,
joy, sadness, surprise}, then the emotional state of a term
t=“kill” is 6-dimensional vector of the proximity of t from
the basic emotions.
anger
sadness
disgust
joy
fear
surprise
kill

Set Proximity Emotion Recognition
Goal: to assess the emotional load of a sentence S by
evaluating the proximity of the set of terms to a set of
basic emotions.
• S={t1, t2, …,tn} terms in the target sentence
• E ={e1, e2, …,em} emotional model
v_S= (all i) f[( ti , e1), ( ti , e2), …, ( ti , em)]
the location of S in vector space model is a
functionof proximities from the reference
emotions of all theelements of the sentence.
•  web based proximity measure
v_emax = max{SEL j1,n {(t1, e1),…, (tn, e1)},...,SEL j1,n {(t1, em),…,(tn, em)}}

Set Proximity Emotion Recognition

Emotive Face Recognition with CNN
 Ekman basic Emotion
 Fine-tune a pre-trained neural network (e.g., AlexNet, GoogleNet)
with a set of images related to the table of emotions (transfer
learning) and save the network
 Automatically detect the position of the person's face in the camera
(e.g., via OpenCV)
 Classify the emotions
 Valid also for dogs

Emotive Face Recognition with CNN
➢ The system is reliable and
useful for the users
➢ CNN are an excellent tool for
processing images in a fast
and accurate way
➢ Even using a commodity
webcam results are very good
➢ The system is made by open
source, well documented and
maintained tools

Emotions Modeling:
The Emotographic Iceberg
Facial Expression
Context
Awareness
Prior
Experience
Words Used Body Language Voice Characteristics
Senses
Understand
Emotion

Computer-generated charisma:
Steve Jobs can take you anywhere!
Text-to-speech
synthesis: Synthetic
robot voice
combined with the
acoustic voice
profiles of Steve
Jobs
& Mark Zuckerberg

30 drivers (University students) knowing the path very well, convinced
or not to take a wrong way by the voices of Steve Jobs and Mark
Zuckerberg.

When the navigation system uses SJ‘s tone of voice…
 26.7 % of all drivers do not abort the test drive at all but follow the
wrong instructions to the very end!
 Drivers show greater trust in the system (= no error) or they do not
blame the system but others!
Perception experiments with
prosodically manipulated TTS
systems are among the first
that clearly show the power
of charismatic speech
melody in perception = not
just on listener ratings but on
listener behavior!

Errors, bias, and overconfidence
in artificial emotional models

Confusion of Emotion Recognition
with Mood or Sentiment Analysis

• Emotions are not only feelings
• Emotions are not thoughts
• Emotions are not moods
• Emotions are transient: they come
immediately from a stimulus, reach
a peak and then gradually
degrade
Confusion of Emotion Recognition with
Mood or Sentiment Analysis

Overestimation of the benefits of
emotional machines

Overconfidence in assuming the
emotional nature of “emotional”
machines
Naming «fear» an
if-then instruction
is functionally
imprecise…

Underestimation of the effects of
well-designed emotional machines
Example: prosodic speech

Underestimation of the effects of
well-designed emotional machines

Duplication of concepts reinventing
the wheel

Oversimplification of
human emotions
Human emotions can list
more than 130 variations.
+surprise=Ekman model
Plutchick model

Acultural models

Gendered models

Open Challenges

Ethical applications

Emotion Recognition for Self-Aid
Possible future directions of multidisciplinary problem-
solving opportunities for in Affective Computing and
Emotion Recognition, as they emerge from the
experience of the ACER-EMORE Workshops Series.
RESEARCH PROBLEM
Enhance user consciousness of emotional states,
ultimately support the development of self-aid
applications.

Problems and solutions
▪ Relevant application domains:
 dependencies treatment (DT) (e.g., workaholism,
sexaholism)
 non-violent communication (NVC) for people in leading
roles using e-mail or chat communication
 empathy learning for parents and teachers in the circle-
of-security (COS) caring environment
▪ Suggested solutions:
 facial ER using Convolutional Neural Networks
 semantic text ER
▪ General framework for Emotion Recognition (ER) in
Self-aid in a human-centered approach.

Why Self-Aid applications?
Ethical goals. Emotion Recognition deeply involves the
user’s intimate believes and states, sometimes even
unknown to the same user who can be unaware of
his/her emotional state.
Sound and Incremental approach. The risk is to create a
lot of models and algorithms that reach the same goal
starting from the same primary resources, instead of
advancing the already existent methods and algorithms.
Practical and Helping applications. Find practical
applications to help human beings to live better:
▪ Shortage of funds
▪ Aids for the weakest
▪ Diminish pain

General Algorithm
 classifying emotions from sensors data
 ranking the relevance of the detected
emotions concerning the application goal and
the current context
 projecting the consequence of emotions
concerning a specific AC model
 eventually triggering a user emotional
feedback in order to call for a self-aid action, if
the detected emotion appears to affect the
application goals

General System Architecture

To start mastering Emotion Recognition, or for
collaboration proposals:
Valentina Franzoni
http://orcid.org/0000-0002-2972-7188
valentina.franzoni@dmi.unipg.it
http://dblp.uni-trier.de/pers/hd/f/Franzoni:Valentina
https://www.researchgate.net/profile/Valentina_Franzoni
Subscribe to (like) the Facebook page of our group:
https://www.facebook.com/affectivecomputing

Appendix:
Achievement of our workshops

2 June 2017, Trieste
International Multidisciplinary
workshop on Emotion Recognition
(EMORE) at the 17th International
Conference on Computational
Science and Its Applications
(ICCSA 2017)
Organized by:
Valentina Franzoni, Daniele Nardi, Alfredo Milani, Jordi Vallverdú

Giulio Biondi, Valentina Franzoni, Valentina Poggioni:
A Deep Learning Semantic Approach to Emotion Recognition Using the IBM
Watson Bluemix Alchemy Language. ICCSA (3) 2017: 718-729
Valentina Franzoni, Giulio Biondi, Alfredo Milani:
A Web-Based System for Emotion Vector Extraction. ICCSA (3) 2017: 653-668
Matteo Riganelli, Valentina Franzoni, Osvaldo Gervasi, Sergio Tasso:
EmEx, a Tool for Automated Emotive Face Recognition Using Convolutional
Neural Networks. ICCSA (3) 2017: 692-704
Banu Yergesh, Gulmira Bekmanova, Altynbek Sharipbay, Manas Yergesh:
Ontology-Based Sentiment Analysis of Kazakh Sentences. ICCSA (3) 2017:669-
677
Zhiyi Ma, Marwa Mahmoud, Peter Robinson, Eduardo Dias, Lee Skrypchuk:
Automatic Detection of a Driver’s Complex Mental States. ICCSA (3) 2017:678-
691
Francesca D’Errico, Isabella Poggi:
“Humble” Politicians and Their Multimodal Communication. ICCSA (3) 2017:705-
717
Published in In Proceedings of ICCSA ’17, Trieste, Italy, July 3-6, 2017. LNCS.

14 July 2017, Perugia University
Valentina Franzoni, chair
Giulio Biondi, co-chair & techs
Jordi Vallverdú, Spain
Fabio Massimo Botti, Italy
Radoslaw Niewiadovmski, Poland
Alfredo Milani, Italy
Lalit Garg, Malta
Valentina Poggioni, Italy
Workgroup on Affective
Computing (remote kick-off)

23 August 2017, Leipzig
International Workshop on
Affective Computing and Emotion
Recognition (ACER) at the
IEEE/WIC/ACM International
Conference on Web Intelligence
(WI 2017)
Organized by:
Valentina Franzoni, Jiming Liu, Alfredo Milani, Daniele Nardi, Roberto Basili

Published in In Proceedings of WI ’17, Leipzig, Germany, August 23-26, 2017.
Valentina Franzoni, Alfredo Milani, Giulio Biondi:
SEMO: a semantic model for emotion recognition in web objects. WI 2017: 953-
958. DOI: 10.1145/3106426.3109417
Valentina Franzoni, Valentina Poggioni:
Emotional book classification from book blurbs. WI 2017: 931-938.
DOI: 10.1145/3106426.3109422
Juan Manuel Mayor Torres, Evgeny A. Stepanov: Enhanced face/audio emotion
recognition: video and instance level classification using ConvNets and
restricted Boltzmann Machines. 939-946. DOI: 10.1145/3106426.3109423
Valentina Franzoni, Alfredo Milani, Jordi Vallverdú:
Emotional affordances in human-machine interactive planning and
negotiation. WI 2017: 924-930 DOI: 10.1145/3106426.3109421
Alejandro Hernández-García, F. Fernández-Martínez, Fernando Díaz-de-María:
Emotion and attention: predicting electrodermal activity through video visual
descriptors. 914-923. DOI: 10.1145/3106426.3109418
Valentina Franzoni, Yuanxi Li, Paolo Mengoni: A path-based model for emotion
abstraction on facebook using sentiment analysis and taxonomy knowledge. WI
2017: 947-952. DOI: 10.1145/3106426.3109420

January 2018
Special issue on
Emotional Machines:
the Next Revolution
in the International Journal on
Web Intelligence (iOS press)
Guest editors:
Valentina Franzoni, Alfredo Milani, Daniele Nardi, Jordi Vallverdù

Published in «Emotional Machines: the Next Revolution», Web Intelligence 2018
https://content.iospress.com/journals/web-intelligence/17/1
Valentina Franzoni, Alfredo Milani, Daniele Nardi, Jordi Vallverdú
Emotional machines: The next revolution (editorial)
https://content.iospress.com/art…/web-intelligence/web190395
Banu Yergesh, Gulmira Bekmanova, Altynbek Sharipbay
Sentiment analysis of Kazakh text and their polarity
Osvaldo Gervasi, Valentina Franzoni, Matteo riganelli, Sergio Tasso
Automating facial emotion recognition
Álvaro García-Faura, Alejandro Hernández-García, et al.
Emotion and attention: Audiovisual models for group-level skin response
recognition in short movies
Karan Sharma, Marius Wagner, Claudio Castellini, Egon L. Van den Broek et al.
A functional data analysis approach for continuous 2-D emotion annotations
Alessandro Ansani, Francesca D’Errico, Isabella Poggi
‘You will be judged by the music I hear’: A study on the influence of music on
moral judgement
Francesca D’Errico, Isabella Poggi
Tracking a leader’s humility and its emotions from body, face and voice
Andrea Gorrini, Luca Crociani, Giuseppe Vizzari, Stefania Baldini
Stress estimation in pedestrian crowds: Experimental data and simulations results

3 July 2018, Sidney, Australia
(EMORE) at the 18th International
Conference on Computational
Science and Its Applications
(ICCSA 2018)
Organized by:
Valentina Franzoni, Alfredo Milani
Co-located with WCES

3 July 2019, Saint Petersburg,
Russia
and Affective Computing
(ACER/EMORE) at the 19th
International Conference on
Computational Science and Its
Applications (ICCSA 2019)
Organized by:
Valentina Franzoni, Alfredo Milani, Giulio Biondi

Valentina Franzoni, Alfredo Milani:
Emotion Recognition for Self-aid in Addiction Treatment, Psychotherapy, and
Nonviolent Communication. ICCSA (2) 2019: 391-404
Giulio Biondi, Valentina Franzoni, Osvaldo Gervasi, Damiano Perri:
An Approach for Improving Automatic Mouth Emotion Recognition. ICCSA (1)
2019: 649-664
Oliver Niebuhr, Jan Michalsky:
Computer-Generated Speaker Charisma and Its Effects on Human Actions in a
Car-Navigation System Experiment - or How Steve Jobs' Tone of Voice Can
Take You Anywhere. ICCSA (2) 2019: 375-390
Francesca D'Errico, Oliver Niebuhr, Isabella Poggi:
Humble Voices in Political Communication: A Speech Analysis Across Two
Cultures. ICCSA (2) 2019: 361-374
Chutisant Kerdvibulvech, Sheng-Uei Guan:
Affective Computing for Enhancing Affective Touch-Based Communication
Through Extended Reality. ICCSA (2) 2019: 351-360
Eoghan Furey, Juanita Blue:
The Emotographic Iceberg: Modelling Deep Emotional Affects Intelligent
Assistants & the IoT. ICCSA (7) 2019: 175-180
Published in In Proceedings of ICCSA’19, S.Petersburg Russia, July 3-6, 2019. LNCS.

14 Oct 2019, Thessaloniki,
Greece
International workshop on
Affective Computing and
Emotion Recognition (ACER)
at ACM/IEEE/WIC Web
Intelligence Conference
(WI2019)
Organized by:
Valentina Franzoni, Alfredo Milani, Daniele Nardi, Jordi Vallverdù

Valentina Franzoni, Jordi Vallverdú, Alfredo Milani:
Errors, Biases and Overconfidence in Artificial Emotional Modeling.
WI (Companion) 2019: 86-90
Valentina Franzoni, Alfredo Milani, Giulio Biondi, Francesco Micheli:
A Preliminary Work on Dog Emotion Recognition.
Sergio Angelastro, Berardina De Carolis, Stefano Ferilli:
Learning and Predicting User Pairwise Preferences from Emotions and Gaze
Behavior.
Berardina De Carolis, Francesca D'Errico, Nicola Macchiarulo, Giuseppe
Palestra:
"Engaged Faces": Measuring and Monitoring Student Engagement from Face
and Gaze Behavior.
Published in In Proceedings of WI ’19, Thessaloniki, Greece, October 14, 2019.

6 Oct 2019, Bari, Italy
International Workshop on Socio-
Affective Technologies: an
interdisciplinary approach (SAT)
at IEEE Systems, Man, and
Cybernetics Conference
(SMC 2019)
Organized by:
Berardina De Carolis, Francesca D’Errico, Veronica Rossano

Valentina Franzoni, Giulio Biondi, Alfredo Milani: Crowd emotional sounds: spectrogram-
based analysis using convolutional neural network. SAT@SMC 2019: 32-36
Concetta Papapicco, Giuseppe Mininni: Impact memes: PhDs humor(e). SAT@SMC 2019:
1-6
Giovanni Boccia Artieri, Gevisa La Rocca: The election day of Pope Francis: between
sentiment and emotions online. SAT@SMC 2019: 7-12
Marinella Paciello, Francesca D'Errico, Giorgia Saleri: Moral struggles in social media
discussion: the case of sexist aggression. SAT@SMC 2019: 13-16
Francesca D'Errico, Manuel Martinez, Carmen D'Anna, Maurizio Schmid, Stella
Mastrobattista, Raffaella Parlongo, Christian Massom: 'Prosocial' virtual reality as tool for
monitoring engagement in intergroup helping situations. SAT@SMC 2019: 17-21
Fabrizio Balducci, Berardina De Carolis, Donato Impedovo, Giuseppe Pirlo: Touch
dynamics for affective states recognition: your smartphone knows how you feel since you
unlock it. SAT@SMC 2019: 22-26
Anne-Marie Brouwer, Ivo V. Stuldreher, Nattapong Thammasan: Shared attention
reflected in EEG, electrodermal activity and heart rate. SAT@SMC 2019: 27-31
Mauro Gaspari, Margherita Donnici: Weekend in Rome: a cognitive training exercise
based on planning. SAT@SMC 2019: 37-41
Berardina De Carolis, Giuseppe Palestra, Olimpia Pino: Facial expression recognition from
nao robot within a memory training program for individuals with mild cognitive
impairment. SAT@SMC 2019: 42-46
Published in In Proceedings of SAT ’19, Bari, Italy, October 6, 2019.

3 July 2020, Cagliari, Italy (online
for Covid-19)
and Affective Computing
(ACER/EMORE) at the 20th
International Conference on
Computational Science and Its
Applications (ICCSA 2020)
Organized by:
Valentina Franzoni, Alfredo Milani, Giulio Biondi

These slides are distributed
under the Creative Commons
licence:
CC BY-SA 4.0
You can use, share and modify
them under the constraints to
cite the original author and
share with the same licence:
https://creativecommons.org
/licenses/by-sa/4.0/deed.en

Emotion Recognition in Artificial Intelligence by Valentina Franzoni, Ph.D. in Engineering for Computer Science, Research Fellow in Artificial Intelligence, adjunct professor in Operative Systems

Recomendados

Recomendados

Más contenido relacionado

Más de Paris Women in Machine Learning and Data Science

Más de Paris Women in Machine Learning and Data Science (20)

Último

Último (20)

Emotion Recognition in Artificial Intelligence by Valentina Franzoni, Ph.D. in Engineering for Computer Science, Research Fellow in Artificial Intelligence, adjunct professor in Operative Systems