3. Why emotion recognition?
• Need more effective human-computer interaction
methods in more intelligent systems
• Video Games which depend on users’ emotions
• Educational packages which depend on users’ emotions
• Healthcare applications for treating patients with PTSD or
depressions.
• Counseling and therapy applications that would help one
to optimize psychologicalogical treatment in patients
• Musical applications to autonomously play the right music
for the user at a certain moment.
4. Limitation of Existing Methods
• Still image emotion classifiers suffers from
weaknesses such as bad lighting, the subject’s
unique appearance, and unique facial expression.
• Other classifiers assume the presence of a fully
frontal face location with previous knowledge of
global facial landmark location.
5. Dense Optical Flow-Based Emotional Recognition
Consists of Training and Testing Phase:
During Training
• we extract dense optical flow-based features from labeled videos of users’
different emotions.
• Use extracted features to train a SVM-based classifier.
During Testing
• Using a sliding window of 15 frames, determine if large displacement of facial
movements can be identified from a video of a user’s unknown emotion
• Then, extract dense optical flow-based features and feed into trained SVM
classifier to determine the emotion type
Training Phase
Testing Phase
6. Dense Optical Flow
• Uses Gunner Farneback's optical flow algorithm to track the
facial movements.
• Accomplishes this by checking the surrounding of each
optical flow point in the grid.
• Relies on the contrast on each frame where if there is a
difference in contrast, that indicates that there are
movement.
7. Advantages of our approach
• Analyzing facial movements in videos as oppose to still
images.
• This method is less vulnerable to numerous factors such
as lighting and different appearance of each people
• Our approach makes it easier to classify emotions since
people generally have similar facial movements and
expression patterns for each emotion
8. Details of Evaluation Experiment
l We collected facial data from all of our subjects
l Perform dense optical flow analysis on the
images
l Train the emotion classifier with SVM with the
optical flow point data
l Predict someone’s emotion by recording them
and analyzing dense optical flow movement.
9. Data Collection
l Asked 6 volunteers to allow us to take pictures of them
with four posed emotions.
l Produced own dataset of 372 videos of expressing each
emotion from 6 different people.
l Emotion labels are neutral, happy, surprised, and angry
l More emotions can be easily added by adding more
video samples
l Data was diverse enough so that each emotion had
different facial movements.
10. Evaluation Results
• Successfully recognized the emotions of a large
majority of the test subjects.
• With a large and diverse dataset, the classifier
achieved an accuracy range of 82-90% with
cross-validation.
• The confusion matrix tells us that the classifier
has a tendency to over guess neutral due to small
facial movements in other emotions. However
sensitive mode overcomes this.
12. Amount of flow points
l We tested to see how our normal 26,912 optical
flow point emotion classifier compares to one with
10,000 optical flow points.
l The one with 26,912 flow points seemed to have
better performance with an accuracy of 81.75%
compared to the other classifier's 74.375%
l
14. Generic vs Per-User Classifier
• Also evaluate differences between a per-user and
generic classifier
• Per-user classifier can achieve an accuracy in the
range of 81-87% compared to the 81.25% with
generic classifier.
l
15. Future Work
• Investigate if using a combination of different
classifiers with each classifier focusing on facial
movements in a particular region e.g. eyebrows or
mouth) gives higher accuracy.
• Investigate the robustness of our scheme when a
user’s facial expression is captured at different
distance from the camera e.g. near or far views,
front or side view of a user’s face.
16. Video of emotion classifier
https://www.youtube.com/watch?v=PM_k_zACWLE
17. Packages used
l OpenCV Python
l Computer Vision tasks such as haar cascade and
dense optical flow
l Scikit-Learn
l Machine Learning tasks such as Support Vector
Machines
l Numpy, matplotlib, etc
l Work with matrices and analyze data
l LibSVM (will add more info once I used it)
