This document discusses using brain signals and non-conventional means for media content analysis and understanding. It notes that the human brain is still more efficient than computers for some tasks like media content analysis. The approach proposed is to use the human brain as a co-processor by applying brain-computer interfaces and social networks to content analysis and annotation. Examples provided include curiosity cloning for deep space exploration by training classifiers on images rated by experts to program a robot's interests, and emotional tagging of media content using EEG signals to classify emotions like valence and arousal. Challenges mentioned include developing more mature and efficient multimodal solutions combining multiple biosignals.
1. 1
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
The Use of Non-Conventional
Means for Media Content
Analysis and Understanding
- Brain Signals -
Touradj Ebrahimi
2. 2
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Observation
• Human brain is still a more efficient
processor for some tasks when compared
to computers
– Media content analysis and understanding
3. 3
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Approach
• Use human brain as a co-processor for
advanced content analysis
– Social networks applied to content analysis
and annotation
– Brain Computer Interface
4. 4
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
BCI versus social tagging
• Social tagging
– Explicit
– Verbal
• BCI
– Implicit
– Non-verbal
5. 5
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
A couple interesting illustrations
• Curiosity cloning
• Emotional tagging
6. 6
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Curiosity cloning in deep space exploration
• Pure pattern matching, Scientific Richness Index or other
classifiers are programmed to find what we already know: the
expected.
Q: Can we code the interest in the unexpected?
(Scientific) Curiosity?
7. 7
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Curiosity cloning in deep space exploration
• An alternative to explicit and specific definition of what we
are looking for ...(e.g., dust-devils)
• Present to experts (e.g., experts on mars geology) a lot of
images and rate them
• Images/Rating pairs can form a training set for a classifier
• The classifier could be programmed on a rover
• Robot’s processor would be a “clone” of the scientist’s
interest, curiosity, expertise
8. 8
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Specialist versus naïve subjects EEG
Averaged PZ electrodes for subjects. Top left: specialist, Others: naïve subjects. Horizontal
axis is the time after stimulus onset and vertical axis amplitude of the P300 signal.
Scientifically interesting Target (red), Non-Target (blue) , Non-obvious target (dashed black)
stimulus.
9. 9
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
BCI for Emotional tagging
AnalysisF
r
Biosignals
acquisition
EEG signals
Feature
vector
Single window
Classification
Classifier
Single window
Aggregation
12. 12
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
EEG Single Trial Classification
• Goal: Predict valence, arousal and stance for each video.
• Threshold subjects' arousal/valence/stance into two classes (e.g.
positive or negative arousal)
• Extract features using common spatial patterns algorithms
• Use linear SVM classifier for classification.
• Segment each video into 10 samples and test using leave-one-
video-out cross-validation.
13. 13
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Results
a
b
a) Single trial classification b) aggregated result
14. 14
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Challenges
• More mature and more efficient solutions
– Fortunately, our community is good at it!
15. 15
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Challenges
• Multimodal approach
EEG Sensor cap
Plethysmograph (bloodflow)
Galvanic skin response
Heart rate
Temperature sensor
Respiration sensor
16. 16
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Signal aquistion
Emotiv Epoc
Dry EEG electrode headset
Designed for games
TU Eindhoven concept
Dry EEG electrode diadem
For rehabilitation
EEG baseball cap
Dry EEG electrode hat
For everyday use
OCZ NIA
Dry EMG+EEG electrode headband For gaming
Integration into headphones
17. 17
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Future biosignals acquisition
Mit MediaLab HandWave
Bluetooth GSR sensors
For man-machine interaction
SenseWear
Measures temperature,
heat flux, GSR, movement
For medical applications
Fraunhofer institute EmoGlove
Measures heart rate/GSR/temperature For man-machine interaction
Biosensor mouse
Measures GSR in the thumb
For stress sensing in offices