My talk at the ACM Multimedia 2010 panel on The Use of Non-conventional Means for Media Content Analysis and Understanding
•
2 recomendaciones•567 vistas
This document discusses using brain signals and non-conventional means for media content analysis and understanding. It notes that the human brain is still more efficient than computers for some tasks like media content analysis. The approach proposed is to use the human brain as a co-processor by applying brain-computer interfaces and social networks to content analysis and annotation. Examples provided include curiosity cloning for deep space exploration by training classifiers on images rated by experts to program a robot's interests, and emotional tagging of media content using EEG signals to classify emotions like valence and arousal. Challenges mentioned include developing more mature and efficient multimodal solutions combining multiple biosignals.
Recomendados
Recomendados
Más contenido relacionado
Destacado
Destacado (6)
Similar a My talk at the ACM Multimedia 2010 panel on The Use of Non-conventional Means for Media Content Analysis and Understanding
Similar a My talk at the ACM Multimedia 2010 panel on The Use of Non-conventional Means for Media Content Analysis and Understanding (20)
Más de Touradj Ebrahimi
Más de Touradj Ebrahimi (20)
Último
Último (20)
My talk at the ACM Multimedia 2010 panel on The Use of Non-conventional Means for Media Content Analysis and Understanding
- 1. 1 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne The Use of Non-Conventional Means for Media Content Analysis and Understanding - Brain Signals - Touradj Ebrahimi
- 2. 2 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Observation • Human brain is still a more efficient processor for some tasks when compared to computers – Media content analysis and understanding
- 3. 3 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Approach • Use human brain as a co-processor for advanced content analysis – Social networks applied to content analysis and annotation – Brain Computer Interface
- 4. 4 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne BCI versus social tagging • Social tagging – Explicit – Verbal • BCI – Implicit – Non-verbal
- 5. 5 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne A couple interesting illustrations • Curiosity cloning • Emotional tagging
- 6. 6 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Curiosity cloning in deep space exploration • Pure pattern matching, Scientific Richness Index or other classifiers are programmed to find what we already know: the expected. Q: Can we code the interest in the unexpected? (Scientific) Curiosity?
- 7. 7 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Curiosity cloning in deep space exploration • An alternative to explicit and specific definition of what we are looking for ...(e.g., dust-devils) • Present to experts (e.g., experts on mars geology) a lot of images and rate them • Images/Rating pairs can form a training set for a classifier • The classifier could be programmed on a rover • Robot’s processor would be a “clone” of the scientist’s interest, curiosity, expertise
- 8. 8 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Specialist versus naïve subjects EEG Averaged PZ electrodes for subjects. Top left: specialist, Others: naïve subjects. Horizontal axis is the time after stimulus onset and vertical axis amplitude of the P300 signal. Scientifically interesting Target (red), Non-Target (blue) , Non-obvious target (dashed black) stimulus.
- 9. 9 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne BCI for Emotional tagging AnalysisF r Biosignals acquisition EEG signals Feature vector Single window Classification Classifier Single window Aggregation
- 10. 10 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Music: The Finest Language of Emotion
- 11. 11 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Emotion representation
- 12. 12 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne EEG Single Trial Classification • Goal: Predict valence, arousal and stance for each video. • Threshold subjects' arousal/valence/stance into two classes (e.g. positive or negative arousal) • Extract features using common spatial patterns algorithms • Use linear SVM classifier for classification. • Segment each video into 10 samples and test using leave-one- video-out cross-validation.
- 13. 13 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Results a b a) Single trial classification b) aggregated result
- 14. 14 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Challenges • More mature and more efficient solutions – Fortunately, our community is good at it!
- 15. 15 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Challenges • Multimodal approach EEG Sensor cap Plethysmograph (bloodflow) Galvanic skin response Heart rate Temperature sensor Respiration sensor
- 16. 16 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Signal aquistion Emotiv Epoc Dry EEG electrode headset Designed for games TU Eindhoven concept Dry EEG electrode diadem For rehabilitation EEG baseball cap Dry EEG electrode hat For everyday use OCZ NIA Dry EMG+EEG electrode headband For gaming Integration into headphones
- 17. 17 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Future biosignals acquisition Mit MediaLab HandWave Bluetooth GSR sensors For man-machine interaction SenseWear Measures temperature, heat flux, GSR, movement For medical applications Fraunhofer institute EmoGlove Measures heart rate/GSR/temperature For man-machine interaction Biosensor mouse Measures GSR in the thumb For stress sensing in offices
- 18. 18 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne More during discussions