SlideShare a Scribd company logo
1 of 19
Download to read offline
Thrust 2: Interaction Modeling
(2013-14 academic years)
Defined last year by Shri:
1. Analyze and model spatio-temporal context
surrounding (leading to, caused by) behavioral
events.
2. Quantitative description and understanding of the
underlying structure (e.g., timing, phasing of
salient events) and emergence and evolution of
interactions.
Progress since Last Thrust 2 Con Call:
(April 4, 2013)
1. CVPR paper
2. Nine Expeditions-related papers (or more?)
in Interspeech 2013 (Lyon, France)
(some are Thrust 2)
3. Synchronization issues for multimodal RABC
data-base (all but Kinect).
Thrust 1 / Thrust 2 dependencies
• Thrust 1 is concerned with extracting
high/low-level features for further analysis.
• Thrust 1’s raison d’être is to serve Thrust 2
with descriptors.
• Thrust 2‘s obligation is to convey its needs to
Thrust 1.
• We have already developed single-mode
processing to analyze behavior; now it is time
to use multi-mode methods.
Thrust 1 / Thrust 2 dependencies
• We need a comprehensive list of Thrust 1
deliverables.
1. Speech and Audio examples:
• Voicing intervals
• Diarization (who is talking when with cross-talk)
• Word locations (KWS)
• Non-linguistic events (laughter, whining, crying, sighs)
• Prosodic features
• Spectral/articulatory features
2. Q-sensors
• EDA
• Temperature
• Accelerometers
• Statistical measures associated with various states and timings
Thrust 1 / Thrust 2 dependencies
• We need a comprehensive list of Thrust 1
deliverables.
3. Vision
• …. (many)
4. Gaze
• --- (many)
5. Kinect
• --- (many)
Thrust 1: Please fill this out for us.
Thrust 2 projects using RABC
(all multi-modal)
1. Engagement score analysis (revisited)
– We have engagement scores on all 157 sessions
(only 43 are fully annotated).
– Develop optimal categorization using “ground
truth” in the annotated sessions.
– Test using non-annotated sessions with Thrust 1
primitives.
Thrust 2 projects using RABC
(all multi-modal)
2. Parsing of stages (revisited)
– The only attempt was using KWS (~80% correct)
– Obvious extension uses Kinect object tracking
– Others include EDA, para-linguistic detection, gaze,
posture, etc.
Thrust 2 projects using RABC
(all multi-modal)
3. Response to name:
– Every RABC session starts with greeting.
– After formal session, child’s name is uttered
twice (once from side, once from front)
a. What responses are predictable? (All modes)
b. Can we ascertain when name was called from
child’s reactions w/o audio cues?
c. This will require gaze, head-orientation, EDA,
posture, etc.
Multimodal Retrieval
• We currently have the ability to retrieve
interesting regions of RABC session through key-
word spotting and diarization.
• Easy extension using any of the other
modalities.
• Challenging extensions would be:
– Find when child is unhappy (requires multi-modes)
– Find when child is not responding.
– …
Multimodal Retrieval
Examples:
Is a child making eye contact while an examiner
talking? (gaze and diarization)
Where is a child looking at while an examiner
talking? (voice active detector + gaze tracker +
head orientation).
Multimodal Linguistic Analysis
Classify examiner’s speech into the four sentence types:
1. Statement
2. Question (“Are you ready to play with new toys?”,
“where is the yellow duck?”, etc.)
3. Exclamation (“It’s a hat! It’s on my head!”)
4. Command (“Look at my book”)
• Useful for retrieval and parsing.
• Child’s responses and turn-taking to each type can be
tracked.
What Happens During Cross-Talk?
1) terminal overlaps: a speaker assumes the other speaker
has or is about to finish their turn. Can include head-
nodding and gestures.
2) continuers: examples of the continuer’s phrases are
“mm hm” or “uh huh.” (speech and para-linguistics)
3) conditional access to the turn: the current speaker
yields their turn or invite another speaker to interject in
the conversation, usually as collaborative effort. (all sorts
of gestures)
4) chordal: a non-serial occurrence of turns, such as
laughter, smiling, and gesturing.
Example of Multi-Modal Retrieval/Analysis
During the ball/book play in RABC, what is a child
response when a ball/book is presented? Is a child
looking at the ball/book and the examiner back and forth?
(word spotting + gaze tracker + head orientation).
Is a child making vocalizations? If so, is it a positive or
negative response? (word spotting + emotion classifier)
Does a child smile when a ball/book is presented? (word
spotting + smile detector)
Previous Project Example
Uni-modal paralinguistic event detection
– Detection of laughter in children’s speech
– Training data from FAU Aibo Emotion Corpus (AEC)
consists of spontaneous vocalizations/verbalizations by
adolescents
– Testing on ~10 Rapid ABC sessions yielded accuracy of
70.58%
– Robust laughter detector with good generalization
properties that can be used for paralinguistic event
diarization
Current Project Example
Multi-modal paralinguistic event detection
– Laughter detection using audio
– Smile detection using Omron.
– These have different time scales and frame rates.
– Fusion of confidence scores is likely to improve
accuracy.
– Might EDA also be useful for this?
Basic Science in Required for Multi-Modal
Analyses: Fusion
Example from USC
Basic Science in Required for Multi-Modal
Analyses: Fusion
Example from GT: binary fusion method with different analysis lengths
Opportunistic use of other data as it may
become available
• BU children’s diagnostic data. (Helen Tager-Flusberg)
– Includes audio and video of at-risk ASD children with more
vocalizations.
• RABC Floor time (natural but very uncontrolled)
• ADOS (only available to USC)
• CFD table sessions.
Thrust 2 action items
• Set up blog like Thrust 1
• Identify who is participating
• Monthly con-call for everyone.
• Set up time-table for new / revisited
challenges.

More Related Content

Similar to Thrust 2 07-aug-2013_v3 (1)

Ud the motion en_jtaboada
Ud the motion en_jtaboadaUd the motion en_jtaboada
Ud the motion en_jtaboadaJordi Taboada
 
2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...
2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...
2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...eMadrid network
 
Pre, while and post listening skills and activities
Pre, while and post listening skills and activitiesPre, while and post listening skills and activities
Pre, while and post listening skills and activitiesAlexander Benito
 
Whakanuia te rerekētanga - Integrated Science plan Year 6-8
Whakanuia te rerekētanga - Integrated Science plan Year 6-8Whakanuia te rerekētanga - Integrated Science plan Year 6-8
Whakanuia te rerekētanga - Integrated Science plan Year 6-8Ruth Lemon
 
Week 7 Lecture Slides
Week 7 Lecture SlidesWeek 7 Lecture Slides
Week 7 Lecture SlidesSerpil
 
LARC: Lesson analysis, Differentiation, Assessment 2011
LARC: Lesson analysis, Differentiation, Assessment  2011LARC: Lesson analysis, Differentiation, Assessment  2011
LARC: Lesson analysis, Differentiation, Assessment 2011Toni Theisen
 
Meghan Shank: How can we be better communication partners?
Meghan Shank: How can we be better communication partners?Meghan Shank: How can we be better communication partners?
Meghan Shank: How can we be better communication partners?Ursula Webhofer
 
individual assessment of each child -etwinning -young scientists
individual assessment of each child -etwinning -young scientists individual assessment of each child -etwinning -young scientists
individual assessment of each child -etwinning -young scientists Sofi Liva
 
Presentation: GETSI-InTeGrate Development Model & Writing Learning Goals
Presentation: GETSI-InTeGrate Development Model & Writing Learning GoalsPresentation: GETSI-InTeGrate Development Model & Writing Learning Goals
Presentation: GETSI-InTeGrate Development Model & Writing Learning GoalsSERC at Carleton College
 
Problem-based Learning & Resource-based Learning two complementary approac...
Problem-based Learning & Resource-based Learning  two complementary approac...Problem-based Learning & Resource-based Learning  two complementary approac...
Problem-based Learning & Resource-based Learning two complementary approac...Wilco te Winkel
 
Educ tech lesson 11 12 13 14 15 16 & 18 copy
Educ tech lesson 11 12 13 14 15 16 & 18   copyEduc tech lesson 11 12 13 14 15 16 & 18   copy
Educ tech lesson 11 12 13 14 15 16 & 18 copyAngel Yuto
 
Sensemaker for Partos Plaza - Irene Guyt
Sensemaker for Partos Plaza  - Irene GuytSensemaker for Partos Plaza  - Irene Guyt
Sensemaker for Partos Plaza - Irene Guytannepartos
 
Ccss ppt
Ccss pptCcss ppt
Ccss pptsueola1
 
From Human Intelligence to Machine Intelligence
From Human Intelligence to Machine IntelligenceFrom Human Intelligence to Machine Intelligence
From Human Intelligence to Machine IntelligenceNUS-ISS
 
IntroductionIn this module, you have learned about the intention.docx
IntroductionIn this module, you have learned about the intention.docxIntroductionIn this module, you have learned about the intention.docx
IntroductionIn this module, you have learned about the intention.docxdoylymaura
 
Sequencingr middleschool
Sequencingr middleschoolSequencingr middleschool
Sequencingr middleschoolKelly Kellogg
 
Theories of student development.chapter.2
Theories of student development.chapter.2Theories of student development.chapter.2
Theories of student development.chapter.2jvirwin
 
The Power of Retrospection
The Power of RetrospectionThe Power of Retrospection
The Power of RetrospectionNaresh Jain
 
Atl structure by Josephine
Atl structure by JosephineAtl structure by Josephine
Atl structure by Josephinesrjosephine1979
 

Similar to Thrust 2 07-aug-2013_v3 (1) (20)

Ud the motion en_jtaboada
Ud the motion en_jtaboadaUd the motion en_jtaboada
Ud the motion en_jtaboada
 
2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...
2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...
2011 10 07 (uam) emadrid aortigosa uam estilos aprendizaje sistemas adaptativ...
 
Pre, while and post listening skills and activities
Pre, while and post listening skills and activitiesPre, while and post listening skills and activities
Pre, while and post listening skills and activities
 
Whakanuia te rerekētanga - Integrated Science plan Year 6-8
Whakanuia te rerekētanga - Integrated Science plan Year 6-8Whakanuia te rerekētanga - Integrated Science plan Year 6-8
Whakanuia te rerekētanga - Integrated Science plan Year 6-8
 
Week 7 Lecture Slides
Week 7 Lecture SlidesWeek 7 Lecture Slides
Week 7 Lecture Slides
 
LARC: Lesson analysis, Differentiation, Assessment 2011
LARC: Lesson analysis, Differentiation, Assessment  2011LARC: Lesson analysis, Differentiation, Assessment  2011
LARC: Lesson analysis, Differentiation, Assessment 2011
 
Meghan Shank: How can we be better communication partners?
Meghan Shank: How can we be better communication partners?Meghan Shank: How can we be better communication partners?
Meghan Shank: How can we be better communication partners?
 
Unit1 revision
Unit1 revisionUnit1 revision
Unit1 revision
 
individual assessment of each child -etwinning -young scientists
individual assessment of each child -etwinning -young scientists individual assessment of each child -etwinning -young scientists
individual assessment of each child -etwinning -young scientists
 
Presentation: GETSI-InTeGrate Development Model & Writing Learning Goals
Presentation: GETSI-InTeGrate Development Model & Writing Learning GoalsPresentation: GETSI-InTeGrate Development Model & Writing Learning Goals
Presentation: GETSI-InTeGrate Development Model & Writing Learning Goals
 
Problem-based Learning & Resource-based Learning two complementary approac...
Problem-based Learning & Resource-based Learning  two complementary approac...Problem-based Learning & Resource-based Learning  two complementary approac...
Problem-based Learning & Resource-based Learning two complementary approac...
 
Educ tech lesson 11 12 13 14 15 16 & 18 copy
Educ tech lesson 11 12 13 14 15 16 & 18   copyEduc tech lesson 11 12 13 14 15 16 & 18   copy
Educ tech lesson 11 12 13 14 15 16 & 18 copy
 
Sensemaker for Partos Plaza - Irene Guyt
Sensemaker for Partos Plaza  - Irene GuytSensemaker for Partos Plaza  - Irene Guyt
Sensemaker for Partos Plaza - Irene Guyt
 
Ccss ppt
Ccss pptCcss ppt
Ccss ppt
 
From Human Intelligence to Machine Intelligence
From Human Intelligence to Machine IntelligenceFrom Human Intelligence to Machine Intelligence
From Human Intelligence to Machine Intelligence
 
IntroductionIn this module, you have learned about the intention.docx
IntroductionIn this module, you have learned about the intention.docxIntroductionIn this module, you have learned about the intention.docx
IntroductionIn this module, you have learned about the intention.docx
 
Sequencingr middleschool
Sequencingr middleschoolSequencingr middleschool
Sequencingr middleschool
 
Theories of student development.chapter.2
Theories of student development.chapter.2Theories of student development.chapter.2
Theories of student development.chapter.2
 
The Power of Retrospection
The Power of RetrospectionThe Power of Retrospection
The Power of Retrospection
 
Atl structure by Josephine
Atl structure by JosephineAtl structure by Josephine
Atl structure by Josephine
 

Recently uploaded

Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 

Recently uploaded (20)

Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 

Thrust 2 07-aug-2013_v3 (1)

  • 1. Thrust 2: Interaction Modeling (2013-14 academic years) Defined last year by Shri: 1. Analyze and model spatio-temporal context surrounding (leading to, caused by) behavioral events. 2. Quantitative description and understanding of the underlying structure (e.g., timing, phasing of salient events) and emergence and evolution of interactions.
  • 2. Progress since Last Thrust 2 Con Call: (April 4, 2013) 1. CVPR paper 2. Nine Expeditions-related papers (or more?) in Interspeech 2013 (Lyon, France) (some are Thrust 2) 3. Synchronization issues for multimodal RABC data-base (all but Kinect).
  • 3. Thrust 1 / Thrust 2 dependencies • Thrust 1 is concerned with extracting high/low-level features for further analysis. • Thrust 1’s raison d’être is to serve Thrust 2 with descriptors. • Thrust 2‘s obligation is to convey its needs to Thrust 1. • We have already developed single-mode processing to analyze behavior; now it is time to use multi-mode methods.
  • 4. Thrust 1 / Thrust 2 dependencies • We need a comprehensive list of Thrust 1 deliverables. 1. Speech and Audio examples: • Voicing intervals • Diarization (who is talking when with cross-talk) • Word locations (KWS) • Non-linguistic events (laughter, whining, crying, sighs) • Prosodic features • Spectral/articulatory features 2. Q-sensors • EDA • Temperature • Accelerometers • Statistical measures associated with various states and timings
  • 5. Thrust 1 / Thrust 2 dependencies • We need a comprehensive list of Thrust 1 deliverables. 3. Vision • …. (many) 4. Gaze • --- (many) 5. Kinect • --- (many) Thrust 1: Please fill this out for us.
  • 6. Thrust 2 projects using RABC (all multi-modal) 1. Engagement score analysis (revisited) – We have engagement scores on all 157 sessions (only 43 are fully annotated). – Develop optimal categorization using “ground truth” in the annotated sessions. – Test using non-annotated sessions with Thrust 1 primitives.
  • 7. Thrust 2 projects using RABC (all multi-modal) 2. Parsing of stages (revisited) – The only attempt was using KWS (~80% correct) – Obvious extension uses Kinect object tracking – Others include EDA, para-linguistic detection, gaze, posture, etc.
  • 8. Thrust 2 projects using RABC (all multi-modal) 3. Response to name: – Every RABC session starts with greeting. – After formal session, child’s name is uttered twice (once from side, once from front) a. What responses are predictable? (All modes) b. Can we ascertain when name was called from child’s reactions w/o audio cues? c. This will require gaze, head-orientation, EDA, posture, etc.
  • 9. Multimodal Retrieval • We currently have the ability to retrieve interesting regions of RABC session through key- word spotting and diarization. • Easy extension using any of the other modalities. • Challenging extensions would be: – Find when child is unhappy (requires multi-modes) – Find when child is not responding. – …
  • 10. Multimodal Retrieval Examples: Is a child making eye contact while an examiner talking? (gaze and diarization) Where is a child looking at while an examiner talking? (voice active detector + gaze tracker + head orientation).
  • 11. Multimodal Linguistic Analysis Classify examiner’s speech into the four sentence types: 1. Statement 2. Question (“Are you ready to play with new toys?”, “where is the yellow duck?”, etc.) 3. Exclamation (“It’s a hat! It’s on my head!”) 4. Command (“Look at my book”) • Useful for retrieval and parsing. • Child’s responses and turn-taking to each type can be tracked.
  • 12. What Happens During Cross-Talk? 1) terminal overlaps: a speaker assumes the other speaker has or is about to finish their turn. Can include head- nodding and gestures. 2) continuers: examples of the continuer’s phrases are “mm hm” or “uh huh.” (speech and para-linguistics) 3) conditional access to the turn: the current speaker yields their turn or invite another speaker to interject in the conversation, usually as collaborative effort. (all sorts of gestures) 4) chordal: a non-serial occurrence of turns, such as laughter, smiling, and gesturing.
  • 13. Example of Multi-Modal Retrieval/Analysis During the ball/book play in RABC, what is a child response when a ball/book is presented? Is a child looking at the ball/book and the examiner back and forth? (word spotting + gaze tracker + head orientation). Is a child making vocalizations? If so, is it a positive or negative response? (word spotting + emotion classifier) Does a child smile when a ball/book is presented? (word spotting + smile detector)
  • 14. Previous Project Example Uni-modal paralinguistic event detection – Detection of laughter in children’s speech – Training data from FAU Aibo Emotion Corpus (AEC) consists of spontaneous vocalizations/verbalizations by adolescents – Testing on ~10 Rapid ABC sessions yielded accuracy of 70.58% – Robust laughter detector with good generalization properties that can be used for paralinguistic event diarization
  • 15. Current Project Example Multi-modal paralinguistic event detection – Laughter detection using audio – Smile detection using Omron. – These have different time scales and frame rates. – Fusion of confidence scores is likely to improve accuracy. – Might EDA also be useful for this?
  • 16. Basic Science in Required for Multi-Modal Analyses: Fusion Example from USC
  • 17. Basic Science in Required for Multi-Modal Analyses: Fusion Example from GT: binary fusion method with different analysis lengths
  • 18. Opportunistic use of other data as it may become available • BU children’s diagnostic data. (Helen Tager-Flusberg) – Includes audio and video of at-risk ASD children with more vocalizations. • RABC Floor time (natural but very uncontrolled) • ADOS (only available to USC) • CFD table sessions.
  • 19. Thrust 2 action items • Set up blog like Thrust 1 • Identify who is participating • Monthly con-call for everyone. • Set up time-table for new / revisited challenges.