[Webinar] SpiraTest - Setting New Standards in Quality Assurance
Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation
1. 2 December 2005
Fusion in Multimodal Interactive Systems:
An HMM-Based Algorithm for User-Induced
Adaptation
Bruno Dumas1, Beat Signer1 and Denis Lalanne2
1 WISE Research Lab, Vrije Universiteit Brussel, Belgium
2 DIVA Research Group, University of Fribourg, Switzerland
2. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 2
June 26, 2012
Outline
• Fusion of multimodal input
• Multimodal interaction: from design to implementation
• Algorithms for fusion of multimodal input
• A Hidden Markov Model-based fusion algorithm
• Instantiation of the algorithm
• Evaluation of the algorithm
• Qualitative test
• Quantitative evaluation
• Performance assessment
• Conclusion
3. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 3
June 26, 2012
Fusion of Multimodal Input
Challenges in multimodal interaction engineering:
continous and probabilistic inputs, real-time…
4. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 4
June 26, 2012
Challenging/Ambiguous Fusion Cases
• “Put-that-there”
Complementarity case
• “Play next track”
Different meanings based on order and context
5. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 5
June 26, 2012
CARE Properties at the Design Level
Describe how modalities can be combined
Complementarity
All modalities are necessary
Assignment
Absence of choice
Redundancy
Same expressive power between modalities
Equivalence
Any modality is sufficient
6. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 6
June 26, 2012
HephaisTK Multimodal Framework
7. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 7
June 26, 2012
Description of Multimodal Dialogues
• Syntactic-level description
• For more details: check
SMUIML language
8. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 8
June 26, 2012
Fusion Algorithms
• Frame-based algorithms & Unification-
based algorithms
• Symbolic approaches
• Both are a de facto standard
• Symbolic-statistical algorithms
9. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 9
June 26, 2012
Symbolic-Statistical Approaches
Symbolic data
Interpretation of low-level data
Statistical processing
Machine learning algorithms
10. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 10
June 26, 2012
Our HMM-Based Fusion Algorithm
Why hidden Markov models?
Very good modelling of time-related events
Takes advantage of input data coming from
probabilistic input modalities
On-the-fly adaptation of the model
- e.g. based on user feedback
11. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 11
June 26, 2012
Put-That-There Example
12. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 12
June 26, 2012
Put-That-There Example
13. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 13
June 26, 2012
Put-That-There Example
14. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 14
June 26, 2012
Put-That-There Example
15. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 15
June 26, 2012
Put-That-There Example
16. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 16
June 26, 2012
Put-That-There Example
17. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 17
June 26, 2012
Instantiation of the Algorithm
18. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 18
June 26, 2012
Evaluation of the Algorithm
Evaluation on three levels
Qualitative “gut check” test
Benchmarks on real-world examples
Performance assessment
19. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 19
June 26, 2012
Evaluation: Qualitative Test
• Three expert users were asked to assess the
behaviour of the algorithm
frames-based and HMM-based fusion
5 minutes per condition + interview
• Coherent behaviour between the algorithms
• No noticeable difference
in responsiveness
20. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 20
June 26, 2012
Evaluation: Benchmark
21. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 21
June 26, 2012
Evaluation: Benchmark
22. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 22
June 26, 2012
Benchmark Results (1)
• CARE properties: equivalence and redundancy tests
23. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 23
June 26, 2012
Benchmark Results (2)
• Sequential and non-sequential complementarity tests
24. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 24
June 26, 2012
Benchmark Results (3)
• “Play next track” example
Meaning
frames
HMM-based
25. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 25
June 26, 2012
Evaluation: Performance
• Per fusion algorithm:
• 5 runs of 40 input pieces
• 5 x 20 expected fusion results
• Frame-based: 18.2 ms
• Standard deviation: 12.7 ms
• HMM-based: 16.6 ms
• Standard deviation: 11.6 ms
26. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 26
June 26, 2012
Conclusion
• A new symbolic-statistical multimodal fusion
algorithm based on Hidden Markov Models
• Integrated into the HephaisTK framework
• Superior recognition rates compared to existing
algorithms
• Handling of ambiguous cases
• Processing of real-time user feedback
http://wise.vub.ac.be/bruno-dumas