by Pedro Mateo (pedromateo@um.es) and Stefan Hillmann

1. Model-based Measurement of Interaction in Mobile
Multimodal Environments
Pedro Mateo – Stefan Hillmann
aMMI - October 2012
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 1 / 17

2. Agenda
Model-based Measurement of Interaction in Mobile Multimodal
Environments:
1 Research Questions
motivation, questions and proposed solutions
2 Research and Development
MMI metrics, MMI model and Android implementation
3 Conclusion
advantages, disadvantages and research directions
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 2 / 17

3. I
Research Questions

4. Motivation
We analyze interaction in different
systems...
...but independently for each system
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 4 / 17

5. Motivation
And now, for each system...
different quantification/evaluation metrics
different representation formats
different interaction analysis tools
multimodal communication is not analyzed as a whole
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 5 / 17

6. Research questions and proposed solutions
Question 1:
MMI assessed on a common level for different modalities?
⇒ Use common and generic metrics for all modalities
Question 2:
Compare interactions observed in different MMI contexts?
⇒ Structure metrics in a common representation (model)
Question 3:
Analyze MMI dinamically, step-by-step, and using real-time data?
⇒ Use a run-time model
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 6 / 17

7. II
Research and Development

8. Defining multimodal metrics
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 8 / 17

9. Modeling multimodal interaction in EMF
Metacommunication
User Metacommunication System Metacommunication
Speech Output
Speech Input
GUI Output
GUI Input User Turn System Turn
Gesture Output
Gesture Input
Cooperativity
Modality Turn Dialog
Synchrony Modality Change Task
Appropriateness
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 9 / 17

10. Modeling multimodal interaction in EMF
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 10 / 17

11. Android implementation
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 11 / 17

12. Android implementation
how does it work?
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 12 / 17

13. III
Conclusion

14. Good stuff...
common metrics + uniform representation
⇒ unified criteria to quantify MMI in different scenarios
comparison is allowed, model transformation is possible
different modalities are on a level ⇒ single thread
dynamic representation of MMI ⇒ run-time applications
reuse of metrics and tools is improved
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 14 / 17

15. Not so good stuff...
some data can’t be automatically collected
criteria to record parameters in different modalities?
lack of plasticity
⇒ user and context information to enrich the analysis process
the HOW of interaction is not described
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 15 / 17

16. And now what?
Integrate this solution in real projects
Use the framework for real analysis
Make the model context-aware, more “mobile”
Explore new possibilities of the model
E.g. behavioral biometric authentication
Project info: http://www.catedrasaes.org/wiki/MIM
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 16 / 17

17. Acknowledgements
Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 17 / 17