Model-driven software engineering is a well-known approach for developing software. It reduces complexity, facilitates maintenance and allows for the simulation, verification, validation and even execution of software models. In this article, we show how model simulation and model execution can be leveraged in the context of human-computer interaction (HCI). We claim that in this application domain, it is beneficial to use heterogeneous models, combining different models of computation for different components of the system. We report on a case study that we have carried out to develop an executable model of a gesture-based application for manipulating 3D objects, using the Kinect sensor as input device, and the OGRE graphical engine as an output device for real-time rendering. This application is fully specified as an heterogeneous model with the ModHel’X modeling environment, allowing the simulation and execution of the model to explore the most appropriate way of implementing the application.

1. Kilix: Heterogeneous Modeling of
Gesture-Based 3D Applications
R. Deshayes and T. Mens
Institut COMPLEXYS, University of Mons, Belgium
C. Jacquet, C. Hardebolle, F. Boulanger
Supelec E3S – Computer Science Dept., France

2. Goals of our research
• Reduce complexity of developing HCI applications
MoDELS 2012 – MPM workshop, Innsbruck
• By using visual modeling instead of programming
• Assess the usability of heterogeneous modeling for
this purpose
• Evaluate the strengths and shortcomings of
ModHel’X, a heterogeneous modeling environment
• Explore and improve its notions of semantic adaptation
wwwdi.supelec.fr/software/ModHelX
wwwdi.supelec.fr/software/ModHelX/Kilix
2

3. Case study
• Gestural interaction with a graphical 3D
MoDELS 2012 – MPM workshop, Innsbruck
application
• Using the Kinect controller
to interact with virtual books
using hands only
3

4. Proposed architecture
• Client-server architecture
MoDELS 2012 – MPM workshop, Innsbruck
• Low coupling between I/O devices
and user interaction models
• Combining different models of computation (MoC)
• choose the most appropriate formalism for the task at hand
• discrete events (DE)
• synchronous data flow (SDF)
• timed finite state machines (TFSM)
4

5. Proposed architecture
• Client-server architecture
MoDELS 2012 – MPM workshop, Innsbruck
• Low coupling between I/O devices
and user interaction models
• Combining different models of computation (MoC)
• choose the most appropriate formalism for the task at hand
• discrete events (DE)
• synchronous data flow (SDF)
• timed finite state machines (TFSM)
5

6. Heterogenous modeling
• Hierarchical architecture
MoDELS 2012 – MPM workshop, Innsbruck
• Top-level model contains 4 blocks
• MoC is discrete events (DE)
• communication through timestamped events
containing data
6

7. Semantic adaptation
• Interface blocks adapt the semantics between
MoDELS 2012 – MPM workshop, Innsbruck
outer and inner models using different models
of computation
• Adaptation can be made to
• Data (which may be represented differently)
• Time (e.g. different time units, different time scales,
continuous vs discrete time)
• Control (trigger observations of the internal model at
instants requested by the internal MoC)
7

8. Sensing block
• Receives data from Kinect and converts it into
MoDELS 2012 – MPM workshop, Innsbruck
hand gesture events
• MoC = synchronous data flow (SDF)
• Processes a chain of sampled signals received from
Kinect at a fixed rate
• Semantic adapter generates DE events when non-
null SDF tokens are produced
DE
8

9. Application Logic block
• Interprets and converts hand gestures into
MoDELS 2012 – MPM workshop, Innsbruck
meaningful actions for 3D object manipulation
• MoC = timed finite state machines (TFSM)
• DE/TFSM adapter converts between DE events and
symbols for the state machine
DE
9

10. Virtual Scene block
• Represents graphical 3D objects (e.g., book)
MoDELS 2012 – MPM workshop, Innsbruck
that interpret the actions as object-specific
behaviour (e.g. opening or closing the book)
• MoC = TFSM
DE
10

11. Heterogenous modeling
• General overview revisited
MoDELS 2012 – MPM workshop, Innsbruck
11

12. Discussion
• Heterogeneous modeling is useful for HCI
MoDELS 2012 – MPM workshop, Innsbruck
applications
• Semantic adaptation can be used
• To adapt between models of computation
• To map application actions (e.g., swipe) to object
behaviors (e.g., open or close)
• To use the same component differently in different
applications
• Leads to less coupling and higher component reusability
• Dynamic modeling is difficult to achieve
• e.g. variable number of users and books at runtime
12

13. Future work
• Compare strengths and weaknesses of
MoDELS 2012 – MPM workshop, Innsbruck
homogenous and heterogenous modeling
• Based on common case study
• Expressed using statecharts only
• Expressed using high-level Petri nets
• Joint work with Ph. Palanque, Toulouse (PetShop tool)
• Expressed using ModHel’X
• ModHel’X improvements
• Performance issues
• Add support for visual editing of models
• Support domain-specific languages to match the
application domain better (work in progress)
• Extend existing MoC (TFSM++)
13

14. References
• For homogeneous modeling
MoDELS 2012 – MPM workshop, Innsbruck
• R. Deshayes and T. Mens. Statechart modelling of interactive
gesture-based applications. In ComDeisMoto satellite event of
INTERACT 2011.
• D. Navarre, et al. ICOs: A model-based user interface description
technique dedicated to interactive systems addressing usability,
reliability and scalability. ACM Trans. Comput.-Hum. Interact.,
16(4), 2009.
• For heterogeneous modeling
• J. Eker et al. Taming heterogeneity - the Ptolemy approach. Proc.
IEEE, 91(1):127–144, 2003.
• C. Hardebolle, F. Boulanger. Exploring multi-paradigm modeling
techniques. SIMULATION: Trans. Society for Modeling and
Simulation International, 85(11/12):688–708, 2009.
• F. Boulanger et al. Semantic Adaptation for Models of Computation.
ACSD 2011: 153-162
• B. Baudry et al. Bridging the Chasm between Executable
Metamodeling and Models of Computation. SLE 2012
14