The development of Augmented Reality (AR) applications is still driven by development at source-code level. Although recent approaches focus to standardize AR functionality, to our knowledge a declarative and model-driven design (MDD) has not been applied for AR development so far. MDD approaches have been successfully applied to model user interfaces for a wide spectrum of modes (such as speech- command interfaces, or remote controls) and media such as HTML, 3D, and smart phone interfaces for instance. In this paper we propose a MDD approach for modeling seamless interaction between Web and AR interfaces. Therefore we implemented a prototype of a web furniture shop that supports arranging furniture in an augmented reality to prove our approach. We then present our approach of modeling a reality spanning Drag-and-Drop interaction between a 2D browser and an AR scene. Finally, we discuss the issues that we were confronted with to support reality spanning interaction like switching interaction modes and coordinate systems and present limitations that we have experienced with using other control modes, like a Wii-Controller or glove-based gesture detection.

1. 1. Juni 2011 Model-based Design of Interactions that can bridge RealitiesThe Augmented “Drag-and-Drop”Sebastian Feuerstack, Allan C. M. de Oliveira, Regina B. Araujo Universidade Federal de São CarlosDepartamento de Computacão

2. The Augmented “Drag-and-Drop” 2 1. Juni 2011 Basic Question How to model Interactions that span Modes and Media ?

3. The Augmented “Drag-and-Drop 3 1. Juni 2011 Which Media ? http://flic.kr/p/6A7D3E

4. The Augmented “Drag-and-Drop” 4 1. Juni 2011 Which Modes? http://flic.kr/p/7sZ6ho http://flic.kr/p/6eeLwg

5. The Augmented “Drag-and-Drop 5 1. Juni 2011 How ? Device ; Modality

6. The Augmented “Drag-and-Drop 6 1. Juni 2011 Why modeling? One specification that is declarative and exact to discuss & execute Is located between Tool and Code can be easily adapted or enhanced to different forms of interaction

7. The Augmented “Drag-and-Drop 7 1. Juni 2011 Use Case: Augmented Drag and Drop Buying Furniture Shopping Cart Drag and Drop Into my environment

8. Video: Connecting the Web with AR The Augmented “Drag-and-Drop 8 1. Juni 2011 VIDEO Mouse-based Drag-and-Drop

9. The Augmented “Drag-and-Drop 9 1. Juni 2011 How to model? Models Interactors(Abstract & Concrete Media, Mode) Static: class diagramm, Behavior: state-chart Mappings Mode-To-Media Synchronization Custom Notation

10. The Augmented “Drag-and-Drop 10 1. Juni 2011 Abstract Media Model

11. The Augmented “Drag-and-Drop 11 1. Juni 2011 Abstract Media Model Single Choice Aggregates a set of Entities from that only one can be chosen at a time Examples: Direction (leftorright), Shopping Cart Further Properties: Output tothe User Container -> Aggregation, Discrete, thatare Inputs Contains Single Choice Elements

12. The Augmented “Drag-and-Drop 12 1. Juni 2011 Abstract Behavior Model

13. Design of Multimodal Interaction 13 1. Juni 2011 Mode Model (Example: Mouse)

14. The Augmented “Drag-and-Drop 14 1. Juni 2011 Multimodal Mapping The Drag-and-Drop

15. The Augmented “Drag-and-Drop 15 1. Juni 2011 Conclusions & Future Work What’s the advantage? Detailed, declarative Modeling of Interactions (Media, Mode) We can combine Modes & Media and bridge Realities What`s still missing is a tool What`s the advantage??? Lets see a demo with different Modes

16. Gesture-based Drag and Drop Control The Augmented “Drag-and-Drop 16 1. Juni 2011 Cam Jessica Pointing Cam Command Monitor

17. Gesture-based Drag and Drop Control Design of Multimodal Interaction 17 1. Juni 2011 VIDEO Gesture-based Drag-and-Drop

18. The Augmented “Drag-and-Drop 18 1. Juni 2011 The End … Thank you for your attention! Your questions please …

19. The Augmented “Drag-and-Drop 19 1. Juni 2011 Abstract Model Reference – Object or Property of an Object to unambigously identify it Examples: Name, Color, Number, Shape, Coordinate Further Properties: Discrete -> Distinguishable Input -> bythe User

20. The Augmented “Drag-and-Drop 20 1. Juni 2011 Multimodal Mapping (Combining Mode with Media)