The rise of gestural interaction led artists to produce shows, or instal-lations based on this paradigm. We present the first stages of the « Sculpture numérique » (Virtual Sculpture) project. This project was born from a collabo-ration with dancers. Its goal is to propose bi-manual interactions in a large augmented space: we aim at giving dancers the possibility to generate and ma-nipulate virtual elements on stage using their hands. The first set of interactions we present in this paper is 3D painting, where the user can generate 3D virtual matter from his hands. The movement of the hand defines a stroke, which shape is controlled by the shape of the hand. Changing the shape and orientation of the hand allows switching between three interaction modes to produce volumes, surfaces or curves in space. We explore the applicative case of dance, with the goal of producing a plastic creation from choreography.

1. Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan
Bi-manual
An interaction paradigm for
augmented reality live
performance
Human-Computer Interaction, Tourism and Cultural Heritage
(HCITOCH 2012)

2. 4th March 2011
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 2

3. 4th March 2011
Feedback from the audience:
80% return rate
90% found it innovative
120 submitted forms
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 3

4. 4th March 2011
Feedback from the audience:
Audience bright interest for 3D
ribbons following the dancer’s
hands
120 submitted forms
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 4

5. 4th March 2011
Feedback from the audience:
>90% wish to witness
the evolution
of the show
120 submitted forms
Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 5

6. Summary
1 VIRTUAL SCULPTURE
Hand Interactions Stereoscopic Display
2 MANUAL INTERACTIONS
3 CAVE 3D PAINTING
Prototype1
CAVE and development tools
4 STAGE 3D PAINTING
Prototype2
5 PERSPECTIVES
Live Performance Application
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 6

7. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
State of the Art
Sketching Surfaces BLUIsculpt system
Deisinger et al - 2000 Arctic Region Super Computing Center - 2002
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 7

8. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
Outcomes
1 Direct Metaphor
 Virtual matter seems to come out
right under the user’s hand
Surface Drawing
Schkölne, Pruett, Schröder- 2001 Freehand Interaction
Large Space Interaction and
visualization
2
Large Space Painting
 Dance application
Sketching Surfaces
Deisinger et al - 2000
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 8

9. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
Our Objective
Hands’ Joints Path
Schkolne et al. Application to Dance
Surface Generation Live Performance
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 9

10. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
Our Objective Credits: Samsung Galaxy SIII advertisement - 2012
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 10

11. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
Metaphor
Generate Virtual Objects Basic Information
Hand conformation
Wrists’ positions
Articulations and tip
positions
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 11

12. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
1rst metaphor: Shapes
Movement parallel to the palm’s surface
Curves Surfaces
 1/2s or 1/10s sample
1  Surface between t and t-1
Shapes of the generated
surface
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 12

13. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
2nd metaphor: Volumes
Movement perpendicularly to the palm’s surface
 Volume section = user hand visual surface
2
2
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 13

14. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
3rd metaphor: Curves
Index/Middle finger conformation
 More accurate
 Same as a tinier brush
3
 Automatic closed surfaces fullfilment
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 14

15. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
Thumb positions Pen-writing metaphor
« Free To Move »
« Draw »
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 15

16. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
Main interests 1
Simple
Direct metaphor
2
Natural modes
Hand orientation / Fingers tucked
3
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 16

17. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE
 Immersion in virtual space  3D Display in a Large Space
 Co-located interactions  Multi viewers
 Display Accuracy  Reduced cost
 Development Tools
 Two users simultaneously max  Dancer Blind From His Creations
 Cost  Occlusion Problem
 3mx3mx3m  One Projection Screen
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 17

18. 1 VIRTUAL 2 MANUAL 4 STAGE 3D
3 CAVE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Instrumented Space
Hand IR Trackers
Virtual Platform
with AR-Dtrack
Glasses with Positions Trackers
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 18

19. 1 VIRTUAL 2 MANUAL 4 STAGE 3D
3 CAVE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
isiVR Scalable and Interactive Virtual Reality
Immersive 3D Visualization FrameWork
OpenSceneGraph
Ready-to-use Immersive Tools
Render Engine
Multi-display Cluster Position Tracking
isiVR Features
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 19

20. 1 VIRTUAL 2 MANUAL 4 STAGE 3D
3 CAVE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
isiVR
OpenSceneGraph
Ready-to-use Immersive Tools
Render Engine
 Facilitate
Additions/Modifications
Inputs Behaviors Outputs
 openSource license
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 20

21. 1 VIRTUAL 2 MANUAL 4 STAGE 3D
3 CAVE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Schkölne Paradigm Application
Flat Polygons
Control Points
Positions:
Bézier Surfaces according to
 Middle Finger Distal Phalanx
Hand’s Movement
 Wrist
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 21

22. 1 VIRTUAL 2 MANUAL 4 STAGE 3D
3 CAVE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Surface Generation by Hand in a CAVE
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 22

23. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING PAINTING
CAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE
 Immersion in virtual space  3D Display in a Large Space
 Co-located interactions  Multi viewers
 Display Accuracy  Reduced cost
 Development Tools
 Two users simultaneously max  Dancer Blind From His Creations
 Cost Effective  Occlusion Problem
 3mx3mx3m  One Projection Screen
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 23

24. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Prototype Scheme Instrumented User
Signals
received are
different
format
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 24

25. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Motion Capture Suit
XSens SIGNALS
Mtx Trackers 17 Motion Trackers
Gyroscope, magnetomet
ers, and accelerometers
signals
Xbus Master Articulated and Rigid
Kinematic Chain
Master Slave Individual
Body Segments
Positions x,y,z(t)
Orientations
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 25

26. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Data Gloves
5DT SIGNALS
1 Thumb_Near 14 Flexion Trackers
2 Thumb_Far
3 Thumb_Index 4 7 10 Erase Objects
4 Index_Near
3 6 9
5 Index_Far 5 8 13
6 Index_Middle 11 12
Point an object
7 Index_Near 1 2
8 Index_Far
9 Index_Ring 0 Choose color
10 Ring_Near
11 Ring Far
Display Palette
12 Ring_Little
13 Ring_Near
14 Ring_Far Choose color
0 < Flexion < 1
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 26

27. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
VIRTOOLS
Behaviors Indirect Basis
Render Engine
Visual Programming Editor
Procedural Approach
SDK
Inputs
Building Blocks
Management
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 27

28. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
VIRTOOLS
Building Blocks
Message Handler
Actions Reactions
Settings
OBJECT
Variable Configuration
Loop Process
Custom Dialog Box
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 28

29. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
VIRTOOLS
COORDINATES Hands
BUFFER x,y,z(t) vertices x,y,z(t+1) vertices
ENTITY CREATION Object Mesh Material Texture Shader
PROCESSING Lights Camera
Main Object Sub Object
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 29

30. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
VIRTOOLS
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 30

31. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 31

32. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Tesselation Post-Process
1)
4.Tip
3.Distal 2)
2.Medial
3)
1.Proximal
Tesselation
More Details
Save CPU Resources
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 32

33. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
Constraint
Occlusion Projection Screen
SCENE
Semi-transparent
projection screen AUDIENCE
(between dancer and
audience)
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 33

34. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
« La Mer », Debussy - 1905
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 34

35. 1 VIRTUAL 2 MANUAL 3 CAVE 3D
4 STAGE 3D PAINTING 5 PERSPECTIVES
SCULPTURE INTERACTIONS PAINTING
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 35

36. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
SCULPTURE INTERACTIONS PAINTING PAINTING 5 PERSPECTIVES
Current Work and Applications
Dancer’s Movement Parameters:
 Space
 Time
 Weight
 Flow
“Computational models of expressive
gesture in multimedia
systems”, G.Volpe et al, 2003
Current Work
Credits: Proteigon - 2011
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 36

37. 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D
SCULPTURE INTERACTIONS PAINTING PAINTING 5 PERSPECTIVES
Perspectives
Creations Haptic Feedback Creations Visual Feedback
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 37

38. Authors
Alexis Clay
Jean-Christophe Lombardo
Nadine Couture
Julien Conan
3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 38

39. Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan
Bi-manual
An interaction paradigm for
augmented reality live
performance
Human-Computer Interaction, Tourism and Cultural Heritage
(ESTIA - INRIA)