Is motion capture based biomechanical simulation valid for hci studies? study and implications
•Descargar como PPSX, PDF•
3 recomendaciones•752 vistas
The presentation at the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI 2014) in Toronto. We have validated Biomechanical Simulation with OpenSim and Full-Body musculoskeletal model for Human-Computer Interaction (HCI)-specific tasks. We have considered specific ranges and types of movements, and experiment scenarios common for HCI. We have highlighted contrasts between HCI and medical or sports settings, as well as their influences on biomechanical simulation predictions validity. Additionally we give recommendation for application of biomechanical simulation in HCI. The paper: http://dl.acm.org/citation.cfm?id=2557027 P.S. The slides contain videos and animations which are not displayed by online-viewer, so it is better to watch them offline after downloading.
Recomendados
Más contenido relacionado
Último
Último (20)
Destacado
Destacado (20)
Is motion capture based biomechanical simulation valid for hci studies? study and implications
- 1. Is Motion Capture-Based Biomechanical Simulation Valid for HCI Studies? Study and Implications Myroslav Bachynskyi Antti Oulasvirta Gregorio Palmas Tino Weinkauf Saarbrücken http://resources.mpi-inf.mpg.de/biomechanics
- 2. There are more opportunities in HCI for gestural and full body controls Larger design space More freedom for interface design Traditional interfaces Modern interfaces
- 3. Physical ergonomics is very important for the success of an interface Gorilla arm Trapezius fatigue Joint stress
- 4. Traditional ergonomics instruments are too expensive, invasive or cumbersome Goniometers Questionnaires sEMG (Surface Electromyography) Needle EMG Subjective Unreliable Cumbersome Not accurate Limited Only surface muscles Cross-talk Unreliable in dynamics Too invasive Only expert use Hard to move
- 5. Motion-capture based biomechanical simulation allows “looking inside the body” Optical motion capture (MoCap) Biomechanical simulation
- 6. Biomechanical simulation produces wide range of inside-body ergonomics indices – Moments – Forces inside joints 1. Model Scaling 2. Inverse Kinematics 3. Inverse Dynamics 4. Static Optimization Output for observable movement: Further processing • Physical work • Energy expenditure • Fatigue index • Per muscle: – Force exerted – Activation by neural system • Per joint: – Angles MoCap data Subject weight
- 7. Performance and ergonomics measured within single experiment synchronously Single HCI experiment with MoCap recording Performance of movements: speed, accuracy, throughput Ergonomics of movements: Joint angles and moments Muscle forces and activations Energy expenditure Synchronized in time and registered in 3D movement space
- 8. Our goal is to adapt biomechanical simulation for HCI scenarios Select HCI task Record MoCap Simulate Analyze the data Another HCI task
- 9. Medicine and sports: • Educated experimenters • Model fine-tuned to subject • Goal: highly focused analysis • Focus on lower body, gait and run HCI: • Non-expert experimenters • No fine-tuning of the model • Multiple user groups • Overview of movement space • Upper extremity and full-body • Specific types of movements Is it valid for HCI? Sources of error: • Marker mapping • Marker drift • Suit drift • Marker trust • Model scaling • Mass distribution • Muscle properties • Activation optimality ?
- 10. Upper extremity model with muscles must be validated for HCI tasks HCI Biomechanics [Honglun2007] [Du2007] [Chang2007] Biomechanicalcoverage Simple ergonomics tools integrated with MoCap Biomechanical simulation with EMG for muscles Complete biomechanical simulation [This paper] Moment at joint [Lloyd2003] Muscle activations[Hamner2010] [Pronost2011] 3 muscles and specific movement [Daly2011] 8 muscles and whole space movements [This paper] Biomechanicalcoverage
- 11. The paper reports 2 experiments Study 1: Applicability across HCI tasks Study 2: Validity against EMG data Small movements Finger and arm Full body Run simulations and qualitatively inspect outputs EMG Simulation Correlation as similarity measure? EMG data Predictions
- 12. Setup PhaseSpace Impulse Force plate OpenSim and Full-Body model
- 13. Main results
- 14. 1. The simulation is successful for movements larger than 4cm Failure Partial success Success Hand model
- 15. 2. Problems caused by strong users Extremely fast movements Inverse Kinematics Static Optimization
- 16. 3. Dynamic contact forces require external force measurements Movement Without external forces the model is up in the air fixed onto pelvis Correct simulation Incorrect simulation Part of parachute harness to fix pelvis
- 17. 4. Large muscles are better predicted Good Bad
- 18. 5. Fast movements are better predicted Good Bad FastMediumSlow
- 19. 6. Large individual differences due to age, gender and anatomy Good Bad
- 20. Conclusions
- 21. Biomechanical simulation is valid for the following HCI scenarios • Movements longer than 4cm • No extreme angles • No strong participants • Recording of external forces, if present • Focus on bigger muscles • Longer and faster movements
- 22. Possible improvements and future work • Computed Muscle Control should produce better results than Static Optimisation • Small finger movements may be successful with more comprehensive motion capture • Simple-to-use index of muscular fatigue needs to be developed based on biomechanical simulation
- 23. Despite the restrictions, biomechanical simulation CAN be effectively applied for a wide range of HCI tasks Select HCI task Record MoCap Run the simulation Analyze the data Another HCI task http://resources.mpi-inf.mpg.de/biomechanics
- 24. Is Motion Capture-Based Biomechanical Simulation Valid for HCI Studies? Study and Implications http://resources.mpi-inf.mpg.de/biomechanics Myroslav Bachynskyi Antti Oulasvirta Gregorio Palmas Tino Weinkauf Saarbrücken
Notas del editor
- Hi all. I’m Myroslav. I'm a second year PhD student from Max Planck Institute for Informatics and Saarland University. Today I will talk about our work on biomechanical simulation and in particular on its validity for HCI tasks. !!!!!
- For decades, HCI was focused on narrow range of devices for computer input, for example keyboard, mouse or touchpad. But recent developments of touch and motion tracking technology give us more freedom !!!!! to broaden interaction space and use not only our hands and fingers, but our whole body for computer input. !!!!! Nobody can be surprised by touchscreen, tabletop, accelerometer or camera based gestural interface. These interfaces give us not only freedom to use their advantages in multiple application scenarios, but also pose big challenges to interface designers due to huge design space of possible movements. !!!!!
- One of the key challenges in design of full-body interfaces is proper physical ergonomics assessment. The traditional input devices and in particular the mouse and the keyboard were thoroughly studied in numerous ergonomics studies. They constrain all movements to be performed in narrow physical space which simplifies the analysis. But with gestural interfaces we don’t have anymore space constraints, which is advantageous for input method, but contains possible risk for ergonomics. It becomes harder to design the interface users will like and use without physiological discomfort, muscular stress or fatigue. As you can see on images, there are multiple ergonomics pitfalls waiting for designers, as gorilla arm, muscular stress or fatigue. Red areas on human body highlight muscles or joints under discomfort or stress in full-body interactions. !!!!!
- Of course you will say there are existing physical ergonomics methods used earlier to assess interfaces, as for example questionnaires, videometry or electromyography. There are numerous studies examining mouse or keyboard input using surveys, or EMG recordings of forearm or shoulder muscles. !!!!! But all these methods have their drawbacks and can not be easily applied for full-body interaction. For example, with surface EMG only close-to surface muscles can be analyzed. It has big problems with reliability of data recorded during dynamic movements, due to muscle movement relative to skin, cross talk or different activation levels during concentric, and eccentric contractions. Additionally it is cumbersome to apply for full-body movements, as even wireless EMG sensors have some cabling and are noticeable when applied densely.
- New approach which can improve ergonomics assessment and fill gaps from previous methods is motion capture based biomechanical simulation. Biomechanics as a discipline is known for decades, but only recent advances in hardware and algorithms made it possible to widely perform the simulations in acceptable time. !!!!! In order to use the method, we first record optical motion capture of movements of our interest. Then we use recorded data !!!!! as input for biomechanical simulation, which processes cloud of points in 3D space and allows us to look on multiple physiological indices inside our body. !!!!!
- As inputs from optical motion capture we receive cloud of points in physical space. Additionally we need to know the mapping between physical and virtual markers and weight of participant. Then we can run biomechanical simulation. The simulation consists of multiple steps. Many of them produce important for ergonomics outputs. !!!!! Scaling automatically updates the model to match proportions and mass of the person. !!!!! Then Inverse Kinematics produces joint angles. !!!!! Inverse Dynamics produces moments and forces at joints. !!!!! And Static Optimisation resolves joint moments to forces exerted by each muscle and corresponding activations. !!!!! By further processing of simulated data, physical work during the movement, energy expenditure and index of fatigue can be estimated. !!!!!
- Big advantage for HCI is that during single experiment we can measure not only ergonomics of the movements, !!!!! but also their performance. These values are synchronized in time and additionally registered in 3D space. !!!!! As result we can analyze in simple way relationships between all these 3 components: performance, ergonomics and movement space. In this paper we focus on !!!!! the ergonomics component. !!!!!
- Here I show you possible example of analyses pipeline for HCI. At first we select an HCI task, which we want to design considering ergonomics. We design experiment with this type of interaction and then perform the experiment in motion capture laboratory. As next step we run the simulation on recorded motion capture data and extract biomechanical indices. Further we analyze these ergonomics indices together with performance and space variables using visualization, statistical, or other tools, and draw practical conclusions how to improve the interface. Our goal is to develop biomechanical simulation as !!!!! a universal method for HCI, which can be applied in all studies, !!!!! where optical motion capture can be performed. !!!!!
- Weak point of biomechanical simulation is numerous sources of error, which can decrease reliability of the outputs. This sources of error come in the game at each step, starting from marker placement and finishing by user-specific non-optimal muscle activation patterns. For successful biomechanical simulation we need to take care about these sources of error. Examples of fields where biomechanical simulation is successfully applied are medicine and sports. !!!!! These fields have their own specifics, which tend to avoid or minimize the errors coming to the simulation from different sources. Goal of our work is to assess whether the simulation can be also successfully applied in HCI setting, !!!!! which is quite different from medicine and sports. Here are main contrasts between HCI versus medicine and sports. At first motion capture recordings and the simulation in medicine and sports are performed by biomechanically educated professionals, who gathers practical experience every day, but in HCI practiotioners and researchers should not be obligated to have deep knowledge in biomechanics or physiology to use the simulation for interface design. Second is that there are more resources in medicine and sports which can be spent to simulate biomechanics of single person. This allows spending more time for accurate marker placement and measurement of the person, taking additional measurements of muscle force, EMG, weight distribution, etc. and fine-tuning the biomechanical model to the person. Third difference are the goals of the simulation, in medicine and sports the goal is deep understanding of specific movement of particular person, while in HCI our goal is to understand whole movement space for the population of users. Fourth, medicine and sports are mostly focused on lower extremities, gait and running, but for HCI main interest is in upper extremities and full-body movements, starting from small multitouch finger movements to full body gestures. The question we answer in our paper is whether biomechanical simulation after such differences will still be valid for HCI tasks?
- The related works can be split into two parts: !!!!! First part is HCI-specific. Currently there were few works which introduce optical motion capture to HCI and integrate it with ergonomics tools, however they used simplified biomechanics, completely without muscles as in Honglun et. Al., or with additional EMG recording for muscle assessment as in Du et al. In our paper we use biomechanical simulation which resolves not only joint angles, but also moments and forces, and muscle forces and activations. Muscle forces and activations are crucial for HCI as they give insights to avoid gorilla arm, muscular stress or fatigue. !!!!! Second part of related work is from general biomechanics. The fact that lower body is more of interest for main biomechanics audience has effect, that muscle activations of lower body models were validated against EMG in numerous studies, for example Hamner. However with upper extremity this is not the case. The single work we found which qualitatively compares predictions of upper extremity muscles against EMG, is PhD thesis of Melissa Daly. In her study she considers EMG of only 3 muscles and records only single specific reaching movement. What than can we say about all variety of movements in reachable space? What about other important muscles? We fill these gaps in knowledge and answer these questions in our paper. !!!!!
- In order to do this we conduct two studies: In first study we examine applicability of biomechanical simulation to variety of HCI tasks. !!!!! They can be ranged on size of movements. We start from small multitouch gestures recruiting only finger or finger and wrist, for example pinch, rotation or tapping. Then we analyze mouse interaction with different control to display ratios and keyboard interaction which includes finger, hand and whole arm movements. Next is flight controls with both arms and torso movements. And last is full-body interaction during dance game. We record motion capture of these interactions. !!!!! Then we try to run biomechanical simulation step by step and qualitatively check realism of outputs. In second experiment we validate predicted muscle activations versus EMG measurements. !!!!! We represent reachable space as half-sphere and uniformly distribute there 25 targets. 16 subjects perform 30 trials each, during which they produce aimed movements between stratified sampled pair of targets. We record motion capture of their movements. !!!!! Synchronously we record EMG of 8 upper extremity muscles. Based on motion capture data we !!!!! run biomechanical simulation. As result we have time series of predicted muscle activation and corresponding EMG signal. !!!!! We process EMG according to recommendations, and then use Pearson correlation between time series of EMG and predictions as similarity measure. !!!!! We avoid or minimize previously mentioned EMG problems with specific experimental design and analysis. !!!!!
- We wanted to study validity of the method in favorable conditions, that’s why we use state of the art motion capture system, simulation software and custom built force plates for our study. More details about studies you can find in the paper. !!!!!
- Now we move to results of our two studies. In the presentation I’ll highlight some of them, the whole list you can find in the paper. !!!!!
- First result concerns size of movement for simulation. We consider it as success if all steps of simulation run without problems and produce realistic results. Failure is when very first step cannot be run, or produces incorrect results. The simulation failed at small multitouch gestures on Inverse Kinematics. Typically full body simulation with mean errors between markers lower than 2 cm and maximal error of 4 cm is considered successful. But whole multitouch movements are in range of 4cm. !!!!! As a result you can see on video how physical markers shown by blue spheres move in pinch, but the simulation treats them in range of error and fingers of the model don’t move. The rest of movements are larger than 4 cm and can be successfully processed, !!!!! this concerns full-body dance, upper body flight control, or mouse with low control to display ratio. This finding poses limitation for usage in current HCI tasks, but we conclude that all interactions that do not involve finger articulation can be analyzed. !!!!!
- Second finding concerns muscles of the user and the model. I’ll describe it on example. !!!!! This video is in real time, without editing. As you can see, one of our participants moved extremely fast. He is kickboxing champion, so he has well trained muscles. !!!!! First step which is inverse kinematics can fit the model to the movement of participant. But the step of static optimization fails. !!!!! The model represents muscles of average adult male, and its muscles cannot produce big enough force and fast enough movement to match the movement of the athlete. To run successfully static optimization, tuning of model muscles to the athlete muscles is obligatory, which is not trivial task for HCI researcher. Good thing is that for most, but extremely strong persons, simulation works correctly. !!!!!
- Next result is about external forces. !!!!! In our experiment we have recorded full-body dance. From video of Inverse Kinematics you can guess that person is dancing on the floor. But in fact we have not recorded ground reaction force, as result our simulation produces outputs as if the person was moving this way !!!!! hanged on the pelvis. Pelvis is the root of the model, and simulation of body part above pelvis is still correct, but legs are not standing on the floor, but hanging in the air. In case of static lower body external forces can be estimated from body weight, but in dynamic setting this doesn’t work. Other point is that external forces improve realism of simulation results, even if they are not very big, for example in case with typing or mouse usage. Within HCI for many types of tasks as mid-air gestures or public display interaction for upper body external forces are either absent or very small. !!!!!
- Next finding considers quality of predictions. We have calculated pearson correlation between time series of predicted activations and EMG for each movement. Here on the boxplot you can observe distributions of similarity between predictions and EMG for different muscles. As you can see, larger muscles are predicted significantly better than smaller ones, with median predictions of trapezius as high as 65%. Possible cause for this effect is not optimal recruitment of smaller musculature, which hinders the assumption of Static Optimisation. Other cause can be related to larger displacements of biceps, triceps and pectoralis major comparing to other muscles during the movements, which influences EMG recordings. !!!!!
- Next result is that faster movements are predicted better than slower ones. As you can see on boxplot, difference is between 60% for fast and 38% for slow ones. This can also be related to not optimal recruitment of muscles during slow movements. !!!!!
- Next result confirms the expectation: predictions are better for persons, who better match the model properties. As I have already told the model represents average adult male with his age and gender-specific properties: weight, mass distribution and specific muscle qualities. 16 different persons participated in our experiment, most of them are 21-28 year old students, 7 of which are females. Still the worst model match has median correlation around 33%, which is not completely bad, while the best match, 36 years old man has median correlation of 61%. Median correlation for whole experiment population is near 50%. !!!!!
- We want to summarize main findings and propose possible solutions to some limitations !!!!!
- As summary I will list the findings of our paper: Presently, biomechanical simulation allows simulating movements longer than 4 cm. We avoid problems without champion participants and movements to extreme angles We focus on bigger muscles and trust more in longer movements We record external force if possible, or we consider only part of body not affected by them !!!!!
- Some restrictions of biomechanical simulation I have mentioned can be avoided or at least improved. Here I will note few such things. At first, there is newer more powerful simulation algorithm than Static Optimisation. It is called Computed Muscle Control. This algorithm consists of Static Optimisation and self corrective feedback loop with forward simulation. Additionally it better considers muscle activation dynamics, so that we cannot instantly activate our muscle to produce maximum force. This algorithm should produce better match for EMG data than Static Optimisation, although it needs more time for computations. Concerning failure at small finger movements, it is not showstopper. The simulation should be able to track fingers correctly with more comprehensive marker setup. In recent study Vignais et al. were able to run the simulation of finger movements using more sophisticated setup. As future work we see also development of an easy-to-use index of fatigue based on the simulation outputs. !!!!!
- As you can see except few restrictions biomechanical simulation can be successfully applied for ergonomics assessment to wide range of HCI tasks. !!!!! If you are interested, you can get a lot of information at our project web page.
- Thank you for attention and welcome to biomechanics.