The work presented in this paper is focused on movement recognition as a first step to achieve the automation of a two-arm-surgical-robotic-system in the laparoscopic surgical environment. In order to accomplish coordination between the surgeon and the robotic assistant, a system able to recognize and differentiate between certain standard surgical maneuvers should be developed. Two different methodologies are proposed to model and identify several surgical maneuvers. The first method is based on Artificial Neural Networks (ANN), by codifying the movements through their Fourier spectra and the second one is based on HMMs which represents the interaction between the surgical tools. The proposed approaches will be tested through a set of experiments that mimic surgical movements as in tissue cutting, suturing and transporting. In this way, the recognition system is able to distinguish between the different maneuvers which have been modeled.

1. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
Irene Rivas Blanco
Department of System Engineering and Automation
University of Malaga
MANEUVERS RECOGNITION IN
LAPAROSCOPIC SURGERY: ARTIFICIAL
NEURAL NETWORK AND HIDDEN
MARKOV MODEL APPROACHES

2. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
I. Introduction
II. Maneuver Recognition System
III. Implementation & Experiments
IV.Conclusions
INDEX

3. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
I. INTRODUCTION

4. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
I. INTRODUCTION
• Surgical assistant robot interfaces:
Direct-Teleoperation Head tracking
Voice commands
Eyes trackingVision algorithms

5. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
I. INTRODUCTION
(HMM) (HMM)
• Intuitive and natural
Human-Machine
Interface: Maneuver
Recognition System
• Based on modeling the
surgeon’s movements
• Comparison between
two modeling
approaches: Artificial
Neural Networks (ANN)
and Hidden Markov
Models (HMM)
Four degrees of freedom

6. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
II. MANEUVER RECOGNITION
SYSTEM

7. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
II. MANUEVER RECOGNITION SYSTEM
Surgical Protocol Maneuver

8. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
II. MANUEVER RECOGNITION SYSTEM
SENSORIAL SYSTEM
PREPROCESSING DATA
CODING DATA
Surgical Tools’ movement
Kalman filter
Two Tracking 3D sensors
ANN Fourier
RECOGNITION SYSTEM
HMM ANN
Maneuver
code
Maneuver
code
Data numerical
description
Modeling of surgeon’s
movements

9. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
II. MANUEVER RECOGNITION SYSTEM
• ANN: Recognizes the trajectory of the tools’ tip
FOURIERTrajectories ANN
Maneuver
Code
Set of
vectors
CODING DATA RECOGNITION
SYSTEM
• HMM: Recognizes the interaction between the tools
ANN
Characteristic
vector
HMM1
Maneuver
Code
Observable
code
CODING DATA
RECOGNITION
SYSTEM
HMM2
HMMn
…
MAX
PROB.

10. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
III. IMPLEMENTATION &
EXPERIMENTS

11. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
III. IMPLEMENTATION & EXPERIMENTS
Polaris
Spectra
Markers
CODE MANEUVER
1 Cutting
2 Suturing
3 Transporting

12. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
III. IMPLEMENTATION & EXPERIMENTS
• Experiments: maneuvers recognition

13. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
Maneuver Training Validation Hits
Suturing 150 35 94,29%
Cutting 150 35 91,43%
Transporting 150 35 94,29%
ANN
Maneuver Training Validation Hits
Suturing 150 35 88,57%
Cutting 150 35 82,86%
Transporting 150 35 85,71%
HMM
III. IMPLEMENTATION & EXPERIMENTS
• Experiments results:

14. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
IV. CONCLUSIONS

15. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
IV. CONCLUSIONS
• Human-Machine interface to recognize
maneuvers during a laparoscopic surgery
• Comparison between two modeling approaches:
Artificial Neural Networks and Hidden Markov
Models
• ANN is an intuitive approach, but it is based on
trajectories analysis in a specific reference
frame.
• HMM provides an independent reference
movement frame recognition system.

16. DepartmentofSysyem
EngineeeringandAutomation
Víctor F. Muñoz Martínez
Research lines
THANK YOU FOR YOUR ATTENTION