Presentation of my senior Project about "A real time automatic eye tracking system for ophthalmology" In the presentation, it briefly explains about conventional object tracking method "template matching" based on Sum-of-Square difference. Therefore we also present the powerful matching technique called Gradient Orientation Pattern Matching (GOPM) proposed by T.Kondo and we proposed an improved version of GOPM called time-vary GOPM to solve a illumination and noise problem.

1. A Real-Time Automatic Eye Tracking
System for Ophthalmology
Mr. Wattanit Hotrakool
Mr. Prarinya Siritanawan
Supervised by Dr.Toshiaki Kondo
Sirindhorn International Institute of Technology
1

2. Outline
 Project Background
 Project Objective
 A eye-tracking technique using Traditional Template
Matching
 A eye-tracking technique using Gradient Orientation
Pattern Matching
 A eye-tracking technique using Time-varying GOPM
 Conclusion
 Question and Answer
2

3. Project Background
 The conventional eye-surgery cameras are manual-
controlled and they reduce the efficiency of surgery.
 In order to reduce the burden of oculist, the automated
camera control is required. The image processing is used
to locate and track the eye’s centroid.
3

4. Project Background
 Many real-time eye tracking techniques used intensity
data as an input; they are very sensitive to changing
lighting condition and result as miss-matching.
 This project proposes new template matching based
technique which robust to changing lighting condition by
using Time-varying Gradient Orientation Pattern
Matching.
4

5. Project Objectives
 To implement matching-based techniques in real-time.
 To verify the robustness to changing lighting condition of
gradient orientation pattern matching.
 To develop new eye-tracking technique; a time-varying
gradient orientation pattern matching.
5

6. Eye-tracking method using template
matching technique
 Template matching is the intensity-based technique for
measuring the similarity between template and
corresponding block of image.
Template
Match
Sample frame
6

7. Simulation Specification
 Simulation System using the following hardware and
software specifications:
Software Specification
Operating System Windows/Linux
Programming Language C/C++
Primary library OpenCV 2.0
Hardware Specification
Processor Intel Core2 Duo
Processor Speed 1.66 GHz
Memory 4GB
7

8. Video sequences used for simulation
 Video sequences used in this simulation can be
categorized into 3 categories:
1. Test video in normal lighting condition
2. Test video in changing lighting condition
3. Actual surgery video from real camera
8

9. Template Initialization
 Template initialization is required before the eye-tracking
method.
 The pupil is specified as a template in order to track the
eye.
 The example of video and corresponding template for
each categories is shown in next page
9

10. Test video in normal lighting condition
Video sequence
Template
10

11. Test video in changing lighting condition
Video sequence
Template
11

12. Actual surgery video from real camera
Video sequence
Template
12

13. A eye-tracking method using
Traditional template matching
13

14. A eye-tracking method using Traditional
template matching
 There are many traditional techniques of template
matching such as sum-of-absolute difference (SAD), sum-
of-squared difference (SSD), or cross-correlation (CC)
technique.
 In this step, we implement the sum-of-squared difference
technique (SSD) to be eye-tracking method.
14

15. Sum-of-squared Difference technique
 SSD is the template matching method done by finding the
lowest difference value between input and template. The
differences are squared in order to remove the sign.
where I1 is the intensity of input image and
I2 is the intensity of template
N is the size of the template
15

16. Procedure
1. Convert template to gray scale image.
2. Convert input frame to gray scale image.
3. For every pixel, compute the SSD between input and
template.
4. Find the minimum difference pixel, which is the best
matching location.
5. For every frame, repeat step 2-4.
16

17. Procedure
17

18. Result
Input Average computation time Precision error
(ms) (%)
Normal light video 78.35 1.33
(resolution: 320x240 px)
Changing light video 78.63 40.47
(resolution: 320x240 px)
Actual surgery video 103.33 0
(resolution: 384x288 px)
Average computation time mostly depends on video
resolution
However, this method currently can process at 10 -12.5 frame/sec
18

19. Result
Input Average computation time Precision error
(ms) (%)
Normal light video 78.35 1.33
(resolution: 320x240)
Changing light video 78.63 40.47
(resolution: 320x240)
Actual surgery video 103.33 0
(resolution: 384x288)
SSD technique can work very well in normal light video.
However, this technique give high error in changing light video
because it uses intensity data which are sensitive to light.
Therefore SSD cannot work in changing light condition
19

20. Result
Properties of eye-tracking using SSD technique
Obstacle robustness Yes
Blur robustness Yes
Light condition robustness No
Scaling robustness No
Average computation time About 50-350 ms
20

21. A eye-tracking method using Gradient
Orientation Pattern Matching
 Presented at ICESIT2010, Chiang Mai, Thailand
21

22. A eye-tracking technique using Gradient
Orientation Pattern Matching
 In order to develop a method that can provide the
robustness to light condition, the new template matching
technique is used.
 The gradient orientation pattern matching (GOPM) is a
new template matching technique proposed by Dr.
Toshiaki Kondo.
22

23. Gradient Orientation Pattern Matching
 GOPM is a template matching method which use the
normalized gradient of the image in place of intensity
data. Thus, it only consider about the shape of the
pattern but not light.
 Gradient vector is the first derivative of intensity. The
gradient in x and y direction are defined as:
where I is a intensity of an image
23

24. Gradient Orientation Pattern Matching
 The gradient in x and y will then be normalized. This
step provides the robustness to light condition. The
normalized gradient in x and y direction are defined
as:
where
And is a small constant used to prevent zero-division.
24

25. Gradient Orientation Pattern Matching
 The normalized gradient in x and y direction of input
frame and template will be match by using SSD
where N1 is the normalized gradient of input image and
N2 is the normalized gradient of template
25

26. Procedure
 For every frame, we can divide the procedure into 2 main
steps;
1. Gradient orientation information (GOI) extraction
2. Gradient orientation pattern matching (GOPM)
26

27. Gradient Orientation Information (GOI)
Extraction
Extract the gradient Images
(Template and sample frame) in
x and y direction.
27

28. Gradient Orientation Pattern Matching
Apply template matching in x and y
direction. Then add the result of x and
y direction
28

29. Result
Input SSD technique GOPM technique
Average Precision Average Precision
computation error (%) computation error (%)
time (ms) time (ms)
Normal lighting condition 78.35 1.33 62.48 0
(resolution: 320x240px)
Change lighting condition 78.63 40.47 63.09 12.87
(resolution: 320x240px)
Actual surgery camera 103.33 0 77.7 0
(resolution: 384x288px)
Average computation time of GOPM slightly inprove from
SSD
Even though the method is more complex, but the computation time
is decrease due to the variable type and internal structure of OpenCV
However, this method still can process at only <15 frame/sec
29

30. Result
Input SSD technique GOPM technique
Average Precision Average Precision
computation error (%) computation error (%)
time (ms) time (ms)
Normal lighting condition 78.35 1.33 62.48 0
(resolution: 320x240px)
Change lighting condition 78.63 40.47 63.09 12.87
(resolution: 320x240px)
Actual surgery camera 103.33 0 77.7 0
(resolution: 384x288px)
In changing light condition, GOPM error is dramatically decrease due to
the normalized process.
Therefore GOPM can provide the robustness to changing light
condition
30

31. Result
Properties of eye-tracking using SSD and GOPM technique
SSD GOPM
Obstacle robustness Yes Yes
Blur robustness Yes Yes
Light condition robustness No Yes
Scaling robustness No No
Average computation time About 50-350 ms About 50-350 ms
31

32. A eye-tracking method using a
Time-varying GOPM
 Accepted by ECTI-CON 2010, Chiang Mai, Thailand
32

33. Time-varying GOPM
 Even though GOPM provides robustness to changing light
condition, however the static template will not
guaranteed that it yields the good result for all condition.
There are many uncontrolled factors such as skin and
noise.
 Time-varying GOPM uses the dynamic template which
update itself automatically in place of static template. It
reduce the difference of template environment in various
period of time.
33

34. Template Update Algorithm
 Step 1 : Perform GOPM, get best matching coordinate
current Template
BEST MATCH
Sample frame
34

35. Template Update Algorithm
 Step 2 : Crop region with the same size of old template
for creating new template
new Template
Sample frame
35

36. Correct-matching criterion
 1st Criterion Equation
where Nxn+1 and Nyn+1 are the normalized gradient of the newly created
template, Nxn and Nyn are the normalized gradient of the current
template, i and j are the size of the template.
is a threshold value defined as
 1st Criterion is used to check the correctness of the updated
template and prevent the jumping coordinate.
36

37. Correct-matching criterion
 2nd Criterion Equation
where Xn and Yn are the location of current matching result,
Xc and Yc are the location of the last known correct result.
T is a threshold value define as
 2nd Criterion is used to double check the jumping real
coordinate.
37

38. 1st Criterion
 STEP 1 : Find the difference b/w gradient component of
old template and new template in X and Y direction
x
- =
y
- =
Old Template New Template Template diff.
38

39. 1st Criterion
 STEP 2 : Combine the difference of x and y
+ =
Diff x Diff y Total Diff
 STEP 3 : Sum all elements and then thresholding
 If summation is less than the threshold function, update
template.
 If summation is more than the threshold function, discard the
new template.
39

40. 31
2nd Criterion
 Using the fact that It is impossible that the eye would
change its position suddenly in next frame.
(400,100)
(300,500)
Frame N Frame N+1
40

41. 2nd Criterion
 STEP 1 : Find the best matching of the frame N.
(x1,y1)
Frame N
41

42. 2nd Criterion
 STEP 2 : If location of N passed the criteria, the location
is used as a latest known correct position C.
Frame N
42

43. 2nd Criterion
 STEP 3 : Find best matching of the frame N+1.
(x2,y2)
Frame N+1
43

44. 2nd Criterion
 STEP 4 : Find Euclidean distance between position C and
position in frame N+1.
Frame N+1
44

45. 2nd Criterion
 STEP 5.1 : If distance more than threshold function,
discard the current location.
Threshold fcn
Frame N+1
45

46. 2nd Criterion
 STEP 5.2 : If distance less than threshold function, mask
the location as new corrected position C.
Threshold fcn
Frame N+1
46

47. Correct-matching criteria with Template
Update
47

48. Procedure
48

49. Downsampling
 In here, we resizes input video sequence and template to
50 percent of the height and width. Hence the
downsampled image is reduced to ¼ of the original size.
 Thus, computation time is 4 times faster.
 No effect to the matching result since both video and
template are downsampled with the same ratio
49

50. Result
Input GOPM technique Time-varying GOPM
Average Precision Average Precision
computation error (%) computation error (%)
time (ms) time (ms)
Normal lighting condition 62.48 0 13.81 0
(resolution: 320x240 px)
Change lighting condition 63.09 12.87 12.92 0
(resolution: 320x240 px)
Actual surgery camera 77.7 0 17.12 0
(resolution: 384x288 px)
Average computation time is decreased by downsampling
Currently this method can process at > 50 frame/sec which is enough
for most of video capture device that run at 25 frame/secs.
50

51. Result
Input GOPM technique Time-varying GOPM
Average Precision Average Precision
computation error (%) computation error (%)
time (ms) time (ms)
Normal lighting condition 62.48 0 13.81 0
(resolution: 320x240 px)
Change lighting condition 63.09 12.87 12.92 0
(resolution: 320x240 px)
Actual surgery camera 77.7 0 17.12 0
(resolution: 384x288 px)
In all cases, time-varying GOPM provide the better result than normal
GOPM. Especially for the case of changing light condition, which error is
decrease to 0%
Therefore time-varying GOPM can provide the robustness to
changing light condition with more precision than normal GOPM
51

52. Result
Properties of eye-tracking using SSD, GOPM , and time-varying GOPM
technique
SSD GOPM Time-varying
Obstacle robustness Yes Yes Yes
Blur robustness Yes Yes Yes
Lighting condition robustness No Yes Yes
Scaling robustness No No No
Average computation time About 50-350 About 50-350 About 10-90
ms ms ms
52

53. Drawback of time-varying GOPM
 In rarely case, when the 2nd criteria drop the frame
repeatedly, it causes the template slightly shifts from the
eye’s centroid.
Fail update
 However for real implementation prospective, if not be
too much, can be tolerated by the surgeon
53

54. Conclusion
54

55. Conclusion
 This work verify that the speed of template matching
technique with downsampling is able to implement in
real-time. (speed > 50 frame/secs).
 In the changing light condition, the result clearly shows
that GOPM is more robust than SSD.
 A time-varying GOPM reduce the difference of
template environment in various time and provides the
higher precision of tracking than normal GOPM.
55

56. Future Work
 Optimize the utilization of the threshold function in
corrected-matching criterion.
 Due to difference in camera specification such as
resolution or sensitivity, it required other advance
method to supervised the threshold function such as
machine learning or neuron network.
56

57. Acknowledgement
 Assist. Prof. Dr. Toshiaki Kondo
 Assoc.Prof. Dr. Waree Kongprawechnon
 Dr. Itthisek Nilkhamhang
 All faculty members and our beloved friends
57

58. Question and Answer
58