2. Tracking
Tracking is the problem of estimating the trajectory of
an object in the image plane as it moves around a
scene.
2 DSG, CEERI Pilani 12/8/2011
3. What is Motion Tracking…?
• Technologies that collect data on human movement
(input) used to control sounds, music, recorded or
projected text, video art, stage lighting (output) via
performer actions / gestures / movements / bio-data.
3 DSG, CEERI Pilani 12/8/2011
4. What is Motion Tracking…?
• other uses:
• Animation modeling (motion capture)
• Scientific research
(musicology, semantics, ergonomics, medicine, sports
medicine, architecture)
• Therapy for physically and mentally handicapped
4 DSG, CEERI Pilani 12/8/2011
5. Motion tracking vs. Motion capture
Motion capture Motion tracking
•Tracks location of fixed •less equipment, less data,
positions on body
•less cost ($1k-2k)
•Highly
accurate, expensive •concerned with motion
($200k-2m) qualities like
dynamic, direction of motion
•Generally not realtime
•real time
•Used for data collection
(research) and for making •used for live applications:
human or animal motion in installation
animations art, dance, theater and
(films, games, etc.) more
5 music
DSG, CEERI Pilani 12/8/2011
6. Motion capture
Motion capture is defined as "The creation of a 3D representation of
a live performance."
Tracks location of fixed positions on body with reflective markers
12-24 cameras, each lens is ringed with infrared lights
6 DSG, CEERI Pilani 12/8/2011
7. Motion capture
Motion capture is used to be considered a tool for
creating animation.
7 DSG, CEERI Pilani 12/8/2011
8. Motion capture
Typical uses
Human movement research
(sports, musicology, ergonomics, medicine,...)
Film and Animation -- often used with 3-D animation (modeling)
programs to make animations
maya (http://www.animationarena.com)
houdini (http://www.sidefx.com)
8 DSG, CEERI Pilani 12/8/2011
9. Motion capture
Vicon is a leading company in motion capture
9 DSG, CEERI Pilani 12/8/2011
10. Motion tracking
media output
sounds, musi
input c, text, projec
physical tions, lighting
human action
10 DSG, CEERI Pilani 12/8/2011
11. Motion tracking
sensor output device
(e.g. video computer (e.g. loud media output
camera) speakers)
input sounds, musi
physical c, text, projec
human action tions, lighting
analogue to digital to
digital analogue
conversion conversion
11 DSG, CEERI Pilani 12/8/2011
12. Digital vs. Analogue
Digital data
Analogue data
• easy to reproduce
• hard to reproduce
• "rich data" (infinite values) • lower resolution, less human-
• very high resolution feel.
• more details • easy to store
• contaminated data (becomes • easy to process
noisy, but rarely fails • contaminated data remains
completely) clean (errors can be filtered) or
signal fails altogether
12 DSG, CEERI Pilani 12/8/2011
13. What you need to track motion
Computer needs video input
Digital video (Firewire, USB2)
+ digital cameras (camcorder, webcams)
+ low noise
+ works with laptops
- latency issues
- image resolution issues (smaller chip sizes)
- limited cable length
Analog video
+ "unlimited" cable length
+ lower latency
+ even digital cams usually have analog output
- cost more (although many older cameras work quite well)
- works less well with laptops i.e. need an external or internal
framegrabber
13 DSG, CEERI Pilani 12/8/2011
14. Who is using motion tracking?
Palindrome Intermedia Performance Group
Krisztina de Chatel
Igloo
Ventura Dance (Pablo Ventura)
Robert Lepage
André Werner
Marlon Barrios Solano
La la la Human Steps
Georg Hobmeier
Leine Roebana Dans Kompanie
Troika Ranch
Blue Man Group
you
14 DSG, CEERI Pilani 12/8/2011
15. The Problem in Motion Tracking
Given a set of images in time which are similar but not
identical, derive a method for identifying the motion
that has occurred (in 2d) between different images.
16. Motion Detection and Estimation
in Literature
Image differencing
based on the threshold difference of successive images
difficult to reconstruct moving areas
Background subtraction
foreground objects result by calculating the difference between
an image in the sequence and the background image
(previously obtained)
remaining task: determine the movement of these foreground
objects between successive frames
Block motion estimation
Calculates the motion vector between frames for sub-blocks of
the image
mainly used in image compression
too coarse
17. What Is Optical Flow?
Optical flow is the displacement field for each of the
pixels in an image sequence.
For every pixel, a velocity vector dx , dy
is found which says: dt dt
how quickly a pixel is moving across the
image
the direction of its movement.
20. Estimation of the optical flow
Sequences of ordered images allow the estimation
of motion as either instantaneous image velocities or
discrete image displacements.
The optical flow methods try to calculate the motion
between two image frames which are taken at times
t and
t + δt at every voxel position.
20 DSG, CEERI Pilani 12/8/2011
21. Voxel Position
A voxel (volumetric pixel or Volumetric Picture
Element) is a volume element, representing a value
on a regular grid in three dimensional space.
A series of voxels in a stack with a single voxel
21 highlighted DSG, CEERI Pilani 12/8/2011
22. Estimation of the optical flow
Optical Flow methods are called differential since they
are based on local Taylor series approximations of the
image signal; that is, they use partial derivatives with
respect to the spatial and temporal coordinates.
In mathematics, a Taylor series is a representation of a
function as an infinite sum of terms that are calculated
from the values of the function's derivatives at a single
point.
22 DSG, CEERI Pilani 12/8/2011
23. Taylor series
The Taylor series of a real or complex function ƒ(x) that is infinitely
differentiable in a neighborhood of a real or complex number a is the
power series
which can be written in the more compact sigma notation as
where n! denotes the factorial of n and ƒ (n)(a) denotes the nth derivative of ƒ
evaluated at the point a. The zeroth derivative of ƒ is defined to be ƒ itself and (x −
a)0 and 0! are both defined to be 1. In the case that a = 0, the series is also called
a Maclaurin series.
23 DSG, CEERI Pilani 12/8/2011
24. Estimation of the optical flow
For a 2D+t dimensional case (3D or n-D cases are similar) a voxel at
location (x,y,t) with intensity I(x,y,t) will have moved by δx, δy and δt
between the two image frames, and the following image constraint
equation can be given:
I(x,y,t) = I(x + δx,y + δy,t + δt)
Assuming the movement to be small, the image constraint at I(x,y,t)
with Taylor series can be developed to get:
H.O.T
(higher-order terms)
24 DSG, CEERI Pilani 12/8/2011
26. Estimation of the optical flow
where Vx,Vy are the x and y components of the velocity or optical flow of
I(x,y,t) and are the derivatives of the image at (x,y,t)
in the
corresponding directions. Ix,Iy and It can be written for the derivatives in
the following.
Thus:
IxVx + IyVy = − It
or
This is an equation in two unknowns and cannot be solved as such. This is
known as the aperture problem of the optical flow algorithms. To find the
optical flow another set of equations is needed, given by some additional
constraint. All optical flow methods introduce additional conditions for
estimating the actual flow.
26 DSG, CEERI Pilani 12/8/2011
27. Our Solution
Optical flow: maximum one pixel large movements
Optical flow: larger movements
Morphological filter
Contour detection (demo purposes)
28. Optical Flow: maximum one pixel large
movements
The optical flow for a pixeli, j given 2 successive
images and k : k 1
mk (i, j ) ( x, y) so that
I k (i, j) I k 1 (i x, j y) (1)
is minimum for 1 x 1, 1 y 1
k k+1
29. Optical Flow: maximum one pixel
large movements
More precision: consider a 3×3 window
around the pixel:
Optical flow for pixel i, j becomes:
mk (i, j ) ( x, y) so that
1 1 1 1
I k (i u, j v) I k 1 (i u x, j v y) (2)
u 1v 1 u 1v 1
is minimum for 1 x 1, 1 y 1
30. Optical Flow: larger movements
Reduce the size of the image
=> reduced size of the movement
Solution: multi-resolution analysis of the images
Advantages: computing efficiency, stability
31. Multi-resolution Analysis
Coarse to fine optical flow estimation:
32 32
64 64
128 128
256 256
Original image k Original image k+1
32. Optical Flow: Top-down Strategy
Algorithm (1/4 scale of resolution reduction):
Step 1: compute optical flow vectors for the highest
level of the pyramid l (smallest resolution)
Step 2: double the values of the vectors
Step 3: first approximation: optical flow vectors for the
(2i, 2j), (2i+1, 2j), (2i, 2j+1), (2i+1, 2j+1) pixels in the l-
1 level are assigned the value of the optical flow
vector for the (i,j) pixel from the l level
Level l Level l-1
33. Optical Flow: Top-down Strategy
Step 4:
adjustment of the vectors of the l-1(one) level in the
pyramid
method: detection of maximum one pixel
displacements around the initially approximated
position
Step 5:
smoothing of the optical flow field (Gaussian
filter)
34. Filtering the Size of the Detected Regions
Small isolated regions of motion detected by the
optical flow method are classified as noise and are
eliminated with the help of morphological
operations:
Step 1: Apply the opening:
Step 2: Apply the B
X closing: B
X B
X B X B B
35. Contour Detection
For demonstration purposes, the contours of the moving
regions detected are outlined
Method: the Sobel edge detector:
Compute the intensity gradient: f f
f x, y , fx, f y (5)
x y
using the Sobel masks: 1 0 1 1 2 1
1 1
Gx 2 0 2 , Gy 0 0 0 (6)
4 4
1 0 1 1 2 1
Compute the magnitude of the gradient:
M x, y f x, y fx
2
fy
2
(7)
if M x, y threshold edge pixel
then
else non-edge pixel
37. Conclusions
What we did:
We managed to estimate the motion with a certain
level of accuracy
The results might be good enough for some
applications, while other applications might require
better accuracy
What remains to be done:
Reduce computational complexity
use the computed background image to separate
foreground objects
Parallelism of the algorithms
Experiment with specific problems, calibrate the
parameters of the algorithms