Computer Vision For Fun and Profit

Computer Vision for Fun
and Proﬁt
Image Processing: The Lowdown
!
!
Steven Mitchell, Ph.D.
Componica, LLC

Copyright 2011 - Componica, LLC (http://www.componica.com/)
What does Componica have to offer?
Decades of Experience and Expertise

A large library of code generated internally

A community of innovative Computer Engineers and Programmers

Access to academic papers and a history of eﬀectively applying archived
research

libSeal - A long term project to take previously written code and turn it into a
single library:

SEAL - Steve's Evil (or Eclectic) Algorithms Library

Image Processing
Any sort of signal processing done to an image

The Acquisition of an image

Compression and storage of an image

Enhancement and restoration

Registration of an image to another.

Measurement of data from the image (height of
building, speed of car)

Interpretation and Recognition of objects.

Getting the Images in the First Place
Standard image files:

GIF, PNG, JPG: Standard. I tend to always use PNGs

TIFF, PDF: More challenging, requires an external library

Video files:

FFMpeg: Without it, it would be a nightmare to read video files but, if it can't read a video
file, I can't read that video file.

Web Cam: OpenCV has a simple way of doing it. Outside OpenCV, there are scattered
libraries

Mobile Devices

Scanner

MRI and CT scanners: Use a file format called DICOM

A Menu of Tools:
Image Enhancement - Computer...uncrop and enhance!
Image Segmentation - These pixels belong to this, those pixels belong to
that.
Image Registration - Line this image up with that image.
Object Recognition - This is a image of a frog and that's an image of a
cheeseburger.
Image Compression - Crush this image and make sure the process is undo-
able. (Not Covered...there are plenty of free libraries to do this.)

Image Enhancement
Simple Pixel Stuﬀ:
Normalizing brightness and contrast.

Gamma correction (the non-linear
eﬀects of TV)

Histogram equalization - maximize the
global contrast.

Color Correction...color temperature
and tint. White balancing.

Geometric Stuﬀ:
Interpolation. Make the image bigger or smaller. This is not as easy as it sounds if you
want non-aliased results. Trips up web developers whenever they try to roll their own
thumbnail generator.

Warping an image from one geometry to another.

Simple rotation, scale, and translation. You need two or three landmark points.

Perspective (aka Projection or Homogeneous) transforms. You need four landmark
points.

Lens distortions. Images pinch or barrel out in camera lens. You can calibrate and
correct for that with enough landmark points.

General warping. Typically used for image morphing in special eﬀects.
Image Enhancement - Geometry

Image Enhancement - Interpolation
AMATEUR
PRO
UPSAMPLING DOWNSAMPLINGORIGINALS

Image Enhancement - Geometry
ORIGINAL IMAGE PERSPECTIVE CORRECTED
The original image is warped to a perspective corrected version.

The four black dots indicated the landmark points used to normalize the image
to this artiﬁcial view.

The black regions are areas that falls outside the original image. Unknown data.

Noise removing. Median ﬁlter, average ﬁlter, etc.

De-blurring. You estimate the cause of the blurring
and then undo it using deconvolution.

Motion blur - Estimate the camera movements
and undo it.

Focus blur - Estimate the lens blur and undo it.
Image Enhancement
http://cg.postech.ac.kr/research/fast_motion_deblurring/

Divide an image into known parts:

It's not quite object recognition because images are typically interpreted
on a pixel basis using edge detection or colors.

Sometimes it's good enough because you just care about the borders,
not content

Border of tumor vs. healthy tissue.

The bright red ball in the color picture.

Is this pixel part of a letter ‘q’ or paper?

Often the ﬁrst step to a bigger solution.
Image Segmentation

Make a decision based on a single pixel:

Simple thresholding - is this pixel darker than 160?
Slightly better - is this pixel red?
Even more better - is this pixel statistically more likely
to be paper or letter based on it's RGB value.
The works - I'm modeling the uneven-ness of the
lighting on this paper and made a statistical model of
RGB, is this damn pixel ink or paper?
The downfall of this is you're looking at single pixels
without understand how it's neighbors relate to it, missing
the whole picture. Sometimes it works. Fast and easy to
do.
Image Segmentation - Pixel Classes
ORIGINAL
SIMPLE
THRESHOLD
ADAPTIVE
BINARIZATION

Minimizing or maximizing a path which borders between
elements. This one is a common technique, with many variants:

Assign a cost to all the pixels. For example, edge detection
- it will cost me a lot to cross an edge. Or color transition, it
will cost me to cross to a diﬀerent color.

Use an optimization technique from classic data structures
(typically used in graph theory if you still remember) to
compute the cheapest path from one location of the image
to a diﬀerent location.

Dynamic Programming - Strange name but all it means is
compute the cheapest path from one side of an image to
another. Works best for paths that tend to be linear.

Minimum Graph Cut - Find the cheapest way to split the
image in two regions. This works well for circular paths and
3D. Tends to be much more complicated than dynamic
programming and slower.

Check out this: http://www.youtube.com/watch?v=6NcIJXTlugc
Image Segmentation - Least Cost Path

Random Decision Forests applied to pixels
Apply twenty questions to pixel and it's
surrounding neighbors to create decision trees
to guess what this pixel is over.

Have a number of these decision trees (forest)
and aggregate the results. Strangely it tends to
works in many cases.
Image Segmentation - Bleeding Edge

I have a source image that I can transform (move, resize,
rotate, or bend). Make it best ﬁt a target image.

Translation Only - Shift the source image until it best
ﬁts the target.

Similarity Transform - Adjust the scale, rotation, and
translation until the source image overlaps the target.

Perspective Transform - Move the source image’s four
corners until it matches the target in a perspective
manner.

Non-Rigid Warping - The source image is on a rubber
sheet. Warp it onto the target image.

Obvious Applications:

Augmented Reality

Image Stitching

Object Detection
Image Registration

The most common way to register as image is the following:

Find the most interesting points on the two images (usually
blobs and corners).

Scale-invariant feature transform - SIFT (Patented and
slow)

Speeded Up Robust Features - SURF (Recently Patented
and fast)

Features from Accelerated Segment Test - FAST (Very
fast but noisy)

With SURF and SIFT you get a position, an angle, scale
factor and a 64 element vector to compare. With FAST
you get a position and maybe a rotation.

Compare all the interesting points from one image to the other
forming matching pairs of points between images. Naive
implementations are SLOW.

Use RANSAC to ﬁnd a consensus (next page).
Image Registration - Interesting Points
SURF
FAST

Matching points locally doesn’t yield global matches.

RANdom Sample Consensus - RANSAC - prunes away the
mismatches and computes the transform that converts the
source image to the target.

It works as followed:

Most transforms can be described by a minimal number of
points. For similarity transforms it’s two points, for
projection transforms it’s four points.

Pick two (or four) matched pair of points at random.

Compute a transform from those two (or four) sets of points.

Transform all the source points using this transform and see
how many points are close to the target.

Repeat this and keep the best transform that matches the
most points.
Image Registration - Interesting Points
Here the green lines indicate pairs of matched points
that ﬁt the transform (looks like a similarity transform)
and the red lines are matched pairs that failed to ﬁt this
transform and therefore rejected.!
!
RANSAC may seem ad hoc, but it works surprisingly
well.

A very accurate way to line up a template onto an
image is to use derivates and linear algebra.

Works very well only when the images are very close
to each other in the ﬁrst place.

It’s usually the polishing step after using an
Interesting Points / RANSAC method.

Known as Lucas-Kanade Tracker -or- Baker-
Matthews Tracker.

It’s the secret sauce to how the “Predator” algorithm
works.
Image Registration - The Mathy Way
http://www.wedesoft.demon.co.uk/lucas-kanade-tracker.html

Making computers recognize objects in images

Many different algorithms, and each algorithm has appropriate uses
depending on the objects being detected. For example:

AdaBoost - Awesome at detecting and locating faces, sucks at
recognizing who’s face it is.

Deep Neural Networks - Works great for highly distorted text, but too
slow for generalized OCR.

Active Appearance Models - Works great for both recognition and
segmentation, but only on generally fixed shapes like faces and hearts.
Sucks for livers and cancer.

Most recognizers depend on training the algorithm on known objects offline
and then testing. Which brings up the topic of data...
Object Recognition

The data can be the most valuable part of a
trainable system as many algorithms will
generally function with somewhat similar hit rates.

Often ideas fail to take into account where the
data comes from. It’s the killer of many ideas.

The basics of training something:

Data is normally split into a training and
testing set. You train a thingy with the
training set and test how well it works with
the testing set.

Why? Most trainable thingies are prone to
overfitting. Splitting the data into two sets
prevents this problem because you use the
test set to know when to stop training.

Disadvantage, you effectively need twice as
much data. Sucks.
Object Recognition - It’s all about the Data
It’s obvious this data is best represented as a line, but!
if the model over-fits the data, it may compute a!
relationship that’s nonsensical.
As your algorithm learns from a training set (blue line),!
the error decreases for that set, but in the testing set,!
it will hit a point where overfitting is happening and!
will increase the overall error in the real world. You stop!
training at the point the testing set starts getting worse.

Steve’s crappy breakdown of object recognition algorithms:

A ship of fools approach - Armies of stupid algorithms that together become smart. Kinda like
democracy sort of...not really.

Let’s create a brain, Igor - So what happens if you simulate brain tissue? It’s grown quite a bit
since the neural network hype in the late 80s / early 90s. Fun Fact: This will eventually kill us all in a
bloody uprising.

If the shoe ﬁts... - Well if this template StarBucks logo ﬁts somewhere on my image using object
registration as described above, then I’m guessing this StarBucks logo is present in the image.
Duh.

Find Features to Filter and Fit in a Feed-forward Fashion - You see this pattern all the time and
it lacks of creativity. Find interesting features in the image, and feed them into a trainable function
like a neural network, a non-linear regression, or a support vector machine. Boring.

I’ve stared at a wall for 20 minutes now and I think everything out there is either one or a combination of
these general ideas. Hmm...That’s it? Disappointed.
Object Recognition - The Menu

Computer Vision For Fun and Profit

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Recently uploaded

Recently uploaded (20)

Computer Vision For Fun and Profit