1. Computer vision:
models, learning and inference
Chapter 10
Graphical Models
Please send errata to s.prince@cs.ucl.ac.uk

2. Models for grids
• Consider models where one unknown world state
at each pixel in the image – takes the form of a grid.
• Loops in the graphical model so cannot use dynamic
programming or belief propagation
• Define probability distributions that favour certain
configurations of world states
– Called Markov random fields
– Inference using a set of techniques called graph cuts
2
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

3. Binary Denoising
Before After
Image represented as binary discrete variables. Some proportion of pixels
randomly changed polarity.
3
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

4. Multi-label Denoising
Before After
Image represented as discrete variables representing intensity. Some
proportion of pixels randomly changed according to a uniform distribution.
4
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

5. Denoising Goal
Observed Data Uncorrupted Image
5
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

6. Denoising Goal
Observed Data Uncorrupted Image
• Most of the pixels stay the same
• Observed image is not as smooth as original
Now consider pdf over binary images that
encourages smoothness – Markov random field
6
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

7. Markov random fields
This is just the typical property of an undirected model.
We’ll continue the discussion in terms of undirected models
7
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

8. Markov random fields
Normalizing constant
(partition function) Potential function
Returns positive number
Subset of variables
(clique)
8
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

9. Markov random fields
Normalizing constant Cost function
(partition function) Returns any number
Subset of variables
Relationship (clique)
9
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

10. Smoothing example
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 10

11. Smoothing Example
Smooth solutions (e.g. 0000,1111) have high probability
Z was computed by summing the 16 un-normalized probabilities
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 11

12. Smoothing Example
Samples from larger grid -- mostly smooth
Cannot compute partition function Z here - intractable
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 12

13. Denoising Goal
Observed Data Uncorrupted Image
13
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

14. Denoising overview
Bayes’ rule:
Likelihoods: Probability
of flipping
polarity
Prior: Markov random field (smoothness)
MAP Inference: Graph cuts
14
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

15. Denoising with MRFs
MRF Prior (pairwise cliques)
Original image, w
Likelihoods
Observed image, x
Inference :
15
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

16. MAP Inference
Unary terms Pairwise terms
(compatability of data with label y) (compatability of neighboring labels)
16
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

17. Graph Cuts Overview
Graph cuts used to optimise this cost function:
Unary terms Pairwise terms
(compatability of data with label y) (compatability of neighboring labels)
Three main cases:
17
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

18. Graph Cuts Overview
Graph cuts used to optimise this cost function:
Unary terms Pairwise terms
(compatability of data with label y) (compatability of neighboring labels)
Approach:
Convert minimization into the form of a standard CS problem,
MAXIMUM FLOW or MINIMUM CUT ON A GRAPH
Polynomial-time methods for solving this problem are known
18
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

19. Max-Flow Problem
Goal:
To push as much ‘flow’ as possible through the directed graph from the source
to the sink.
Cannot exceed the (non-negative) capacities cij associated with each edge.
19
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

20. Saturated Edges
When we are pushing the maximum amount of flow:
• There must be at least one saturated edge on any path from source to sink
(otherwise we could push more flow)
• The set of saturated edges hence separate the source and sink
20
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

21. Augmenting Paths
Two numbers represent: current flow / total capacity
21
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

22. Augmenting Paths
Choose any route from source to sink with spare capacity and push as much flow as
you can. One edge (here 6-t) will saturate. 22
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

23. Augmenting Paths
Choose another route, respecting remaining capacity. This time edge 5-6 saturates.
23
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

24. Augmenting Paths
A third route. Edge 1-4 saturates
24
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

25. Augmenting Paths
A fourth route. Edge 2-5 saturates
25
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

26. Augmenting Paths
A fifth route. Edge 2-4 saturates
26
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

27. Augmenting Paths
There is now no further route from source to sink – there is a saturated edge along
every possible route (highlighted arrows) 27
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

28. Augmenting Paths
The saturated edges separate the source from the sink and form the min-cut solution.
Nodes either connect to the source or connect to the sink. 28
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

29. Graph Cuts: Binary MRF
Graph cuts used to optimise this cost function:
Unary terms Pairwise terms
(compatability of data with label w) (compatability of neighboring labels)
First work with binary case (i.e. True label w is 0 or 1)
Constrain pairwise costs so that they are “zero-diagonal”
29
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

30. Graph Construction
• One node per pixel (here a 3x3 image)
• Edge from source to every pixel node
• Edge from every pixel node to sink
• Reciprocal edges between neighbours
Note that in the minimum cut
EITHER the edge connecting to
the source will be cut, OR the
edge connecting to the sink, but
NOT BOTH (unnecessary).
Which determines whether we
give that pixel label 1 or label 0.
Now a 1 to 1 mapping between
possible labelling and possible
minimum cuts
30
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

31. Graph Construction
Now add capacities so that
minimum cut, minimizes our cost
function
Unary costs U(0), U(1) attached
to links to source and sink.
• Either one or the other is paid.
Pairwise costs between pixel
nodes as shown.
• Why? Easiest to understand
with some worked examples.
31
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

32. Example 1
32
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

33. Example 2
33
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

34. Example 3
34
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

35. 35
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

36. Graph Cuts: Binary MRF
Graph cuts used to optimise this cost function:
Unary terms Pairwise terms
(compatability of data with label w) (compatability of neighboring labels)
Summary of approach
• Associate each possible solution with a minimum cut on a graph
• Set capacities on graph, so cost of cut matches the cost function
• Use augmenting paths to find minimum cut
• This minimizes the cost function and finds the MAP solution
36
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

37. General Pairwise costs
Modify graph to
• Add P(0,0) to edge s-b
• Implies that solutions 0,0 and
1,0 also pay this cost
• Subtract P(0,0) from edge b-a
• Solution 1,0 has this cost
removed again
Similar approach for P(1,1)
37
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

38. Reparameterization
The max-flow / min-cut algorithms
require that all of the capacities are
non-negative.
However, because we have a
subtraction on edge a-b we cannot
guarantee that this will be the
case, even if all the original unary and
pairwise costs were positive.
The solution to this problem is
reparamaterization: find new graph
where costs (capacities) are different
but choice of minimum solution is the
same (usually just by adding a constant
to each solution)
38
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

39. Reparameterization 1
The minimum cut chooses the same links in these two graphs
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

40. Reparameterization 2
The minimum cut chooses the same links in these two graphs 40
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

41. Submodularity
Subtract constant
Add constant,
Adding together implies
41
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

42. Submodularity
If this condition is obeyed, it is said that the problem is “submodular” and it can
be solved in polynomial time.
If it is not obeyed then the problem is NP hard.
Usually it is not a problem as we tend to favour smooth solutions.
42
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

43. Denoising Results
Pairwise costs increasing
Original
Pairwise costs increasing 43
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

44. Plan of Talk
• Denoising problem
• Markov random fields (MRFs)
• Max-flow / min-cut
• Binary MRFs – submodular (exact solution)
• Multi-label MRFs – submodular (exact solution)
• Multi-label MRFs - non-submodular (approximate)
44
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

45. Multiple Labels
Construction for two pixels
(a and b) and four labels
(1,2,3,4)
There are 5 nodes for each
pixel and 4 edges between
them have unary costs for
the 4 labels.
One of these edges must be
cut in the min-cut solution
and the choice will
determine which label we
assign. 45
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

46. Constraint Edges
The edges with infinite
capacity pointing upwards
are called constraint
edges.
They prevent solutions
that cut the chain of
edges associated with a
pixel more than once (and
hence given an ambiguous
labelling)
46
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

47. Multiple Labels
Inter-pixel edges have costs
defined as:
Superfluous terms :
For all i,j where K is number of labels47
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

48. Example Cuts
Must cut links from before cut on pixel a to after cut on pixel b. 48
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

49. Pairwise Costs
Must cut links from before cut on
pixel a to after cut on pixel b.
Costs were carefully chosen so
that sum of these links gives
appropriate pairwise term.
If pixel a takes label I and pixel b takes label J
49
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

50. Reparameterization
50
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

51. Submodularity
We require the remaining inter-pixel links to be
positive so that
or
By mathematical induction we can get the more
general result
51
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

52. Submodularity
If not submodular then the problem is NP hard. 52
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

53. Convex vs. non-convex costs
Quadratic Truncated Quadratic Potts Model
• Convex • Not Convex • Not Convex
• Submodular • Not Submodular • Not Submodular
53
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

54. What is wrong with convex costs?
Observed noisy image Denoised result
• Pay lower price for many small changes than one large one
• Result: blurring at large changes in intensity 54
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

55. Plan of Talk
• Denoising problem
• Markov random fields (MRFs)
• Max-flow / min-cut
• Binary MRFs - submodular (exact solution)
• Multi-label MRFs – submodular (exact solution)
• Multi-label MRFs - non-submodular (approximate)
55
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

56. Alpha Expansion Algorithm
• break multilabel problem into a series of binary problems
• at each iteration, pick label and expand (retain original or change to )
Initial Iteration 1
labelling (orange)
Iteration 2 Iteration 3
(yellow) (red)
56
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

57. Alpha Expansion Ideas
• For every iteration
– For every label
– Expand label using optimal graph cut solution
Co-ordinate descent in label space.
Each step optimal, but overall global maximum not guaranteed
Proved to be within a factor of 2 of global optimum.
Requires that pairwise costs form a metric:
57
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

58. Alpha Expansion Construction
Binary graph cut – either cut link to source
(assigned to ) or to sink (retain current label)
Unary costs attached to links between
58
source, sink andSimon J.D.nodes appropriately.
Computer vision: models, learning and inference. ©2011
pixel Prince

59. Alpha Expansion Construction
Graph is dynamic. Structure of inter-pixel links
depends on and the choice of labels.
There are four cases.
59
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

60. Alpha Expansion Construction
Case 1:
Adjacent pixels both have label already.
Pairwise cost is zero – no need for extra edges.
60
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

61. Alpha Expansion Construction
Case 2: Adjacent pixels are .
Result either
• (no cost and no new edge).
• (P( ), add new edge).
61
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

62. Alpha Expansion Construction
Case 3: Adjacent pixels are . Result either
• (no cost and no new edge).
• (P( ), add new edge).
• (P( ), add new edge).
62
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

63. Alpha Expansion Construction
Case 4: Adjacent pixels are . Result either
• (P( ), add new edge).
• (P( ), add new edge).
• (P( ), add new edge).
63
• (no cost Prince no new edge).
Computer vision: models, learning and inference. ©2011 Simon J.D.
and

64. Example Cut 1
64
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

65. Example Cut 1
Important!
65
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

66. Example Cut 2
66
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

67. Example Cut 3
67
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

68. Denoising
Results
68
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

69. Conditional Random Fields
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 69

70. Directed model for grids
Cannot use graph cuts as three-wise term. Easy to draw samples.
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 70

71. Applications
• Background subtraction
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 71

72. Applications
• Grab cut
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 72

73. Applications
• Stereo vision
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 73

74. Applications
• Shift-map image editing
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 74

75. Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 75

76. Applications
• Shift-map image editing
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 76

77. Applications
• Super-resolution
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 77

78. Applications
• Texture synthesis
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 78

79. Image Quilting
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 79

80. Applications
• Synthesizing faces
Computer vision: models, learning and inference. ©2011 Simon J.D. Prince 80