Super Resolution in Digital Image processing

1. Super-Resolution

2. Super-resolution
• convolutions, blur, and de-blurring
• Bayesian methods
• Wiener filtering and Markov Random Fields
• sampling, aliasing, and interpolation
• multiple (shifted) images
• prior-based methods
• MRFs
• learned models
• domain-specific models (faces)- Gary
3/7/2003
Super-Resolution
2

3. Linear systems
Basic properties
• homogeneity T[a X]
T[X1+X2]
• additivity
= a T[X]
= T[X1]+T[X2]
• superposition T[aX1+bX2] = aT[X1]+bT[X2]
Linear system ⇔ superposition
Examples:
• matrix operations (additions, multiplication)
• convolutions
3/7/2003
Super-Resolution
3

4. Signals and linear operators
Continuous
Discrete
Vector form
I(x)
I[k] or Ik
I
Discrete linear operator
y=Ax
Continuous linear operator:
convolution integral
g(x) = s h(ξ,x) f(ξ) dξ, h(ξ,x): impulse response
g(x) = s h(ξ-x) f(ξ) dξ= [f * h](x) shift invariant
3/7/2003
Super-Resolution
4

5. 2-D signals and convolutions
Continuous
Discrete
I(x,y)
I[k,l] or Ik,l
2-D convolutions (discrete)
g[k,l] = ∑m,n f[m,n] h[k-m,l-n]
= ∑m,n f[m,n] h1[k-m]h2[l-n]
separable
Gaussian kernel is separable and radial
h(x,y) = (2πσ2)-1exp-(x2+y2)/σ2
3/7/2003
Super-Resolution
5

6. Convolution and blurring
3/7/2003
Super-Resolution
6

7. Separable binomial low-pass filter
3/7/2003
Super-Resolution
7

8. Fourier transforms
Project onto a series of complex sinusoids
F[m,n] = ∑k f[k,l] e-i 2π(km+ln)
Properties:
• shifting
g(x-x0) ⇔ exp(-i 2πfxx0)G(fx)
• differentiation dg(x)/dx ⇔ i 2πfxG(fx)
• convolution
3/7/2003
[f * g](x) ⇔ [F G] (fx)
Super-Resolution
8

9. Blurring examples
Increasing amounts of blur + Fourier transform
3/7/2003
Super-Resolution
9

10. Sharpening
Unsharp mask (darkroom photography):
• remove some low-frequency content
y’ = y + s (y – g * y)
spatial (blur, sharp)
3/7/2003
Super-Resolution
freq (blur,sharp)
10

11. Sharpening - result
Unsharp mask: original, blur (σ=1),
sharp(s=0, 1, 2)
3/7/2003
Super-Resolution
11

12. Deconvolution
Filter by inverse of blur
• easiest to do in the Fourier domain
• problem: high-frequency noise amplification
3/7/2003
Super-Resolution
12

13. Bayesian modeling
Use prior model for image and noise
• y = g * x + n, x is original, y is blurred
• p(x|y) = p(y|x)p(x)
= exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2)
• -log p(x|y) ∝ |y – g*x|2σn-2 + |x|2σx-2
where the norm || is summed squares over all
pixels
3/7/2003
Super-Resolution
13

14. Parseval’s Theorem
Energy equivalence in spatial ↔ frequency
domain
• |x|2 = |F(x)|2
• -log p(x|y) ∝ |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2
• least squares solution (∂/∂X = 0)
X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
3/7/2003
Super-Resolution
14

15. Wiener filtering
Optimal linear filter given noise and signal
statistics
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
• low frequencies:
X(f) ≈ G-1(f)Y(f)
boost by inverse gain (blur)
X(f) ≈ G(f) σn-2σx2 Y(f)
• high frequencies:
attenuate by blur (gain)
3/7/2003
Super-Resolution
15

16. Wiener filtering – white noise prior
Assume all frequencies equally likely
• p(x) ~ N(0,σx2)
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
• solution is too noisy in high frequencies
3/7/2003
Super-Resolution
16

17. Wiener filtering – pink noise prior
Assume frequency falloff (“natural statistics”)
• p(X(f)) ~ N(0,|f|-βσx2)
• X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2]
• greater attenuation at high frequencies
G(f)
3/7/2003
H(f)
Super-Resolution
17

18. Markov Random Field modeling
Use spatial neighborhood prior for image
i
• -log p(x) = ∑ij∈Cρ(xi-xj)
where ρ(v) is a robust norm:
•
•
•
•
j
ρ(v) = v2: quadratic norm ↔ pink noise
ρ(v) = |v|: total variation (popular with maths)
ρ(v) = |v|β: natural statistics
ρ(v) = v2,|v|: Huber norm
[Schultz, R.R.; Stevenson, IEEE TIP, 1996]
3/7/2003
Super-Resolution
18

19. MRF estimation
Set up discrete energy (quadratic or non-)
• -log p(x|y) ∝ σn-2 |y – Gx|2 + ∑ij∈Cρ(xi-xj)
where G is sparse convolution matrix
• quadratic: solve sparse linear system
• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient
descent, conjugate gradient, …)
3/7/2003
Super-Resolution
19

20. Sampling a signal
• sampling:
• creating a discrete signal from a continuous signal
• downsampling (decimation)
• subsampling a discrete signal
• upsampling
• introducing zeros between samples
• aliasing
• two sampled signals that differ in their original
form (many → one mapping)
3/7/2003
Super-Resolution
20

21. Sampling
interpolation
3/7/2003
Super-Resolution
21

22. Nyquist sampling theorem
Signal to be (down-) sampled must have a
bandwidth no larger than twice the sample
frequency
ωs = 2π / ns > 2 ω0
3/7/2003
Super-Resolution
22

23. Box filter (top hat)
3/7/2003
Super-Resolution
23

24. Ideal low-pass filter
3/7/2003
Super-Resolution
24

25. Simplified camera optics
1.
2.
3.
4.
Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian
Integrate = box filter
Sample = produce single digital sample
Noise = additive white noise
3/7/2003
Super-Resolution
25

26. Aliasing
Aliasing (“jaggies” and “crawl”) is present if
blur amount < sampling (σ = 1)
• shift each image in previous pipeline by 1
3/7/2003
Super-Resolution
26

27. Aliasing - less
Less aliasing (“jaggies” and “crawl”) is present if
blur amount ~ sampling (σ = 2)
• shift each image in previous pipeline by 1
3/7/2003
Super-Resolution
27

28. Multi-image super-resolution
Exploit aliasing to recover frequencies above
Nyquist cutoff
∀ ∑kσn-2 |yk – Gkx|2 + ∑ij∈Cρ(xi-xj)
where Gk are sparse convolution matrices
• quadratic: solve sparse linear system
• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient
descent, conjugate gradient, …)
• projection onto convex sets (POCS)
3/7/2003
Super-Resolution
28

29. Multi-image super-resolution
Need:
• accurate (sub-pixel) motion estimates
(Wednesday’s lecture)
• accurate models of blur (pre-filtering)
• accurate photometry
• no (or known) non-linear pre-processing
(Bayer mosaics)
• sufficient images and low-noise relative to
amount of aliasing
3/7/2003
Super-Resolution
29

30. Prior-based Super-Resolution
“Classical” non-Gaussian priors:
• robust or natural statistics
• maximum entropy (least blurry)
• constant colors (black & white images)
3/7/2003
Super-Resolution
30

31. Example-based Super-Resolution
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,
IEEE Computer Graphics and Applications, March/April, 2002
• learn the association between low-resolution
patches and high-resolution patches
• use Markov Network Model (another name for
Markov Random Field) to encourage adjacent
patch coherence
3/7/2003
Super-Resolution
31

32. Example-based Super-Resolution
William T. Freeman, Thouis R.
Jones, and Egon C.
Pasztor,
IEEE Computer Graphics
and Applications,
March/April, 2002
3/7/2003
Super-Resolution
32

33. References – “classic”
Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image
Processing, 53(3), May 1991, 231-239.
Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE
Trans. Image Proc., 5(6), Jun 1996, 996-1011.
Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy,
and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 16461658.
Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep
1999, 817-834.
Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000,
I-600-605 vol. 1.
Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001.
3/7/2003
Super-Resolution
33

34. References – strong priors
Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution,
IEEE Computer Graphics and Applications, March/April, 2002
Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88.
Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces:
global parametric model and local nonparametric model. CVPR 2001. I-192-8.
08/03/2014
Super-Resolution
34