Signal Processing Course : Compressed Sensing

Compressive
Sensing
Gabriel Peyré
www.numerical-tours.com

Overview

•Shannon’s World
•Compressive Sensing Acquisition
•Compressive Sensing Recovery
•Theoretical Guarantees
•Fourier Domain Measurements

Discretization

Sampling:
˜ acquisition N
f L ([0, 1] )
2 d f R
device

Idealization: ˜
f [n] ⇡ f (n/N )

Pointwise Sampling and Smoothness
Data aquisition: ˜
f [i] = f (i/N )

Sensors

˜
f L2 f RN

Data aquisition: ˜
f [i] = f (i/N )

Sensors

˜
f L2 f RN
ˆ
˜
Shannon interpolation: if Supp(f ) [ N ,N ]
˜ sin( t)
f (t) = f [i]h(N t i) h(t) =
i
t

Data aquisition: ˜
f [i] = f (i/N )

Sensors

˜
f L2 f RN
ˆ
˜
˜ sin( t)
f (t) = f [i]h(N t i) h(t) =
i
t
Natural images are not smooth.

Data aquisition: ˜
f [i] = f (i/N )

Sensors JPEG-2k
0,1,0,. . .

˜
f L2 f RN
ˆ
˜
˜ sin( t)
f (t) = f [i]h(N t i) h(t) =
i
t
Natural images are not smooth.
But can be compressed e ciently.
Sample and compress simultaneously?

Sampling and Periodization

(a)

(b)

1

(c) 0

(d)

Sampling and Periodization: Aliasing

(a)

(b)

1

(c) 0

(d)

Single Pixel Camera (Rice)
˜
f

˜
f

y[i] = f, i
P measures N micro-mirrors

˜
f

y[i] = f, i
P measures N micro-mirrors

P/N = 1 P/N = 0.16 P/N = 0.02

CS Hardware Model
˜
CS is about designing hardware: input signals f L2 (R2 ).
Physical hardware resolution limit: target resolution f RN .

array micro
˜
f L2 f RN mirrors y RP
resolution
K
CS hardware

CS Hardware Model
˜

array micro
˜
resolution
K
CS hardware

,
,
...

,

CS Hardware Model
˜

array micro
˜
resolution
K
CS hardware

,
Operator K
, f
...

,

Inversion and Sparsity
Operator K
Need to solve y = Kf .
More unknown than equations. f

dim(ker(K)) = N P is huge.

Inversion and Sparsity
Operator K
Need to solve y = Kf .
More unknown than equations. f

dim(ker(K)) = N P is huge.

Prior information: f is sparse in a basis { m }m .

J (f ) = Card {m | f, m |> } is small.

f f, m

Convex Relaxation: L1 Prior
J0 (f ) = # {m ⇥f, m⇤ = 0}
J0 (f ) = 0 null image.
Image with 2 pixels: J0 (f ) = 1 sparse image.
J0 (f ) = 2 non-sparse image.

q=0

J0 (f ) = # {m ⇥f, m⇤ = 0}

q=0 q = 1/2 q=1 q = 3/2 q=2
q
priors: Jq (f ) = | f, q
m ⇥| (convex for q 1)
m

J0 (f ) = # {m ⇥f, m⇤ = 0}

q=0 q = 1/2 q=1 q = 3/2 q=2
q
priors: Jq (f ) = | f, m ⇥|
q
(convex for q 1)
m

Sparse 1
prior: J1 (f ) = | f, m ⇥|
m

Sparse CS Recovery
f0 RN
f0 RN sparse in ortho-basis

N
x0 R

Sparse CS Recovery
f0 RN

(Discretized) sampling acquisition:
y = Kf0 + w = K (x0 ) + w
=

N
x0 R

Sparse CS Recovery
f0 RN

y = Kf0 + w = K (x0 ) + w
=
K drawn from the Gaussian matrix ensemble
Ki,j N (0, P 1/2
) i.i.d.
drawn from the Gaussian matrix ensemble
N
x0 R

Sparse CS Recovery
f0 RN

y = Kf0 + w = K (x0 ) + w
=
K drawn from the Gaussian matrix ensemble
Ki,j N (0, P 1/2
) i.i.d.
drawn from the Gaussian matrix ensemble
N
Sparse recovery: x0 R
||w|| 1
min ||x||1 min || x y|| + ||x||1
2
|| x y|| ||w|| x 2

CS Simulation Example

Original f0
= translation invariant
wavelet frame

CS with RIP

1
recovery:
y= x0 + w
x⇥
argmin ||x||1 where
|| x y|| ||w||

Restricted Isometry Constants:
⇥ ||x||0 k, (1 k )||x||2 || x||2 (1 + k )||x||2

CS with RIP

1
recovery:
y= x0 + w
x⇥
argmin ||x||1 where
|| x y|| ||w||

Restricted Isometry Constants:
⇥ ||x||0 k, (1 k )||x||2 || x||2 (1 + k )||x||2

Theorem: If 2k2 1, then [Candes 2009]
C0
||x0 x || ⇥ ||x0 xk ||1 + C1
k
where xk is the best k-term approximation of x0 .

Singular Values Distributions
Eigenvalues of I I with |I| = k are essentially in [a, b]
a = (1 ) 2
and b = (1 )2
where = k/P
When k = P + , the eigenvalue distribution tends to
1
f (⇥) = (⇥ b)+ (a ⇥)+ [Marcenko-Pastur]
1.5
2⇤ ⇥ P=200, k=10

P=200, k=10

f ( )
1.5
1

1
0.5

P = 200, k = 10
0.5
0
0 0.5 1 1.5 2 2.5
0
0 0.5 1 P=200, k=30 1.5 2 2.5
1
P=200, k=30
0.8
1

0.6
0.8

0.4

k = 30
0.6

0.2
0.4

0
0.2
0 0.5 1 1.5 2 2.5
0
0 0.5 1 P=200, k=50 1.5 2 2.5

0.8 P=200, k=50

0.6
0.8

0.6
0.4
Large deviation inequality [Ledoux]
0.2
0.4

RIP for Gaussian Matrices

Link with coherence: µ( ) = max | i, j ⇥|
i=j
2 = µ( )
k (k 1)µ( )


i=j
2 = µ( )
k (k 1)µ( )

For Gaussian matrices:
µ( ) log(P N )/P


i=j
2 = µ( )
k (k 1)µ( )

For Gaussian matrices:
µ( ) log(P N )/P
Stronger result:
C
Theorem: If k P
log(N/P )
then 2k 2 1 with high probability.

Numerics with RIP
Stability constant of A:
(1 ⇥1 (A))|| ||2 ||A ||2 (1 + ⇥2 (A))|| ||2

smallest / largest eigenvalues of A A

Numerics with RIP
Stability constant of A:
(1 ⇥1 (A))|| ||2 ||A ||2 (1 + ⇥2 (A))|| ||2

smallest / largest eigenvalues of A A

Upper/lower RIC:
ˆ2
k
i
k = max i( I)
|I|=k
2 1 ˆ2
k
k = min( k, k)
1 2

Monte-Carlo estimation:
ˆk k k
N = 4000, P = 1000

Polytopes-based Guarantees
Noiseless recovery: x argmin ||x||1 (P0 (y))
x=y

= ( i )i R 2 3
3 2

1

x0 x0
1
y x
3
B = {x ||x||1 } 2
(B )
= ||x0 ||1

Polytopes-based Guarantees
Noiseless recovery: x argmin ||x||1 (P0 (y))
x=y

= ( i )i R 2 3
3 2

1

x0 x0
1
y x
3
B = {x ||x||1 } 2
(B )
= ||x0 ||1

x0 solution of P0 ( x0 ) ⇥ x0 ⇤ (B )

L1 Recovery in 2-D
= ( i )i R 2 3

C(0,1,1) 2
3
K(0,1,1)
1

y x

2-D quadrant 2-D cones
Ks = ( i si )i R
3
i 0 Cs = Ks

Polytope Noiseless Recovery
Counting faces of random polytopes: [Donoho]
All x0 such that ||x0 ||0 Call (P/N )P are identiﬁable.
Most x0 such that ||x0 ||0 Cmost (P/N )P are identiﬁable.

Call (1/4) 0.065
1

0.9

Cmost (1/4) 0.25 0.8

0.7

0.6

Sharp constants. 0.5

0.4

No noise robustness. 0.3

0.2

0.1

0
50 100 150 200 250 300 350 400

RIP
All Most

Polytope Noiseless Recovery
Counting faces of random polytopes: [Donoho]
All x0 such that ||x0 ||0 Call (P/N )P are identiﬁable.
Most x0 such that ||x0 ||0 Cmost (P/N )P are identiﬁable.

Call (1/4) 0.065
1

0.9

Cmost (1/4) 0.25 0.8

0.7

0.6

Sharp constants. 0.5

0.4

No noise robustness. 0.3

0.2
Computation of 0.1

“pathological” signals 0
50 100 150 200 250 300 350 400

[Dossal, P, Fadili, 2010]
RIP
All Most

Tomography and Fourier Measures

Tomography and Fourier Measures
ˆ
f = FFT2(f )

k

Fourier slice theorem: ˆ ˆ
p (⇥) = f (⇥ cos( ), ⇥ sin( ))
1D 2D Fourier

t R
Partial Fourier measurements: {p k
(t)}0 k<K
Equivalent to: ˆ
Kf = (f [!])!2⌦

Regularized Inversion
Noisy measurements: ⇥ ˆ
, y[ ] = f0 [ ] + w[ ].
Noise: w[⇥] N (0, ), white noise.
1
regularization:
1 ˆ[⇤]|2 +
f = argmin
⇥
|y[⇤] f |⇥f, ⇥m ⇤|.
f 2 m

MRI Imaging
From [Lutsig et al.]

MRI Reconstruction
From [Lutsig et al.]
randomization
Fourier sub-sampling pattern:

High resolution Low resolution Linear Sparsity

Radar Interferometry
CARMA (USA)

Fourier sampling Linear
(Earth’s rotation) reconstruction

Structured Measurements
Gaussian matrices: intractable for large N .

Random partial orthogonal matrix: { } orthogonal basis.
Kf = (h'! , f i)!2⌦ where |⌦| = P uniformly random.

Fast measurements: (e.g. Fourier basis)



⌅ ⌅
Mutual incoherence: µ= N max |⇥⇥ , m ⇤| [1, N]
,m



⌅ ⌅
Mutual incoherence: µ= N max |⇥⇥ , m ⇤| [1, N]
,m

Theorem: with high probability on , =K
CP
If M 2 log(N )4
, then 2M 2 1
µ
[Rudelson, Vershynin, 2006]
not universal: requires incoherence.

Conclusion
Sparsity: approximate signals with few atoms.

dictionary

Conclusion

dictionary

Compressed sensing ideas:
Randomized sensors + sparse recovery.
Number of measurements signal complexity.
CS is about designing new hardware.

Conclusion

dictionary

Compressed sensing ideas:
Randomized sensors + sparse recovery.
Number of measurements signal complexity.
CS is about designing new hardware.

The devil is in the constants:
Worse case analysis is problematic.
Designing good signal models.

RESULTS ARE THOSE GIVEN BY MCAULEY AND AL [28] WITH THEIR “3 3 MODEL.” THE TOP-RIGHT RESULTS ARE THOSE O
RAINED DICTIONARY. THE BOTTOM-RIGHT ARE THE IMPROVEMENTS OBTAINED WITH THE ADAPTIVE APPROACH WITH 20 IT
CALE K-SVD ALGORITHM [2] ON EACH CHANNEL SEPARATELY WITH 8

EPRESENTATION FOR COLOR IMAGE RESTORATION
DENOISING ALGORITHM WITH 256 ATOMS OF SIZE 7 7 3 FOR
color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposed (
uced with our proposed technique (
in the new metric).
in our proposed new metric). Both images have been denoised with the same global dictionary.
Some Hot Topics
bias effect in the color from the castle and in some part of the water. What is more, the color of the sky is piecewise constant when
ch is another artifact our approach corrected. (a) Original. (b) Original algorithm, dB. (c) Proposed algorithm,

Dictionary learning:
dB.
with 256 atoms learned on a generic database of natural images, with two differental.: SPARSE REPRESENTATION FOR COLOR IMAGE RESTORATION
MAIRAL et sizes of patches. Note the large number of color-less atoms. 57
have negative values, the vectors are presented scaled and shifted to the [0,255] range per channel: (a) 5 5 3 patches; (b) 8 8 3 patches.

OR IMAGE RESTORATION 61
Fig. 7. Data set used for evaluating denoising experiments.

learning

ing Image; (b) resulting dictionary; (b) is the dictionary learned in the image in (a). The dictionary is more colored than the global one.
TABLE I

g. 7. Data set used for evaluating denoising experiments. with 256 atoms learned on a generic database of natural images, with two different sizes of patches. Note the large number of color-less atoms.
Fig. 2. Dictionaries
Since the atoms can have negative values, the vectors are presented scaled and shifted to the [0,255] range per channel: (a) 5 5 3 patches; (b) 8 8 3 patches.

color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposed ( in the new metric).
duced with our proposed technique (
TABLE I our proposed new metric). Both images have been denoised with the same global dictionary.
in
TH 256 ATOMS OF SIZE castle 7 in3 FOR of the water. What is more, the color of the sky is .piecewise CASE IS DIVIDED IN FOUR
a bias effect in the color from the 7 and some part AND 6 6 3 FOR EACH constant when
ch MCAULEY AND approach corrected. (a)HEIR “3(b) Original algorithm, HE TOP-RIGHT RESULTS ARE THOSE OBTAINED BY
Y is another artifact our AL [28] WITH T Original. 3 MODEL.” T dB. (c) Proposed algorithm,
dB.
8 ATOMS. THE BOTTOM-LEFT ARE OUR RESULTS

2] ON EACH CHANNEL SEPARATELY WITH 8 8 ATOMS. THE BOTTOM-LEFT ARE OUR RESULTS OBTAINED
AND 6

OTTOM-RIGHT ARE THE IMPROVEMENTS OBTAINED WITH THE ADAPTIVE APPROACH WITH 20 ITERATIONS.
H GROUP. AS CAN BE SEEN, OUR PROPOSED TECHNIQUE CONSISTENTLY PRODUCES THE BEST RESULTS
6 3 FOR

Fig. 3. Examples of color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposed ( in the new metric).
Color artifacts are reduced with our proposed technique ( in our proposed new metric). Both images have been denoised with the same global dictionary.
In (b), one observes a bias effect in the color from the castle and in some part of the water. What is more, the color of the sky is piecewise constant when
(false contours), which is another artifact our approach corrected. (a) Original. (b) Original algorithm, dB. (c) Proposed algorithm,
dB.
. EACH CASE IS DIVID


in the new metric).
Some Hot Topics

Image f =
dB.

have negative values, the vectors are presented scaled and shifted to the [0,255] range per channel: (a) 5
MAIRAL et sizes of patches. Note the large number of color-less
5 3 patches; (b) 8 8
atoms.
3 patches.
57
x

learning

TABLE I

Analysis vs. synthesis:

Js (f ) = min ||x||1
TABLE I in the new metric).
a bias effect in the color from the 7
and some part AND 6 6 3 FOR EACH constant when f= x
dB.

AND 6


Coe cients x
6 3 FOR

dB.


in the new metric).
Some Hot Topics

Image f =
dB.

atoms.
3 patches.
57
x

learning

D
TABLE I


Js (f ) = min ||x||1
J (f ) = ||D f ||
dB.

a 1
AND 6


Coe cients x c=D f
6 3 FOR

dB.


in the new metric).
Some Hot Topics

Image f =
dB.

atoms.
3 patches.
57
x

learning

D
TABLE I


Js (f ) = min ||x||1
J (f ) = ||D f ||
dB.

a 1
AND 6


Other sparse priors:

Coe cients x c=D f
6 3 FOR

dB.

|x1 | + |x2 | max(|x1 |, |x2 |)


in the new metric).
Some Hot Topics

Image f =
dB.

atoms.
3 patches.
57
x

learning

D
TABLE I


Js (f ) = min ||x||1
J (f ) = ||D f ||
dB.

a 1
AND 6



Coe cients x c=D f
6 3 FOR

dB.

2 1
|x1 | + |x2 | max(|x1 |, |x2 |) |x1 | + (x2
2 + x3 ) 2


in the new metric).
Some Hot Topics

Image f =
dB.

atoms.
3 patches.
57
x

learning

D
TABLE I


Js (f ) = min ||x||1
J (f ) = ||D f ||
dB.

a 1
AND 6



Coe cients x c=D f
6 3 FOR

dB.

2 1
|x1 | + |x2 | max(|x1 |, |x2 |) |x1 | + (x2
2 + x3 ) 2 Nuclear

Signal Processing Course : Compressed Sensing

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Signal Processing Course : Compressed Sensing

Similar a Signal Processing Course : Compressed Sensing (20)

Más de Gabriel Peyré

Más de Gabriel Peyré (20)

Signal Processing Course : Compressed Sensing