Compressed Sensing In Spectral Imaging

Introduction to Compressive Sensing
Compressive Spectral Imaging
Low-rank Anomaly Recovery in (CASSI)


Gonzalo R. Arce

Department of Electrical and Computer Engineering
University of Delaware
Email:arce@ece.udel.edu

Distinguished Lecture Series
Aristotle University of Thessaloniki

October 19th - 2010

Gonzalo R. Arce Compressive Spectral Imaging -1


Outline

Sparsity and ℓ1 Norm
Incoherent Sampling
Sparse Signal Recovery
Single Shot CASSI System
Spectral Selectivity in (CASSI)
Random Convolution SSI (RCSSI)


Introduction to Compressive Sensing Sparsity and ℓ1 norm
Compressive Spectral Imaging Incoherent Sampling
Low-rank Anomaly Recovery in (CASSI) Sparse Signal Recovery

Traditional signal sampling and signal compression.

Nyquist sampling rate gives exact reconstruction.

Pessimistic for some types of signals!



Sampling and Compression

Transform data and keep important coefﬁcients.
Lots of work to then throw away majority of data!.
e.g. JPEG 2000 Lossy Compression: A digital camera can
take millions of pixels but the picture is encoded on a few
hundred of kilobytes.



Problem: Recent applications require a very large number of
samples:
Higher resolution in medical imaging devices, cameras,
etc.
Spectral imaging, confocal microscopy, radar arrays, etc.

y
λ

x

Spectral Imaging
Medical Imaging



Fundamentals of Compressive Sensing

Donoho † , Candès ‡ , Romberg and Tao, discovered
important results on the minimum number of data needed
to reconstruct a signal
Compressive Sensing (CS) uniﬁes sensing and
compression into a single task
Minimum number of samples to reconstruct a signal
depends on its sparsity rather than its bandwidth.

†
D. Donoho. "Compressive Sensing". IEEE Trans. on Information Theory. Vol.52(2), pp.5406-5425, Dec.2006.
‡
E. Candès, J. Romberg and T. Tao. "Robust Uncertainty Principles: Exact Signal Reconstruction from Highly
Incomplete Frequency Information". IEEE Trans. on Information Theory. Vol.52(4), pp.1289-1306, Apr.2006.



Sparsity
Signal sparsity critical to CS
Plays roughly the same role in CS that bandwidth plays in
Shannon-Nyquist theory
A signal x ∈ RN is S-sparse on the basis Ψ if x can be
represented by a linear combination of S vectors of Ψ as
x = Ψα with S ≪ N
At most S non-zero components

x Ψ

α


The ℓ1 Norm and Sparsity
Sparsity of x is measured by its number of non-zero
elements, the ℓ0 norm

x 0 = #{i : x(i) = 0}

The ℓ1 norm can be used to measure sparsity of x

x 1 = |x(i)|
i

The ℓ2 norm is not effective in measuring sparsity of x

x 2 =( |x(i)|2 )1/2
i

The ℓ0 and ℓ1 norms promote sparsity


Why ℓ1 Norm Promotes Sparsity?

Given two N -dimensional signals:
x1 = (1, 0, ..., 0) → "Spike" signal
√ √ √
x2 = (1/ N , 1/ N , ..., 1/ N ) → "Comb" signal

x 2
x1 and x2 have the same ℓ2
norm:
x1 2 = 1 and x2 2 = 1.
x 1
However, x1
√ 1 = 1 and
x2 1 = N .



Compressive Measurements
Measurements in CS are different than samples taken in
traditional A/D converters.
The signal x is acquired as a series of non-adaptive inner
products of different waveforms {φ1 , φ2 , ..., φM }
yk =< φk , x >; k = 1, ..., M ; with M ≪ N

y Φ x

Mx1
MxN
Measurements
Sampling Operator

Nx1
Sparse Signal



Recoverability

yk =< φk , x >; k = 1, ..., M ; with M ≪ N

Recovering x from yk is an inverse problem.
Need to solve an under determined system of equations
y = Φx.
Inﬁnitely solutions for the system since M ≪ N .
Amplitude

Amplitude

Original sparse signal Compressed measurements Reconstructed signal using least-squares.
Solution not sparse



Recoverability: Incoherent Sampling

The number of samples required to recover x from M samples
depends on the mutual coherence between Φ and Ψ
Mutual Coherence
√
µ(Φ, Ψ) = N max{| < φk , ψ j > | : φk ∈ Rows(Φ), ψ j ∈ Columns(Ψ)};

where, ψj 2 = φk 2 =1

The coherence µ(Φ, Ψ) satisﬁes:
√
1 ≤ µ(Φ, Ψ) ≤ N



Recoverability: Incoherent Sampling

The random measurement matrix Φ has to be incoherent
to the dictionary Ψ and x can be recovered from M
samples exactly when M satisﬁes:
M ≥ C · µ2 · S · log(N ), C ≥ 1

(a) (b)

(a) Very sparse vector.
(b) Examples of pseudorandom, incoherent test vectors φk † .
†
J. Romberg. "Imaging Via Compressive Sampling". IEEE Signal Processing Magazine. March,2008.



Compressive Sensing Signal Reconstruction
Goal: Recover signal x from measurements y
Problem: Random projection Φ not full rank (ill-posed
inverse problem)
Solution: Exploit the sparse/compressible geometry of
acquired signal x

y Φ x



Reconstruction Algorithms

Different formulations and implementations have been
proposed to find the sparsest x subject to y = Φx
Those are broadly classified in:
Regularization formulations (Replace combinatorial
problem with convex optimization)
Greedy algorithms (Iterative refinement of a sparse
solution)
Bayesian framework (Assume prior distribution of sparse
coefficients)


Introduction to Compressive Sensing Single Shot Coded Aperture System (CASSI)
Compressive Spectral Imaging Spectral Selectivity in (CASSI)
Low-rank Anomaly Recovery in (CASSI) Random Convolution SSI (RCSSI)


Collects spatial information from across the
electromagnetic spectrum.
Applications, include wide-area airborne surveillance,
remote sensing, and tissue spectroscopy in medicine.




Spectral Imaging System - Duke University†

†
A. Wagadarikar, R. John, R. Willett, D. Brady. "Single Disperser Design for Coded Aperture Snapshot Spectral Imaging."
Applied Optics, vol.47, No.10, 2008.
A. Wagadarikar and N. P. Pitsianis and X. Sun and D. J. Brady. "Video rate spectral imaging using a coded aperture
snapshot spectral imager." Opt. Express, 2009.



Single Shot Compressive Spectral Imaging
System design

With linear dispersion:

f1 (x, y; λ) = f0 (x, y; λ)T (x, y)

f2 (x, y; λ) = δ(x′ − [x + α(λ − λc )]δ(y ′ − y)f1 (x′ , y ′ ; λ))dx′ dy ′

= δ(x′ − [x + α(λ − λc )]δ(y ′ − y)f0 (x′ , y ′ ; λ)T (x, y))dx′ dy ′

= f0 (x + α(λ − λc ), y; λ)T (x + α(λ − λc ), y)



Experimental results from Duke University

Original Image

Reconstructed image cube of size:128x128x128.
Measurements Spatial content of the scene in each of 28
spectral channels between 540 and 640nm.
† A. Wagadarikar, R. John, R. Willett, D. Brady. "Single Disperser Design for Coded Aperture Snapshot Spectral Imaging."
Applied Optics, vol.47, No.10, 2008.



Simulation results in RGB

Original Image Measurements

R
Reconstructed
¡ Image




Object with spectral information only in (xo , yo )
Only two spectral component are present in the object




Object with spectral information only in (xo , yo )




One pixel in the detector has information from different spectral
bands and different spatial locations




Each pixel in the detector has different amount of spectral
information. The more compressed information, the more
difﬁcult it is to reconstruct the original data cube.



Each row in the data cube produces a compressed
measurement totally independent in the detector.




Undetermined equation system:
Unknowns = N × N × M and Equations: N × (N + M − 1)



Complete data cube 6 bands
The dispersive element shifts each spectral band in one
spatial unit
In the detector appear the compressed and modulated
spectral component of the object
At most each pixel detector has information of six spectral
components



We used the ℓ1 − ℓs reconstruction algorithm † .
†
S. J. Kim, K. Koh, M. Lustig, S. Boyd and D. Gorinevsky. "An interior-point method for large scale L1 regularized least
squares." IEEE Journal of Selected Topics in Signal Processing, vol.1, pp. 606-617, 2007.



Coded Aperture Snapshot Spectral Image System
(CASSI)(a)

Advantages:
Enables compressive spectral imag-
ing
Simple
Low cost and complexity

Limitations:
Excessive compression
Does not permit a controllable SNR
May suffer low SNR gmn = f(m+k)nk P(m+k)n + wnm
Does not permit to extract a speciﬁc k
subset of spectral bands = (Hf )nm + wnm = (HW θ)nm + wnm

A. Wagadarikar, R. John, R. Willett, and D. Brady. "Single disperser design for coded aperture snapshot spectral imaging."
Appl. Opt., Vol.47, No.10, 2008.



Bands Recovery

Typical example of a measurement of CASSI system. A set of bands
constant spaced between them are summed to form a measurement



Multi-Shot CASSI System
Multi-shot compressive spectral imaging system

Advantages:
Multi-Shot CASSI allows
controllable SNR
Permits to extract a hand-
picked subset of bands
Extend Compressive Sens-
ing spectral imaging capabil-
ities
L
gmni = fk (m, n + k − 1)Pi (m, n + k − 1)
k=1
L
i
= fk (m, n + k − 1)Pr (m, n + k − 1)Pg (m, n + k − 1)
k=1

Ye, P. et al. "Spectral Aperture Code Design for Multi-Shot Compressive Spectral Imaging". Dig. Holography and
Three-Dimensional Imaging, OSA. Apr.2010.



Mathematical Model of CASSI System

L
gmni = fk (m, n + k − 1)Pi (m, n + k − 1)
k=1
L
i
= fk (m, n + k − 1)Pr (m, n + k − 1)Pg (m, n + k − 1)
k=1
where i expresses ith shot

Each pattern Pi is given by,
i
Pi (m, n) = Pg (m, n)xPr (m, n)
i 1 mod(n, R) = mod(i, R)
Pg (m, n) =
0 otherwise
One different code aperture is used for each shot of CASSI system


Code Apertures

Code patterns used
in multishot CASSI
system

Code patterns used in multishot CASSI system


Cube Information and Subsets of Spectral Bands

Spectral axis, Spatial
L bands axis, N
Spectral data cube → L bands
pixels R subsets of M bands each one
Complete
Spectral (L = RM ) Each component
Data Cube of the subset is spaced by R
Spatial
bands of each other
axis, N
pixels

Subset 1
M bands
R R
Subset 1 Subset 2 Subset 3 ... Subset R
M=bands M=bands M=bands M bands



Cube Information and Subsets of Spectral Bands

Spectral axis, Spatial
L bands axis, N Spectral data cube → L bands
pixels
R subsets of M bands each one
Complete (L = RM ) Each component
Spectral of the subset is spaced by R
Data Cube
Spatial bands of each other
axis, N
R R
pixels Subset 2
M bands

Subset 1 Subset
£ Subset
¢ ... Subset R
M=bands M=bands M=bands M=bands



Multi-Shot CASSI System

First shot and Second shot and R shot and
measurement measurement measurement



Single Shot
Multi-Shot
One shot of CASSI Information of all band exists in all shots
system. One high
compressing
measurement.

First shot Second shot Third shot
Reconstruction
Algorithm
Re-organization
algorithm

Reconstructed
spectral data
cube.

Bands 1,4,7 Bands 2,5,8 Bands 3,6,9



Multi-Shot
Reorder Process R R R

′ L
gmnk = j=1 fj (m, n + j − 1)Pi (m, n + j − 1)
L i First shot Second shot Third shot
= j=1 fj (m, n + j − 1)Pr (m, n + j − 1)Pg (m, n + j − 1) Re-organization
algorithm
= mod(n+j−1,R)=mod(i,R) fk (m, n + k − 1)Pr (m, n + j − 1)
= (Hk Fk )mn




Multi-Shot
Reorder Process R
R R

′ L
gmnk = j=1 fj (m, n + j − 1)Pi (m, n + j − 1)
L i First shot Second shot Third shot
= j=1 fj (m, n + j − 1)Pr (m, n + j − 1)Pg (m, n + j − 1) Re-organization
algorithm
= mod(n+j−1,R)=mod(i,R) fk (m, n + k − 1)Pr (m, n + j − 1)
= (Hk Fk )mn




Multi-Shot

Recover any of the subsets
independently
Recover of complete spec-
tral data cube is not neces-
sary



Multi-Shot

High SNR in each re-
construction
Enable to use paral-
lel processing
To use one proces-
sor for each indepen-
dent reconstruction



Multi-Shot
Single Shot

One shot of CASSI
system. One high
compressing
measurement.

Reconstruction
Algorithm

Reconstructed
spectral data
cube.



Multi-Shot Reconstruction

Reconstructed image of one spec-
tral channel in 256x256x24 data
cube from multiple shot measure-
ments.

(a) One shot result,PSNR
(a) One shot (b) 2 shots
P SN R = 17.6dB
(b) Two shots result,PSNR
P SN R = 25.7dB
(c) Eight shots result,PSNR
P SN R = 29.4
(d) Original image

(c) 8 shots (d) Original



Multi-Shot Reconstruction
Reconstructed image for dif-
ferent spectral channels in the
256x256x24 data cube from
six shot measurements.

(a) Band 1
(b) Band 13
(c) Band 8
(d) Band 20
(a) and (b) are recon-
structed from the ﬁrst
group of measurements
(c) and (d) are recon-
structed from the second
group of measurements



Random Convolution Spectral Imaging



Random Convolution Imaging

J. Romberg. "Compressive Sensing by Random Convolution." SIAM Journal on Imaging Science, July,2008.



Random Convolution Imaging
Random Convolution
Circularly convolve signal x ∈ Rn with a pulse h ∈ Rn , then
subsample.
The pulse is random, global, and broadband in that its energy is
distributed uniformly across the discrete spectrum.

x ∗ h = Hx
where
H = n−1/2 F ∗ ΣF
Ft,ω = e−j2π(t−1)(ω−1)/n , 1 ≤ t, ω ≤ n
Σ as a diagonal matrix whose non-zero entries are the Fourier
transform of h.


Random Convolution

 
σ1 0 · · ·
 0 σ2 · · · 
Σ=
 
.
. .. 
 . . 
σn

ω=1 : σ1 ∼ ±1 with equal probability,
2 ≤ ω < n/2 + 1 : σω = ejθω , where θω ∼ Uniform([0, 2π]),
ω = n/2 + 1 : σn/2+1 ∼ ±1 with equal probability,
n/2 + 2 ≤ ω ≤ n : ∗
σω = σn−ω+2 , the conjugate of σn−ω+2 .



Random Convolution

H
The effect of H on a signal x can be broken down into a
discrete Fourier transform, followed by a randomization of
the phase (with constraints that keep the entries of H real),
followed by an inverse discrete Fourier transform.
Since F F ∗ = F ∗ F = nI and ΣΣ∗ = I,

H ∗ H = n−1 F ∗ Σ∗ F F ∗ ΣF = nI

So convolution with h as a transformation into a random
orthobasis.



Randomly Pre-Modulated Summation

ym×1 = Φm×n xn×1 = Pm×n Θn×n Hn×n xn×1
where
 
ones(n/m, 1) 0 0 0
 0 ones(n/m, 1) 0 0 
Pm×n =
 
.. 
 0 0 . 0 
0 0 0 ones(n/m, 1) m×n
 
±1 0 0 0
 0 ±1 0 0 
Θn×n =
 
.. 
 0 0 . 0 
0 0 0 ±1 n×n



Main Result

H will not change the magnitude of the Fourier transform,
so signals which are concentrated in frequency will remain
concentrated and signals which are spread out will stay
spread out.

The randomness of Σ will make it highly probable that a
signal which is concentrated in time will not remain so after
H is applied.



Main Result

(a) A signal x consisting of a single Daubechies-8 wavelet.
(b) Magnitude of the Fourier transform F x.
(c) Inverse Fourier transform after the phase has been
randomized. Although the magnitude of the Fourier transform is
the same as in (b), the signal is now evenly spread out in time.
J. Romberg. "Compressive Sensing by Random Convolution." SIAM Journal on Imaging Science, July,2008.



Fourier Optics

Fourier optics imaging experiment.
(a) The 256 × 256 image x.
(b) The 256 × 256 image Hx.
(c) The 64 × 64 image P θHx.


(a) The 256 × 256 image we wish to acquire.
(b) High-resolution image pixellated by averaging over 4 × 4 blocks.
(c) The image restored from the pixellated version in (b), plus a set of
incoherent measurements. The incoherent measurements allow us to
effectively super-resolve the image in (b).



Fourier Optics

a) b) c)

d) e) f)
Pixellated images: (a) 2 × 2. (b) 4 × 4. (c) 8 × 8. Restored from: (d) 2 × 2 pixellated
version. (e) 4 × 4 pixellated version. (f) 8 × 8 pixellated version.


Random Convolution Spectral Imaging



20

40

60

80

100

120

20 40 60 80 100 120




Spectral video analysis
Video surveillance: Anomaly detection
Stationary background corresponds to low-rank contribution
and the moving objects corresponds to sparse data.



Connection Between Low-Rank Matrix Recovery and
Compressed Sensing

Low-rank Rank miniz. Convex Relax.
Recovery min rank(X)
L min L
s.t. M=S+L s.t. M=S+L

Compressed Rank miniz. Convex Relax.
Sensing

B. Recht, M. Fazel and P. Parrilo, "Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm
Minimization," SIAM Review, Aug. 2010.



Low-Rank Anomaly Recovery in (CASSI)
Problem Description
(i)
Consider the video surveillance of Fk,n1,n2 ∈ RN1 ×N2 ×K ,
i = 1, ..., N frames.
The ith scene is assumed to be composed by a stationary
background L(i) and an event changing in time S(i) ,
(i) (i) (i)
Fk,n1,n2 = Lk,n1,n2 + Sk,n1 ,n2
CASSI encodes both 2D spatial information and spectral
information in a 2Dsingle measurement G(i) for
i = 1, ..., N .
GOAL: recover anomalies occurring in both time and spectra
from a sequence of spectrally compressed video frames
G(1) , G(2) , ..., G(N ) .



Recovering anomalies:
Form G as the large data matrix G = [g(1) , g(2) , . . . , g(N ) ],
where g(i) is the column representation of G(i) .
G = L + S where L is the stationary background and S is
sparse capturing the anomalies in the foreground



Principal Component Pursuit
The matrix G is decomposed into a low-rank matrix L and a
sparse matrix S, such that
G =L+S (1)
Using Principal Component Pursuit.
Principal Component Pursuit

min L ∗ +λ S 1

n
L ∗ = i=1 σi (L), is the nuclear norm of L.
S 1 = ij Sij is the ℓ1 -norm of the matrix S

E. J. Candès, X. Li, Y. Ma, and J. Wright. "Robust Principal Component Analysis?," Submitted to Journal of the ACM.
2009.


Spectral recovery of anomalies.
Coded measurements in S have been biased by the
background reconstruction
ˆ
Identify spatial location of the anomalies in S by:
ˆ
Filter |S| with a Weighted Median (WM) ﬁlter as
(i) ˆ (i)
Mn1 ,n2 = MEDIAN{Tv,w ⋄ |Sn1 +v,n2 +w | : (v, w) ∈ [−3, 3]}

where T is a WM ﬁlter of size (L × L) with centered weight
(L + 1)/2, and linearly decreasing weights
Spectrally coded measurements of anomalies denoted by
˜
G(i) are estimated as
G(i) = G(i) ⊙ U(M(i) − Th )
˜
Th is a thresholding parameter that extracts the pixels that
are most likely to be in the region of interest



ˆ ˜
Recover S(i) from G(i) by

ˆ(i) = Ψ min( g(i) − HΨθ (i)
s ˜ 2
2 + τ θ (i) 1 ) (2)
θ

s ˜ ˜
where ˆ(i) and g(i) are the column representation of S(i)
˜
and G (i) , respectively.



(video) (video)



Summary

Compressive Sensing
Spectral Imaging
Low-Rank Recovery

´
Euχαριστ ω !


Compressed Sensing In Spectral Imaging

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Compressed Sensing In Spectral Imaging

Similar a Compressed Sensing In Spectral Imaging (20)

Más de Distinguished Lecturer Series - Leon The Mathematician

Más de Distinguished Lecturer Series - Leon The Mathematician (10)

Último

Último (20)

Compressed Sensing In Spectral Imaging