Privacy protection of visual information

1

Privacy protection
of visual information

Touradj Ebrahimi
touradj.ebrahimi@epfl.ch
MediaSense 2012
Dublin, Ireland
21-22 May 2012

Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne

Video surveillance popularity 2

• Rise in terrorism and crime
– Globalization of good and bad
• Political
– Perception that the problem of crime and terrorism is
addressed
• Business and economy
– New revenue models
– Cost issues


Potential abuses in video surveillance 3

• Criminal abuse
– Criminal misuse by law enforcement officers
– Police official gathering information on a gay club to blackmail patrons
• Institutional abuse
– Spy upon and harass political activists (Civil Rights, Vietnam war)
– Surveillance of political demonstrations
• Personal usage
– Police officers helping friends stalk women, track estranged girlfriends/
spouses
• Discrimination
– Racial discrimination towards people of color
• Voyeurism
– Bored male operators spying on women
– Footage of public cameras made publicly available


Civil liberty and right to privacy 4

• Increased resistance to video surveillance
• Several countries have set up or are in the process of
setting up directives and guidelines to regulate video
surveillance
– EU – Directive 95/46/EC


World Trade Center, 9/11 5

filmed by a Gas Station
surveillance camera on
September 10, 2001

filmed by an ATM
surveillance camera on
September 10, 2001
Mohamed Atta


Attack on London underground, July 7, 2005 6

On a reconnaissance mission two
weeks before the attack

Plot to attack trains in Germany, August 2006 7

Two unexploded bombs found in
luggage aboard two trains

Both terrorists have been arrested
thanks to the video footage


Proliferation of video surveillance applications 8

• Surveillance of sensitive locations
– Embassies, airports, nuclear plants, military zone, border
control, …
• Intrusion detection
– Residential surveillance, retail surveillance, …
• Traffic control
– Speed control
• Access to places
– Car license plate recognition in cities
• Event detection
– Child/Elderly care
• Marketing/statistics
– Customers habits
– Number of visitors
• …

Forensic video – legal admissibility 9

• If the image is not inherently reliable, its admissibility in court is
questionable
• If a poor image is ruled admissible, it will be afforded little or no
weight
• For an image to be admissible, the prosecutor must prove that the
image has not been altered
– Lossy compression
– Conditional replenishment
– Enhancement
• Original versus copy
– Any digital image can be thought of as being ‘the original’


10
Video surveillance dimensions

• Technology
• Business
• Legal
• Social


Video surveillance technologies 11

• First generation
– Analog
– CCTV
– Recording
• Second generation
– Digital/Hybrid
– Recording
– Computerized
– IP wired/wireless


Video surveillance technologies 12

• Third generation
– Content analysis
– Biometrics
– Search
– Unusual event detection
• Forth generation
– Pervasive
– Distributed
– Invisible
– Multi-view
– Ultra high definition


13
Main security tools in video surveillance

• Encryption
– Secure communication
– Conditional access
• Integrity verification
– Digital signature
– Proof for lack of manipulation after capture


14
Alternatives to implement video surveillance with privacy

• Fully automatic surveillance without intervention of
human operators
– False positives and false negatives
• Encrypting the whole video
– Not good for monitoring
• Distorting/blocking sensitive regions
– Impact on intelligibility
• Reversible encryption/scrambling of sensitive regions
with a key
– Identification can take place when crime happens
• Legal and best practices in video surveillance
– Recorded materials locked in secure locations
• Only extract/record needed information from the scene
– MPEG-7 visual descriptors

Smart video surveillance 15

Video +
Metadata
Recording


Smart video surveillance 15

Video +
Metadata
Recording

[…011001…]


Example: Smart video surveillance 16


17
Privacy-sensitive visual information

• Predefined zones
– Windows, doors
– Bank teller
– Casino playing tables
– …

• Automatic identification of Regions of Interest (ROI)
– People in the scene
– Human faces
– Cars license plates
– Moving objects
– …


18
Legacy solutions to visual privacy protection

• Masking
• Blur
• Pixelization


19
Masking


20
Blur


21
Pixelization


22
More recent solutions for privacy protection

• (ROI) Encryption
• (ROI) Scrambling


23
ROI selective encryption


24
ROI selective decryption


ROI selective scrambling 25


Bitstream encryption 26

• Selective encryption of the bitstream at packet level
• One or more secret keys
• Symmetric encryption
– Packet body
– Block cipher: e.g. AES

packet

private key
encrypted
packet


Scrambling approaches 27

• Image-domain
– Randomly flip bits in one or more bit planes

image Scrambling Transform Entropy Coding bitstream

Encoder

• Pros
– Very simple
– Independent from the subsequent encoding scheme
– Does not affect the bitstream syntax → standard compliance
• Cons
– Significantly alter statistics of video signal
– Ensuing compression less efficient



• Transform-domain
– Randomly flip sign of transform coefficients

image Transform Scrambling Entropy Coding bitstream

Encoder

• Pros
– Does not adversely affect subsequent entropy coding
– Strength of scrambling can be controlled
– Does not affect the bitstream syntax → standard compliance
• Cons
– Must be integrated inside the encoder



• Bitstream-domain
– Randomly flip bits in bitstream

image Transform Entropy Coding Scrambling bitstream

Encoder

• Pros
– Applied on bitstream after encoding
• Cons
– Require parsing of bitstream
– Difficult to guarantee syntax remains compliant and will not crash a
decoder


30
Scrambling in JPEG

(a) (b) DC

pseudo-randomly
PRNG
inverse sign

seed

assymetric scrambled
encryption codestream

public key

(c) DC (d) DC

pseudo-randomly pseudo-randomly
PRNG PRNG
inverse sign inverse sign

seed seed

assymetric scrambled assymetric scrambled
encryption codestream encryption codestream

public key public key

Figure 4 – AC coefficients scrambling:
(a) 63 AC coefficients, (b) 60 AC coefficients, (c) 55 AC coefficients, (d) 48 AC coefficients.

Straightforwardly, as the scrambling is merely flipping signs of selected coefficients, the technique requires negligible
Swiss Federal Institute of Technology, scrambled region is restricted to match the 8x8 DCT blocks
computational complexity. Clearly, the shape of the
Lausanne
boundaries.

31
Scrambling in JPEG


Scrambling in JPEG 2000 (JPSEC) 32

• Codeblock-based bitstream domain scrambling
image
wavelet quantizer selective arithmetic Encoder codestream
selective scrambled Decoder
transform scrambling coder scrambling codestream
scrambled
codestream
PRNG JPSEC JPSEC PRNG JPSEC JPSEC
syntax codestream syntax codestream
encrypted encrypted
seed seed

seed encryption seed encryption

Preserve the markers in the bitstream; do not introduce erroneous markers

x=current byte, y=preceding byte
1. If x=0xFF, no modification
2. If y=0xFF
where m is an 8-bit pseudo-
random number in [0x00,0x8F]

3. Otherwise
where n is an 8-bit pseudo-
random number in [0x00,0xFE]


Scrambling in JPEG 2000 (JPSEC) 33

• ROI-based wavelet domain scrambling
– Arbitrary-shape regions
• Exploit ROI mechanisms in JPEG 2000
wavelet quantizer
Encoder
image
transform

no resolution yes
level l
< TI ? keys ROI-based scrambled Decoder
Decoder
up-scale JPSEC code-stream
code-block
distortion decrypt arithmetic
seeds decoder

foreground foreground
resolution seeds objects
yes objects segmentation background yes resolution background
level l level l coefficient
≥ TS ? mask ? ≥ TS ? < 2s ?
scramble down-shift unscramble up-shift
PRNG PRNG
wavelet no wavelet wavelet wavelet
no
coefficient coefficient coefficient coefficient
seeds

encrypt arithmetic inverse inv. wavelet
seeds coder quantizer transform
ROI-based sc
rambled
keys JPSEC code-stream image


Scrambling in JPEG 2000 34


35
Scrambling in MPEG-4


36
Scrambling in MPEG-4


Scrambling in MPEG-4 37


38
Scrambling in H.264/AVC


Scrambling in H.264/AVC 39


40
An existing product

Scrambler Unscrambler


41
Scrambling in DVC
• Key frame privacy (JPEG)
– Scrambling in the transform domain on the DCT coefficients.
– Driven by a Pseudo-Random Number Generator (PRNG) to pseudo-
randomly invert the sign of the DCT Coefficients.
• WZ frames

DCT scrambler
DVC scheme with privacy protection

42
Scrambling in DVC

a) Key frame (JPEG). b) Wyner-Ziv transform domain scrambling.


MPEG-7 camera 43

The MPEG-7 camera describes a scene in terms of
semantic objects and of their properties

XML scene description


MPEG-7 camera 44

– Image analysis: segmentation, change detection, and tracking
(implemented on the camera DSP).
– MPEG-7 coder: scene description represented using MPEG-7 (XML).
– MPEG-7 decoder: MPEG-7 description is parsed. Extraction of the
information related to the specific applications.


MPEG-7 camera 45

<!-- ################################################## --!>
<!-- DDL output for object 4 --!>
<!-- ################################################## --!>
<Object id="4">
<RegionLocator>
<BoxPoly> Poly </BoxPoly>
<Coords1> 237 222 </Coords1>
</RegionLocator>
XML scene <DominantColor>
description <ColorSpace> YUV </ColorSpace>
<ColorValue1> 143.4 </ColorValue1>
</DominantColor>
<HomogeneousTexture>
<TextureValue> 9.02 </TextureValue>
</HomogeneousTexture>
<MotionTrajectory>
<TemporalInterpolation>
<KeyFrame> 100 </KeyFrame>
<KeyPos> 268.6 251.7 </KeyPos>
...
</TemporalInterpolation>
</MotionTrajectory>
</Object>

MPEG-7 camera for video surveillance 46

original frame segmentation mask bounding box


47
Existing product


48
Evaluation of privacy protection in video surveillance

• Serious study of performance analysis of privacy protection solutions is
lacking
• It is paramount to validate privacy protection solutions against user and
system requirements for privacy
• Two approaches can be used
– Performance analysis using subjective evaluations
– Performance analysis using objective metrics


49
Pixelization

• Naïve approach for privacy protection
– Commonly used in television news and documentaries in
order to obscure the faces of suspects, witnesses or
bystanders to preserve their anonymity
– Also used to censor nudity or to avoid unintentional product
placement on television.
• Consists in noticeably reducing resolution in ROI
• Can be achieved by substituting a square block of pixels
with its average
• Drawback
– Integrating pixels along trajectories over time may allow to
partly recovering the concealed information
– Irreversible


50
Gaussian Blur

• Naïve approach for privacy protection
• Removes details in ROI by applying a Gaussian low pass
filter
• Image is convolved with a 2D Gaussian function

• Drawback
– Irreversible


51
Scrambling by Random Sign Inversion

• ROI-based transform-domain scrambling method
• Scrambles the quantized transform coefficients of each 4x4
block of the ROI by pseudo-randomly flipping their sign

• Advantages
– Fully reversible
– Same scrambled stream is transmitted to all users
– Small impact in terms of coding efficiency
– Requires a low computational complexity


52
Scrambling by Random Permutation

• ROI-based transform-domain scrambling method
• Random permutation to rearrange the order of transform
coefficients in 4x4 blocks corresponding to ROI
– Knuth shuffle to generate a permutation of n items with uniform
random distribution

• Advantages
– Fully reversible
– Same scrambled stream is transmitted to all users
– Small impact in terms of coding efficiency
– Requires a low computational complexity


53
Face Recognition - PCA

• Principal Components Analysis (PCA)
– Also known as eigenfaces
– A linear transformation is applied to rotate feature vectors from the initially large
and highly correlated subspace to a smaller and uncorrelated subspace
– PCA has shown to be effective for face recognition
– Firstly, it can be used to reduce the dimensionality of the feature space
– Secondly, it eliminates statistical covariance in the transformed feature space
– In other words, the covariance matrix for the transformed feature vectors is
always diagonal


54
Face Recognition - LDA

• Linear Discriminant Analysis (LDA)
– LDA aims at finding a linear transformation which stresses differences between
classes while lessening differences within classes (a class corresponds to all
images of a given individual)
– The resulting transformed subspace is linearly separable between classes
– PCA is first performed to reduce the feature space dimensionality
– LDA is then applied to further decrease the dimensionality while safeguarding the
distinctive characteristics of the classes
– The final subspace is obtained by multiplying the PCA and LDA basis vectors.


55
Face Identification and Evaluation System

• Preprocessing
– Reduces detrimental variations between images
– Face alignment aligned using eye coordinates
– Pixel values equalization, contrast and brightness normalization
• Training
– Create the subspace into which test images are subsequently projected and
matched
– Performed using a training set of images
• Testing
– A distance matrix is computed in the transformed subspace for all test images
– Euclidian distance for PCA and soft distance for LDA
– Two image sets are defined:
– gallery set is made of known faces
– probe set corresponds to faces to be recognized.


56
Face Identification and Evaluation System

• Performance analysis
– Generate cumulative match curve
– For each probe image, the recognition rank is computed
– rank 0 means that the best match is of the same subject
– rank 1 means that the best match is from another person but the second best
match is of the same subject
– etc.
– The cumulative match curve is obtained by summing the number of correct
matches for each rank


57
Test Data

• Grayscale Facial Recognition Technology (FERET)
– Although it is not representative of typical video surveillance footage, this
database is widely used for face recognition research
– We consider a subset of 3368 images of frontal faces for which eye coordinates
are available
– Images have 256 by 384 pixels with eight-bit per pixel
– We further consider two series of images denoted by ‘fa’ and ‘fb’
– ‘fa’ indicates a regular frontal image
– ‘fb’ indicates an alternative frontal image, taken within seconds of the
corresponding ‘fa’ image, where a different facial expression was requested
from the subject.
• Standard training, gallery and probe sets from the FERET test
– Training set: 501 images from the ‘fa’ series
– Gallery set: 1196 images from the ‘fa’ series
– Probe set: 1195 images from the ‘fb’ series


58
Performance Analysis – Attack #1

• Simple attack
– Training and gallery sets are made of unaltered images
– Probe set corresponds to images with privacy protection
– In other words, altered images are merely processed by the face recognition
algorithms without taking into account the fact that privacy protection tools have
been applied.

PCA LDA


59

• For both PCA and LDA schemes applied on original images, recognition rate
is superior to 70% at rank 0 (i.e. the best match is of the same subject as the
probe), and superior to 90% at rank 50
• When applying a Gaussian blur, the performance drops radically for LDA.
However, recognition rate remains high for PCA with 56% success at rank 0
• Pixelization fares worse. The recognition rate is 56% and 13% at rank 0 for
PCA and LDA respectively
• Results clearly show that both region-based transform-domain scrambling
approaches are successful at hiding identity. The recognition rate is nearly 0%
at rank 0, and remains below 10% at rank 50, for both PCA and LDA
algorithms. In addition, it can be observed that both random sign inversion
and random permutation schemes achieve nearly the same performance


60

• More sophisticated attack
– Privacy protection tools are now applied to all images in the training, gallery and
probe sets
– This corresponds to an attacker which gets access to protected data
– Alternatively, an attacker may attempt replicating the alteration due to privacy
protection techniques on his own training and gallery sets

PCA LDA


61

• With Gaussian blur, the performance remains nearly identical. It even
improves slightly for LDA
• Pixelization is not much better at hiding facial information. The recognition
rate is still 45% and 17% at rank 0 for PCA and LDA respectively
• Finally, both region-based transform-domain scrambling approaches are
again successful at hiding identity. The recognition rate is nearly 0% at rank 0
for both PCA and LDA algorithms.


62
Thanks for your attention!


Privacy protection of visual information

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (14)

Similar a Privacy protection of visual information

Similar a Privacy protection of visual information (6)

Más de Touradj Ebrahimi

Más de Touradj Ebrahimi (20)

Último

Último (20)

Privacy protection of visual information

Notas del editor