5. B. Rinner 5
Sensor Node Platforms
• From research prototypes to commercial products
The Vision
„Smart Dust“ UC
Berkeley
late 1990‘s
Commercial Products
„Mote-on-a Chip“
Dust Networks, 2010
Research Prototypes
„Mica-2“
Crossbow 2004
6. B. Rinner 6
Some Applications of Sensor Networks
• Health
• Structural Monitoring
• Agriculture
• Environmental
[(c) University of Ghent][Kim et al. ACM SenSys, 2006
[AgriNet][M. Welsh, Harvard 2007]
7. B. Rinner 7
Communication is Key
• Wireless communication is an enabling technology
– Eases deployment
– Enables mobility
– Increases flexibility
– Reduces costs
• Communicate on demand (ad hoc, spontaneous) with dynamic
infrastructure
– Nodes organize themselves into network
– Data is transferred via multiple hops
source destination
9. B. Rinner 999
The “Energy Problem”
• Major sources of energy consumption
– Sensing, computing, communication
– High temporal variation
• Energy is the scarce resource for WSN. Several challenges
– What energy reservoirs to exploit?
Constraints: availability, max. power, size, …
– How to distribute power over the network?
Energy provider and consumer might be dislocated.
– How to control the distribution?
“The proper amount of energy in the right place at the right time”
• Sensor networks have always been a “green” technology!
10. B. Rinner 10
Alternative Energy Reservoirs
• Maybe micro-heat engines
– Exploit MEMS technology
to build „internal combustion engines“
– Expected power: 10-20 W
– Still in early research/development
phase
• Or harvest energy from environment
– Organic semiconductors for exploiting indoor ambient light
– Thin film batteries for storing energy
– EnHANTs : energy harvesting
networked active tags
[Handbook of Sensor Networks, Wiley]
[Columbia University, CLUE]
12. B. Rinner. 12
Principle of Smart Cameras
• Smart cameras combine
– sensing,
– processing and
– communication
in a single embedded device
• perform image and video analysis in real-time closely
located at the sensor and transfer only the results
• collaborate with other cameras in the network
TrustEYE.M4 prototype
on top of RaspberryPI
13. B. Rinner. 13
Differences to traditional Cameras
Traditional Camera
– Optics and sensor
– Electronics
– Interfaces
delivers data in form of
(encoded) images and videos,
respectively
Smart Camera
– Optics and sensor
– Onboard computer
– Interfaces
delivers abstracted image data and
is configurable and programmable
Sensor
Electronics
Image enhancement/
Compression
Image
Video
Sensor
Embedded
Computer
Image analysis
„Events“
Programming
Configuration
Light Light
15. B. Rinner 15
Be aware of scarce Resources
• Major resource limitations
– Processing power
– Communication bandwidth
– Onboard memory
– Energy
• Various Prototypes (with decreasing performance)
Sony XCISX100C/XP
x86 VIA Eden ULV @ 1 GHz
TrustEYE.M4
ARM Cortex@ 168MHz
SLR Engineering
Atom Z530@ 1.6 GHz
CITRIC
PXA 270@ 13-640MHz
[Rinner et al. The Evolution from Single to Pervasive Smart Cameras. Proc. ICDSC 2008]
17. B. Rinner 17
Video Surveillance Network
• 3rd generation
– all-digital systems
• 3+ generation
– smart cameras
– surveillance tasks run on-site on smart cameras, e.g.,
• video compression traffic statistics
• accident detection wrong-way drivers
• stationary vehicles (tunnels) vehicle tracking
• 1st and 2nd generation
– primarily analog frontends
– backend systems are digital
[Regazzoni, Ramesh, Foresti. Special Issue on Video Communications, Processing and Understanding
for Third Generation Surveillance Systems. Proceedings of the IEEE. October 2001]
18. B. Rinner 18
Video Surveillance Network (2)
• Even third generation networks rely on “heavy” infrastructure.
– Camera nodes: sensor, onboard processing (encryption)
– Network: hierarchically structured, wired, large bandwidth
– Energy: dedicated supply
• Surveillance networks typically consist of large number of
cameras
• Processing in network is fixed; (compressed) data is streamed
to control center
19. B. Rinner 19
Characteristics of VSN
• Visual sensor networks lie somewhere in between wireless
sensor networks (WSN) and multi-camera/surveillance
networks.
• VSN have unique characteristics (wrt. traditional WSN)
• Resource limitations
– Need to process and transfer large amounts of data
– Energy and bandwidth
• On-board processing (cp. Smart cameras)
– Challenging vision algorithms
– Adaptive behavior
[Soro et al. A Survey of Visual Sensor Networks. Advances in Multimedia 2009]
20. B. Rinner 20
Characteristics of VSN (2)
• Real-time operation
– Most applications require real-time analysis (camera to user)
• Location and orientation information (spatial calibration)
– Absolute or relative coordinates and orientations
– (Multi-)camera calibration
• Time Synchronization (temporal calibration)
• Data Storage
– Access to historic data necessary, eg., frame buffer, detected events
– Stored data may be discarded over time
• Autonomous Camera Collaboration
– cp. Distributed smart cameras (DSCs)
21. B. Rinner. 21
(Selected) VSN Problems
• Sensor Placement
– Eg., dynamic setting of PTZ parameters
• Clustering, cluster head election
– Eg., what cameras should “work together”, who is the “leader”
• Synchronization and calibration
– Eg., establish temporal and spatial correlation
• Data (and energy) distribution
– Eg., when and what data to exchange
• …
23. B. Rinner 23
Configuring Smart Camera Networks
• Smart camera networks process data onboard can modify their
functionality/execute actions during runtime to reflect changes
– to the state of the environment
– to the user criteria
• A configuration describes what is processed/executed where;
specified by
– Description of camera network (including the available actions/tasks)
– Specification of the objective
• We study configuration methods to use scarce resources in
these networks more efficiently
24. B. Rinner 24
Configuration Problem (example)
• Configuring a camera network
– Select a set of cameras to monitor an area of interest
– Set the sensor (frame rate, resolution, PTZ) to achieve QoS
– Assign monitoring functions to cameras
– Optimize wrt. multiple criteria
– Dependent on dynamics of environment
s1
s2 s3
s4
s5
t1
t2
t3
p1, p2
p3
p4, p5
25. B. Rinner 25
Configuration Design Space
Design space for configuration methods is given by:
• Dynamics of environment (static vs. mobile observation points)
• Configuration algorithm (centralized vs. distributed)
• Tasks and sensors (homogeneous/heterogeneous; static/mobile cameras)
• A priori knowledge (complete vs. no knowledge of environment/VSN)
• Various alternatives for solving this optimization problem, eg.
– Centralized configuration algorithms
– Distributed configuration algorithms
focus on resource-aware approaches
26. B. Rinner 26
Centralized Configuration with EA
• Approximation with evolutionary algorithm satisfying all
requirements along multiple criteria (eg., energy, data, QoS)
• Smart Camera Network
– Set of cameras at known position with fixed FoV
– Sensor configurations (frame rate, resolution)
• Observation Area
– Static set of observation points with monitoring activity a
at required QoS (pot, fps)
• Monitoring tasks
– Assign procedures for achieving
– Required resources for
},...{ 1 nSSS =
[Dieber, Micheloni, Rinner. Resource-Aware Coverage and Task Assignment in Visual Sensor Networks
IEEE Transactions on Circuits and Systems for Video Technology, Aug 2011]
},...,{ 1 ki ddD =
},...,{ 1 mttT =
},...,{; 1 apaa ppPAa =∈
),,(),( iiiii emcdPr →
27. B. Rinner 27
Self-aware Configuration
• Adopted from proprioceptive computing systems
– use proprioceptive sensors to monitor “one self”
(concept from psychology, robotics/prosthetics, …, fiction)
– reason about their behavior (self-awareness)
– effectively and autonomously adapt their behavior to
changing conditions (self-expression)
• Demonstrate autonomous multi-object tracking in camera
network
– Exploit single camera object detector & tracker
– Perform camera handover
– Learn camera topology
28. B. Rinner 28
Bid C4
Virtual Market-based Handover
• Initialize auctions for exchanging tracking responsibilities
– Cameras act as self-interested agents, i.e., maximize their own utility
– Selling camera (where object is leaving FOV) opens the auction
– Other cameras return bids with price corresponding to “tracking” confidence
– Camera with highest bid continues tracking;
trading based on Vickrey auction
Camera 1 Camera 2
Camera 3
Camera 4
Init
auction
Bid C3
Fully distributed approach
no a-priori topology knowledge required
29. B. Rinner 29
Market-based Tracking Handover
• Utility function (each camera) rpjvcOU
iOj
ijjii +−Φ⋅⋅= ∑∈
)]([)(
tracking decision
visibility
confidence
payments made
payments received
Simulation
green: tracking
yellow: shared FOV
red: trading (handover)
30. B. Rinner 30
Tracking Performance
• Tradeoff between utility and communication effort
Scenario 1 (5 cameras, few objects) Scenario 2 (15 cameras, many objects)
• Emerging Pareto front
[Esterle et al. Socio-Economic Vision Graph Generation and Handover in
Distributed Smart Camera Networks. ACM Trans. Sensor Networks. 2013]
31. B. Rinner 31
Learn Neighborhood Relationships
• Gaining knowledge about the network topology (vision graph) by
exploiting the trading activities
• Temporal evolution of the vision graph
32. B. Rinner 32
Learning Heterogeneous Strategies
• Heterogeneous strategies at cameras may improve Pareto front
• Adapt camera behaviour by online learning using bandit solvers
Homogeneous vs. heterogeneous
handover strategies (offline)
Online learning strategies with
different bandit solvers
[Lewis et al. Learning to be different: Heterogeneity and Efficiency in Distributed
Smart Camera Networks. In Proc. IEEE SASO. 2013]
34. B. Rinner 343434
#1 Trustworthy Cameras
• Smart cameras
– Highly capable embedded systems (on-board video analysis)
– Large software stacks
– Networked devices using closed (CCTV) and public networks
• Applications no longer only in public but also in private
areas (assisted living, home monitoring, …)
• Protection of sensitive image data
– Protection against manipulation (e.g., enforcement applications;
evidence at court)
– Privacy of monitored people
35. B. Rinner 353535
Goals and Assumptions
• We present a system level approach that addresses the following
security issues:
– Integrity: detect manipulation of image and video data
– Authenticity: provide evidence about the origin of image and videos
– Confidentiality: make sure that privacy sensitive image data cannot be
accessed by an unauthorized party
– Multi-level Access Control: support different abstraction levels and
enforce access control for confidential data
• Security and privacy protection as inherent features of the
camera
• Considered attack types: only software attacks
[Winkler, Rinner. Securing Embedded Smart Cameras with Trusted Computing.
EURASIP Journal on Wireless Communications and Networking, 2011 ]
36. Approach
• Bringing of Trusted Computing concepts into cameras
• Trusted Platform Modules (TPMs) are well defined, readily
available and cheap
• TC is an open industry standard
• TPMs are available from many manufacturers
B. Rinner 36
37. Hardware Security Anchor
37
• Trusted Platform Module (TPM) at a glance
– Secure storage for cryptographic keys
– Data encryption, digital signatures
– System status monitoring and reporting (measurement + attestation)
– Unique platform ID
Security Chip
(TPM)
Image Sensor CPU RAM
Bootloader
Operating System (e.g., Embedded Linux)
Software Libraries and Middleware
Image Processing and Analysis Communication…
Software
Hardware
B. Rinner
38. Implemented Security Features
38
• Trusted boot where camera software stack is “measured” and
the status is securely reported to operator
• Integrity and authenticity guarantees using non-migratable,
TPM-protected RSA keys
• Freshness/timestamping for outgoing images via TPM-
protected tick (counter) sessions
B. Rinner
39. B. Rinner 393939
Hardware Prototype
• TI OMAP 3530 CPU:
ARM @ 480MHz and
DSP @ 430MHz
• 256MB RAM,
SD-Card as mass storage
• VGA color image sensor
• wireless: 802.11b/g WiFi
and 802.15.4 (XBee)
• LAN via USB
(primarily used for debugging)
• Atmel hardware TPM
on I2C bus
40. Privacy Protection Approaches
40
• Protection as an inherent
feature of the camera
• Object-based protection:
Identification of sensitive
data (e.g., human faces)
• Data abstraction and
obfuscation
• Global protection techniques: Uniform protection of entire
frames (insensitive to misdetections of computer vision)
B. Rinner
41. Multi-Level Protection
41
• Video stream contains sub streams
• Every sub stream is encrypted
– Hardware-bound cryptographic keys
• Recovery of identities only via four eyes
principle
Video Stream
Smart
Camera
Sub Streams
B. Rinner
43. B. Rinner 434343
Privacy-aware Camera Networks
• What about users (i.e., monitored people)?
• users usually do not care much about integrity, authenticity of time
stamping
• users (hopefully!) care about confidentiality and privacy!
• Question 1: How can we increase privacy awareness?
• Question 2: How can we demonstrate that (our) cameras
protect the privacy of users?
44. B. Rinner 444444
Raising Privacy Awareness
• Let users know if there are cameras in their environment
• Use user's handheld (e.g., smart phone) for location-based
notifications
45. User Feedback
• Goal: Trustworthy feedback to monitored
persons about camera’s privacy protection
• Visual communication for authentication
– Direct line of sight
– Intuitive way to select intended camera
• Operator discloses applications to TrustCenter
T. Winkler and B. Rinner, “User Centric Privacy Awareness in Video Surveillance,”
Multimedia Systems Journal, vol. 18, no. 2, pp. 99–121, 2012.
B. Rinner
47. B. Rinner 47
#2 Aerial Cameras for Disaster Mgm.
• Develop autonomous multi-UAV system for aerial
reconnaissance
• Up-to-date aerial overview images are helpful in many
situations:
“Google Earth with up-to-date images in high resolution”
• Small-scale quadcopter platform with
onboard sensors and computation
• GPS receiver for autonomous
waypoint flights
• Generic framework not bound
to specific UAV
48. B. Rinner 48
Key Challenges
• Increase autonomy
– Control and coordination of multiple UAVs
– High-level interaction with user
• Provide prompt response to user
– Provide preliminary results fast and improve over time
• Deal with strong resource limitations
– Flight time, payload, computation and communication
– Limited sensing capabilities
50. B. Rinner 50
Key Questions
• How to generate and update movement routes for the UAVs?
– Achieve multiple optimization goals
– Deal with changes in the environment
• How to setup a wireless UAV network?
– Provide networking coverage
• How to generate the mosaic image?
– Apply incremental image stiching
– Combine RGB and thermal images
• System integration and demonstration
52. B. Rinner. SCVSN Tutorial (Chapter 3) 525252
Research Directions of
Visual Sensor Networks
53. B. Rinner. 53
#1: Architecture
• Low-power (high performance) camera nodes
– Dedicated platforms: vision processors, PCBs, systems
– Many examples: CITRIC, NXP
• Visual/Multimedia Sensor Networks
– Topology and (multi-tier) architecture
– Multi-radio communication
• Dynamic Power Management
– For sensing, processing and communication
How to design resource-aware nodes and networks
54. B. Rinner. 54
#2: Networking
• Ad hoc, p2p communication over wireless channels
– Providing RT and QoS
– Eventing and/or streaming
• Dynamic resource management
– (local) computation, compression, communication, etc.
– Degree of autonomy: dynamic, adaptive, self-organizing
– Fault tolerance, scalability
– Network-level software, middleware
How to process and transfer data in the network
55. B. Rinner 55
#3: Deployment, Operation, Maintenance
• Development support for applications
– Model/simulate the application (function, resources, QoS)
– Reuse/exchange of software/libraries
– Software updates, debugging etc.
• Autonomous calibration and scene adaption
– Avoid manual procedures
– Adapt to different scenes and settings
• Network configuration
Consider the entire life cycle of the camera network
56. B. Rinner. 56
#4: Distributed Sensing & Processing
• Sensor placement, calibration & selection
– Optimization problem
– Distributed approaches eg., consensus, game theory, multi-agent systems
• Compressive Sensing
• Collaborative data analysis
– Multi-view, multi-temporal, multi-modal
– Sensor fusion
• Online/real-time processing
– Can not effort to store large amounts of data
Where to place sensors and analyze the data
57. B. Rinner 57
#5: Mobility
• Mobile cameras are ubiquitous
– PTZ, vehicles, robotics etc.
– Mobile phones
• Advanced vision algorithms
– Ego motion, online calibration
– Closed-loop control, active vision
How to exploit networks of mobile cameras
58. B. Rinner 58
#6: Usability
• Ease of deployment, maintenance
– Self-* functionality
– “Smart cameras for dumb people”
• Privacy and Security
– Trust of the user
– Control the privacy setting
• Interaction with the camera network
How to provide useful services to people
59. B. Rinner 59
#7: Applications
• Demonstrations
– Large scale networks eg., for surveillance
– Small scale networks eg., for entertainment, home environments
– Only single camera application?
• Market opportunities
• Killer Application
What applications can (only) be solved by DSC
60. B. Rinner. SCVSN Tutorial (Chapter 3) 60
Summary
• VSNs exploit various advantages of distributed camera sensors
such as increased coverage, redundancy and 3D information.
• Distributed cameras impose various challenges such as huge
amount of data, required infrastructure and (network)
topology.
• VSN have unique characteristics (wireless sensor networks vs.
surveillance camera networks)
• Current research addresses signal processing,
communications, architecture and middleware issues.
61. B. Rinner 61
Acknowledgements & Further Info
Pervasive Computing @ AAU UAV Research
http://pervasive.aau.at http://uav.aau.at
• Tutorial site
Most recent course material is available at
http://pervasive.aau.at/S5-tutorial
