The document discusses embedded computer vision and presents examples of embedded computer vision systems developed by Wang, Yuan-Kai and his team. It describes research in embedded computer vision using CPUs, DSPs and FPGAs. It also outlines challenges in embedded computer vision and provides examples of projects including an entertainment robot, vision sensor network, video surveillance system, and wearable camera.
1. Wang, Yuan-Kai(王元凱)
Electrical Engineering Department,
Fu Jen University (輔仁大學電機工程系)
Email:
ykwang@mail.fju.edu.tw,
URL:
http://www.ykwang.tw
Slideshare:
Search
"Towards
Embedded
Computer
Vision"
Towards
Embedded
Computer Vision
1
2. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Context Awareness
2
人
時
物
地
事
Situation
High-Level Context Low-Level Contexts
3. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
3
MIT Oxygen
http://oxygen.lcs.mit.edu/ 1999
4. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
4
The Aware Home
http://awarehome.imtc.gatech.edu/
5. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Ambient Intelligence v.s.
Object Intelligence
Embedded system + Sensors
5
Wearable Computing
Smart Home, Smart …
6. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
66
Embedded Systems
"Without" Sensors
資料來源:資策會MIC
ITIS計畫整理
7. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
77
Embedded Systems
"With" Sensors
GPS
Navigation
Wii
Roomba
Exoskeleton
8. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
8
Driver assist
Front view
Night vision
Blind corner
Blind spot
Rear view
Embedded Systems
"With" Image Sensors
9. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
99
Embedded Systems
"With" Image Sensors
Google Glass
10. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Smart Watch
10
11. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Microsoft
SenseCam
Embedded Systems
"With" Image Sensors
11
12. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Observation
Miniaturization
High Resolution
3D Capability
12
13. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
13
Embedded Computer Vision
Embedded System + Camera
+ Computer Vision Algorithm
Image
Processing
Image
Recognition
Image
Capturing
14. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
14
Embedded Computer Vision
Embedded compute vision
Implements computer vision algorithms
on low-cost, low-power,
constrained hardware
Constrained hardware
Low-speed CPU
Low capacity memory
No floating-point (FPU)
Low-resolution image sensor
15. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
A
B
Research Status (1/3)
15
16. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Research Status (2/3)
H
K
K
16
17. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
1717
Research Status (3/3)
DARPA
Grand
Challenge
Google
Car
Google
Taxi
18. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
18
International Activities (1/2)
Special conferences
IEEE Int. Workshop on Embedded Computer
Vision
IEEE Int. Conf. Distributed Smart Cameras
Special journal issues
IEEE Journal of Selected Topics in Signal
Processing, vol. 2, no. 4, Aug. 2008
EURASIP Journal on Embedded Systems
Short courses in important CV conferences
CVPR07&08: Distributed vision processing in
smart camera networks
ESC07: Embedded CV and smart cameras
ICASSP09: Distributed processing in smart
cameras
19. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
19
International Activities (2/2)
Research projects and Lab.
Princeton Univ./Georgia Tech.:
Embedded Systems Lab., Wayne Wolf
Stanford Univ.
Wireless sensor networks Lab.
UCLA, CMU, MIT
Delft Univ. of Technology
SmartCam Project
Graz Univ. of Technology
20. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
20
Three Ways for ECV
CPU (Central Processing Unit)
ARM, PowerPC
DSP (Digital Signal Processor)
TI, ADI, NXP
FPGA (Field Programmable Gate Array)
Altera, Xilinx
21. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
21
CPU
Stanford MeshEye
ARM 7 (55MHz), ZigBee node
3 image sensors
30x30 grayscale x 2
640x480 color x 1
Object detection
(http://wsnl.stanford.edu/smartcam.html)
"MeshEye:a hybrid-resolution
smart camera mote for applications
in distributed intelligent
surveillance", IPSN-SPOTS, 2007
22. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
22
DSP
"Visual surveillance on DSP-based
embedded platform," Graz Univ. of
Technology, 2008(Phd. dissertation)
TI C6414 (600MHz)
Applications: Adaboost face detection, vehicle
detection, license plate detection
23. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
23
FPGA (2/3)
"Hardware, Design and
Implementation Issues on a
FPGA-Based Smart Camera,"
ICDSC, 2007
24. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
24
Hybrid: CPU+DSP
"Distributed Embedded Smart Cameras
for Surveillance Applications,"
IEEE Computer, 2006.
Developed for traffic surveillance
25. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hybrid: CPU+DSP by SoC
TI OMAP
25
26. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
26
Hybrid: CPU+FPGA
UCLA Cyclops
MCU: Atmega128
CPLD: image processing
ECV app.: Hand gesture recognition
http://research.cens.ucla.edu/
projects/2007/
Multiscaled_Actuated_Sensing/
Cyclops/
27. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hybrid: CPU+FPGA by SoC
Xilinx Zynq
27
28. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
28
Challenges for ECV
Algorithm refinement
Parallel computation
Function partition, Multi-threading
Stream processing
Memory hierarchy optimization
Hardware design
Pipeline, SIMD, board design
Optimized programming skills
Fixed-point arithmetic
Memory management
Intrinsic commands
29. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
29
Our Experiences
Entertainment Robot (CPU)
Vision Sensor Network (CPU)
Smart Video Surveillance (DSP)
Wearable Camera (DSP)
30. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
30
Case 1
Entertainment
Robot
CPU
31. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
31
Sony AIBO
32. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Rolling Dice
32
33. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
33
Face Detection
34. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
34
Face Recognition
35. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
35
Facial Expression Recognition
36. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
36
Case 2
Embedded Vision
Sensor
CPU
37. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
37
FJUCam
• Power
consumption:
1W
• Power sources:
• 5V DC current
• 5V Cell Battery
• Weight: 35gm
• Size:
6 x 4.5 x 5 (cm)
(W x H x D)
Three Modules
1. Main board, 2. Lens module
3. Storage module
38. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
38
Face Detection
FJUCam
Face Detection
Image
Display
39. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
39
Cyclops Robot
Color tracking
40. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
40
Case 3
Video Surveillance
FPGA
41. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
41
Mobile Video Surveillance
FPGA
行動通訊網路
(3G/GPRS)
網際網路
網路攝影機
監控伺服器
行動裝置
(3G行動電話)GSM
數據機
Wap Push
傳遞視訊串流
PCBackground subtraction, ...
• 2.8 GHz Intel CPU
• Software: C/C++
• Frame rate: 10 fps
42. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
42
Background Subtraction
Current
Frame
-
Background
Image Update
Post Processing
Motion Object Image
Background Image
1+kB
kB
1+kP
1+kM
),(),(
),(
1
1
yxByxP
yxM
kk
k
−= +
+
),()1(),(
),(
1
1
yxPyxB
yxB
kk
k
+
+
−+= αα
Speed up by (1) Circuit design, (2) Paralization
43. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
43
FPGA Speed Up
Comparisons
PC: 2.8GHz CPU, C implementation
FPGA can speed up 500 times
Clock(Hz) FPS
CPU
FPGA
2.8G
25M
51
10
44. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
44
Case 4
Wearable Camera
X-EYE
ARM + DSP
45. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
45
Goal
45
46. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Demo Video
46
47. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
47
Development
滑鼠
鍵盤
SD卡
USB-WIFI卡
讀卡機
USB-RS232
控制線
微投影機
自製
外殼
USB
連接線
顯示器
觸控面板
筆電
移動電源
BeagleBoard
Camera
USB
Hub
48. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
48
Hardware
4848
Peripheral I/O
• USB Host
• JTAG
• DVI-D video out
• S-Video out
• SD/MMC+
• Stereo in/out
• RS-232 serial1
• Alternate power
• USB 2.0 HS OTG
USER
OMAP3530 Processor
• 600MHz Cortex-A8
• NEON+VFPv3
• 16KB/16KB L1
• 256KB L2
• 430MHz C64x+ DSP
• 32K/32K L1
• 48K L1D
• 32K L2
• Power VR SGX GPU
• 64K on-chip RAM
POP Memory
• 256MB LPDDR RAM
• 256MB NAND flash
RESET
7.6 cm
49. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
49
System
2010.04.25
49
50. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
50
Software
51. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : UAV
51
52. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : Sleep Monitoring
52
53. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : Vehicular
Object Detection
53
54. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
54
Future of
Embedded Vision
55. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
55
Summary
Embedded CPU is not appropriate for
ECV
Pros
Great flexibility
Fast development
Cons
No float-point arithmatics
Interrupt-based architecture
Smart phone with 4/8 cores is OK?
56. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
56
I Think …
Embedded compute vision
Low-cost, low-power,
minimal hardware
High-Performance
computer vision
Fast speed without cost,
power, and hardware
constraints
From contradiction to convergence ?!
Constrained
Resource
Abundant
Resource
Yes by multicore
Contradiction
57. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
57
Challenges (1/2)
Algorithm decomposition
Function decomposition
Partition serial part and parallel part
Data flow analysis and
data dependency analysis
Parallelism
Loop unrolling
Multithreading
Pipeline
58. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
58
Challenges (2/2)
Performance analysis method
For efficiency improvement
Implementation efforts
Choose a good embedded platform for
computer vision
Software issues
Hardware issues
Programming skills
Multi-threading
59. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
59
Future Research Directions
Heterogeneous
Multicore
ARM
+ DSP
ARM +
FPGA
ARM +
GPGPU
60. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
I Think Again …
Is there a development platform for
embedded computer vision with
Built-in camera
AE, AF, AWB
High-performance processor
Heterogeneous multicore
Software development environment:
C, Matlab
OpenCV
Sample codes
Labs
60
61. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Embedded Vision Alliance
61
62. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Ramesh Raskar
MIT Platform
62
63. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Andrew Adams
Stanford Platform
63
64. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Taiwan
Platform
for
Embedded
Vision
?
64
65.
66. 王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
本簡報授權聲明
此簡報內容採用
Creative Commons 「姓名標示
- 非商業性台灣
3.0 版」
授權條款
歡迎非商業目的的重製、散布或修改本簡報的內容,但
請標明:
(1)原作者姓名:王元凱;
(2)圖標示:
簡報中所取用的部份圖形創作乃截取自網際網路,僅供
演講者於自由軟體推廣演講時主張合理使用,請讀者不
得對其再行取用,除非您本身自忖亦符合主張合理使用
之情狀,且自負相關法律責任。
66