For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit-montgomery For more information about embedded vision, please visit: http://www.embedded-vision.com Clay D. Montgomery, Freelance Embedded Multimedia Developer at Montgomery One, presents the "Building Complete Embedded Vision Systems on Linux—From Camera to Display" tutorial at the May 2019 Embedded Vision Summit. There’s a huge wealth of open-source software components available today for embedding vision on the latest SoCs from suppliers such as NXP, Broadcom, TI and NVIDIA, at lower power and cost points than ever before. Testing vision algorithms is the first step, but what about the rest of your system? In this talk, Montgomery considers the best open-source components available today and explains how to select and integrate them to build complete video pipelines on Linux—from camera to display—while maximizing performance. Montgomery examines and compares popular open-source libraries for vision, including Yocto, ffmpeg, gstreamer, V4L2, OpenCV, OpenVX, OpenCL and OpenGL. Which components do you need and why? He also summarizes the steps required to build and test complete video pipelines, common integration problems to avoid and how to work around issues to get the best performance possible on embedded systems.

1. © 2019 Montgomery One
Building Complete Embedded
Vision Systems on Linux - Camera
to Display
Clay D. Montgomery
Montgomery One
May 2019

2. © 2019 Montgomery One
The Big Picture - Overview
• Start on a Ubuntu Linux PC to develop and prototype new CV algorithms and applications
- Install, build and explore the OpenCV sample programs on a Ubuntu system
• Select your Open-Hardware ARM SoC (System on Chip) board and cameras carefully
- A good BSP with camera drivers and Yocto support are critical!
• Build a custom OS with Yocto Open Embedded Linux for your target board
- Use the same open-source libraries on Ubuntu and target systems, if at all possible
• Build your application with the Yocto toolchain for your ARM target system
• Select Open-Source component libraries to use
- Which do you actually need and which are the best for your application?
- V4L2, FFmpeg, GStreamer, OpenCV, OpenGL ES, OpenCL and OpenVX?
• Explore the options for acceleration on your ARM SoC
- Acceleration is required for most CV applications, so plan for it
2

3. © 2019 Montgomery One
Popular ARM SoC Boards for Vision
3
RaspberryPi 3 TI BeagleBoard
NXP i.MX8
Nvidia Jetson TX2
i.MX6 WandBoard with MIPI Camera

4. © 2019 Montgomery One
Exploring Open Source Video
Components for Linux

5. © 2019 Montgomery One
An Example - Vision Components for Lightwing
5

6. © 2019 Montgomery One
V4L2 – Video for Linux (Version 2)
6
• Negotiates compatible video formats and manages buffers and events between apps and drivers
• The de-facto standard low-level video API for Linux
• The most widely used API for video input on Linux systems
• Most Linux distros and BSPs provide drivers for cameras and TV tuners for V4L2
• Version 1 is obsolete, but still used by the OpenCV VideoCapture class
• Well supported by most SoC vendors that have integrated MIPI camera support
• The UVC standard (USB Video Class for web cams) is newer and still uses V4L2 on Linux

7. © 2019 Montgomery One
GStreamer
• Negotiates compatible video formats and builds pipelines and filter-graphs to connect plug-in
components and apps to the V4L2 API
• The de-facto standard high-level audio/video API for Linux
• The most widely used API for video pipelines on Linux systems
• Used by OpenCV's VideoCapture class to work around video format issues, such as color space
and RGB/BGR mismatch issues
gst-launch-1.0 autovideosrc ! videoconvert ! autovideosink
./videocapture_basic "imxv4l2videosrc device=/dev/video0 ! videoconvert ! appsink"
• Well supported by some SoC vendors that have integrated VPUs and codecs
- VPUs can be very difficult to exploit without good drivers for GStreamer
- This is a major differentiator among SoC vendors and should be evaluated carefully
7

8. © 2019 Montgomery One
OpenCV
• 2D and 3D feature toolkits
• Egomotion estimation
• Facial recognition system
• Gesture recognition
• Human–computer interaction (HCI)
• Motion tracking and understanding
• Object segmentation and recognition
• Depth perception from 2 cameras
• Structure from motion (SFM)
• Statistical machine learning and DNN
• Support for OpenCL and CUDA accelerators
8
• Not a Khronos standard, but mostly open-source
• Began as Intel Research open initiative in 1999
• Intel donated OpenCV to non-profit OpenCV.org
• Most comprehensive and widely used CV library
• Available on most platforms and languages today
• Lots of sample programs in C/C++ and Python
• Great place to start to prototype a new design
• Latest 4.x available, but not much help on ARM
FeaturesOrigin and Status
Issues
• Typically too slow for camera video (on ARM)
• Samples require X11 on Linux
• Bugs in VideoCapture class on Linux
• Requires video in BGR format

9. © 2019 Montgomery One
FFmpeg
• A set of libraries for encoding, decoding and converting audio and video files and
streams, such as JPEG, MPEG and H.264, etc.
• Supported on a very wide range of platforms
• Very limited support for accelerators on ARM SoCs (NEON)
- Most ARM SoC vendors do not provide drivers to accelerate FFmpeg
- FFmpeg codecs are typically too slow for camera video on ARM SoCs
- But, still useful for de-muxing, removing containers, etc.
9

10. © 2019 Montgomery One
OpenGL ES
• The original Khronos consortium standard for 3D graphics on SoCs
• One of the most successful and widely adopted APIs for embedded multimedia ever
• Version 1.1 is obsolete
• Versions 2.0 - 3.0 are widely supported today by most SoC vendors with GPUs
• Shader code can be used to accelerate some CV algorithms
• Version 3.1 added more abilities to do general-purpose compute, including CV
• Driver support is more mature and stable than for OpenCL and OpenVX
• GLSL coding is familiar to more developers than OpenCL, NEON or CUDA
• Lightwing uses version 2.0 GLSL for faster motion tracking than is possible with OpenCV
10

11. © 2019 Montgomery One
OpenCL
11
• Khronos consortium standard for general-purpose compute acceleration on SoCs and FPGAs
• Framework for general-purpose compute acceleration across heterogeneous processors
including CPUs, GPUs, DSPs and FPGAs
• Based on C99 and C++11 languages
• Provides a standard interface for parallel computing using task- and data-based parallelism
• Some mid to high-end SoC and FPGA vendors support OpenCL for CV applications
• Open alternative to Nvidia's CUDA
• Not as mature on most platforms as OpenGL ES or CUDA
• Will accommodate a wider range of algorithms than OpenGL ES shader code

12. © 2019 Montgomery One
OpenVX
12
• Khronos standard for cross-platform acceleration of computer vision applications
• A higher level of abstraction for programming CV than OpenCL
• Based on a connected graph of vision nodes that execute a preferred chain of operations
• Complementary to OpenCV, but can offer more optimized graph management
• Many SoC vendors are quickly developing support for OpenVX
• Available now on NVIDIA
• Supports face, body and gesture tracking, smart video surveillance, advanced driver assistance
systems (ADAS), object and scene reconstruction, augmented reality, visual inspection, robotics, etc.

13. © 2019 Montgomery One
Basic Camera Video Pipelines
13

14. © 2019 Montgomery One
Your First Camera Video Pipeline (V4L2 and GStreamer)
• Attach a USB Camera to a Ubuntu PC (or your target system) and try:
dmesg | grep camera
- Displays kernel error messages about your camera driver's initialization
ls /dev/video0
cat /dev/video0
- Shows if your camera driver is installed and working (producing data)
lsmod
- Lists all installed kernel drivers
gst-launch-1.0 videotestsrc ! autovideosink
- Test to see if gstreamer is installed and working (should display a color bars test pattern)
gst-launch-1.0 autovideosrc ! autovideosink
- Initialize and run a complete video pipeline (camera to display) using V4L2 and gstreamer
14

15. © 2019 Montgomery One
OpenCV Sample Programs

16. © 2019 Montgomery One
Explore the OpenCV Sample Programs
• Install OpenCV 3.2 on Ubuntu (14 - 16) and build the sample apps:
sudo apt-get install opencv
sudo apt-get install build-essential
sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
cd ~/opencv-3.2.0/opencv
mkdir release
cd release
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
make
sudo make install
cp -r /usr/local/lib/libopencv* /usr/lib/.
• Run some sample apps on Ubuntu:
cd /usr/share/OpenCV/samples/bin
./cpp-example-videocapture_basic
• Combine sample source codes with 'videocapture_starter' to test an algorithm with camera video
- cpp-example-videocapture_starter.cpp + cpp-example-edge.cpp
16

17. © 2019 Montgomery One
Yocto Open Embedded Linux

18. © 2019 Montgomery One
Build a Yocto Linux System for Your Target Board
• Choose a board with a Yocto BSP and cameras with recipes for Yocto
• Test the camera drivers provided in your BSP well
- You will likely have to create recipes and/or a new V4L2 driver for your camera
- An example of a working driver from your camera vendor is a minimum requirement
• Install Yocto (Pyro, Rocko or Sumo) on Ubuntu and build a bootable OS:
repo init -u https://github.com/Freescale/fsl-community-bsp-platform -b pyro
repo sync
MACHINE=wandboard DISTRO=fslc-x11 source setup-environment build
bitbake fsl-image-machine-test-full (or, core-image-basic, fsl-image-x11-qt5, etc.)
• Build and install cross-development toolchain to build your app and any additional packages:
bitbake fsl-image-machine-test-full -c populate_sdk
./build/tmp/deploy/sdk/fslc-x11-glibc-x86_64-fsl-image-multimedia-armv7at2hf-neon-toolchain-2.3.4.sh
bitbake -s
18

19. © 2019 Montgomery One
Integrating OpenCV with
Yocto Linux

20. © 2019 Montgomery One
An Example - Vision Components for Lightwing
20

21. © 2019 Montgomery One
Build and Run OpenCV Samples for Your Target Board
• Add packages for OpenCV, X11, gstreamer, etc. to Yocto's config file:
buildconflocal.conf
CORE_IMAGE_EXTRA_INSTALL += "opencv x11 gstreamer"
• Build and test the OpenCV 3.2 samples using Yocto toolchain, sysroot and CMake in your Yocto tree
(mostly the same as on Ubuntu):
source /opt/fslc-x11/2.3.4/environment-setup-arm7at2hf-neon-fslc-linux-gnueabi
cd release
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
make
• Running the sample apps on your target requires initialization of an X window (X11):
export DISPLAY=:0.0
xhost +SI:localuser:root
cd /usr/share/OpenCV/samples/bin
./cpp-example-videocapture_basic
21

22. © 2019 Montgomery One
VideoCapture Class for Linux
• The camera interface for OpenCV
• About 40 implementations are provided for all major OS platforms
• Three implementations are provided for Linux:
cap_v4l.cpp - Original, for obsolete V4L version 1
cap_libv4l.cpp – Updated replacement for V4L2, with many patches
cap_gstreamer.cpp – Optional extension convert formats and accepts GStreamer pipeline syntax:
cpp-example-videocapture_basic 'videotestsrc ! videoconvert ! appsink'
• Required some debugging to work on newer (4.x) Linux kernels
- If V4L2 fails to initialize, it then tries V4L (version 1), which produces very confusing error messages.
- Removed the xioctl() call macro because it was failing when calling to inquire the video format
(VIDIOC_S_FMT)
• Some improvements are available in OpenCV 4.1.0 for the Android Media NDK
22

23. © 2019 Montgomery One
What Else Can (and Did) Go Wrong?
• OmniVision 5640 camera discontinued
–Was a popular camera supported by the FSL Community BSP
–Use new OV5645 driver from Rocko version instead
–Only older versions of Yocto (Pyro) actually work on most i.MX6/8 boards
• Yocto only supports OpenCV 3.2 because of CMake compatibility issues
–CMake installed by Yocto will not build OpenCV 3.4
• OpenCV samples require X Windows on Linux
–Either use X11 or eliminate the use of the High-GUI module in samples
• OpenCV VideoCapture class requires GStreamer on Linux
–Unnecessarily complex and broken on the i.MX6/8 platform
–Fixed the bugs in V4L2 implementation, so it works without GStreamer
23

24. © 2019 Montgomery One
Working Around Component Limitations
• OpenCV (VideoCapture class) requires the antiquated 24 bit BGR pixel format !
–Many cameras only provide 16 bit RGB 565, YUV or RAW formats
–RAW format is 10 bit RGB (Bayer), but requires software conversion (very slow)
• Official solution is to use gstreamer (with an appsink) element to convert YUV to BGR:
videocapture_starter 'imxv4l2videosrc ! videoconvert ! appsink'
- Can be accelerated by the IPU on i.MX6/8 by using NXP's plugins for gstreamer
- libgstapp.so must be installed manually for this to work due to a bug in some builds for i.MX6/8
• Other ways to adapt the V4L2 VideoCapture class and get better performance:
–Use the i.MX6/8 IPU without GStreamer. Red and Blue are swapped, but for many CV apps, that is fine
–Use luminance data directly (dropping color data), without conversion to RGB
• i.MX6 IPU will not scale video beyond 1024 x 1024 pixels (fixed on the i.MX8)
• Lightwing solution - Use the GPU (OpenGL ES) to convert and scale video, instead of the IPU or GStreamer
- Basic motion tracking algorithm is done in shader code, instead of OpenCV
24

25. © 2019 Montgomery One
Lightwing Motion Tracking
Demo

26. © 2019 Montgomery One
Lightwing Motion Tracking 3D GUI Demo
Motion tracking on the NXP i.MX6 GPU enables interactive control of 3D assets for
mixed-reality video walls, without VR headsets or hand controllers
26

27. © 2019 Montgomery One
Conclusions

28. © 2019 Montgomery One
Lessons Learned
• Start on a desktop PC with Ubuntu and the OpenCV samples
- Develop your vision algorithms there, first
• Use Yocto Open Embedded Linux to build your custom platform
• Use the same open source libraries on desktop and target systems
- Hacking OpenCV's VideoCapture class will likely be required
• Understand which vision components you actually need and why
• Select cameras carefully, considering software support
- You really need a good V4L2 driver with Yocto recipes
- Use a MIPI camera if possible, instead of USB
• Explore the acceleration options for your SoC (GPU, IPU, NEON, etc)
- Acceleration is required for most CV applications of camera video on ARM
28

29. © 2019 Montgomery One
Resources
29
OpenCV Installation Tutorial
https://www.docs.opencv.org/2.4.13/doc/tutorials/intr
oduction/linux_install/linux_install.html#linux-
installation
Yocto Open Embedded Linux
https://www.yoctoproject.org/
GStreamer Documentation
https://gstreamer.freedesktop.org/documentation/
V4L2 Documentation
https://www.kernel.org/doc/html/v4.9/media/uapi/v4l
/v4l2.html
Khronos OpenGL ES, OpenCL and OpenVX
Documentation
https://www.khronos.org/
Lightwing Open Mixed-Reality Platform
https://montgomery1.com/lightwing/
NXP i.MX6/8 Development Boards
https://www.wandboard.org/
OpenCV Status on Github
https://github.com/opencv/opencv/wiki/ChangeLog

30. © 2019 Montgomery One
Appendix

31. © 2019 Montgomery One
OpenCV C++ Example - Edge Detection on Camera Video
int iEdgeThreshSobel = 1;
Mat mImageFrame, mImageGray, mImageGrayBlur, mImageEdgeMask, mImageEdgeResult;
const char* WindowName = "Canny edge map with Sobel gradient";
static void onTrackbar(int, void*)
{
cvtColor(mImageFrame, mImageGray, COLOR_BGR2GRAY); // Create blurred gray scale image for edge detection.
blur(mImageGray, mImageGrayBlur, Size(3, 3));
Canny(mImageGrayBlur, mImageEdgeMask, iEdgeThreshSobel, iEdgeThreshSobel * 3, 3); // Canny detector with sobel filter.
mImageEdgeResult = Scalar::all(0); // Clear to black.
mImageFrame.copyTo(mImageEdgeResult, mImageEdgeMask);
imshow(WindowName, mImageEdgeResult); // Display image frame in window.
}
int main(int argc, char** argv)
{
VideoCapture capture;
capture.open(0); // Open camera device through V4L2.
namedWindow(WindowName, WINDOW_KEEPRATIO); // Create window and slider control.
createTrackbar("Canny threshold Sobel", WindowName, &iEdgeThreshSobel, 100, onTrackbar);
char key = 0;
while (key != ‘q’) // Capture frames from the camera and display them.
{
capture >> mImageFrame; // Capture another image frame from camera.
if (mImageFrame.empty())
break;
onTrackbar(0, 0); // Show the image.
key = (char)waitKey(30); // Wait 30 milliseconds for a key press.
}
return 0;
}
31

32. © 2019 Montgomery One 32
Lightwing – Open Mixed-Reality Video Wall Platform
• Build custom digital signs and video walls with interactive 3D content
• Camera motion tracking controls 3D assets without VR headsets or hand controllers
• Supports both touch screens and camera motion tracking input
• Multipanel video wall dimensions of any size for public spaces
• Built-in 3D animations, fonts, effects, audio, video, images and web RSS feeds
• GPU and IPU acceleration of camera video on the NXP i.MX6/8 series SoCs
• Scriptable, browserless architecture built on open tools for Windows and Linux
https://montgomery1.com/lightwing/

33. © 2019 Montgomery One
Lightwing on Linux verses Android and Windows
33

34. © 2019 Montgomery One
About Clay D. Montgomery
• 30+ Years of Embedded Multimedia Software Development
• C/C++, OpenGL 3D, Audio/Video, Linux, Windows and Android
• Previously worked at STB (3Dfx), VLSI (NXP), Nokia, TI and AMX (Harmon)
• Authored on Embedded OpenGL ES for Intel Developer Zone, TI, Montgomery One
• Freelance Embedded Linux Developer since 2010
• Now working on Open Embedded Linux (Yocto) on ARM almost Exclusively
• Active in Tech Start Up Community in Texas
• Created Lightwing Mixed-Reality Engine for Interactive Digital Signage, Video Walls and
Touch-Kiosks on NXP i.MX6/8 SoCs
• Currently adding CV Object Detection and Motion Tracking Features to Lightwing
34