For the full video of this presentation, please visit: http://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2015-embedded-vision-summit-opencv For more information about embedded vision, please visit: http://www.embedded-vision.com Gary Bradski, President and CEO of the OpenCV Foundation, presents the "OpenCV Open Source Computer Vision Library: Latest Developments" tutorial at the May 2015 Embedded Vision Summit. OpenCV is an enormously popular open source computer vision library, with over 9 million downloads. Originally used mainly for research and prototyping, in recent years OpenCV has increasingly been used in deployed products on a wide range of platforms from cloud to mobile. The latest version, OpenCV 3.0 is currently in beta, and is a major overhaul, bringing OpenCV up to modern C++ standards and incorporating expanded support for 3D vision. The new release also introduces a modular “contrib” facility that enables independently developed modules to be quickly integrated with OpenCV as needed, providing a flexible mechanism to allow developers to experiment with new techniques before they are officially integrated into the library. In this talk, Gary Bradski, head of the OpenCV Foundation, provides an insider’s perspective on the new version of OpenCV and how developers can utilize it to maximum advantage for vision research, prototyping, and product development.

1. OpenCV 3.0
Gary Bradski
Chief Scientist, Perception and AI at Magic Leap
CEO, OpenCV.org

2. OpenCV at glance
• BSD license, 10M downloads, 500K+ lines of code
• Huge community involvement, automated patch testing and
integration process
• Runs everywhere
SSE, NEON, IPP, OpenCL, CUDA,
OpenCV4Tegra, …
core, imgproc, objdetect …
OpenCV HAL
OpenCV
face, text, rgbd, …
OpenCV
Contrib
Bindings: Python,
Java
Samples, Apps,
Solutions
• Find more at http://opencv.org (user)
• Or http://code.opencv.org (developer)

3. Recent Stats
NOTE:
This is only for source
forge. Many more
downloads come from
Git and many more
come on Unix distros.

4. OpenCV History
Willow Support
OpenCV
Foundation
Intel Support
Google Summer of Code
Nvidia Support
Renewed
Intel Support
Magic Leap

5. OpenCV Algorithm Modules Overview
5
Image Processing
Object recognition
Machine learning
Transforms
Calibration Features
VSLAM
Fitting Optical Flow
Tracking
Depth, Pose
Normals, Planes,
3D Features
Computational
Photography
CORE:
Data structures, Matrix math, Exceptions etc
Segmentation
HighGUI:
I/O, Interface

6. Recent Contributions 2013
https://www.youtube.com/watch?v=_TTtN4frMEA

7. Recent Contributions 2014
https://www.youtube.com/watch?v=3f76HCHJJRA

8. OpenCV 3.0 at glance
• Mostly compatible with OpenCV 2.x; OpenCV
1.x C API is deprecated and partially removed
• Highlights:
– even more modular and extendible
– very stable API tailored for a long-term support
– decent out-of-box performance thanks to IPP,
OpenCL (T-API) and NEON instructions
– lot’s of new functionality!
Aug’14 Nov’14 Dec’14 May’15
3.0 alpha 3.0 beta 3.0rc 3.0 3.1
Q3’15
Goal of 3.0: make a better OpenCV 2.0, cleanup API, get better
performance (with T-API, IPP, NEON), shift to modular structure and
enable user contributions. – Vadim Pisarevsky

9. OpenCV => OpenCV + opencv_contrib
OpenCV
2.x
OpenCV 3.x
contributions
OpenCV
3.x
• OpenCV 3.0 includes mature and clean algorithms that
are fully supported
• A separate contribution repository is for new CV
algorithms that people want to share:
http://github.com/itseez/opencv_contrib
• Patches to the contrib repository are tested as well by
our buildbot to ensure integrity!

10. Using opencv_contrib
The modules in contrib have the same structure as the standard ones:
$ cmake –D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules …
opencv/
modules/
core/
include/, doc/, src/, test/, …
CMakeLists.txt
imgproc
…
opencv_contrib/modules
text/
include/, doc/, src/, test/, …
CMakeLists.txt
…
Path to the contrib modules can be passed to cmake to build them
together with OpenCV:

11. New-style C++ API
• All the high-level vision algorithms (face detectors, optical flow estimators,
stereo matchers etc.) are declared as pure abstract classes, similarly to interfaces
in Java
• Smart pointers to instances are retrieved using special “factory” methods
• All the implementation details are hidden
• The saved space in headers is taken up instead by the verbose Doxygen comments in
order to generate an always up-to-date reference manual.
class StereoMatcher : public Algorithm // Algorithm is used as a base
{
public: // common methods for all stereo matchers
virtual void compute(InputArray left, InputArray right,
OutputArray disparity) = 0;
…
};
class StereoBM : public StereoMatcher
{
public: // specific methods for particular algorithm
virtual void setUniquenessRatio(int ratio) = 0;
…
// factory method
static Ptr<StereoBM> create(…);
};
…
Ptr<StereoBM> matcher = StereoBM::create(…);
matcher->compute(left, right, disparity);

12. Transparent API (T-API) for HW
Specializations
• single API entry for each function/algorithm – no
specialized cv::Canny, ocl::Canny, gpu::Canny etc.
• uses dynamically loaded OpenCL runtime if
available; otherwise falls back to CPU code.
Dispatching is at runtime, no recompilation needed!
• minimal or no changes in user code
– CPU-only processing – no changes should be required
• includes the following key components:
– new data structure UMat
– simple and robust mechanism for async processing
– bonus: very convenient API for implementing custom
OpenCL kernels

13. UMat
• UMat is new type of array that wraps clmem when OpenCL is available
• Replacing Mat with UMat is often the only change needed
#include "opencv2/opencv.hpp"
using namespace cv;
int main(int argc, char** argv)
{
Mat img, gray;
img = imread(argv[1], 1);
imshow("original", img);
cvtColor(img, gray,
COLOR_BGR2GRAY);
GaussianBlur(gray, gray,
Size(7, 7), 1.5);
Canny(gray, gray, 0, 50);
imshow("edges", gray);
waitKey();
return 0;
}
#include "opencv2/opencv.hpp"
using namespace cv;
int main(int argc, char** argv)
{
UMat img, gray;
img = imread(argv[1]).getUMat();
imshow("original", img);
cvtColor(img, gray,
COLOR_BGR2GRAY);
GaussianBlur(gray, gray,
Size(7, 7), 1.5);
Canny(gray, gray, 0, 50);
imshow("edges", gray);
waitKey();
return 0;
}

14. Transparent API: under the hood
bool _ocl_cvtColor(InputArray src, OutputArray dst, int code) {
static ocl::ProgramSource oclsrc(“//cvtcolor.cl source coden …”);
UMat src_ocl = src.getUMat(), dst_ocl = dst.getUMat();
if (code == COLOR_BGR2GRAY) {
// get the kernel; kernel is compiled only once and cached
ocl::Kernel kernel(“bgr2gray”, oclsrc, <compile_flags>);
// pass 2 arrays to the kernel and run it
return kernel.args(src, dst).run(0, 0, false);
} else if(code == COLOR_BGR2YUV) { … }
return false; // OpenCL function does not have to support all modes
}
void _cpu_cvtColor(const Mat& src, Mat& dst, int code) { … }
// transparent API dispatcher function
void cvtColor(InputArray src, OutputArray dst, int code) {
dst.create(src.size(), …);
if (useOpenCL(src, dst) && _ocl_cvtColor(src, dst, code)) return;
// getMat() uses zero-copy if available; and with SVM it’s no op
Mat src_cpu = src.getMat();
Mat dst_cpu = dst.getMat();
_cpu_cvtColor(src_cpu, dst_cpu, code);
}

15. OpenCV+OpenCL execution model
• One queue and one OpenCL device per CPU thread
• Different CPU threads can share a device, but use different queues.
• OpenCL kernels are executed asynchronously
• cv::ocl::finish() puts the barrier in the current CPU thread.
• It’s rarely needed to call cv::ocl::finish() manually.
…
ocl::Queue
ocl::Device
ocl::Queue ocl::Queue
ocl::Device
…
…
ocl::Context
CPU threads

16. IPP + OpenCV
= v. fast OpenCV
• Intel gave us and our users free (as in “beer”) and
royalty-free subset of IPP 8.x (IPPICV).
• IPPICV is linked into OpenCV at compile stage and
replaces the corresponding low-level C code.
• Our buildbot ensures that all the tests pass

17. New Functionality
and other improvements
• Results from 20+ successful projects
from GSoC 2013, 2014:
– Computational photography, Text detection,
Object Tracking, Matlab bindings etc.
• 1000+ Pull Requests @ github (200+
PR’s between alpha & beta)
• 18 new OpenCV modules! (mostly in
opencv_contrib)

18. OpenCV QA
Contribution/patch workflow:
see OpenCV wiki
build.opencv.org: buildbot with 50+ builders
pullrequest.opencv.org: tests each pullrequestgithub.com/itseez/opencv

19. OpenCV test suite
• GoogleTest-based + set of Python
scripts
• Thousands of unit tests
• Accuracy tests
• Performance tests
python ../modules/ts/misc/summary.py core*.xml -f "add:.*C4" -u s
Geometric mean
Name of Test core core core
posix posix posix
x64 x64 x64
6693M 6695 6695
2011-09-08--13-13-41 2011-09-08--13-30-06 2011-09-08--13-30-06
vs
core
posix
x64
6693M
2011-09-08--13-13-41
core_arithm__add::Size_MatType::(127x61, 8UC4) 0.000 s 0.000 s 1.00
core_arithm__add::Size_MatType::(1280x720, 8UC4) 0.004 s 0.004 s 0.99
core_arithm__add::Size_MatType::(1920x1080, 8UC4) 0.009 s 0.009 s 1.02
core_arithm__add::Size_MatType::(640x480, 8UC4) 0.001 s 0.001 s 1.00

20. OpenVX (Khronos HAL)
OpenCV was one of
the key contributors
to the new Khronos
accelerated vision
API: OpenVX
(Hardware Acceleration Library)

21. The HAL + Accelerators
• opencv_hal - IPP-like, fastcv-like low-level API to accelerate
OpenCV for different platforms. To be extended in 3.X…
• As stated, opencv_ocl module (OpenCL acceleration) is now
used in other modules to give transparent acceleration (T-API)
• Universal Umat structure now can be used instead of cv::Mat
and OclMat.
• OpenVX, as it matures, may be used as one of the HW
accelerators
SSE, NEON, IPP, OpenCL, CUDA,
OpenCV4Tegra, …
core, imgproc, objdetect …
OpenCV HAL
OpenCV
face, text, rgbd, …
OpenCV
Contrib
Bindings: Python,
Java
Samples, Apps,
Solutions

22. New from Google Summer of Code 2015
• Deep network optimized execution and
interoperability to existing libraries
• Stereo matching improvements
• Projection mapping
• Improved Camera Calibration
• Better AR fiducial support
• Improvements to text detection and tracking
• Neon optimization

23. Other Initiatives:
CVPR State of the Art Vision Challenge
State of the Art Vision Challenge at CVPR 2015
Our aim is to make available state of the art vision in OpenCV. We thus ran a
vision challenge to meet or exceed the state of the art in various areas. We
will present the results.
The contest details are available at:
http://code.opencv.org/projects/opencv/wiki/VisionChallenge
Prizes:
1. Win: $1000; Submit code: $3000
2. Win: $1000; Submit code: $3000
3. Win: $1000; Submit code: $3000
4. Win: $1000; Submit code: $3000
5. Win: $1000; Submit code: $3000

24. Other Initiatives:
People’s Choice: Best Paper
People’s Choice: Best paper
We will tally the people’s vote for best paper/paper you’d most like to see
implemented. We’ll present the histogram of results which is an indication of
the algorithms people are interested in overall and then list the 5 top
winners.
Prizes will be awarded in two stages:
A modest award for winning and
a larger award for presenting the code w/in 5 months as a pull request to
OpenCV as Detailed here:
http://code.opencv.org/projects/opencv/wiki/How_to_contribute
Prizes:
1. Win: $500; Submit code: $6000
2. Win: $300; Submit code: $4000
3. Win: $100; Submit code: $3000
4. Win: $50; Submit code: $3000
5. Win: $50; Submit code: $3000

25. Functional Language Exploration
• Proliferation of new hardware makes it hard
to support code.
– Let the compiler port to different hardware using a
no-side effects functional language “NUML.”
– Can compile NUML to C++, C, Java and Python.
• NUML is an array/image comprehending
functional language.
– https://github.com/vpisarev/numl/tree/alt_syntax/src

26. Learning OpenCV 3.0
• Out with 3.0/CVPR … or late summer!
For Version 3.0
Other news:
Intel is now supporting OpenCV Optimizations
• https://software.intel.com/en-us/opencv
OpenCV release candidate is now out!
• http://opencv.org/opencv-3-0-rc1.html

27. 27
Photo: Gary Bradski
Questions?
http://youtu.be/LE7aiONMjK4