Presentation PL-4043, Accelerating OpenVL for Heterogeneous Platforms, by Gregor Miller at the AMD Developer Summit (APU13) Nov. 11-13, 2013.

1. OpenVL: A Developer-Friendly Abstraction for
Computer Vision on APU, GPU and CPU
Gregor Miller and Sidney Fels
Human Communication Technologies Laboratory
University of British Columbia
Vancouver, Canada
13th November 2013
AMD Developer Summit

2. Vision Everywhere
OpenVL
Source: AFDS Keynote by Phil Rogers, AMD Inc. (12/06/12)
2
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

3. What’s happening in the real world?
OpenVL
Quote from Hacker News about FastCV and OpenCV (28/10/11):
“I hate how this (and to some extent OpenCV) expects
you to understand computer vision to use the library.
I sympathise with your frustration, but being in vision myself, I have an idea why this is
the case: vision only barely works!
What this means is that most things you would want to do are only barely possible, if
at all, and it is highly application- and input-dependent. To get it to work the way you
want, you have to understand some vision, so that you can tweak things the right way.
I think once vision matures more and things become more ‘off-the-shelf’, you'll find
more libraries aimed at non-experts.”
Source: http://news.ycombinator.com/item?id=3167295
3
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

4. Mainstream Developers are Key to Technology Adoption
OpenVL
WHAT WILL IT TAKE TO HAVE MORE PROJECTS LIKE “CINEMADNG->OPENEXR CONVERSION”?
HOW DO WE ATTRACT THE MAINSTREAM DEVELOPER?
(Performance, Features, etc.)
Developer Return
GPU DEVELOPERS
(NINJAs)
~100K
GPU
Coders
~200+
Apps
Significant
Niche Value
MAINSTREAM DEVELOPERS
~10+M*
CPU
Coders
~4M+
Apps
Good User
Experiences
Developer Investment
(Effort, Time, New skills)
*IDC
Source: AFDS Keynote by Tom Malloy, Senior Vice President and Chief Software Architect, Adobe Systems Inc. (12/06/12)
4
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU
13 | The Promise of Parallel: Today’s State of Heterogeneous Computing | June 12, 2012

5. Higher-Level Abstractions are Needed
OpenVL
SOFTWARE CONSIDERATIONS
PLATFORMS (AND STANDARDS)
Claim 3:
 OpenCL is a valuable standard for ninjas, but …
 Higher level abstractions that provide performance portability
needed for mainstream developers
 May be evolutionary
 See Khronos’ OpenCL Overview
http://www.khronos.org/assets/uploads/developers/library/overview/opencloverview.pdf
 … or it may be something completely different
Source: AFDS Keynote by Tom Malloy, Senior Vice President and Chief Software Architect, Adobe Systems Inc. (12/06/12)
5
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

6. Current Computer Vision Frameworks
OpenVL
• OpenCV: Probably the most widely-used
vision framework
• Not aimed at mainstream developers:
use face detection as an example
• We do not consider it developer-friendly
for three reasons:
– The specific algorithm is used as the function
name - cvHaarDetectObjects
– The arguments are specific to the algorithm
– No indication of the conditions under which it
operates: e.g. how much of a rotation is
allowed, what’s the smallest/largest face, etc.
• Most common level of abstraction for
vision, similar found in Matlab IPT/
CVT, ...
6
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

7. HCI for Vision, focussing on developers; Why?
OpenVL
• Example from FastCV: Qualcomm’s vision library optimised for mobile and embedded platforms
– fcvCornerFast9InMaskScoreu8(const uint8_t *__restrict src, unsigned int srcWidth, unsigned int srcHeight,
unsigned int srcStride, int barrier, unsigned int border, uint32_t *__restrict xy, uint32_t *__restrict scores, unsigned int
nCornersMax, uint32_t *__restrict nCorners, const uint8_t *__restrict mask, unsigned int maskWidth, unsigned int
maskHeight)
– fcvNCCPatchOnCircle8x8u8_v2(const uint8_t *__restrict patch, const uint8_t *__restrict src, unsigned short
srcWidth, unsigned short srcHeight, unsigned short search_center_x, unsigned short search_center_y, unsigned short
search_radius, int filterLowVariance, uint16_t *best_x, uint16_t *best_y, uint32_t *bestNCC, int findSubPixel, float *subX,
float *subY)
– fcvTrackBMOpticalFlow16x16u8(const uint8_t *__restrict src1, const uint8_t *__restrict src2, uint32_t srcWidth,
uint32_t srcHeight, uint32_t srcStride, uint32_t roiLeft, uint32_t roiTop, uint32_t roiRight, uint32_t roiBottom, uint32_t
shiftSize, uint32_t searchWidth, uint32_t searchHeight, uint32_t searchStep, uint32_t usePrevious, uint32_t *numMv,
uint32_t *locX, uint32_t *locY, uint32_t *mvX, uint32_t *mvY)
– fcvDescriptor17x17u8To36s8(const uint8_t *__restrict patch, int8_t *__restrict descriptorChar, int32_t *__restrict
descriptorNormSq)
– fcvMserExtu8(void *mserHandle, const uint8_t *__restrict srcPtr, unsigned int srcWidth, unsigned int srcHeight,
unsigned int srcStride, unsigned int maxContours, unsigned int *__restrict numContours, unsigned int *__restrict
numPointsInContour, unsigned int *__restrict pointsArray, unsigned int pointsArraySize, unsigned int *__restrict
contourVariation, int *__restrict contourPolarity, unsigned int *__restrict contourNodeId, unsigned int *__restrict
contourNodeCounter)
7
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

8. A Case for the Problem Description Layer
OpenVL
Applications
Consumer
Computer Vision Problem Description
Mainstream Developer
Computer Vision Solutions
Vision Ninja
Computer Vision Frameworks
Vision Researcher
Hardware Acceleration API
Physical Hardware
Each of these layers plays a role for
widespread adoption of computer vision.
8
Optimisation Specialists
Users
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

9. Higher-level abstraction for computer vision
OpenVL
?
Power
Power
Power
Precision
Precision
Precision
Performance
Performance
Performance
9
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

10. Analysis using OpenVL: ‘What’ vs ‘How’
OpenVL
• Analysis using OpenVL
– High-level abstraction targeted towards mainstream
developers
• API based on problem description
– The description is interpreted to select algorithm(s) to
find the required solution
• Expressive interface: ‘what’ vs. ‘how’
– Describe what the task is, not how to solve it
• Example: face detection
– Description: pose, 2D rotation, expression, age,
occlusion, size quantity, species
10
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

11. OpenVL
OpenVL
• Input / Output
• Operations
– Images: input from developer (via VU)
– Segments: basic unit; distinct regions within the image
e.g. red circle
– Matches/Detections: representation of matches and
detections e.g. a face
– Transforms: geometric representation e.g. vector of
motion between segments
• Descriptors
Segment(constraints, images, segments)
Detect(constraints, template, segments, detections)
Match(constraints, variances, segments, matches)
Solve(constraints, matches/detections, transforms)
Select(constraints, conditions, segments, segments)
• Constraints
– Template: Description of well-known (and usually
arbitrary) ‘object’ to find in images e.g. face, eyes,
hand, body
– Variances: Differences between segments such as
intensity
– Conditions: Property-based requirements such as
colour == red
11
–
–
–
–
–
– Operation-specific hints e.g.
– Quantity of segments/detections/matches
– Shape/regularity of segments
– Type of match e.g. unique
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

12. Using Segments to Abstract Pixels
OpenVL
Definition:
A segment is a distinct continuously bounded
region within a bounded 2D plane
12
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

13. Segmentation using a Problem Description
13
OpenVL
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

14. Image Registration Example
OpenVL
Segment - Match - Solve(Image)
Input
14
Visualized Output
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

15. Flexibility of Operation Sequences
OpenVL
• Segment - Match(variances) - Solve(Image)
defines image registration
– Match variances:
 Intensity: HDR registration
 Blur: Multi-focal registration
• Segment - Match(variances)
defines correspondence, as shown
15
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

16. OpenVL Matching Demo
16
OpenVL
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

17. Optical Flow (Video Motion)
OpenVL
Segment - Match - Solve(Segment)
• Change Solve(Image) to Solve(Segment) defines Optical Flow
– Find the transform of each pair of matched segments
17
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

18. Combined Example: Stitching and Detection
18
OpenVL
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

19. Example accelerated back-end to OpenVL using HSA
19
OpenVL
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

20. Summary and Future Work
OpenVL
• A problem description layer fills the gap
between applications and algorithms
– Critical for widespread adoption of computer vision
– Allows replacement of implementation without impacting developer
– High-level abstraction aimed at mainstream developers - not ninjas!
• Expressive API (based on ‘What’, not ‘How’)
• An abstraction for all platforms
• Prototype reference implementations for
CPU, GPU and HSA platforms
20
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

21. OpenVL has been released! (Image segmentation)
OpenVL
gregor@ece.ubc.ca | http://www.openvl.org
21
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

22. Acknowledgements
22
OpenVL
OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU