TULIPP's tutorial presentation about the Image Processing Library HiFflipVX

1. This project has received funding from
the European Union’s Horizon 20 20
research and innovation programme
under grant agreement No 688403
www.tulipp.eu
TULIPP
Title :
Place :
Date :
Towards Ubiquitous Low-power Image Processing Platform –
project overview
HiPEAC, Valencia, Spain
22nd of January 2019
Topic : Image Processing Library (HiFlipVX)
Ariel Podlubne & Diana Göhringer (ADS - TUD)

2. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
• Includes 21 image processing functions
• It is parametrizable and highly optimized for High-Level Synthesis
• Functions are based on the OpenVX specification with some
extensions
• Supports auto-vectorization (1, 2, 4, 8) & more data types (8, 16, 32-bit
signed/unsigned)
• Filters support different kernel sizes (3-11)
• Border handling (e.g. constant and replication)
• No extra libraries (e.g. OpenCV) required
• Independent of OS
• Three types of functions implemented:
• Pixel-wise operations
• Filters
• Analysis & Image conversion
More information in "HiFlipVX: an Open Source High-Level Synthesis FPGA Library for Image Processing“ @ ARC2019 (accepted paper)

3. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Pixel-wise Operations:
• Binary and Arithmetic operations
• Same data types (Signed or unsigned) for input and output
• Bit-width of 8, 16 or 32-bit (1, 2, 4 or 8 pixel computation in parallel in
vector) of input/output
• Template based implementation
of functions
• Optimized for resource
utilization on FPGA
Pixel-wise Operations
Bitwise AND Bitwise OR Bitwise XOR
Bitwise NOT
Arithmetic
Addition
Arithmetic
Subtraction
Absolute
Difference
Magnitude
Pixel-wise
Multiplication

4. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Filters:
• Kernel sizes (Ks) is selected by the user
• Ks of 3, 5, 7, 9 and 11 are supported for all filters
(Ks of 3 for Scharr and Sobel filter)
• Three border behaviors are defined
• Undefined
• Constant zero
• Replicated
• Separable Kernel implementation for
Gaussian and Box filter
• To reduce resource utilization
Image Filter Functions
Median Filter Gaussian Filter
Sobel Filter (3x3) Scharr Filter (3x3)
Custom
Convolution
Box Filter

5. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Image Conversion and Analysis Functions:
• Implementation of special functions used for
• Image format conversion
• Image analysis
Image Conversion and Analysis Functions
Convert Bit Depth Color Convert
Histogram Table Lookup
Integral Image

6. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Developer‘s function
input
Example Application: Gradient Magnitude of a Gaussian smoothed image
Main Function
output

7. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Developer‘s function
ImgGaussian
input input outputlsmooth
Smooth the image
Configurable Parameters
Example Application: Gradient Magnitude of a Gaussian smoothed image
Main Function

8. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Developer‘s function
ImgGaussian imgScharr3x3
input input output outputlsmooth
lx
ly
Smooth the image Compute the derivatives in X and Y
Configurable Parameters Configurable Parameters
Example Application: Gradient Magnitude of a Gaussian smoothed image
Main Function

9. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Developer‘s function
ImgGaussian imgScharr3x3 imgMagnitude
input input output outputlsmooth
lx
ly
Smooth the image Compute the derivatives in X and Y Compute the Gradient Magnitude
Configurable Parameters Configurable Parameters Configurable Parameters
Example Application: Gradient Magnitude of a Gaussian smoothed image
Main Function

10. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Example Application: Code (available when cloning the HiFlipVX)
FIFOs for internal connection
HW built around the function

11. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Standalone Example: Filter (demo_test) without acceleration
Go to demo on Virtual Machine

12. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Standalone Example: Filter (demo_test) without acceleration
• Standalone Profiling

13. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Standalone Example: Filter (demo_test) without acceleration
• Standalone Profiling Results

14. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Exercise 1:
Compute median filter using HiFlipVX following the same structure shown
in previous filters.
RGBXtoGrayscale Median Filter GrayscaletoRGBX
input output
Configurable Parameters Configurable Parameters Configurable Parameters

15. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)

16. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Exercise 2:
Move hwMedian to hardware and profile it using Sthem.

17. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)

18. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Exercise 2: Median Filter with HW acceleration
• Standalone Profiling
No HW acceleration

19. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Exercise 2: Median Filter with HW acceleration
• Standalone Profiling

20. TULIPP‘s Survey
http://bit.ly/Tulipp_Survey
Please take the survey titled "TULIPP Autonomous Robotics and Computer Vision Survey".
Your feedback is important!

21. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Tomorrow’s Agenda:
• 10:00: HIPPEROS Introduction and Lab1: using the OS
• 11:00: Break
• 11:30: HIPPEROS – Lab 2, accelerating with OpenMP
• 13:00: Launch
• 14:00: HIPPEROS – Lab 3, accelerating with SDSoC
• 15:30: Break
• 16:00: Analysis tool, advanced features.
• 17:00: Closing notes
• 17:30: End

22. Backup Slides

23. Image Processing Library
Open Source High-Level Synthesis FPGALibrary for Image Processing
(HiFlipVX)
Latency evaluation of HiFlipVX with xfOpenCV
Reference: http://tulipp.eu/wp-content/uploads/2019/01/d44-final.pdf
• More information in "HiFlipVX: an Open Source
High-Level Synthesis FPGA Library for Image
Processing“ @ ARC2019 (accepted paper)