- Dynamic neural networks (DNNs) can adapt to varying resource availability on edge devices through techniques like incremental training and group convolution pruning. This allows meeting requirements for timing, power/energy, and accuracy.
- Experiments on two embedded platforms showed that dynamic DNNs combined with DVFS and task mapping can reduce energy consumption while maintaining classification accuracy compared to static DNNs.
- Runtime power management is needed to coordinate heterogeneous processors, respond to environmental factors, balance power consumption and battery life, and meet requirements for concurrently executing tasks and applications under varying conditions on edge devices.
2. • Innovate UK drives productivity and economic growth by supporting businesses
to develop and realise the potential of new ideas, including those from the UK’s
world-class research base.
• Knowledge Transfer Network (KTN) is Innovate UK’s Network partner
• We help business to grow the economy and improve people’s lives by capturing
maximum value from innovative ideas, scientific research and creativity.
• KTN combines in-depth expertise in all sectors with the ability to cross
boundaries.
• Nigel Rix, Head of Enabling Technology: nigel.rix@ktn-uk.org
3. eFutures aims to strengthen and support a network of people
working in electronic systems across the UK
• Building new links and increasing involvement with industry
• Mapping the national electronics research, to ensure the work across the UK is known and noted
• Encouraging and funding innovative multi-disciplinary/multi-university proposals
• Working to improve, encourage and support equality, diversity and inclusion across our sector
• Communicating with our network via a monthly magazine & social media
• Running regular events that support our network & strategy
• Launching a Big Ideas Challenge
Twitter @efuturesuk
Sign up to our mailing list: efutures@qub.ac.uk
4. Next webinar: Friday 3rd July
Vision & Imaging Systems
AI: Vision Systems
Speakers include Xilinx; University of Edinburgh;
Sensing Feeling and AAEON Technology
7. 3
THE AIOT IS APPLICABLE ACROSS MARKETS
ENABLING HIGH PERFORMANCE, ACROSS VERTICALS, ECONOMICALLY
Smart speaker
Audio visual
Appliances
Lighting
Security
Fitness
Care
Diagnostics &
monitoring
MHealth
Traffic &
parking
Environmental
Utilities
Public safety &
security
TAM
Operations
Tracking
Safety
Maintenance
Energy
management
Asset tracking &
predictive
maintenance
In car people
tracking
Autonomous L1
driving & safety
500M
UNITS
500M
UNITS
650M
UNITS
450M
UNITS
90M
UNITS
8. 44
CHALLENGES OF THE AIoT REVOLUTION
45% DATA SECURITY AND AUTONOMY
38% BANDWIDTH
32% LATENCY
24% SCALABILITY
24% CLOUD INFRASTRUCTURE LIMITATIONS
BASED ON PRIMARY RESEARCH WITH ELECTRONICS ENGINEERS
9. WHAT’S NEEDED?
AIoT devices demand a processor with
high-performance compute, efficient energy
usage and a low eBOM.
10. A NEW KIND OF PROCESSOR
Fast, flexible and economical, xcore.ai puts
intelligence at the core of smart products,
combining AI, DSP, control and IO compute
in a one dollar device.
11. 77
FAST, FLEXIBLE AND ECONOMICAL
32 x 16 x
15 x 21 x
ARM Cortex M7 @ 600MHzxcore.ai
AI performance faster I/O processing
DSP performance more 16-bit MACs
Benchmarked 18 Nov 2019. Preliminary information subject to change without notice
DELIVERING STANDOUT PERFORMANCE
12. 88
FLEXIBLE & SCALABLE ARCHITECTURE
DRIVING FAST TIME TO MARKET, ENABLING COST EFFECTIVE SOLUTIONS
xcore device families
xcore Tools
xcore Libraries
3rd Party
Libraries
xcore LibrariesFreeRTOS
Custom platform solutions
xcore Libraries
USB
Audio
Voice
Human
Presence
Smart
Home
Connect
Health
Smart
Mobility
IndustryIoT
SmartCities
Solutions
13. 99
STATE OF THE ART ARCHITECTURE
HIGH PERFORMANCE AND ENERGY EFFICIENCY CONVERGE IN A LOW eBOM CLASS LEADER
c
hardware ports
IO pins
switch
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xtime scheduler
hardware ports
xtime scheduler IO pins
SRAMSRAM
ALU (FP + int)
vector unit
ALU (FP + int)
vector unit
High-Speed USB PHY MIPI D-PHY
external
LPDDR
interface
JTAG
core PLL
app
PLL
OTP OTP
oscillator reset16 real-time logical cores,
with support for scalar /
float / vector instructions
Vector processing unit,
supports 8-bit and binarised
neural network inferences
Extended memory support
for large applications
Flexible IO ports with
nano-second latency;
create interfaces in software
High performance instruction
set for DSP, ML and
cryptographic functions
Integrated MIPI interface
for imaging support
Example software tasks
14. 1010
MAPPING REAL-TIME TASKS, APP TASKS, AND INFERENCING TASKS
Neuralnetmodel
c
Hardware Ports
IO pins
Switch
xTIME scheduler
Hardware Ports
xTIME scheduler IO pins
High speed USB PHY MIDI D-PHY
External
LPDDR
interface
JTAG
Core PLL App PLL
Oscillator Reset
FreeRTOS and app
tasks dynamically
share fixed number of
thread contexts
Inferencing and real time tasks
allocated fixed threads at compile time
I2SLEDdrivers
PDMPDM
c
Far-fieldmicrophone
processing
Applicationtask
Applicationtask
…
Applicationtask
Keyworddetection
FreeRTOS
I2C
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
xcore logical core
Internal
SRAM
Internal
SRAM
ALU (FP + int)
Vector unit
ALU (FP + int)
Vector unit
OTP OTP
PDM
Far-field microphone
processing
Keyword detection
Free
RTOS
I2S, I2C, LED drivers
Apptask
PDM
Apptask
Apptask
Apptask
Neural net model
15. 1111
FOUR CLASSES OF COMPUTE, ONE DEVELOPMENT PLATFORM
“USING XMOS WE WERE ABLE TO REPLACE THREE SEPARATE DEVELOPMENT SYSTEMS”
Richard Hollinshead, Meridian
Embedded
code
DSP
code
NN
Model
Cortex-M DSP core NPU Hardware
gates
IO &
accelerators
Cortex SoC development
Embedded
code
DSP
code
NN
Model
xcore
IO &
accelerators
xcore development
16. 1212
PROGRAMMABLE USING INDUSTRY STANDARD TOOLS
ENABLING RAPID DEPLOYMENT AND SHORTENING TIME TO MARKET
Example software tasks
• Industry-standard TensorFlow Lite
workflow
• Automatic model translation
• Community support
Applicationtask
Applicationtask
…
Applicationtask
FreeRTOS
• Familiar, real-time, industry-standard
development environment
• Community support
• Wide variety of third party applications
FFT
FFT
QSPI
Filter
Filter
• High performance, predictable DSP
• Accessed using industry standard tools
• Highly optimised library kernels access
xcore.ai processing
CONTROL AI DSP
17. 1313
AI USER WORKFLOW
Trained floating
point network
Lite convertor
(python API)
Run TFL to
xcore.ai
convertor
Key
TensorFlow component
XMOS component
User component
Key
TensorFlow component
XMOS component
User component
ONNX componentAlternative framework flow
trained network
my_model.tflite to TensorFlow
convertor
xcore.ai
micro Runtime
my_model.tflite
lib_xs3_ai
18. 1414
PROGRAMMING – PULLING IT ALL TOGETHER
xmos
compiler
3rd party
Libraries
Executable
Control
source code
Neural net
model
Dataflow
source code
XMOS
Libraries
TensorFlowLite
to xcore.ai
convertor
Applicationtask
Applicationtask
…
Applicationtask
FreeRTOS
FFT
FFT
QSPI
Filter
Filter
19. 1515
IN SUMMARY
• The AIoT industry has reached a tipping point that will
radically transform our way of life
• Success depends on being able to drive one of the most
impressive feats of electronics engineering
• xcore.ai is that feat
21. 1
Adapting AI to Available Resource
in Mobile/Embedded Devices
Geoff Merrett
Implementing AI: Running AI at the Edge
12 June 2020 | KTN & eFutures Online Webinar
spatialml.net
22. 2
WHY AI AT THE EDGE?
Data Privacy
• Increased privacy if data never leaves the edge
Sending data to a central location consumes energy. Once there, the
temptation is great to keep crunching them 1
Network Latency/Bandwidth/Connectivity
• Cloud AI requires good networking
Self-driving cars need very fast-reacting connections and cannot
risk being disconnected; computing needs to happen in the car itself 1
Traffic lights in Las Vegas generate 60 terabytes a day (10% of the
amount Facebook collects in a day) 1
• (the edge must fulfil requirements instead though!)
1 https://www.economist.com/special-report/2020/02/20/should-data-be-crunched-at-the-centre-or-at-the-edge
“
“
“ ”
23. 3
WHY AI AT THE EDGE?
Power Consumption of AI
• Cloud AI consumes considerable natural resource.
The carbon footprint of training a single AI is up to 284 tonnes of
CO2 equivalent – 5x the lifetime emissions of an average car 2
An estimate puts the energy used to train the model at over 3x the
yearly consumption of the average American 3
From the earliest days, the amount of computing power required
by the technology doubled every two years. But from 2012
onwards, the computing power required for today’s most-vaunted
machine-learning systems has been doubling every 3.4 months 3
• An indirect benefit of moving computation to the
edge, is that it has to be more efficient
2 https://www.newscientist.com/article/2205779-creating-an-ai-can-be-five-times-worse-for-the-planet-than-a-car/
3 https://www.theguardian.com/commentisfree/2019/nov/16/can-planet-afford-exorbitant-power-demands-of-machine-learning
“
“
”
“
24. 4
PERFORMANCE METRICS
Inference at the Edge (/End)
• Connectivity, latency; privacy…
• …but constrained platforms
Inference
Test Data
result Inference
Trained
model
Servers Servers
Training
Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Optimising Resource Management for Embedded Machine Learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20).
25. 5
EMBEDDED AI ACCELERATION
• General/specialist compute units for AI rapidly increasing
• Some mobile/embedded AI systems are
reasonably static…
• …however, others aren’t
– General purpose systems
– Multi-tenant systems
– ‘Adaptive’ AI/event-driven operation
– etc
26. 6
• Complexity of hardware-software interaction has grown
• Managing resources is no longer
trivial, yet is increasingly needed
1 CPU
Core
SYSTEM RESOURCE MANAGEMENT
n CPU1
Cores
n GPU
Cores
n FPGA
Cores
n CPU2
Cores
n T/NPU
Cores
n Device
Variants
n
Workloads
Samsung Exynos 5422 Xilinx Zynq Ultrascale+
HiSiliconKirin9905G
NVIDIA
Xavier NX
27. 7
DESIGN-TIME CHALLENGES
PlatformDiversity
How can we develop DNN models that can:
1. operate across a wide range of different
heterogeneous platforms, and
2. meet diverse application requirements?
• Existing design-time approaches such
as static model pruning compress the
model to approximately the ‘right size’.
Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Optimising Resource Management for Embedded Machine Learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20).
28. 8
RUN-TIME CHALLENGES
WorkloadDiversity
How can we perform inference while:
1. meeting timing requirements?
2. meeting power/energy requirements?
3. meeting accuracy requirements?
How can we do this:
• while executing another DNN model at
the same time?
• while executing other foreground/
background tasks at the same time?
We need dynamic DNNs…
Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Optimising Resource Management for Embedded Machine Learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20).
CPU
Type1
CPU
Type2
GPU
CPU
Type1
CPU
Type1
CPU
Type1
CPU
Type2
CPU
Type2
CPU
Type2
NPU
DNN 1 DNN 2 VR/AR
30. 10
DYNAMIC DNNs
ExperimentalSetup
Model: Modified AlexNet (~320kB)
Dataset: CIFAR10
– 32*32*3 images in 10 classes
– 50,000 training and 10,000 testing images
Framework: Caffe
Hardware:
• Odroid XU3
– CPU: 4x Arm A15 ( f = 0.2–2 GHz ) + 4x Arm A7 ( f = 0.2–1.4 GHz )
– GPU: Mali-T628 ( not used in these experiments )
• Nvidia Jetson Nano
– CPU: 4x Arm A57 ( f = 0.9, 1.4 GHz )
– GPU: 128x CUDA core Maxwell ( f = 0.6, 0.9 GHz )
Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Incremental Training and Group Convolution Pruning for
Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
31. 11
DYNAMIC DNNs
Results:DVFSandTaskMapping(OdroidXU3)
Energy Consumption Top-1 Accuracy
Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Incremental Training and Group Convolution Pruning for
Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
33. 13
DYNAMIC DNNs
Results:DVFSandTaskMapping(JetsonNano)
Energy Consumption Top-1 AccuracyEnergy Consumption
Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Incremental Training and Group Convolution Pruning for
Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
34. 14
RUNTIME POWER MANAGEMENT
www.prime-project.org
Runtime Management (RTM)
• System software to react and predict
• Controls/’knobs’
• ‘Monitors’/sensors
RTM to coordinate/balance…
• Mapping to heterogeneous PEs
• Response to environmental factors
• Power consumption/battery life
• (concurrently) Executing tasks
• Application(s) requirements
• User requirements/QoE
Bragg, Graeme McLachlan, Leech, Charles R., Balsamo, Domenico, Davis, James J., Weber Wachter, Eduardo, Merrett, Geoff, Constantinides, George A. and Al-Hashimi, Bashir (2018) An application- and platform-agnostic
control and monitoring framework for multicore systems. 3rd International Conference on Pervasive and Embedded Computing, Portugal. 29 - 30 Jul 2018.
35. 15
CONCLUSIONS
• AI is moving to the edge…
If machine learning is going to be deployed at a global
scale, most of the computation will have to be done in
users’ hands, ie in their smartphones 3
• …but available resources on edge platforms
are typically both constrained and time-varying
• We need improved approaches to manage
resources in systems while providing
acceptable performance
Companies will learn to make trade-offs between
accuracy and computational efficiency, though that will
have unintended, and antisocial, consequences too 3
3 https://www.theguardian.com/commentisfree/2019/nov/16/can-planet-afford-exorbitant-power-demands-of-machine-learning
Photo by Patrick Schneider on Unsplash
“
“
”
36. 16
ACKNOWLEDGEMENTS
Lei Xun (PhD student)
w https://www.ecs.soton.ac.uk/people/lx2u16
@XunLei_CHN
spatialml.net
International Centre for Spatial Computational Learning (EPSRC)
w https://spatialml.net/
@spatialmlnet
Power and Reliability in Many-Core Embedded Systems (EPSRC)
w https://www.prime-project.org
@prime_programme
37. 17
YOUR QUESTIONS
Professor Geoff Merrett
Head of Centre for IoT and Pervasive Systems
e: gvm@ecs.soton.ac.uk
w: www.geoffmerrett.co.uk
@g_merrett
39. Andrew Swirski,
Founder and Executive Director,
a.swirski@beetlebox.org
Real Time Low Latency Computer Vision
Unit 1. 10,
Chester House,
Kennington Park,
1-3 Brixton Road,
London,
SW9 6DE
40. • Meeting the increasing demands of Computer Vision
• What are FPGAs and why do they perform better than
CPUs
• Why in previous years FPGAs have been restricted to
hardware engineers
• The new generation of Xilinx software development tools:
• Vitis Unified Software Development platform
• Vitis Vision Library
• Vitis AI
• ClickCV: Beetlebox’s computer vision library
• Electronic Image Stabilisation with Sundance
• Getting involved with our Early Access
Introduction
41. Our vision of the future puts enormous
demand on embedded devices to understand
the world around them
Autonomous delivery drone
Our vision of the future
42. What do we mean by high performance?
1. Throughput:
• Achievable Resolution
• Achievable Frames Per Second (FPS)
2. Latency:
• Consistent end-to-end latency
3. Power Consumption:
• Battery Life
• Heat Production
What is high performance?
45. What happens on a FPGA
int main() {
int a[5], b[5], output[5];
for(int i=0;i<5;i++){
output[i]=a[i]+b[i];
}
} + +
a[0] b[0] a[1] b[1] a[2] b[2] a[3] b[3] a[4] b[4]
+ + +
output[0] output[1] output[2] output[3] output[4]
• Control over the design
• Control over the latency
• Control over the clock (power consumption)
46. With great control comes great responsibility
• Need hardware specialists:
• Hardware Description Languages: Verilog or VHDL
• Design and Verification is a time-consuming
process
• Newer tools (High Level Synthesis) do allow for
the uses of C/C++, but:
• Limited subset of C
• No standard libraries
• Need to still understand the hardware
Restricted to Hardware Engineers
47. The new generation of tools are now focused on System-on-
Chip (SoC)
• Xilinx’s Zynq series and the Zynq Ultrascale+ series contain
ARM cores
• Benefits of a CPU host code + FPGA accelerated code
The new generation of tools
CPU FPGA
48. Providing a familiar software development
environment
• Develop, compile and debug code using C, C++ or OpenCL
either through the provided IDE or using a makefile build
flow
• Embedded platforms run a version of Linux known as
PetaLinux
• Use Standard Libraries and Tools such as GStreamer,
OpenCV and FFMPEG
• Gain access to open-source Xilinx Vitis Accelerated
Libraries
Xilinx Vitis Unified Software Platform
49. Vitis Vision library provides a library of low-
level computer vision kernels
• Provides a strong video pipeline framework
• Great for forming basic pipelines: e.g. colour thresholding
Xilinx Vitis Vision Library
CPU FPGA CPU
Collect
Data
Computer
Vision
Pipeline
Output
Data
30 FPSVideo
In
50. What we can do with Vitis Vision
Xilinx Vitis Vision Library
Edge Detection Colour Detection
51. Allows the deployment of models from deep
learning frameworks such as Tensorflow and
Caffe on to a specialized processor
• Optimizes your model to run on FPGAs:
• Prune
• Quantisation
Xilinx Vitis AI Development Environment
CPU FPGA CPU
Collect
Data Pre-process
Output
Data
30 FPSVitis AI
DPU
52. What we can do with Vitis AI
Xilinx Vitis AI
Object prediction Bounding Boxes Semantic Segmentation
53. High performance, high level FPGA kernels without the need
for hardware expertise
• Provides high-level, out-of-the-box functionality using
software only to get computer vision developed faster
• Also provides the low-level kernels needed to build
custom systems or for building test systems
• Integrates with industry standard software such as
OpenCV and GStreamer
Beetlebox ClickCV: Accelerated Computer Vision
54. Partnered with Sundance Microprocessor technologies
• Robotics and vision hardware experts
• Robots and drones often suffer from
shaky camera, especially when there
is no room or power for a gimbal
• Cameras with in-built stabilization,
such as GoPros are often impractical
• Stabilise video purely through the
video itself
• No sensor data
Electronic Image Stabilisation (EIS)
VCS-1
56. Currently running the ClickCV Early Access Programme
• High Performance with no need for computer vision hardware
specialists:
• Out-of-the-box functionality
• We can develop bespoke hardware for a specific solution
• Provide custom systems to get testing on FPGAs up and
running fast
• Let your software engineers use industry-standard tools, such
as OpenCV, GStreamer and FFMPEG
• Areas where we are looking next:
• Super Resolution
• Template Matching
• SLAM
• Deep Learning
ClickCV Early Access
57. Our Team
Andrew Swirski
MD
• MEng Electronic
Engineering at Imperial
College London
• Design Engineer at Intel
Matthew Simpson
Head of Systems
Dr Christos-Savvas Bouganis
Consultant
• Reader and Director of the
Integrated Digital Systems Lab
at Imperial College:
• Computer Vision, Image
Processing, Machine Learning
and SLAM on FPGAs
• MSc Electronic
Engineering at Imperial
College London
• CMOS Engineer at NXP
Ashley Unitt
Advisor
• CTO and CSO of
NewVoiceMedia, a SaaS
contact centre tech provider
• Acquired by Vonage
Peter Collins
Advisor
• CEO of Permasense Ltd-
sensor systems; acquired by
Emerson
• Chairman Inflowmatix Ltd
• Chariman Guided Ultrasonics
Ltd
Dr Marlon Wijeyasinghe
Head of HLS
• PhD in Electronic
Engineering at Imperial
College London
58. Contribute back and grow the community
• Technical tutorials: Vitis Vision Library tutorial:
• https://beetlebox.org/getting-started-with-computer-vision-for-
vitis-embedded-systems/
• Explainer articles: What is Computer Vision and why are
Neural Networks so important:
• https://beetlebox.org/what-is-computer-vision-and-why-are-
neural-networks-important/
• Soon launching a tutorial on Vitis AI, where we explore
sign language recognition
• Open Source
• Soon launching our github.io page:
• beetlebox.github.io
Free resources
59. • Previously FPGAs have always been a powerful but
obscure chip that only a few hardware specialists can use
• With Xilinx’s new focus on SoCs and Vitis software
development environment, embedded FPGAs have never
been more accessible
• ClickCV Computer Vision library provides out-of-the-box
high performance functionality allowing the fast
development of systems without the need for hardware
expertise
• Currently running an Early Access Programme. Come talk
to us!
• Check out our website
Conclusion
60. Andrew Swirski,
Founder and Executive Director,
a.swirski@beetlebox.org
Any Questions?
Unit 1. 10,
Chester House,
Kennington Park,
1-3 Brixton Road,
London,
SW9 6DE
61. EDGE AI CASE-STUDY:
RADAR GESTURES
W W W . I M A G I M O B . C O M
ALEXANDER SAMUELSSON, CTO/CO-FOUNDER
JUNE 2020
62. • Specialized in Edge AI
• Experience from 20+ Edge AI customer projects
• We offer
• Imagimob AI – Software-tools-as-a-Service
• Edge AI expertise
• Based in Stockholm, Sweden
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep learning
GestureRecognitiononTheEdge|2020
Introduction Imagimob
63. • What is Edge AI?
• Imagimob AI
• Case study – radar gestures in earphones
• Edge AI opportunities for the future
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep learning
GestureRecognitiononTheEdge|2020
What I will talk about
64. So, WHAT IS
EDGE AI?
W W W . I M A G I M O B . C O M
COMPANYPRESENTATION||JUNE2019
Placing AI at the very Edge of the network
• Where data is collected
• Autonomous
• Real time
• Low power
• Strong privacy
• We are democratizing Edge AI
Cloud aggregated raw data
• Needs connection
• High latency
• High communication cost
• High power consumption
• Weak Privacy
EDGE AI Cloud aggregated
curated data
• Autonomous
• Real time
• Low communication cost
• Low power consumption
• Strong privacy
CLOUD AI
RAW DATA
65. EDGE AI APPLICATION DEVELOPMENT
IMAGIMOB AI SUPPORTS THE FULL CYCLE FROM DATA COLLECTION TO FINISHED
EDGE AI APPLICATION
W W W . I M A G I M O B . C O M
COMPANYPRESENTATION||2019
DATA COLLECTION &
LABELLING
DATA MANAGEMENT
MODEL BUILDING
PREPARED FOR EDGE
MODEL EVALUATION EDGE OPTIMIZATION AND
VERIFICATION
APPLICATION PACKAGING
Imagimob Capture
(Android + Sensor)
MCUTraining Service
(Cloud)
Imagimob Studio
(PC)
Imagimob Studio
(PC)
Imagimob Studio
(PC)
Train
Validation
Test
66. Lot’s of applications depending on sensors
• Predictive maintenance
• Anomaly detection
• Human activity recognition
• Wake-word detection (Hi Alexa)
• …
Radar + Edge AI
• Material or surface recognition
• Detect deviations/defects in manufacturing
• Object detection
• Detect hand gestures in headphones
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
What can you do with Edge AI?
67. • Working proof of concept shown at CES with Acconeer
• Application running in real time on the actual radar module
• ARM-Cortex M4 processor with only 256KB shared with BLE
stack, firmware, other applications.
• Impossible without Edge AI, the data generated by the sensor
and the almost infinite variations in how a gesture can be
performed demands an AI solution.
• Sending the data off device would drain the battery and be
impossible over bluetooth
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
Radar gestures in headphones
68.
69. W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
Challenge #1 – Data collection
USB
WIFI
Acconeer radar
70. W W W . I M A G I M O B . C O M
GestureRecognitiononTheEdge|2020
71. Radar output: 30 KB data per second
Model predictions: 14.3 Hz
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
Challenge #2 - Preprocessing
Learned
Preprocessing
Manualpreprocessing
Avg FFT
Hanning
Window
Abs Sum
Sliding
Window
72. W W W . I M A G I M O B . C O M
GestureRecognitiononTheEdge|2020
73. • Testing and verifying Edge AI models is a REAL pain
• To test on device you would have to go all the way to C code
• Moreover you would have to reflash the firmware of your headphones
• To get a really good test we would have to do this on several headphones in different locations
multiple times each week
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
Challenge #3 – Testing and verifying
74. W W W . I M A G I M O B . C O M
GestureRecognitiononTheEdge|2020
75. W W W . I M A G I M O B . C O M
GestureRecognitiononTheEdge|2020
76. • End-to-end
• We have solutions to the major problems
• Data collection
• Model evaluation/testing
• Computation designed for the Edge all the way
W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
How are we different?
77. W W W . I M A G I M O B . C O M
Imagimob AI
Edge AI | Software-tools-as-a-Service | Deep Learning
GestureRecognitiononTheEdge|2020
The future (opportunities)
78. THANK YOU
W W W . I M A G I M O B . C O M
FOLLOW US
@imagimob
@samuelsson_al
SIGN UP FOR OUR IMAGIMOB AI EARLY ACCESS PROGRAM
GET IMAGIMOB AI FOR A FULL MONTH + EDUCATION
LIMITED SEATS
JOIN THE WORKSHOP LATER TODAY