In this deck, Pradeep Dubey from Intel discusses AI & The Virtuous Cycle of Compute. Traditionally, there has been a division of labor between computers and humans where all forms of number crunching and bit manipulation are left to computers, whereas intelligent decision-making is left to us humans. We are now at the cusp of a major transformation that can disrupt this balance. This disruption is triggered by an unprecedented convergence of massive compute with massive data, and some recent algorithmic advances. This confluence has the potential to spur a virtuous cycle of compute. Deep Learning was recently scaled to obtain 15PF performance on the Cori supercomputer at NERSC. Cori Phase II features over 9600 KNL processors. It can significantly impact how we do computing and what computing can do for us. In this talk, I will discuss some of the application-level opportunities and system-level challenges that lie at the heart of this intersection of traditional high performance computing with emerging data-intensive computing. Dr. Pradeep Dubey is an Intel Fellow and Director of Parallel Computing Lab (PCL), part of Intel Labs. His research focus is computer architectures to efficiently handle new compute- and data-intensive application paradigms for the future computing environment. Watch the Video: https://wp.me/p3RLHQ-hRv Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. Intel Confidential
CNDA Required
Virtuous Cycle of Compute

2. AI And The Virtuous Cycle of Compute
Cloud &
DATA Center
Things &
devices
Dense
Compute
Scripting
Frameworks
Training
INFERENCING
2* Source: IDC Worldwide Semiannual Cognitive/Artificial Intelligence Systems Spending Guide, Oct 2016
AI Needs More Compute Faster: 55% revenue CAGR … >$47 billion in 2020 *

3. Science:
Cognition and
Neuroscience
Computer Science:
Scalable Algorithms for
learning
and Decision making
Computer Systems:
End-to-end pipeline
data management, model
training and deployment
Intel’s AI R&D Stack
3

4. AI: What makes it hard and fun!
4
Compute architecture needs of AI
REDUCING ARITHMETIC PRECISION WHILE PRESERVING ACCURACY: ALL 32 à 16, 8, 4, 2 …
DOMAIN-SPECIFIC ARCHITECTURES à TRADITIONAL, NEUROMORPHIC, QUANTUM
STRONG-SCALING AI TO HPC SCALE: LARGER BATCH SIZES AND HIGHER ORDER METHODS
DELIVERING PERFORMANCE-PRODUCTIVITY: SCRIPTING LANGUAGES AND SPATIAL ARCHITECTURES
Productivity and Scaling needs of AI

5. Deep Learning at 15PF*
(with NERSC, Stanford, and Univ of Montreal)
Scientific Achievement
§ Signal vs. Background classification for LHC datasets
exceeds physics cuts
§ Pattern discovery for Climate data
Methods Achievement
§ Hybrid parameter update strategy
§ Supervised and semi-supervised architectures
CS Achievement
§ IntelCaffe + MLSL optimized on KNL
§ ~2TF peak on single KNL node
§ ~15 PF peak on ~9300 nodes
* “Petascale Deep Learning” Thorsten Kurth, Jian Zhang, Nadathur Satish, Ioannis Mitliagkas, Evan Racah, Mostofa Patwary, Tareq Malas, Narayanan Sundaram, Wahid
Bhimji, Mikhail Smorkalov, Jack Deslippe, Mikhail Shiryaev, Srinivas Sridharan, Prabhat, and Pradeep Dubey, accepted at Supercomputing 2017

6. Reduced precision needs for DL Training
What motivates us to look beyond FP16 *:
Horizontal bars indicating the dynamic range covered by the 16-bit
mantissa
* https://arxiv.org/pdf/1711.02213.pdf

7. Productivity Challenge
Motivation
Challenge#1:
Domain experts are not professional
software programmers.
Adoption of Python
continues to grow among
domain experts and
developers for its productivity
benefits
Challenge#2:
Python performance limits migration
to production systems
State of the Art algorithms are
changing so fast, that 50% of ML
programmers implement
algorithms * “from scratch” *
* Source: State of the Developer Nation Q!’2017, http://vmob.me/DE1Q17Mobile

8. Celeste: 1st Julia application to hit 1PF*
(In collaboration with NERSC, UCB, MIT and Julia Computing)
* Cataloging the Visible Universe through Bayesian Inference at Petascale in Julia; https://www.youtube.com/watch?v=uecdcADM3hY&feature=youtu.be
• Scientific Achievement
• First catalog with parameter & uncertainty estimates for over 300M
objects
• 55 TB SDSS dataset processed in 15 minutes
• DESI instrument will use catalog for target selection
• Methods Achievement
• Bayesian Inference on world’s largest generative model (in science)
• Joint estimation of Billions of parameters
• CS Achievement
• Code written in Julia, optimized for execution on KNL
• Code scaled on 9300 Cori KNL nodes

9. 9
Architectural Choices Growing …
1D SIMD
Such as: AVX-512
For: Short vector processing
2D Tile
Such as: Nervana Lake Crest
For: Dense Matrix Algebra
Configurable Fabric:
Such as: Stratix-10 FPGA
For: Irregular Spatial Parallelism