AI Developments Aug 2017

AI Developments
1
AI Developments v0.10 Peter Morgan August 2017

Outline
• Concepts
• AI Market
• Data
• Algorithms
• Hardware
• Conferences
• New Developments
AI Developments v0.10 Peter Morgan August 2017 2

Types of Intelligence

The Intelligence Revolution
5AI Developments v0.10 Peter Morgan August 2017

Deep Learning/AI Frameworks - The Big Picture
AI Frameworks
Cognitive
Architectures
ML Frameworks
Supervised,
Unsupervised &
Reinforcement
Deep Learning
Frameworks
Neural Networks

What is Intelligence?
• Intelligence is an Agent’s ability to adapt to and to
achieve goals within its Environment
• Human vs machine intelligence – ultimately the same
• “The term artificial intelligence is somewhat
nonsensical. Something is either intelligent or it isn’t.
Just as something either flies or it doesn’t. We don’t
talk about artificial flying” - Zoubin Ghahramani,
Cambridge University
• Information processing, computation, physics,
hardware
• Exploration vs exploitation
• Biological (any species) versus machine (any type)
•Does substrate matter - carbon vs silicon?

What is Learning?
• Learning algorithms – system gets better
with more data until no further
improvement
• Train the system – just like animals learn
• Supervised, unsupervised and
reinforcement learning
• Physically, it is the strengthening of
connections (synapses) between nodes
(neurons)
• Memory (short and long term) is involved
• Deep learning is a step towards the goal of
artificial general intelligence (AGI)
• Ensemble of techniques

What is Deep Learning?

Deep Learning = Neural Networks
• Refers to systems that learn from data
• These systems are based on artificial neural networks (ANNs), which in
turn are based on biological neural networks (BNN), such as the human
brain
• In practice such learning systems consist of data, multiple layers, nodes,
weights and optimisation algorithms

Biological Neuron

Deep Learning Is Eating the World
• What about the “AI winters”? 1974–80 and 1987–93, where AI companies
over-promised and under-delivered https://en.wikipedia.org/wiki/AI_winter
• Due to more labeled data, more compute power, better optimization
algorithms, and better neural net models and architectures, deep learning
has started to supersede humans when it comes to image recognition and
classification
• Work is being done to obtain similar levels of performance in natural
language processing and understanding
• According to Jeff Dean in a recent interview, Google have implemented DL in
over one hundred of their products and services including search and photos
• AI is enjoying a renaissance now, not simply because of the promise it holds
for the future but because of the impact it is having on businesses today

AI Market

Deep Learning Startups

• Hardware (compute) – Nvidia GPU, Intel (Nervana), AMD Radeon
• Data available - structured and unstructured
• Research activity
• Conference attendance (e.g., NIPS)
• Meetup groups
• PhD enrolments in CS and machine learning
• Performance measures - chess, Jeopardy, Go, computer vision,
language processing, ...
• Number of papers being published in AI/ML
AI Trends 1

• Availability of open source deep learning frameworks, including
TensorFlow, Mxnet, etc.
- Number of packages
- GitHub commits
- Contributors etc.
• For example, TF is most downloaded repo from GitHub in under a year
• Fact that major corporates open sourced their AI frameworks, starting
with Google (TF, etc.)
• Number of AI related jobs on job boards
• Salaries for AI experts
AI Trends 2

• Backing and realignment by corporates to rebrand as AI companies -
Microsoft, IBM, Amazon (Google and Facebook were already there)
• For example:
- IBM Watson HQ in NYC
- Microsoft announcing 5000 strong AI division
- Apple announcing at NIPS that it would be open sourcing its AI research
- Siri, Cortana, Alexa are all NLP apps using neural nets
• Number of AI products and apps
• Number of AI/deep learning startups
AI Trends 3

• Venture capital investment in AI startups
• Number of press/news articles
• Announcements from AI experts with 30 years experience confirming that
this time is for real - there will be no more AI winters
• Number of professors being hired way from academia to join AI
companies , e.g. Uber and CMU, Google and Oxford, Facebook, Apple, etc.
• Government level panels on the development and impact of AI on jobs,
society and policy, e.g., Whitehouse and U.K. parliament
• AI Safety consortium announced last month between Google, Microsoft,
IBM and Amazon to track developments in AI
• Recent books published by professors and engineers on AI development
AI Trends 4

Image classification
Progress in machine classification of images - error rate by year. Red line is the
error rate of a trained human. 2.25% as of July 2017 (ImageNet).

DL Outperforms ML

Computer Vision Accuracy

GPU Faster than Moore’s Law

As of May 2016, Aug 2017 > 66,000

Growth of Deep Learning atGoogle
and many more . . ..
Directories containing model description files

Data

Where does the data come from?
• Science – particle, astrophysics
• Industry – oil, finance, telecom (all verticals)
• Social – Facebook, LinkedIn, Twitter
• Medicine – genome, neuroscience
• Government – census, education, police
• Sports – statistics
• Environment – weather, sensors

Data Sets
• Raw data input into the neural network can originate from any environmental source
• It can be recorded and stored in a database (e.g., text, images, audio, video), or live
(incident directly from the environment) streaming data
• Examples of recorded data sets include MNIST, Labeled Faces in the Wild (LFW),
ImageNet, CIFAR and YouTube-8M
MNIST LFW

Data Sets
• Images: MNIST, CIFAR-10, ImageNet, PASCAL VOC,
Mini-Places2, Food 101
• Text: IMDB, Penn Treebank, Shakespeare Text,
bAbI, Hutter-prize
• Video: UCF101, Kinetics, YouTube-8M
• Others: flickr8k, flickr30k, COCO

Algorithms

Deep Learning Evolution

Convolutional Neural Networks
• First developed in 1970’s
• Widely used for image recognition and classification
• Inspired by biological processes, CNN’s are a type of feed-forward ANN
• The individual neurons are tiled in such a way that they respond to overlapping
regions in the visual field

Recurrent Neural Networks
• First developed in 1970’s
• RNN’s are neural networks that are used to predict the next element in a
sequence or time series
• This could be, for example, words in a sentence or letters in a word
• Applications include predicting or generating music, stories, news, code,
financial instrument pricing, text, speech, in fact the next element in any event
stream

LSTM and NTM
• Long Short Term Memory (LSTM)
• LSTM (Schmidhuber, 1997) is an RNN architecture that contains blocks that can
remember a value for an arbitrary length of time
• It solves the vanishing or exploding gradient problem when calculating back
propagation
• An LSTM network is universal in the sense that given enough network units it can
compute anything a conventional computer can compute, provided it has the proper
weight matrix
• LSTM outperforms alternative RNNs and Hidden Markov Models and other sequence
learning methods in numerous applications, e.g., in handwriting recognition, speech
recognition and music composition
• Neural Turing Machines (NTM)
• NTMs are a method of extending the capabilities of recurrent neural networks by
coupling them to external memory resources

Optimizations
• Initializers
Constant, Uniform, Gaussian, Glorot Uniform, Xavier, Kaiming, IdentityInit, Orthonormal
• Optimizers
Gradient Descent with Momentum, RMSProp, Adadelta, Adam, Adagrad, MultiOptimizer
• Activations Rectified Linear, Softmax, Tanh, Logistic, Identity, ExpLin
• Layers Linear, Convolution, Pooling, Deconvolution, Dropout, Recurrent, Long Short-
Term Memory, Gated Recurrent Unit, BatchNorm, LookupTable, Local Response Normalizat
ion, Bidirectional-RNN, Bidirectional-LSTM
• Costs Binary Cross Entropy, Multiclass Cross Entropy, Sum of Squares Error
• Metrics, Misclassification
(Top1, TopK), LogLoss, Accuracy, PrecisionRecall, ObjectDetection

GANs
• Introduced by Ian Goodfellow et al in 2014 (see references)
•A class of artificial intelligence algorithms used in unsupervised deep
learning
• A theory of adversarial examples, resembling what we have for
normal supervised learning
• Implemented by a system of two neural networks, a discriminator, D
and a generator, G
• D & G contest with each other in a zero-sum game framework
• Generator generates candidate networks and the discriminator
evaluates them

Stacked Generative Adversarial Networks
https://arxiv.org/abs/1612.04357v1AI Developments v0.10 Peter Morgan August 2017 40

Monet Paintings to Photos

Collection Style Transfer

Object Transfiguration

Season Transfer

Deep Learning Frameworks

TensorFlow
• TensorFlow is the newly (Nov 2015) open sourced deep learning library
from Google
• It is their second generation system for the implementation and
deployment of large-scale machine learning models
• Written in C++ with a python interface, it is borne from research and
deploying machine learning projects throughout a wide range of
Google products and services
• Initially TF ran only on a single node (your laptop, say), but Google have
now released a version that runs on a distributed cluster
• Available in the cloud on GCP
• https://www.tensorflow.org/

47

TensorFlow supports manyplatforms
Raspberry Pi
AndroidiOS
1st-gen TPU
GPUCPU
Cloud TPU AI Developments v0.10 Peter Morgan August 2017 48

TensorFlow supports manylanguages
Java

Cognitive Toolkit
• Microsoft open source deep learning framework (Jan 25, 2016)
• Version 2.0 released Oct 25, major upgrade
• Renamed CNTK to Microsoft Cognitive Toolkit
• Announced partnership with Nvidia and OpenAI (Elon Musk backed
AI startup), Nov 16
• Languages are Python, C++ or BrainScript
• Can run on Azure GPU’s
• https://www.microsoft.com/en-us/research
/product/cognitive-toolkit/

Torch
• First released in 2000, with over 50,000 downloads, company users
include Google, Facebook, Twitter
• The goal of Torch is to have maximum flexibility and speed in building
scientific algorithms while making the process extremely simple
• Torch is a neural network library written in Lua with a C/CUDA interface
originally developed by a team from the Swiss institute EPFL
• At the heart of Torch are popular neural network and optimization
libraries which are simple to use, while being flexible in implementing
different complex neural network topologies
• http://torch.ch/

Hardware is Hot Again

Types of Hardware
• Sensors, processors, storage, memory, network
• Processors - CPU, GPU, FPGA, ASIC, NPU, QPU
• GPU - Graphics Processing Units were first brought to market by Nvidia in 2007 to
meet the demands of the gaming market
• Massively parallel processing (MPP)
• 100 x speedup compared with CPU’s
• Widespread application – science, industry, government
• Nvidia www.nvidia.com
• Intel Xeon Phi http://www.intel.com/content/www/us/en/processors/xeon/xeon-
phi-detail.html
• AMD Radeon www.amd.com

CPU v GPU Architecture

CPU – Intel Xeon

CPU – AMD RyZen

GPU’s

Nvidia GPU Exponentials

New Hardware – Nvidia Volta V100

Volta V100 Specs
•Pairs NvidiaCUDA and Tensor Cores to deliver the performance of an
AI supercomputer
•Over 21 billion transistors
•With 640 Tensor Cores, Volta delivers over 100 TFLOPS of deep
learning performance
•Over a 5X increase compared to prior generation NVIDIA Pascal
architecture (last year)
•Next generation of Nvidia NVLink connects multiple V100 GPUs at up
to 300 GB/s

Volta V100 Benchmarks

HGX-1 - For AI Cloud Computing
•Purpose-built for cloud computing
•Eight NVIDIA Tesla GPUs interconnected
with an NVLink hybrid cube
•Applications include including deep
learning training, inference, advanced
analytics, and HPC
•Faster and cheaper than legacy
CPU-based servers
•Extract the full AI performance that
Tesla V100 provides

Self-driving Cars – Nvidia Drive PX2
• Delivers 20 TFLOPS of
performance
• Consumes only 20W of power
• Packed with 7 billion
transistors

AMD Radeon Vega GPU

Intel Nervana Hardware

Google TPU v1

Google TPU v2

Fujitsu DLU

Graphcore - IPU

Neuromorphic – HBP Spinnaker

Neuromorphic – IBM True North

Quantum - DWave
73

Deep Learning in the Cloud

DLaaS - Cloud Services
• DL/ML as a Service is offered by all the major cloud providers
• AWS https://aws.amazon.com/machine-learning/
• Azure https://azure.microsoft.com/en-us/services/machine-learning/
• GCP https://cloud.google.com/products/machine-learning/
• Bluemix https://www.ibm.com/cloud-computing/bluemix/watson

TPU as a Service

AI Conferences – Business Focussed
• O’Reilly AI http://conferences.oreilly.com/artificial-intelligence/ai-ny
• AI Frontiers http://www.aifrontiers.com/
• AI World http://aiworldexpo.com/program/
• AI Europe http://ai-europe.com/
• AI Summit https://theaisummit.com/london/
• Re:Work https://www.re-work.co/events/
• World of Watson https://www-01.ibm.com/software/events/wow/

AI Conferences – Research Focussed
• NIPS = Neural Information Processing Systems https://nips.cc/
• IJCNN = International Joint Conference on Neural Networks http://www.ijcnn.org/
• IJCAI = International Joint Conference on Artificial Intelligence http://ijcai.org/
• ICANN = International Conference on Artificial Neural Networks http://www.icann2017.org/
• IWANN = International Work-Conference on Artificial Neural Networks http://iwann.uma.es/
• ICONIP = International Conference on Neural Information Processing http://www.iconip2017.org/papers.html
• ICAART = International Conference on Agents and Artificial Intelligence http://www.icaart.org/
• ISIS = International Symposium on Advanced Intelligent Systems http://isis2017.org/
• AAAI = Association of Advancement of Artificial Intelligence
http://www.aaai.org/Conferences/conferences.php
• ACM = Association of Computing Machinery https://www.acm.org/conferences
• AGI = Artificial General Intelligence Conference http://agi-conf.org/
• TensorCon = TensorFlow Conference https://ti.to/TensorCon/

AI Research Centers
• Nvidia AI Lab - Donated DGX-1’s plus research funding to twenty
universities
• MIT CSAIL – Image classification
• Stanford SAIL – Robotics
• Toronto – Self-driving cars
• Montreal MILA – Disease prediction
• Berkeley BAIR – Robotics planning
• NYU – Cancer screening
• Oxford – Lip reading
• IDSIA – AGI

New Developments
• Multi-modal learning, Transfer learning, One-shot learning, GANs
• Better reinforcement learning / integration of deep learning and
reinforcement learning
• Better generative models. Algorithms that can reliably learn how to
generate images, speech, text that humans can’t tell apart from the
real thing
• Learning to learn and ubiquitous deep learning: algorithms that
redesign their own architecture, tune their own hyperparameters,
etc. Right now it still takes a human expert to run the learning-to-
learn algorithm, but in the future it will be easier to deploy, and all
kinds of business that don’t specialize in AI will be able to leverage
deep learning

New Developments (cont.)
• Sample-efficient learning algorithms that learn from as few labeled
examples as humans do
• Semi-supervised learning and one-shot learning will reduce the
amount of data needed to train several kinds of models and make AI
use more widespread
• Research will focus on making extremely robust models that almost
never make a mistake, for use in safety-critical applications
• Deep learning will continue to spread out into general culture and
we’ll see artists and meme creators using it to do things that we
never would have anticipated

Where are we headed?
World Economic Forum (WEF) Report, 2016:
Today, we are at the beginning of a Fourth Industrial Revolution.
Developments in genetics, artificial intelligence, robotics, nanotechnology, 3D
printing and biotechnology, to name just a few, are all building on and
amplifying one another. This will lay the foundation for a revolution more
comprehensive and all-encompassing than anything we have ever seen
Deepmind Mission:
Solve intelligence then use it to solve everything
else

AI Related Books
• Bengio, Yoshua et al, Deep Learning, MIT Press, 2016
• Brynjolfsson, Erik and Andrew McAfee, The Second Machine Age, W.W. Norton &
Co., 2014
• Chollet, Francois, Deep Learning with Python, Manning, Oct 2017
• Domingos, Pedro, The Master Algorithm, Basic Books, 2015
• Ford, Martin, Rise of the Robots: Technology and the Threat of a Jobless Future,
Basic Books, 2015
• Kaku, Michio, The Future of the Mind, Doubleday, 2014
• Kurzweil, Ray, How to Create a Mind, Penguin Books, 2013
• Russell and Norvig, Artificial Intelligence, A Modern Approach, Pearson, 2009
• Shanahan, Murray, The Technological Singularity, MIT Press, 2015
• Yampolskiy, Roman, Artificial Superintelligence, A Futuristic Approach, CRC, 2015

References
• LeCunn, Y., Unsupervised Learning: the Next Frontier in AI [video], Nov 2016
https://www.aices.rwth-aachen.de/charlemagne-distinguished-lecture-series
• LeCun, Y., Bengio, Y., and Hinton, G., Deep Learning, Nature, v.521, p.436–444,
May 2016
http://www.nature.com/nature/journal/v521/n7553/abs/nature14539.html
• Goodfellow, I. et al, Generative Adversarial Networks, in NIPS 2014
• Radford, A., Metz, L., and Chintala, S., Unsupervised Representation Learning
with Deep Convolutional Generative Adversarial Networks, Jan 2016
https://arxiv.org/abs/1511.06434
• CycleGAN https://arxiv.org/abs/1703.10593
• Mathieu, M., Couprie, C., and LeCun, Y., Deep Multi-Scale Video Prediction
Beyond Mean Square Error, ICLR 2016 conference paper, Feb 2016

References
• Brtiz, D. et al, Massive Exploration of Neural Machine Translation Architectures,
Mar 2017 https://arxiv.org/abs/1703.03906
• Johnson, M. et al, Google's Multilingual Neural Machine Translation System:
Enabling Zero-Shot Translation, Nov 2016 https://arxiv.org/abs/1611.04558
• Gehring, J. et al, Convolutional Sequence to Sequence Learning, May 2017
• Feedback Networks
http://feedbacknet.stanford.edu/feedback_networks_2016.pdf
• AI safety discussion https://www.facebook.com/groups/467062423469736/
• Google ICML 2017 Publications
https://research.googleblog.com/2017/08/google-at-icml-2017.html

Questions?
86AI Developments v0.10 Peter Morgan August 2017 86

AI Developments Aug 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AI Developments Aug 2017

Similar to AI Developments Aug 2017 (20)

More from Peter Morgan

More from Peter Morgan (9)

Recently uploaded

Recently uploaded (20)

AI Developments Aug 2017