Refresher: Machine Learning
A computer program is said to learn from experience (E) with respect to some class of tasks (T) and
performance measure (P), if its performance at tasks in T, as measured by P, improves with
experience E — T. Mitchell 1997
• Machine learning generally breaks down into two application areas which are closely
related: classification and regression.
• Some Intuitive Examples of Machine learning:
Learning by labelled example (supervised learning) e.g. An email spam
Discovering patterns (unsupervised learning) e.g. data clustering, pattern
Feedback right/wrong: reinforcement learning e.g. repetitive game play with
defined win/loss rules
• Most machine learning methods work well because of human-designed/hand
engineered features (representations) .
• It can be seen as curve fitting, optimising weights in order to make a prediction.
Unsupervised Learning can also be viewed as a way to create a higher level
representation of data.
Neural Networks (Basics)
• A basic neural network takes some numbers for input, looks at
some example answers and guesses new numbers for output
where there are no example answers (e.g. of supervised
• Transforms through function such as the sigmoid, logistic
function, hyperbolic tangent etc.
• In simple neural networks there is an input layer, a hidden layer,
and an output layer.
• A typical neural network consists of a set of layers with nodes
that take input from the preceding layer, multiply these
numbers by weights, sum up different inputs, and pass the
result to the next layer.
• Multiple layers can be stacked one after the next, and each
layer can have different numbers of nodes as per design.
• Imagine that you don’t speak a word of Chinese, but your company is moving you to China next
month. Company will sponsor Chinese speaking lesson for you once you are there, but you
want to prepare yourself before you go.
• You decide to listen to Chinese radio station
• For a month, you bombard yourself with Chinese radio.
• This can be seen as unsupervised learning experience as you don’t know the meaning of
• Lets think that somehow your brain develops capacity to understand few commonly occurring
patterns without meaning. In other words, you have developed a different level of
representation for some part of Chinese by becoming more tuned to its common sounds and
• Hopefully, when you arrive in China, you’ll be in a better position to start the lessons.
Example loosely taken from Lecture series by Prof. Abu Mustafa
Image from blog http://www.datarobot.com/blog
What changed ?
• More data (Big Data) – Better performance is observed when you feed lots
and lots of unlabelled data.
• Faster hardware: GPU’s, multi-core CPU’s.
• Some key technical breakthroughs after 2006 (Currently cutting edge and
active research areas.)
• Stacked Restricted Boltzman Machines
• Stacked Autoencoders
• Pretrained layers
• Sparse Representations
• Working ideas on how to train deep architectures
What deep learning refers to
• Large Neural networks used to learn feature hierarchies
• Simply a continuation of the multi-decade advancement in our ability to make use of large scale
Deep Learning in a nutshell
Unsupervised training tries to get the network to learn statistical regularities in the input space.
To achieve this you need to define an objective function to maximize, which typically revolves
around some kind of sparseness or information preservation criteria within the context of
dimension reduction. You try to get the network to learn the best way to maintain as much
information about the input as possible despite a data bottleneck. In this way the network
hopefully learns a sort of conceptual vocabulary which is used to best describe the typical data.
Train the networks with data in layer-wise fashion: train the first layer, and then train the
second layer on inputs that are pre-processed by the first, and so on. Once you have a network
that has been trained on unlabelled data, then build a classifier (just an example) by passing
the output of the network through a support vector machine , or by using other techniques.
The final layer that generates the classifications is trained by using a fully-labelled data set
which can have fewer examples than the data set used for the unsupervised pass.
This classifier ends up working very well because its input is has been already trained to
represent the important statistical features of the input space.
Why deep learn?
• Don't need to hand engineer features. Manually designed features are often over-specified,
incomplete and take a long time to design and validate
• Theoretically, a good lower level representation can be used for many distinct tasks. In deep
learning words, pre-trained deep network weights can be used as the starting point for further
training on labelled data. This has shown better performance that other weight initialization
• For some problems like computer vision, the learnt features are better than the best hand-
engineered features, and can be so complex that it would take way too much human time to
• Use of unlabelled data to pre-train the network. In addition, in some domains we have so much
unlabelled data but labelled data is hard to find.
• It’s a latest machine learning technique that is producing significant increase in quality of
Deep Learning Impact
Image recognition (e.g. Tagging faces in photos)
Voice recognition (e.g. Voice based search, Siri)
Natural Language Processing
Pattern detection (e.g. Handwriting recognition)
Deep Learning Myths
Deep learning outperforms all other algorithms in confined settings and is ‘cure all’ ML.
Deep learning is taking advantage of an understanding of how the brain processes information,
learns, makes decisions, or copes with large amounts of data. *
Deep Learning is commonplace - just another algorithm.
Deep Learning works out of box. **
Some cool demos and DIY
• Handwritten digit recognition - http://www.cs.toronto.edu/~hinton/adi/index.htm
• open-source implementations of various state-of-the-art neural networks (c.f. Caffe,cuda-
covnet, Torch, Theano)
• Deep Learning Tutorial - http://deeplearning.net/tutorial/
• Deep Learning in java - http://deeplearning4j.org/
• Deep Learning in R - http://cran.r-project.org/web/packages/deepnet/deepnet.pdf
• Microsoft Research free book
Parece que tiene un bloqueador de anuncios ejecutándose. Poniendo SlideShare en la lista blanca de su bloqueador de anuncios, está apoyando a nuestra comunidad de creadores de contenidos.
¿Odia los anuncios?
Hemos actualizado nuestra política de privacidad.
Hemos actualizado su política de privacidad para cumplir con las cambiantes normativas de privacidad internacionales y para ofrecerle información sobre las limitadas formas en las que utilizamos sus datos.
Puede leer los detalles a continuación. Al aceptar, usted acepta la política de privacidad actualizada.