Machine Learning from a Software Engineer's perspective

1. Machine learning from a
software engineer’s perspective
Marijn van Zelst
AMSTERDAM 16 - 17 MAY 2017

2. About me
@marijnvanzelst
Marijn van Zelst
• Software engineer at Luminis Amsterdam

3. Goal:
• Practical knowledge about machine learning
• What kind of problems it can solve
• Do your own experiment

4. Why is machine learning
interesting for Software Developers?
• New opportunities
• Builds on existing knowledge
• Open-source communities and libraries

5. Tensorflow
• Developed by Google
• Open-source
• Has api’s for most major programming languages
• High level api

6. Machine learning
Learn from examples

7. Handwritten digit classification
MNIST dataset

8. What is the difference between a
2 and a 3?

9. Neural networks
Image recognition
Speech recognition
Natural language recognition
Churn prediction
Fraud detection
Anomaly detection

10. Neural networks
Pixel 1
Pixel 2
Pixel 3
Pixel 784
Zero
Two
One
Nine

11. Neurons
x3
x1
x2
w1
w2
w3 b
((x1 * w1) + (x2 * w2) + (x3 * w3)) * b

12. Activation function
x3
x1
x2
w1
w2
w3
((x1 * w1) + (x2 * w2) + (x3 * w3)) * b
b
Input
Output (activation)

13. Learning algorithm
• While not done
• pick a training example (input data, class)
• run it through the neural network which produces a prediction
• modify the weights in the neural network so that the prediction is closer to the actual class

14. Agile approach to machine learning
• Theory takes a lot of time to explain and learn
• Using high level api’s you can do experiments without knowing all the mathematical details
• When you have your first results and want to improve
• Learn more
• Ask for expert help

15. Using a high level api
• Which features do I use?
• Do I need more data?
• How many neurons and layers?
• How many training cycles?

16. Distinguish 0s and 1s

17. Just two features
blackwhite

18. Two dimensional feature space
white black
black 0
1
y
x
y
x
y
x

19. white black
black 0
1
y
x

21. white black
black
y
x
x
y

22. w1
w2 b
w1=0.7
w2=0.62
b=0.085

23. w1
w2 b
w1=0.7
w2=0.62
b=2
w1=0.7
w2=0.62
b=-1
w1=0.5
w2=0.2
b=0.085
w1=0.2
w2=0.5
b=0.085

24. Neural nets intuition
• Input:
• Features (coordinates in feature space)
• Output:
• A predicted class at every coordinate
in the feature space
white black
black
y
x

25. More training examples
x
y
x
y
x
y
x
y

26. Two dimensional feature space
white black
black 0
1
y
x
y
x
y
x
0 & 1

27. Neural nets intuition
• The combination of several features make classes distinguishable
• Training on more examples doesn’t always work
• Neural networks can handle thousands of features
white black
black 0
1
y
x
0 & 1

28. white black
black
y
x
x
yx y
x
y
x
y

29. y
x
y
x

30. Neural nets intuition
Adding a feature can make other features more useful

32. 50 steps 600 steps5 steps
train - 72.5% train - 88.8% train - 92.2%

34. 50 steps 600 steps5 steps
train - 72.5% train - 88.8% train - 92.2%
test - 76.5% test - 82.6% test - 80.3%

36. Neural nets intuition
• Keep a test set to verify the accuracy of your training algorithm
• Keep an eye on the test set accuracy during training to decide when to stop

37. 78% 88% 71%

38. 78%88%
71%

39. Neural nets intuition
• How many neurons?
• Depends on the complexity of the data
• Start small, gradually ramp up
• Check for overfitting

40. Insights
• High dimensional
• Can handle and find correlations between lots of features
• They do probabilistic predictions
• They require a training set and a test set for validation

41. from tensorflow.contrib.learn import infer_real_valued_columns_from_input, DNNClassifier
from tensorflow.examples.tutorials import mnist
mnist_data = mnist.input_data.read_data_sets("MNIST_data/")
train_data, test_data = mnist_data.train.images, mnist_data.test.images
train_labels, test_labels = mnist_data.train.labels.astype('int32'),
mnist_data.test.labels.astype('int32')
estimator = DNNClassifier(
feature_columns=infer_real_valued_columns_from_input(mnist_data.train.images),
hidden_units = [2500, 1000, 1500, 2000, 500],
n_classes = 10)
estimator.fit(x = train_data, y = train_labels, steps = 1000)
print(estimator.evaluate(x = test_data, y = test_labels)['accuracy'])
# 97.5% accuracy in ~15min
Handwritten digit classifier in TensorFlow

42. Conclusion
• You can use machine learning using a high level api
• It can make complex decisions on lots of features
• Using high level api’s you can do experiments without knowing all the mathematical details
• When you have your first results and want to improve
• Learn more
• Ask for expert help

43. Resources on machine learning
• Tensorflow high level api quickstart: https://www.tensorflow.org/get_started/tflearn
• Other resources:
• My blog on the text classification with some more in depth code: https://github.com/luminis-ams/
blog-text-classification
• For a theoretical understanding of the backpropagation algorithm: Machine learning course
Coursera - Andrew Ng
• More practical deep learning course: Udacity deep learning course
• Visualization of a neural network (used in the presentation)
• http://playground.tensorflow.org

44. Dijkstra probably hates me - Linus
Torvalds, in kernel/sched.c
?Thank you
Any
Questions?