Swift for tensorflow

1. Swift for Tensorflow
Next-Generation Platform
Tensorflow Python Swift

2. •
•
• Tensorflow
•
• Tensorflow goal
• Why Swift for Tensorflow?
• Swift Python
• Tensorflow Python Swift
•
•

3. Swift for Tensorflow
Google I/O 2019 5. 8.TensorFlow Dev Summit 2019 3. 6.

4. • Python Tensorflow Swift "Next-generation platform" .
• Tensorflow Swift ?

5. -
.
- Swift Python (Expressive, Writability) ?
.
- Swift Python ?
Tensorflow Python Ecosystem( , , ) Swift ?

6. Tensorflow

7. Tensorflow
• Tensorflow
Python Javascript Swift
• CPU/GPU . Google TPU(Tensor Processing Unit)
. .
• Tensorflow Data Flow Graph .

8. Data Flow Graph
• (Node) (Edge)
(Directed Graph) .
• , , / .
• .
• (=Tensor) ,
(Tensor Flow) .

9. Tensorflow goal
Tensorflow .
• High Performance
• Writability(Expressiveness)
• Performance Predictability
• Debuggability and Introspection
• High-End User Experience
• Best-of-class Automatic Differentiation (AD)

10. Swift vs Python

11. Python
• Community: .
• Open-source and cross-platform
• Shallow learning curve: , .
• High productivity
• Ecosystem: Numpy, Pandas, …

12. Python
• Performance: Python C++ Tensorflow .
• Concurrency: GIL(Global Interpreter Lock) .
Mutex lock.
• Deployment: Python . .
Tensorflow .(Caffe-C++, TF Lite-Android, CoreML-iOS)
• Custom Operations: C++, CUDA .
From WhySwiftForTensorFlow.md

13. Swift
• Swift iOS macOS , 2014 6 2 WWDC .
• OS Objective-C
LLVM(Low-Level Virtual Machine) .
• Strong type language, Optional Type .
• 2019 4 18 Swift 5.0

14. Swift
• Type Safe
• Enable Automatic Differentiation in compile time
• Python Interoperability
In machine learning aspect

15. Type safe - Optional
• Compile-time .
• .
• Type '?'
• : , (nil)
• NullPointerException compile-time .
var optionalInt1: Int = nil // error
var optionalInt2: Int? = nil // correct

16. Automatic Differentiation ( )
• .
• Automatic Differentiation .
• (derivative) . (Runtime)
• gradient-based . ex) (Linear Regression)

17. Automatic Differentiation
• .
• Swift Compile-time

18. @differentiable
@differentiable
func square1(_ x: Float) -> Float {
return x * x
}
@differentiable
func square2(_ x: Float) -> Float {
return x * x * x * x
}
let x: Float = 3.0
print(gradient(at: x, in: square1)) // x^2, 6.0
print(gradient(at: x, in: square2)) // 4*x^3, 108.0

19. Python Interoperability
// NumPy example:
import Python
let np = Python.import("numpy") // import numpy as np
let a = np.arange(15).reshape(3, 5) // a = np.arange(15).reshape(3, 5)
let b = np.array([6, 7, 8]) // b = np.array([6, 7, 8])
// Pickle example:
let gzip = Python.import("gzip") // import gzip as gzip
let pickle = Python.import("pickle") // import pickle as pickle
let file = gzip.open("mnist.pkl.gz", "rb") // file = gzip.open("mnist.pkl.gz", "rb")
// (images, labels) = pickle.load(file)
let (images, labels) = pickle.load(file).tuple2
print(images.shape) // (50000, 784) print(images.shape)

20. Swift
• Performance
• Concurrency
Swift C-based low-level .
Swift compile , .
Tensorflow Swift Tensor compile-time .
Swift pthread API . Python .
Swift Dispatch() (async/await) .

21. Disadvantages of Swift
• New language: Swift , Python native .
• Windows support: BSD, Linux, UNIX .
• Small data-science community: Python data-science .

22. Swift vs Python in MNIST Example

23. MNIST
• Softmax-regression .
• Layer vector .
• bias .
• Softmax-regression .

24. Softmax Regression
• Multi-class Classification Model
• Probability = [0.1, 0.2, 0.8, … 0.6, 0.1, 0.0] 1 .

25. Softmax Regression

26. Gradient Descent
=

27. Swift
struct MNISTParameters : ParameterGroup {
var w1 = Tensor<Float>(randomNormal: [784, 30])
var w2 = Tensor<Float>(randomNormal: [30, 10])
var b1 = Tensor<Float>(zeros: [1, 30])
var b2 = Tensor<Float>(zeros: [1, 10])
}
let iterationCount = Int32(batchSize) / minibatchSize
for i in 0..<iterationCount {
let images = minibatch(images, index: i)
let numericLabels = minibatch(numericLabels, index: i)
let labels = minibatch(labels, index: i)
let z1 = images • parameters.w1 + parameters.b1
let h1 = sigmoid(z1)
let z2 = h1 • parameters.w2 + parameters.b2
let predictions = sigmoid(z2)
let dz2 = predictions - labels
let dw2 = h1.transposed() • dz2
let db2 = dz2.sum(squeezingAxes: 0)
let dz1 = matmul(dz2, parameters.w2.transposed()) * h1 * (1 - h1)
let dw1 = images.transposed() • dz1
let db1 = dz1.sum(squeezingAxes: 0)
let gradients = MNISTParameters(w1: dw1, w2: dw2, b1: db1, b2: db2)
parameters.update(withGradients: gradients) { param, grad in
param -= grad * learningRate
}
let correctPredictions = predictions.argmax(squeezingAxis: 1).elementsEqual(numericLabels)
correctGuesses += Int(Tensor<Int32>(correctPredictions).sum())
totalGuesses += Int(minibatchSize)
}

28. Python
X = tf.placeholder(tf.float32, [None, 784])
Y = tf.placeholder(tf.float32, [None, 10])
W1 = tf.Variable(tf.random_normal([784, 256], stddev=0.01))
L1 = tf.nn.relu(tf.matmul(X, W1))
W2 = tf.Variable(tf.random_normal([256, 256], stddev=0.01))
L2 = tf.nn.relu(tf.matmul(L1, W2))
W3 = tf.Variable(tf.random_normal([256, 10], stddev=0.01))
model = tf.matmul(L2, W3)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=model, labels=Y))
total_batch = int(mnist.train.num_examples / batch_size)
for epoch in range(15):
total_cost = 0
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(batch_size)
_, cost_val = sess.run([optimizer, cost], feed_dict={X: batch_xs, Y: batch_ys})
total_cost += cost_val

29. https://youtu.be/5ttySH7CppA
Swift for TensorFlow MNIST Example

30. Which one is Next-generation platform?
Swift is Next-generation platform of TensorFlow
But, Python is still dominient.
Python and Swift are in a complement relationship

31. • Tensorflow github : https://github.com/tensorflow/
• Swift for Tensorflow github : https://github.com/tensorflow/swift
• Swift for Tensorflow : https://www.tensorflow.org/swift
• Swift for Tensorflow Design Overview : https://github.com/tensorflow/swift/blob/master/docs/
DesignOverview.md
• Why Swift for Tensorflow? : https://github.com/tensorflow/swift/blob/master/docs/
WhySwiftForTensorFlow.md
• Swift example : https://github.com/tensorflow/swift/blob/master/docs/site/tutorials/
model_training_walkthrough.ipynb
• https://github.com/bbvch13531/tf.swift