This document provides an overview of PyTorch, describing it as an ndarray library with GPU support and automatic differentiation that can be used for deep learning and reinforcement learning applications. It also discusses PyTorch's NumPy-like tensor operations, seamless GPU integration, neural network and optimization packages, and its ecosystem of complementary libraries for tasks like probabilistic programming, Gaussian processes, and natural language processing.
3. What is PyTorch?
Ndarray library
with GPU support
automatic differentiation
engine
gradient based
optimization package
Deep Learning
Reinforcement Learning
Numpy-alternative
Utilities
(data loading, etc.)
19. Distributed PyTorch
• MPI style distributed communication
• Broadcast Tensors to other nodes
• Reduce Tensors among nodes
- for example: sum gradients among all nodes
20. Distributed Data Parallel
for epoch in range(max_epochs):
for data, target in enumerate(training_data):
output = model(data)
loss = F.nll_loss(output, target)
loss.backward()
optimizer.step()
21. Distributed Data Parallel
for epoch in range(max_epochs):
for data, target in enumerate(training_data):
output = model(data)
model = nn.DistributedDataParallel(model)
loss = F.nll_loss(output, target)
loss.backward()
optimizer.step()
22. P Y T O R C H 1 . 0
Distributed Training Performance – ResNet101
0
1
2
3
4
5
6
7
8
9
1 Node (8 GPUs) 2 Nodes (16 GPUs) 4 Nodes (32 GPUs) 8 Nodes (64 GPUs)
Speedups
ResNet-101 on NVIDIA V100 GPUs
100 Gbit TCP 4 x 100Gbit Infiniband Ideal Speedup
23. Use via DataBricks MLFlow
•mlflow.pytorch
- saves and loads models
•More resources:
- https://docs.databricks.com/spark/latest/mllib/mlflow-pytorch.html
- https://www.mlflow.org/docs/latest/models.html
36. fast.ai 1.0
• High-level library on PyTorch: http://docs.fast.ai
• Built by Jeremy Howard, Rachel Thomas and many
community members
37. fast.ai 1.0
• High-level library on PyTorch: http://docs.fast.ai
• Built by Jeremy Howard, Rachel Thomas and many
community members
• an online course accompanies the library
38. fast.ai 1.0
• High-level library on PyTorch: http://docs.fast.ai
• Built by Jeremy Howard, Rachel Thomas and many
community members
• an online course accompanies the library
• Read more at http://www.fast.ai/2018/10/02/fastai-ai/
41. fast.ai 1.0
• state-of-the-art models in few lines
• fine-tune on your own data
data = data_from_imagefolder(Path('data/dogscats'),
ds_tfms=get_transforms(), tfms=imagenet_norm, size=224)
learn = ConvLearner(data, tvm.resnet34, metrics=accuracy)
learn.fit_one_cycle(6)
learn.unfreeze()
learn.fit_one_cycle(4, slice(1e-5,3e-4))
Near State-of-the-art Image Classifiers
42. fast.ai 1.0
Models and Transforms for Tabular Data
• state-of-the-art models in few lines
• fine-tune on your own data