Monitoring of GPU Usage with Tensorflow Models Using Prometheus

MONITORING OF GPU USAGE
WITH TENSORFLOW MODEL TRAINING USING PROMETHEUS
Diane Feddema, Principal Software Engineer
Zak Hassan, Senior Software Engineer
#RED_HAT #AICOE #CTO_OFFICE

YOUR SPEAKERS
DIANE FEDDEMA
PRINCIPAL SOFTWARE ENGINEER - ARTIFICIAL INTELLIGENCE CENTER OF EXCELLENCE, CTO OFFICE
● Currently focused on developing and applying Data Science and Machine Learning techniques for performance
analysis, automating these analyses and displaying data in novel ways.
● Previously worked as a performance engineer at the National Center for Atmospheric Research, NCAR, working on
optimizations and tuning in parallel global climate models.
ZAK HASSAN
SENIOR SOFTWARE ENGINEER - ARTIFICIAL INTELLIGENCE CENTER OF EXCELLENCE, CTO OFFICE
● Leading the log anomaly detection project within the aiops team and building a user feedback service for improved
accuracy of machine learning predictions.
● Developing data science apps and working on improved observability of machine learning systems such as spark and
tensorflow.
#RED_HAT #AICOE #CTO_OFFICE

Outline
● Story
● Concepts
○ Comparing CPU vs GPU
○ What Is Cuda and anatomy of cuda on kubernetes
○ Monitoring GPU and custom metrics with pushgateway
○ TF with Prometheus integration
○ What is Tensorflow and Pytorch
○ A Pytorch example from MLPerf
○ Tensorflow Tracing
● Examples:
○ Running Jupyter (CPU, GPU, targeting specific gpu type)
○ Mounting Training data into notebook/tf job
○ Uses of Nvidia-smi
● Demo
○ Running Detectron on a Tesla V100 with Prometheus & Grafana
monitoring

“Design the factory like you
would design an advanced
computer… In fact use
engineers that are used to doing
that and have them work on
this.”
-- Elon Musk (2016)
https://youtu.be/f9uveu-c5us
Source: https://flic.kr/p/chEftd

• unlocking
phones
WHY IS DEEP LEARNING A BIG
DEAL ?
MobileOnline
• Netflix.com
• Amazon.com
• Targeted ads
Automotive
• self driving
• voice assistant

Source: https://bit.ly/2I8zIcs

Source: https://bit.ly/2HVCaUC

PARALLEL PROCESSING
MOST LANGUAGES
SUPPORT
● MODERN HARDWARE SUPPORT
EXECUTION OF PARALLEL
PROCESSES/THREADS AND HAVE APIS
TO SPAWN PROCESSES IN PARALLEL
● YOUR ONLY LIMITS IS HOW MANY CPU
CORES YOU HAVE ON YOUR MACHINE
● CPU USED TO BE A KEY COMPONENT OF
HPC
● GPU HAS DIFFERENT ARCHITECTURE &
# OF CORES
CPU
INSTRUCTION
MEMORY
DATA
MEMORY
Input/Output
ARITHMETRIC
LOGIC UNIT
CONTROL
UNIT

Hardware accelerators
● GPU
○ CUDA
○ OpenCL
● TPU

Performance Goals
Latency
Decreased
Bandwidth
Increased
Throughput
Increased

WHAT IS CUDA?
PROPRIETARY TOOLING
● hardware/software for HPC
● prerequisite is that you have nvidia cuda supported graphics cards
● ML frameworks like tensorflow, theanos, pytorch utilize cuda for leveraging
hardware acceleration
● You may get a 10x faster performance for machine learning jobs by utilizing
cuda

ANATOMY OF A CUDA
WORKLOAD ON K8S
TENSORFLOW
CUDA LIBS
CONTAINER RUNTIME
NVIDIA LIBS
HOST OS
SERVER
/dev/nvidaX
GPU
CONTAINER
HARDWARE
JUPYTER

Cli monitoring tool
Nvidia-Smi
● Tool used to display
usage metrics on
what is running on
your gpu.

TFJob + Prometheus
PROMETHEUS
ALERT
MANAGER
PULL
PUSH
PUSH
GATEWAY
NOTIFICATION
EMAIL
MESSAGING
WEBHOOK
TENSORFLOW
JOBS
TRAINING
DATA
GPU NODE
EXPLORER

Idle GPU Alert
● Alert Manager can
notify:
○ slack chat notification
○ email
○ web hook
○ more
● Get notified when your
GPU isn’t being utilized
and shut down your
VM’s in the cloud to
save on cost.
groups:
- name: nvidia_gpu.rules
rules:
- alert: UnusedResources
expr: nvidia_gpu_duty_cycle == 0
for: 10m
labels:
severity: critical
annotations:
description: GPU is not being utilized you
should scale down your gpu node
summary: GPU Node isn't being utilized

Jupyter +TF on CPU
apiVersion: v1
kind: Pod
metadata:
name: jupyter-tf-gpu
spec:
restartPolicy: OnFailure
containers:
- name: jupyter-tf-gpu
image: "quay.io/zmhassan/fedora28:tensorflow-cpu-2.0.0-alpha0"

Jupyter+TF on GPU
apiVersion: v1
kind: Pod
metadata:
spec:
restartPolicy: OnFailure
containers:
image: "tensorflow/tensorflow:nightly-gpu-py3-jupyter"
resources:
limits:
nvidia.com/gpu: 1

Specific GPU Node Target
apiVersion: v1
kind: Pod
metadata:
spec:
containers:
image: "tensorflow/tensorflow:nightly-gpu-py3-jupyter"
resources:
limits:
nvidia.com/gpu: 1
nodeSelector:
accelerator: nvidia-tesla-v100

Relabel kubernetes node
kubectl label node <node_name>
accelerator=nvidia-tesla-k80
# or
kubectl label node <node_name>
accelerator=nvidia-tesla-v100

Mount Training Data
AzureDisk
GlusterFS
NFS
AzureFile
Gce Persistent Disk
Aws Elastic Block
Storage
CephFS
… more

Persistent Volume Claim
● Native k8s resource
● lets you access pv
● can be used to share
data cross different
pods.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 100Gi

Persistent Volume
● native k8s resource
● can be readonly,
readWriteOnce or
readwritemany
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
nfs:
server: 0.0.0.0
path: "/"

Mounting Training Data
● use persistent
volume claims to
access your data
● in this example we
us nfs but you can
choose another
type.
apiVersion: v1
kind: Pod
metadata:
name: jp-notebook
spec:
containers:
- name: jp-notebook
image: tensorflow/tensorflow:nightly-gpu-py3-jupyter
volumeMounts:
- name: my-pvc-nfs
mountPath: "/tf/data"
volumes:
- name: my-pvc-nfs
persistentVolumeClaim:
claimName: nfs

Additional Tips
● Kubernetes doesn’t support sharing gpu’s
● If your running in cloud you should look at
stopping your VM if there is no workloads
being used. Restart it when you need it. The
costs can add up.
● Use volumes to mount your data for training
and share it across your environment

Monitoring and Performance
of ML on GPUs
● Benchmarking ML on GPUs
○ Monitoring
○ Performance
● Example using MLperf together with Prometheus
and Grafana
● Computing requirements & why GPU’s for ML

Why do we need gpus to
solve these problems
● Neural Networks rely heavily on floating point matrix
multiplication
● These algorithms also require a lot of data to train
large memory (GBs) and high speed networks to
complete in a reasonable amount of time
● Faster Deep Learning training

Nvidia DGX-2
GPUGPU GPU GPU GPU GPU GPU GPU
DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM
DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM
GPUGPUGPUGPUGPUGPUGPUGPU
Source: Nvidia
V100V100 V100V100 V100 V100V100V100
V100V100 V100V100 V100 V100V100V100

Benchmarks in MLPerf
Application
Area
Vision Language Commerce
Reinforcement
Learning
Problem
Image classification
Object Detection (light weight and
heavy weight)
Translation Recommendations
Games
Go
Datasets
ImageNet
COCO
WMT
English-German
MovieLens-20M Go
Models
ResNet-50
Detectron
Transformer
OpenNMT
Neural Collaborative
Filtering
Mini Go
Metrics COCO mAp
Prediction accuracy
BLEU Prediction Accuracy
Prediction accuracy
Win/Loss

MLPerf Project Sponsors
University research contributors
Industry contributors

What is Tensorflow?
● Open source Python library used to implement
deep neural networks (released from Google in
2015)
● A machine learning framework
● Tools to write your own models in Python,
JavaScript or Swift
● Collection of datasets ready to use with tensorflow
● TF run in Eager and Graph mode
● TF can run on CPUs or GPUs

What is Pytorch?
● Python-based open source deep learning library
● Used to build Neural Networks
● Replacement for NumPy for use with GPUs
● Can run on CPUs or GPUs
● Uses GPUs to accelerate numerical computations
● Pytorch performs computations

85,000 Images
Identify 91 objects
Source: Cornell Project
COCO Dataset

MLPerf Results
[c
Source: Nvidia Developer News Dec 2018

MLPerf Results - Single Node
[c
Source: Nvidia Developer News Dec 2018

How to monitor gpus with
nvidia-smi
$ nvidia-smi
--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.
link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,ut
ilization.memory,memory.total,memory.free,memory.used
--format=csv -l 5

Monitoring GPUs with nvidia-smi$ nvidia-smi
--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gp
memory,memory.total,memory.free,memory.used --format=csv -l 5
2019/04/17 14:41:35.223, Tesla V100-SXM2-32GB, 00000000:06:00.0, 418.40.04, P0, 3, 3, 44, 100 %, 0 %, 32480 MiB, 24052 MiB, 8428 MiB
2019/04/17 14:41:35.227, Tesla V100-SXM2-32GB, 00000000:0A:00.0, 418.40.04, P0, 3, 3, 47, 100 %, 0 %, 32480 MiB, 15773 MiB, 16707 MiB
2019/04/17 14:41:35.229, Tesla V100-SXM2-32GB, 00000000:0B:00.0, 418.40.04, P0, 3, 3, 43, 100 %, 0 %, 32480 MiB, 14363 MiB, 18117 MiB
2019/04/17 14:41:40.241, Tesla V100-SXM2-32GB, 00000000:0B:00.0, 418.40.04, P0, 3, 3, 43, 100 %, 1 %, 32480 MiB, 14363 MiB, 18117 MiB
timestamp
pstate
driver_versionpci.bus.id
pcie.link.gen.current
utilization GPU [%]
memory.used [MB]
memory.free [MB]
memory.total [MB]
utilization memory [%]
temperature GPU
pcie.link.gen.max
name

How to log nvidia-smi metric
data (long/short term logging)
[cephagent@asgnode021 object_detection]$ nvidia-smi --query-gpu=index,timestamp,power.draw,clocks.sm,clocks.mem,clocks.gr
--format=csv
index, timestamp, power.draw [W], clocks.current.sm [MHz], clocks.current.memory [MHz], clocks.current.graphics [MHz]
0, 2019/04/17 15:25:33.862, 68.71 W, 1530 MHz, 877 MHz, 1530 MHz
1, 2019/04/17 15:25:33.865, 77.53 W, 1530 MHz, 877 MHz, 1530 MHz
2, 2019/04/17 15:25:33.868, 74.54 W, 1530 MHz, 877 MHz, 1530 MHz
3, 2019/04/17 15:25:33.870, 146.91 W, 1530 MHz, 877 MHz, 1530 MHz
4, 2019/04/17 15:25:33.873, 143.57 W, 1530 MHz, 877 MHz, 1530 MHz
5, 2019/04/17 15:25:33.875, 76.06 W, 1530 MHz, 877 MHz, 1530 MHz
6, 2019/04/17 15:25:33.878, 77.58 W, 1530 MHz, 877 MHz, 1530 MHz
7, 2019/04/17 15:25:33.881, 74.15 W, 1530 MHz, 877 MHz, 1530 MHz

Tensorflow Tracing
import tensorflow as tf
import numpy as np
from tensorflow.python.client import timeline
shape = (5000, 5000)
device_name = "/gpu:0"
random_matrix = tf.random_uniform(shape=shape, minval=0, maxval=1)
random_matrix2 = tf.random_uniform(shape=shape, minval=0, maxval=1)
dot_operation = tf.matmul(random_matrix, tf.transpose(random_matrix2))
with tf.Session() as sess:
# add options to trace the session execution
options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
result = sess.run(dot_operation, options=options, run_metadata=run_metadata)
print(result)
# Create the Timeline object and write it to a json file
fetched_timeline = timeline.Timeline(run_metadata.step_stats)
chrome_trace = fetched_timeline.generate_chrome_trace_format()
with open('timeline_01.json', 'w') as f:
f.write(chrome_trace)

Monitoring of GPU Usage with Tensorflow Models Using Prometheus

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Monitoring of GPU Usage with Tensorflow Models Using Prometheus

Similar to Monitoring of GPU Usage with Tensorflow Models Using Prometheus (20)

More from Databricks

More from Databricks (20)

Recently uploaded

Recently uploaded (20)

Monitoring of GPU Usage with Tensorflow Models Using Prometheus