O'Reilly AI Meetup sponsored by IBM

1. @seldon_io
hello@seldon.io
bit.ly/SeldonCoreSlack

2. Machine Learning
Deployment for Enterprise.
O’Reilly AI for IBM: Deploying ML Models at Scale

3. We help people shape the
future by bringing
machine learning
deployment to life.

4. Meet the Team.
Alex Housley
CEO & Founder
Clive Cox PhD
CTO
Gurminder Sunner
VP Engineering
Lee Baker
Commercial Director
Giovanni Vacanti PhD
Machine Learning
Engineer
Inga Veidmane
Operations Manager
Andrew Turner
Sales Manager
Janis Klaise PhD
Data Scientist
Arnaud Van Looveren
Data Scientist
@seldon_io hello@seldon.io bit.ly/SeldonCoreSlack

5. Mapping a global network of Seldon
installs
Organisers of the UK’s largest applied
meetup focused on their AI/ML tools
- now over 2000 members. Supported
by Google.
Organic traction and
a global community.
● Over 120,000 installs (Docker Hub pulls) since Dec ‘17.
● Community across 90 countries and across industries.
● 21% MoM growth in clusters, 138% MoM growth in
nodes.
● 98% growth in nodes per cluster is an indicator that people
are using it for more demanding workloads.
bit.ly/SeldonCoreSlack

6. Seldon serves the world’s largest companies in
technology and financial services.
Applied data science project followed by managed service.
Enterprise Sales
- Recommend products
and supporting
documentation to
sales reps.
- Increase performance
of sales reps and
campaigns.
1% increase = £52m
revenue.
Credit Risk
- Reduce credit default
amongst UK Card
customers.
- Support the customer
before they miss
payments.
- Aiming to reduce
default rate by 2%.
FX Trading
- Identify which way
currency markets
are likely to move
within a short
timeframe.
- 5% uplift over
baseline model
worth estimate £3m
/ month.
Churn Prediction
- Improve customer
retention by
identifying customers
likely to cancel.
- Aiming to increase
accuracy of churn
prediction by 10%.
NA Bank

7. @seldon_io
4th Industrial
Revolution

8. Source: Notes from the AI frontier: Applications and value of deep learning - McKinsey April 2018 - https://mck.co/2I5tMld
A Small Number of Machine Learning Techniques work across
many use cases and industries.

9. Source: Notes from the AI frontier: Applications and value of deep learning - McKinsey April 2018 - https://mck.co/2I5tMld
Deep learning and Reinforcement learning deliver 62%
incremental value on average.

10. High value ML use cases
“$20.2bn machine learning budget in banking in 2018.”
McKinsey 2018
Fraud Detection Compliance & Risk Capital Markets Customer
Experience
Customer
Engagement
Identify fraudulent
behaviour
● First and third
party
● Identify bad
actors
● KYC
● Outlier detection
Reduce risks and
streamline reporting
● Audit trails
● Early warning and
response to cyber
breach
● Automate risk
controls
AI in the investment
sector
● Fundamental and
technical trading
strategies
● Market
predictions
● FX, equities and
derivatives
Identifying patterns of
behaviour
● Product
recommendation
● Customer
retention and
churn prediction
● Customer journey
Customized and intelligent
products & services
● Chatbots & virtual
agents
● Task management
● CRM
● Personalisation

11. Finance + tech
niche skill
sets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Regulatory
requirements
Auditability &
provenance
6 Machine Learning Trends Seldon observe

12. Finance + tech
niche skillsets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Regulatory
requirements
Auditability &
provenance
6 Machine Learning Trends in Financial Services
Evolving Standards
R Regulatory compliance a group effort.
ML without a ‘human in the loop’ will require
Explainable AI, or ‘XAI’
Expect to see compliance teams adopt ML
Possible driver of M&A within the banking
community

13. Finance + tech
niche skillsets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Regulatory
requirements
Auditability &
provenance
6 Machine Learning Trends in Financial Services
Data management
● R Transparency of data governance
Regulate the Algo - ‘Play the ball, not the man’
Cloud portability - *Euro bank example of conflict
Vital nature of successful Data strategies -
"We don’t have better algorithms, we just have more
data."
Peter Norvig, Google's director of research

14. Finance + tech
niche skillsets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Auditability &
provenance
6 Machine Learning Trends in Financial Services
Security
● R As Cloud becomes standard, data & training will
become increasingly separate
Initiatives like the CNCF* will ally with traditional
regulatory bodies
Models will be a hybrid of open (Risk, KYC) and
proprietary (Trading, Fraud)
*Cloud Native Computing Foundation https://www.cncf.io

15. Finance + tech
niche skillsets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Auditability &
provenance
6 Machine Learning Trends in Financial Services
Citizen ‘Data Scientist’
$120,000 - Data Scientist annual salary*
Best tool for the job - commoditization of ML frameworks
ForwardLane seeing portfolios of >200 under
management
Curation will be the key skillset
*Glassdoor Salary Estimates FSI

16. Finance + tech
niche skillsets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Regulatory
requirements
Auditability &
provenance
6 Machine Learning Trends in Financial Services
Speed to market
Cost & speed of compute is your new competitive
advantage
New Revenue models - Can you extract meaning from
your data?
Model iteration needs to be rapid and optimal

17. Finance + tech
niche skillsets
Low latency & high
data volumes
Integration with
legacy ecosystem
Security & data
governance
Regulatory
requirements
Auditability &
provenance
6 Machine Learning Trends in Financial Services
Legacy tech
Key value of ‘Challenger Bank’ community
Growth in outsourced banking engine
New IT landscape demands ‘plug & play’ architecture
Open Source adoption to escalate

18. 2018 Market Analysis
Machine Learning Deployment
● 25000 banks forecasted to spend $519bn
on IT in 2018 (Gartner)
● 26% early adopters of ML spent 15% of
their IT budget on ML
● $20.24bn ML budget in banking 2018
● 52% initial and broad ML implementation
● 71% Kubernetes adoption in
orchestration
● $7.47bn current ML budget for
Kubernetes users in banking

19. Seldon Core
Open platform for deploying
machine learning models

20. Seldon Focus
Seldon Core
Building
a
Model
Serving OptimizationMonitoringRoll-out
Data
Ingestion
Data
Analysis
Data
Transform
-ation
Data
Validation
Data
Splitting
Model
Validation
Trainer
Training
At Scale
Logging

21. Goal: Help Business succeed with Data Science
Data Scientist
● Analyzes the data
● Builds the predictive model
● Optimizes the model
DevOps Engineer
● Manages infrastructure
● Monitors the model in
production
● First response on issues
Business Manager
● Decides the project goals
● Defines business KPIs
● Evaluates ROI
● Provides Approval/Audits

22. Clive Cox
CTO, Seldon

23. Seldon Deploy
UI, Collaboration, Control, Audit
MAB
(Multi-Arm Bandits)
Outlier Detection Explanation
Seldon Core
(ML Control Plane)
80% of enterprises are hybrid or multi-cloud.
Over half use containers, of which 75% use Kubernetes.
Seldon Technology Stack
Cloud Native Tools : e.g., Ambassador, Argo, Istio
Bias Detection,
Concept Drift

24. Seldon Core : ML Tool Agnostic
Use any
ML Tool
Dockerise REST and gRPC
Handle
Seldon APIs
Deploy!
Deploy!

25. Seldon Core : ML Tool Agnostic
Any Language
Current wrappers incl:
● Python
● R
● Java
● NodeJS
Any ML Library
Current examples incl:
● SKLearn
● TensorFlow
● PyTorch
● H2O
Any ML Inference Engine
Current examples incl:
● Seldon wrapped inference code
● NVIDIA TensorRT Server
● TensorFlow Serving
● Intel nGraph

26. Title
Type Here Type Here Type Here Type Here
Seldon Core Wrappers : Source-to-Image
https://github.com/openshift/source-to-image
from sklearn.externals import joblib
class IrisClassifier(object):
def __init__(self):
self.model = joblib.load('/mnt/model/IrisClassifier.sav')
def predict(self,X,features_names):
return self.model.predict_proba(X)
User Source Code (Github/Local) Seldon Builder Images
seldonio/seldon-core-s2i-python2
seldonio/seldon-core-s2i-python3
seldonio/seldon-core-s2i-r
seldonio/seldon-core-s2i-java-build
seldonio/seldon-core-s2i-nodejs
● ./assemble
● ./run
● ./usage
Dependencies
S2I
User Image
Ready to deploy on seldon-core
#> s2i build .

27. Rich Processing Components
● Routing requests
○ AB Tests
○ Multi-Armed Bandit
● Transformations
○ Feature Normalization
○ Ensembles results
● Metrics
○ Concept drift
○ Outlier detection
Seldon Core : Powerful Inference Graphs
Flexible
● Custom
○ Build your own
○ 3rd Party
● Dynamic
○ Change graph at inference
time
● Pluggable
○ Reuse components in
different inference graphs

28. Title
Type Here Type Here Type Here Type Here
Seldon Core Inference Graphs
Seldon/3rd party
component
Model C
Model A
Feature
Transformation
Client component
API
(REST, gRPC)
Model B
A/B Test
Multi Armed
Bandit
Direct traffic to
the most optimal
model
Outlier
Detection
Key features to
identify outlier
anomalies (Fraud, KYC)
Explanation
Why is the model doing
what it’s doing?

29. Title
Type Here Type Here Type Here Type Here
Seldon Core Example Inference Graph
Seldon/3rd party
component
Tensorflow
model served
via NVIDIA
TensorRT Server
ONNX model
served via
Intel nGraph
Feature
Transformation in
Java
Client component
API
(REST, gRPC)
R Model
A/B Test
Multi Armed
Bandit
(e-greedy,python)
Outlier
Detection
Custom Router
using business
logic

30. Model Explanation
For example, a machine learning model flags a bank transaction as fraudulent;
it becomes necessary to explain what factors contributed to the decision.
Explaining complex "black box" models is a key problem.
Understanding why a model output a given prediction can….
1. Provide insight into the model
behaviour. Guide decision making
processes.
2. Facilitate compliance with
regulation.
Lime
● Locally interpretable
● Suited for images and text
Representation erasure
● Suited for run time explanations
● Return features importance for any
“black box” model
● Based on how much the model
prediction will change when then
input features are changed
Lime2
● Updated version of Lime

31. Title
Type Here Type Here Type Here Type Here
Multi-Armed Bandits
Model C
Model AAPI
Model B
A/B Test
Multi Armed
Bandit
Direct traffic to
the most optimal
model
● Deploy several models in parallel
● Optimize between models at inference time
● Generalizes A/B testing
● Exploration vs Exploitation
○ ε-greedy
○ Thompson Sampling
● State dependency
○ Contextual bandits

32. ● Outlier Detection
○ Identify data anomalies
○ Measure the distance of an observation to the center of the distribution
○ Online learning, starts without knowledge of the feature distribution
● Concept Drift
○ Change in the relationship between input and output leads to lower prediction accuracy
○ Detection of drift allows to adjust the predictive models in time
● Bias Detection
○ Human bias in the data like gender bias or machine bias leads to biased predictions
○ Neutralizing bias avoids pitfalls of human decision making
Monitoring Machine Learning Models

33. Seldon Core : Production Deployment Options
Update Inference Graphs
● Rolling
● Canary
● Blue-Green*
● Shadow*
Resource Control
● GPUs, CPUs, Memory
● Data Volume Attachment
● Sidecar containers
○ DBs, Other Business logic
● Separate k8s deployments for
parts of graph
* roadmap

34. Example Seldon Deployment Manifest (Kubernetes Custom Resource)
Graph Definition
Pod Specification
Replicas
List of predictors

35. Title
Type Here Type Here Type Here Type Here
1. Package
Create REST or gRPC
dockerized microservice
2. Describe Deployment
Create/update Kubernetes
resource manifest for
deployment graph
3. Deploy
Manage and analyze the
performance of live deployments
Seldon Deployment Workflow

36. Title
Type Here Type Here Type Here Type Here
Get Started with Seldon Core
1. Install
a. https://github.com/SeldonIO/seldon-core
b. Helm Charts or Ksonnet Registry
c. Via GCP MarketPlace
d. Via Kubeflow
e. Via IBM FfDL
2. Run example charts
a. Helm :
https://github.com/SeldonIO/seldon-core/blob/master/notebooks/helm
_examples.ipynb
b. Ksonnet:
https://github.com/SeldonIO/seldon-core/blob/master/notebooks/kson
net_examples.ipynb
c. Provides example graphs - models, AB tests, multi-armed bandits,
Outlier detections
3. Customize to your models
a. Wrap components using S2i
b. Create your own graphs

37. Seldon Deploy
Machine Learning Deployment
for Data Science Teams.
Development in progress
Pilots Q4 2018, launch early 2019
>500 companies applied for beta
● Modern UX/UI for model management
● Manage multiple Seldon Core clusters
● Team workflows, approvals and audit trails
● Advanced experiments and CI/CD (GitOps)
● Explanations and compliance (GDPR)

38. Thank You.
Please get in touch with any further questions.
@seldon_io
hello@seldon.io
http://bit.ly/SeldonCoreSlack

39. Title
Type Here Type Here Type Here Type Here
Seldon Core Architecture
Data scientists,
engineers and
managers
Deployment Controller
(kubectl, CI/CD, Seldon Deploy)
Business
Applications
Pluggable
Authentication
REST API or gRPC
Kubernetes clusters
running Seldon Core
Service
Orchestrator
Kubernetes
API
Operator
1. N deployment
graphs
Reverse Proxy
(Ambassador)
Public docker
registry
Client docker
registry
Seldon docker
registry