Best practices to start, develop and ship a data science product

1. Data Science @ PMI
Tools of The Trade
Best Practices to Start, Develop and Ship a Data Science Product
Manuel Valverde
Tokyo WebHack, 17th January 2019

2. • PhD.@Granada U. Spain: Physics modelling and MC simulations for
SuperKamiokande
• PostDoc@Osaka U. Osaka: Nuclear Structure Calculations. Think Gaussian
processes
• DataScientist@Rakuten, Tokyo: Search Relevancy for e-commerce
• DataScientist@PMI, Tokyo: Fraud prevention
2
About Me

3. About Philip Morris International
3
• Founded in 1847
• No. 108 in the 2018 Fortune 500
• 80,000 employees, 180+ markets, 150M consumers
• 6 of the world's top international 15 brands, including
Shifting from combustible cigarettes to smoke-free, reduced risk products (RRP)
https://www.pmi.com/smoke-free-products

4. • We are part of PMI's Enterprise Analytics and Data (EAD) group
• 40+ Data Scientists across 4 hubs
• Offices in Amsterdam (NL), Kraków (PL), Lausanne (CH) and Tokyo (JP)
• Profiles
• Education: 30% PhD, 70% MSc/BSc
• Data Science Experience: 7.4 yrs on average
• Experience in PMI: 88% under 2yrs
• Expertise in Machine Learning, Big Data Engineering, Insights Communication
• SCRUM certified (Professional Scrum Developer)
4
Data Science @ PMI

5. 5
2 Labs
LA
2 Labs
North
America
Add 1
Lab
EU
2 Labs
EE
Add 2
Labs
Asia
A
( Data x Science x Communication ) = Insight
Data is only one part of the equation. We bring the scientific method. It materializes in the analytical code we
write. It is as valuable as the data itself.
B We are business driven
Whatever we do, it contributes to the business. We are diligent about making an impact.
C
We invest in people
We invest in the ability to ask questions. It can’t be achieved with tools only. Tools are for generating answers,
but questions are posed by people.
D
We self-organize
We choose coordination & cultivation over command & control. We believe this approach allows for the best
solutions to emerge.
E We iterate and improve
We embrace lean development, we learn from mistakes and we do it together with business.
F We co-create
Data insights ecosystem requires collaboration among all parties. We want to be active contributors.
Data Science Principles @ PMI

6. 6
PMI’s Data Ocean

7. Why are we here?
Because a data scientist is not just someone who knows more statistics than a programmer
Data Science is Software.
The product of a Data Science effort (a Model or a Report)
is essentially a small but critical part of a large,
sophisticated business software. Data Products must
therefore be designed to play well with systems up- and
downstream.
Remember that the system can work without a model, but
a model is pretty much worthless without the system.

8. Writing code for implementing machine learning
algorithms is getting easier every year.
Building a scikit-learn Pipeline to implement a Random
Forest model with GridSearch is less than twenty lines of
code today. AutoML is around the corner.
We need to acknowledge and understand two things:
 The code, or even the model is not our end-goal.
 We're in the business of building intelligent
applications,
or data products.
Why are we here?
Because a data scientist is not just someone who knows more programming than a statistician

9. 9
• Obtain
connect to DBs,
download flat files
• Scrub
outliers/missing data,
aggregations
• Explore
statistical analysis,
feature engineering
• Model
learning algorithms,
parameter optimization
• INdustrialize
reports, APIs
ExploratoryProduction
Smart
Application
An OSEMN Data Science Process
Explore, Model, Iterate.
Create a Data Product.

10. 10
We define a data product as a system that
 takes raw data as input, 📲
 applies a machine-learned model to it, 🤖
 produces data as output to another system 💻
Additionally, a data product must
 be dynamic and maintainable,
allowing periodic updates 🏃
 be responsive, performant and scalable 👨👨👦👦
What is a Data Product? 🤔
In a nutshell, it’s a software product with an ML Engine.
Examples
Amazon’s Product Recommendation Engine
LinkedIN’s “People You May Know”
Autonomous Vehicles
The Classic Data Science Workflow
Data Product Development Workflow

11. 11
Challenges in Data Product Development 🤔
“Team programming isn’t a divide and conquer problem.
It is a divide, conquer, and integrate problem.”
1. The Process
Infrastructure Setup > Code > Build > Test >
Package > Release > Monitor
2. The Team
Cross-functional group of businesspeople, data
scientists, engineers and developers.
3. The Challenge
As an example, consider we have 2 groups,
 Team A consists of data engineers and
scientists and works on the Prediction Engine.
👨🤖💻👨🤖🎓
 Team B consists of software engineers and
front-end developers working on the UI. 👨🤖🎨
👨🤖🔬
The goal is that every piece in the product should
integrate well into a larger codebase. 🍻

12. 12
Continuous Integration (CI), Delivery (CD) and
DeploymentDevelopment practices for overcoming integration challenges and moving faster to delivery
The CI/CD Cycle
 Continuous Integration requires multiple developers to
integrate code into a shared repository frequently.
Requested merges are automatically tested and
reviewed.
 Enabled by git-flow, code standards and
automated testing
 Continuous Delivery makes sure that the code that we
integrate is always in a deploy-ready state.
 Enabled by agile (iterative) methods,
testing and build automation
 Continuous Deployment is the actual act of pushing
updates out to the user – think of your iPhone apps or
Desktop browser that prompt for updates to be installed
periodically.

13. 14
The Role of Data Scientists
Learn best practices to contribute effectively to data products
Write code that is
 Readable,
so others can understand and add to it
 Testable
so others can verify it does what it advertises
and integrate it into their work
 Reusable
so it may be included in other projects
 Reproducible
uses libraries/packages that are available on
production environments
 Usable
don’t write code in SAS or R,
most engineers don’t speak those languages.
Joel’s Tests
 Do you use source control?
 Can you make a build in one step?
 Do you make daily builds?
 Do you have a bug database?
 Do you fix bugs before writing new code?

14. 15
Data Science Best Practices @ PMI
Python
Style
Guides
Notebooks
to Modules
Testing
Code
Reviews
Docker
Virtual
Environments
Version
Control
Project
Templates

15. 16
Data Science Best Practices @ PMI
Python
Style
Guides
Notebooks
to Modules
Testing
Code
Reviews
Docker
Virtual
Environments
Version
Control
Project
Templates

16. Agile Data Science Workflow

17. Our building blocks
Ocean Components

18. To create a workflow that is …
Our Vision
• Flexible
Adapts to specific needs of every use-case
Accommodates changing requirements
• Inspection
Transparency at all times
Artifacts can be audited at any time.
• Reproducible
Out-of-the-box dependency management
No more ‘But-it-works-on-my-machine’ or ‘Please-industrialize-this-
model’
• Easy to use
Frictionless development experience
Freedom to experiment
🔥

19. Some things we always need to be mindful of.
Our Principles
 Sensitive Data must never leave the Ocean
 Restricted Open-Source libraries must be avoided
 Every use-case must be industrialization-ready

20. DS Prod Lab
Scanned by BlackDuck
Automation
On-demand
infrastructure
Data Read/Write
Data Product
Reproducible Containers
Version Control
System Architecture
The dots, connected.

21. We organize our workflow in 3 phases – Start, Develop and Ship
3 Steps to a Data Product
• Get Infrastructure
• DS Prod Lab
• Docker Container
• Python Environments
• Get Data
• Flat Files
• Database Connections
• Get Code
• Project repo
• Cookiecutter template
• Start Docker container
• Check out a Branch
• For each task in OSEMN,
write
Exploratory code in NBs,
• Standard Code Styles
• Documentation, Tests
• Maintain
dependencies
• Refactor into Modules
• Push
• Review, Merge
• Package Python code,
publish to PyPi on
Artifactory
• Persist models
• Build an API to industrialize
the model.
• Provide endpoints for
app-health checks.
• Set up Jenkins pipeline for
continuous integration
• Plan for the next iteration
Start Develop Ship

22. For Reproducibility
Docker Containers

23. Docker for Containerized Data Science
All your dependencies in one place.
Code guaranteed to run anywhere.
A container is a lightweight, stand-alone package of a software that
includes everything needed to run it: code, runtime, system tools,
system libraries, settings.
Containerized software will always run the same, regardless of the
environment.
Benefits for Data Scientists
 Freedom, install all your favorite tools and libraries
 Ease of installation, set up your toolbox once and it will always work
 Reproducibility and Portability,
your development environment can be reproduced anywhere
 Isolation, your Py2 setup doesn’t mess up your Py3 setup, installing
a new library doesn’t mess up system Python
 Speed, get up and running in minutes with images optimized for
specific applications like time-series analysis or deep-learning.

24. For organization and predictability
Project Templates

25. CookieCutter
Everything has a place and a purpose
The idea is borrowed from popular web-frameworks like Rails and Django
where each developer uses the same template when starting a new project.
This makes it easier for everyone on the team to figure out where they
would find or put the various moving parts.
We will use a standard project skeleton tailed for data science projects so
that every scientist knows where to put their code, notebooks, data, models,
figures and references.
Benefits of a standardized directory structure:
 allows people to collaborate more easily
 empowers reproducible analysis
 enforces a "data as immutable" design philosophy
Cookiecutters help us generate this folder structure automatically.

26. CookieCutter
The standard folder structure enforces a design philosophy for faster delivery
Treat Data as Immutable
Raw data should be stored inside /data/raw and should never be modified
by hand. The code you write should ingest the data from /raw and cleaned
or processed data should be written to /processed.
Reproducibility
Everyone on the team should be able to reproduce your analysis with
 the code in src/
 the data in data/raw/
 the dependencies in Dockerfile, requirements file
Notebooks for Exploration, Scripts for Production Code
Jupyter is great for exploratory analysis, but quite challenging for version
control (they're stored as json files.) Once your code works well, move it
from notebooks/ to src/ and package the functions and classes into
modules.

27. For being deploy-ready
Moving code from
Notebooks to Source Code

28. Notebooks for Exploration. Files for Production.
The case against Notebooks
 The main cause of unmaintainable code and bad structure in Data Science is the mixing
of exploratory "throw away" code with production code. Notebooks are being used to write
code that ultimately would be deployed in production.
 This is not what notebooks where invented for;
they are essentially browser-based shells and presentation tools with charts and code
blocks.
 Notebooks do not have refactoring tools, code structuring tools and are
notorious for version control management.
Motivation for Organizing Code
 Extract text and plots from notebooks into Markdown Reports for a business audience
 Notebooks with minimal code and clear narrative can be used as Technical Reports
 Move the core functionality into Python modules to speed up subsequent exploration

29. In the exploratory phase,
the code base is expanded through data analysis, feature
engineering and modelling.
In the refactoring phase,
the most useful results and tools from the exploratory phase are
translated into modules and packages.
The Production Codebase grows across sprints.

30. For integration and deployment
Automated Testing

31.  If your code is not performing as expected, will you
know?
 If your data are corrupted, do you notice?
 If you re-run your analysis on different data,
are the methods you used still valid?

32. Automated Testing
“Why do most developers fear to make continuous changes to their code? They are afraid they’ll break
it!
Why are they afraid they’ll break it? Because they don’t have tests!”
Two Types of Tests useful for DS
 Unit Testing to make sure individual pieces of code work
 Integration Testing to make sure your code works with everyone else's
Challenge with writing Tests for Data Science
For most software, the output is deterministic - a function for averaging numbers can be
Unit tested with a simple function that checks if result is accurate. You can then check your
changes in, and Integration tests can run against the new build with a fabricated set of
results to ensure that everything works as expected.
But not so with Data Science work – the output is probabilistic.
You can't always put in a 2 and 4 and expect a 3 to come out.

33. Automated Testing for Data Science
 First, implement a Unit Test framework within your code; use pytest or nose
 In some cases, you can set a deterministic value like number of rows or the
expected data type from a function, and write a test for it.
 But if you can't - pick the performance metric (p-value, F1-score, or AUC, etc.)
and check if it lies within an acceptable range.
Test-Driven Development (TDD)
First the developer writes an (initially failing)
automated test case that defines a desired
improvement or new function, then produces the
minimum amount of code to pass that test.” So, before
actually writing any code, you should write your tests.
All tests should go into the tests/ subdirectory of the
specific package. Write tests in three steps
 Get/make the input data
 Manually construct the result you expect
Compare the actual result
to the expected correct result

34. In Conclusion

35.  Engineering smart systems around a machine-learned
core is difficult
 It requires teams of exceptionally talented individuals to
work together.
 What makes data scientists special is their ability to work
with both business leaders and technology experts.
 We must acknowledge that we are a part of something
much bigger and learn to play well with each other and
with all parties involved.
Our hope is that these systems, principles and best
practices will help you take the first steps in that direction

36. Questions?