Scalable Machine Learning solution for email targeting

1. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics
Modoop
Scaling Machine Learning for Marketing at LinkedIn
Yan Liu
TDWI Solution Summit San Diego 2015

2. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 2
Yan Liu
Manager, Business Analytics Data Mining
LinkedIn Corporation
https://www.linkedin.com/in/yanliu7

3. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 3
Members
360,000,000
+

4. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 4
LinkedIn for members
The professional profile
of record
Connect all of the
world’s professionals
The definitive
professional
publishing platform
Identity Network Knowledge

5. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 5
LinkedIn for customers
Enable passive
recruiting at
massive scale
Identify and engage
professionals with
relevant content
Transform cold
calls into warm
prospects
Hire Market Sell

6. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 6
Critical mass of dataRelevant and valuable
products and services
Technology
platform
Member growth and engagement
LinkedIn Business Model & why analytics is important

7. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 7
Overview of Business Analytics Team
Biz Analytics
Engineering
Global
Sales
Organization
Operations
Sales Ops,
Ad Ops,
Biz Ops
Product
Global
Customer
Organization
Marketing
Talent
Solutions
Marketing &
Sales
Solutions
Premium
Subscriptions
Consumer
Marketing

8. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 8
Email Marketing

9. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 9
Premium Subscriptions

10. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 10
Identify Potential Subscribers
300M+ Members
Qualified Members
Potential Subscribers

11. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 11
Identity
Data
Social
Data
Behavioral
Data
DM2 - Data Mart for Data Mining

12. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 12
4-6 weeks
Propensity
Score
Ground Truth
Model
Validation
Feature
Engineering
Features
(DM2)
Scoring
Model
Development
Data Partition
training/validatio
n/(test)
Model Selection
Ground Truth
for Testing
Feature
Engineering
Propensity Model Workflow

13. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics
Model
week 1 week 6
13
Photo credit: asmfoto Marcell Mizik, photo license with Depositphotos File Purchase Agreement #41549281

14. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 14
Model Refreshment
Initial Model
Baseline
Model
Champion
Model
Challenger
Model
Performance
A/B Test
Winning
Model

15. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 15
Challenger
Baseline
Champion
Time
Performance
GAIN
Performance Gain

16. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 16
Model
Refreshment
>> 4-6 weeks
Propensity
Score
Ground Truth
Model
Validation
Feature
Engineering
Features
(DM2)
Scoring
Model
Development
Data Partition
training/validatio
n/(test)
Model Selection
Ground Truth
for Testing
Feature
Engineering
Propensity Model Workflow

17. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 17
Who will most likely buy Recruiter Lite/Sales Sub/Job Seeker?
Who will most likely edit Profile?
Can we re-build models to include NEW features?
How about segmentation models by GEO/Industry/etc.?
Can I (non data mining person) build predictive models by
myself?
Scale
Up
Increasing Demands

18. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics
Input
Output
Input
Input
Input
Input
18
Propensity
Score
Ground Truth
Model Validation
Feature
Engineering
Features
(DM2)
Scoring
Model
Development
Data Partition
training/validation
/(test)
Model Selection
Ground Truth
for Testing
Feature
Engineering
Model
Refreshment
Modoop
2-3 days

19. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 19
Modoop
Hadoop/Spark
DM2 (Data Mart for
Data Mining)
Application 1 Application 2 Application N… …
Workflows Workflows Workflows
Feature Engineering
Libraries
Command line,
Python ...
Machine Learning
Libraries
Workflow Scheduler & Manager
Ground Truth
Models
Scores
Hive
Pig
Drivers
Web UI

20. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics 20
Personalization
Emily
Score = 0.9
Steve
Score = 0.9
Search Score = 0.5
Connection Score = 0.3
Profile View Score = 0.1
Search Score = 0.2
Connection Score = 0.3
Profile View Score = 0.4
Our premium
subscription
enables you do
advanced searches
for better results …
... Do you know
you can view
unlimited profile up
to 3rd degree by
purchasing our
premium
subscription …

21. ©2015 LinkedIn Corporation. All Rights Reserved. Biz Analytics
Summary
 Modoop – Model on Hadoop
 Member targeting for email marketing
 Highlighted features
 Easy-to-use web UI
 Built-in data mart (DM2) and feature engineering
 Powered by the machine learning library
 Model refreshment framework
 Automatic model deployment
 Request minimal user input and machine learning knowledge
 Other Use Case
21

22. Thank you!