Feb. 27, 2019 4pm-7pm, MSBA Industry Seminar, Anderson School of Management, UCLA. Presenter: Jimmy Wong from LinkedIn. As a professional social network, LinkedIn leverages data throughout the entire organization to fulfill its mission to connect the world’s professionals to make them more productive and successful. In this presentation, Jimmy Wong, Senior Manager of Business Analytics at LinkedIn, will share with aspiring business analytics students the role that analytics plays at LinkedIn. Part 1: People and roles: As every team relies on data to run the business, what’s the specific role of the business analytics team? Which types of job openings should you apply to? Data systems: What systems power the analytics framework at LinkedIn and what technical skills can you learn to prepare for it? Case studies: Example analytics projects and business problems solved. Part 2: Interactive workshop on interpreting and presenting data to management. Exercise: Within your group of 6 people, analyze and interpret a small set of data points to produce a chart of the key insights, and then deliver your interpretation and recommendations to the whole class

1. UCLA MS Business Analytics
Industry Seminar 2/27/19
Jimmy Wong
Senior Manager of Data Science, LinkedIn

2. Today’s
agenda
4:00 Introductions
4:10 LinkedIn Overview
4:15 Analytics and Data Science at LinkedIn
4:30 Analytics Case Studies
5:00 Q&A
5:30 Team Workshop
6:00 Workshop Review
Feb. 27, 2019

3. Jimmy’s Professional Journey
BS Mechanical
Engineering, ‘94
UCLA
Business Analytics
Data Science
2011-Present
LinkedIn
Process
Engineering
1994-1995
Hughes
MS Engineering,
‘96
Stanford
CIM Engineering
Information
Systems
BI/Data Warehouse
1996-2011
Trimble

4. LinkedIn Overview

5. Create economic opportunity for every
member of the global workforce
OUR VISION

6. MEMBERS COMPANIES JOBS SKILLS SCHOOLS KNOWLEDGE

7. Connect the world’s professionals to
make them more productive and
successful
OUR MISSION

8. FOR OUR MEMBERS
ADVANCE MY CAREER
FOR OUR CUSTOMERS
WORK SMARTER
OUR VALUE PROPOSITION
Connect to Opportunity

9. FOR OUR CUSTOMERS
WORK SMARTER
FOR OUR MEMBERS
ADVANCE MY CAREER
OUR VALUE PROPOSITION
Connect to Opportunity

10. FOR OUR MEMBERS
ADVANCE MY CAREER
Connect to Opportunity
Stay well-informed
Build meaningful
relationships
Get the right job
Establish and manage my reputation
Research and contact people
Keep tabs on my activity

11. ADVANCE MY
CAREER
Get the right job

12. ADVANCE MY
CAREER
Build meaningful
relationships

13. ADVANCE MY
CAREER
Stay well informed

14. ADVANCE MY
CAREER
Establish &
manage my
reputation

15. ADVANCE MY
CAREER
Research &
contact people

16. ADVANCE MY
CAREER
Keep tabs on
my activity

17. FOR OUR CUSTOMERS
WORK SMARTER
Hire
Premium
Connected Enterprise
Market Sell Learn
Connect to Opportunity

18. WORK SMARTER
Hire
Talent Solutions

19. WORK SMARTER
Market
Marketing Solutions

20. WORK SMARTER
Sell
Sales Solutions

21. WORK SMARTER
Learn
Learning Solutions

22. WORK SMARTER
Premium

23. Analytics and Data Science
at LinkedIn

24. Critical mass of
data
Relevant and
valuable products
and services
Technology
platform
Member growth and engagement
LinkedIn’s business model & why analytics is important

25. Relevance & AI
Other data infrastructure and
apps
Data Team
LinkedIn Engineering
Data Organization at LinkedIn
See https://engineering.linkedin.com/teams/data
Analytics & Data Science

26. Analytics and Data Science Team

27. Job Qualifications
Data Scientist Roles
• SQL
• Programming Language
• Statistical Modeling
• Data Visualization/Dashboarding
• Communications
• And others…

28. Technical Skills

29. Data Visualization

30. Story-Telling

31. Leadership

32. Analytics Case Studies
Product Analytics
for Jobs Tab
Case 1
Optimization
via Machine Learning
Case 2

33. Case 1:
Product Analytics for Jobs
Tab

34. Data driven product innovation
Data-Driven Product Innovation Framework
Use data to ask, measure, understand, and improve the product experience
Ask
Actionable
Insights Lead to
Production
Ideation
Measure
Success Metrics
Definition
Develop
Tracking
Instrumentation
Specification
Test
Experimentation
and Iteration
Learn
Release & Post-
Launch Insights
34

35. Job Applications Website Funnel Analysis
LinkedIn WAUs
Successful Hires
Jobs WAUs
Job Viewers
Job Applicants
1. Grow
Drive Jobs WAUs
2. Discover
From Jobs WAUs to Job Viewers &
Applicants
3. Get
From Interested Job Viewers to
Applicants and ultimately
successful hires

36. Actionable Insights
Lead to Product Ideation
Success Metric
Definition
Tracking
Instrumentation
specification
Experimentation
and Iteration
Release & Post-
Launch Insights
Ask Measure Develop Test Learn
Move Profile
Entry Point
New Jobs
Entry Point
36

37. Hypothesis:
• Improving awareness of jobs in mobile app
• Building a consistent experience between Desktop and Mobile
Actionable Insights
Lead to Product Ideation
Success Metric
Definition
Tracking
Instrumentation
specification
Experimentation
and Iteration
Release & Post-
Launch Insights
Ask Measure Develop Test Learn
Invest in developing the right success metric.
37

38. What to consider when testing such a change?
Hypotheses & Key Metrics Impacted
↗ Overall LinkedIn ecosystem
→ Metrics: UU’s, sessions, revenue
↘ “Profile”
→ Metrics: Self profile views, edits
↗ “Jobs” and drive a lot of job applications
→ Metrics: Jobs UUs, job views, job applies
A/B Testing
Control Treatment

39. 1. Collaborate with product manager to draft tracking specs
2. Align with engineers on what will be tracked and how the data
will flow
3. Make sure all the needed data will be available at launch
Actionable Insights
Lead to Product Ideation
Success Metric
Definition
Tracking
Instrumentation
specification
Experimentation
and Iteration
Release & Post-
Launch Insights
Ask Measure Develop Test Learn
Need accurate reliable standardized data logging to enable metric computation.
39

40. Rigorously set up, then identify whether the feature increased the success metric.
Actionable Insights
Lead to Product Ideation
Success Metric
Definition
Tracking
Instrumentation
specification
Experimentation
and Iteration
Release & Post-
Launch Insights
Ask Measure Develop Test Learn
Portion of users will have the new experience rolled out to
their app
How can we go fast while controlling risk and improving decision
quality?
1. Launch to a small portion of members to mitigate risks
2. Reach maximum statistical power to analyze the impact
3. Based on the results: Launch to 100% OR roll back
40

41. Actionable Insights
Lead to Product Ideation
Success Metric
Definition
Tracking
Instrumentation
specification
Experimentation
and Iteration
Release & Post-
Launch Insights
Ask Measure Develop Test Learn
Hypotheses verified by A/B test
Overall LinkedIn ecosystem → Sessions
“Profile” ↘ Profile Edits ↘ Self Profile Views
“Jobs” ↗ Jobs UUs ↗ Jobs Views ↗ Jobs Applications
Recommended next steps
- Ramp the jobs tab to 100%
- [Profile] Build an onboarding tutorial that points out the new location of the ME tab
- [Profile] Add an edit profile promo on the jobs tab
- [Jobs] Improve the tab by adding different type of modules to drive more downstream engagement
41

42. The journey is not done! Keep on improving
Identify
opportunities to
continuously
improve the
experience

43. Case 2: Optimization
via Machine Learning

44. 610M
+
Members
30M
Companies
90K
Schools
LinkedIn’s Social Network

45. Machine Learning is a “Process”Prob
lem
For
mula
tion
Labe
l
Prep
arati
on
Feat
ure
Engi
neeri
ng
Mod
el
Lear
ning
Mod
el
Man
age
ment
45

46. Problem Formulation
Skills Clustering
Engineering
Art
Business
Marketing
Estimating Customer
Spending Predicting Churn
Churn
No
Churn
CLUSTERING REGRESSION CLASSIFICATION
46

47. Problem Formulation
Example: Job Seeker Subscription Model
Assume we periodically send marketing promotions /
campaigns to LinkedIn members for job-seeker subscriptions.
How do we decide who we should send these emails to?
Senior Data Scientist
Linkedin
Mountain View, CA, US
12 connections work here
------------------------------------------
Posted 2 days ago
Job 1 Job 2 Job 3
Job Seeker Subscription
Reaching out
• Who's viewed your profile
• InMail™ messages
Finding the right people
• Premium search
Job seeking
• Featured applicant
• Applicant insights
• Salary data
Standing out
• Premium Profile
• Larger search listings
Binary classification problem: let 𝑦i represents the product
subscription status of member
47

48. Model Management
Photo credit:asmfoto Marcell Mizik, photo license with Depositphotos File Purchase Agreement #41549281
Day 1 Day 30
48

49. Model Management - Monitoring
● Business customers evolve
dynamically
● Products update periodically
 Centralized model repo with standard format
 Monitor both feature/model performance changes
over time
 Feed in new training data to generate “challenger
models” to compete
● Model refresh
● Feature diagnosis
feature
monitoring
performance
monitoring
inherent temporal nature
● Performance degradation
● Failure/Outlier examples
● Feature statistics over time:
○ non-null count, sum, medium
○ coefficient of variation for
volatility evaluation
Monitoring
49

50. Model Management - Refresh
Feed in new training data to
generate new model periodically
Assign versions to models built
over time
Monitor changes over time
Ensemble historical models as one
of the candidate models
50

51. Performance Measurement via A/B Test
Algorithm B
20%
Algorithm A
80%
Collect results to determine which one is better
51

52. Contact:
Jimmy Wong
https://www.linkedin.com/in/jimmywong
@JimmyWongUSA
Questions?

53. Team Workshop

54. Successful Charts for Communicating to Executives
Be relevant
• Why is it important
to the audience?
• Replace slide title
with actionable
headline
“So what?”
Easy to understand
• Compare against
something
• Highlight area of
interest, quotable
“magic number”
“2-second rule”
Remove distractions
• Remove clutter
• Label your units
“When in doubt, cut it out”

55. Prioritize by these display features
Pre-attentive Visual Attributes
1. Length/Size: bar/column chart,
ordered
2. Position/Orientation: X-Y scatter
plot to show outliers, or line chart
to show trend
3. Shape: add call-outs
4. Color/Intensity: highlight the
outliers
Make Impactful Data Visualizations

56. * Fictitious data