Data is king! Get ready to understand how a successful analytics team can empower managers from product, marketing, and other areas to make effective, data-driven decisions.
Louis Cialdella, a data scientist at ZipRecruiter, shared some case studies and successful strategies that he has used at ZipRecruiter as well as previous experiences. The purpose of this data talk was to enlighten people on how to make sure that analysts can successfully partner with other departments and get them the information they need to do great things.
5. 5
Today’s topics
About me and ZipRecruiter
What does the analytics team do? What are our projects like?
Strategies for successful partnerships
Some things I wish I knew when I started
7. 7
About Me
Columbia University - Computer Science and Machine Learning
Worked at startups in New York, work on the credit risk analytics system at JP
Morgan
Currently a data scientist on the analytics team at ZipRecruiter
8. 8
ZipRecruiter
Short version: We help people find jobs
Employers post jobs, we help them find qualified candidates
#1 job search app on iPhone!
Located in Santa Monica
We’re hiring! For analytics and product positions - Resumes to
louisc@ziprecruiter.com
9. 9
Some challenges we face at ZipRecruiter
How can we match employers with jobseekers in a way that benefits both?
Who should we market our service to?
How can we improve our user experience?
How can we guard against fraud on our platform?
10. 1
0
What we do and how we do it
Analytics at ZipRecruiter
11. 1
1
The Role of analytics at ZipRecruiter
Help business stakeholders make data driven decisions
Other departments have domain knowledge and problems to solve, we
supply statistical skills
Define metrics
Answer vague business questions with well-defined data analysis
13. 1
3
ZipRecruiter’s Analytics team
Follows the “centralized” model
Independent department which provides data and statistical analysis
Advisory capacity - help other teams understand their data and figure out
which decisions will benefit them the most
14. 1
4
ZipRecruiter’s Analytics team
Follows the “centralized” model
Pros:
Makes it easy to build up institutional knowledge
Can build and share analytics tools
Independent incentive structure
Cons:
Requires skilled analytics-specific leadership
Further from domain experts
My opinion: Given the rapidly changing state of industrial analytics, pros outweigh
cons
16. 1
6
Analytics can help make great products
Product managers have vision, and data analysis can help advise on the best
way to execute it
Two main ways:
Product Optimization:
Example: A/B testing different user experiences
Machine intelligence Integration:
Example: Making recommendations to users, dynamic pricing, fraud
detection
17. 1
7
Successful partnerships
A balancing act:
Product people are incentivized
to do cool things as fast as
they reasonably can
Analytics people incentivized to
be rigorous and careful
Pragmatism vs rigor
Fast vs slow
18. 1
8
Analytics projects
It helps to understand how an analytics project is structured
For a data-driven project to succeed, we need to:
Collect the data
Run experiments, look at historical data, etc
Analyze the data
Build a model, extract insights, etc
19. 1
9
Analytics projects
What insight will
be extracted?
What information
will the model give
us?
What mechanism
collects the data?
Where will it be
stored and
accessed?
What is the
question we want
to answer?
What data will be
used?
What modelling
approach will be
used?
What software will
be used?
Data Collection
Data Analysis
Design Implementation
20. 2
0
Analytics projects
Some anti-patterns:
“Crunch these numbers for me”
Analysts need context
“Do some data stuff with all our historical data”
Analysts benefit from a clear product vision
“We ran this experiment, what does it tell you”
If an analyst did not help design the experiment, the data may not
be useful
22. 2
2
A typical request
ZipRecruiter has a free trial for our employer subscription product
During the free trial, employers post jobs and get applications
If we knew how the number of applications related to the conversion rate, we
could design our experience to improve this
How can we understand the relationship between number of
applications per job and likelihood of conversion?
23. 2
3
A first look
We have historical examples of how many applications/job were received, as
well as whether or not a user converted
We could compare the response counts of the convert/non-convert groups
to describe the data
But this doesn’t allow us to make predictions - what we want is a model!
24. 2
4
Understanding our data
We have a few years of historical data about:
Hundreds of thousands of free trials
How many jobs each one posted, which jobs received applications
Whether or not they converted
Geographic and industry information
Our model will need to relate the features (geo/industry info, number of
applications) to the output (conversion event)
25. 2
5
Error bars
Just specifying the expected value of the conversion rate at each response
count can give a false impression of precision
We want to quantify our uncertainty
Solution: Add confidence intervals to estimates (via bootstrap)
26. 2
6
Analytics projects
Can we produce a
curve defining the
FT conversion vs
response
relationship?
The data used is
collected in our
SQL (Redshift)
database
What is the
relationship
between FT
conversion and
responses?
Use historical data
Model: Logistic
regression
Software: SQL
and Python
Data Collection
Data Analysis
Design Implementation
27. 2
7
The result
The curve which describes the
relationship between application
count and conversion
Can use this to optimize the free trial
give our customers the best
experience (for example, tuning
the length of time the trial lasts)
29. 2
9
Uncertainty is the only certainty
Analytics helps us understand where there is uncertainty, but it usually can’t
be brought to zero
Nate Silver: “[Some forecasters] see uncertainty as the enemy...this tends to
leave us less prepared when a deluge hits.”
Analysts deal with uncertainty by mitigating it where possible, and
communicating it where not possible
Use confidence intervals and similar techniques, which provide best/worst
cases
30. 3
0
Don’t be afraid to experiment
Experiments have become common in the industry (A/B tests)
But there is a cost associated with running them - when should we run them?
Answer - whenever possible!
Product can often get great insight by running an experiment
Analytics can often provide much more definitive results and mitigate
uncertainty as much as possible
31. 3
1
Understand Everyone’s Incentives
Analysts should report revenue/profit/cost impact, dollars and cents
Product folks can help by making it clear what needle you want to move,
even if it’s a big picture metric
Analysts are responsible for translating their findings into a language that a
business user can use for decision making
32. 3
2
Understand the choices, make a recommendation
Try and help people make data driven decisions by understanding the
choices they want to evaluate
Product - present the strategies you are considering
Analytics - focus on making specific recommendations, rather than simply
conveying the results of number crunching
33. 3
3
Focus on the small picture
It’s tempting to totally overhaul a system and replace it with DEEP
LEARNING THE BIG DATA
But big overhauls are risky
At each step in modelling, you may make bad assumptions - rolling out
incremental improvements allows you to check these assumptions
When trying to improve a process, don’t overhaul it from the very beginning -
start with small improvements to the existing method
34. 3
4
Summary
● ZipRecruiter’s centralized analytics team model has a lot of
advantages
● Analytics + Product = 😊
● Uncertainty is part of the process, but we can do a lot to mitigate it
and communicate it clearly
35. 3
5
About my work? About ZipRecruiter? About working in data science?
Q & A
37. Part-time Product Management Courses in
Silicon Valley, New York, Los Angeles, and
Orange County
www.productschool.com
Notas del editor
When you checked in tonight, you got an email inviting you to join our slack community
In that community, we have 12k product people who have come through different companies like google, facebook, uber
Sharing information about events, job offers from our partner companies, and valuable online content
Please check your email and join - it’s free
Opening remarks:
Good evening
Hope you’re doing well
Name, rank, serial number
Thanks for inviting meEvangelizing for the ZipRecruiter analytics model - the industry is figuring itself outQuestions at the end, pleaseWho is in the audience? Analysts? Product? Marketing or Finance?
Go over the analytics at ziprecruiter - it’s a good model
Talk about how to avoid friction, which can show up even when everyone is acting in good faith
Context is important
Background is more engineering than stats
Experience at a small company was good to learn about poorly-defined problems, experience at JPMC was good to learn about scale
ZipRecruiter is a little of both
Fast growing, founded in 2010
Major player in the employment/recruiting space
Raised millions of dollars in funding
Great place to work!
The experience needs to be good for both employers and jobseekers, but finding good matches is hard, as tinder users know
Analytics support is also needed to determine strategy for marketing and product
We can improve operations by automating fraud detection
Gatekeepers of statistical decisions making
Work with everyone
Know what is and is not feasible
Organizations do analytics differently
Sometimes analysts are “embedded”
Sometimes they are independent advisors which are separated
Early and mid-stage companies are still often figuring this out! The structure can change too
Up next:
How can analytics and product work together? Avoid friction?
Data scientist, not a tech talk, one equation
How can analytics help? What does a successful partnership look like?
Analysis can reveal what the best product decisions are from multiple alternatives
Automating analysis can lead to efficiency gains and product improvements
It’s not a consulting gig, it’s a partnership! Both teams should trust and support each otherThis is related to incentives:
For product, the key is a successful, on-time, on-budget shipment of something that people like. A project manager can afford to be wrong and iterate quickly.
For analytics, the key is applying the right techniques to reliably get correct answers. An analyst would rather be slow but cautiously correct
For the partnership to be successful, both sides need to respect the goals of the other!
If we have a roadmap for the project, we can figure out who needs to be involved at each step
Not having a clear division of responsibility leads to confusion and stepping on toes
Analysts care about all of these, product only needs to care about the left side of the line.
If we have a roadmap for the project, we can figure out who needs to be involved at each step
This is a parable about partnership
This demonstrates, I think, the success of our model
The conversion rate at N responses is not exactly 30.4532324234234%
Remember: Analysts care about all of these, product only needs to care about the left side of the line.
Product needs to help us understand:
What data we can use
What the specific problem is
What the metric they want to move is
If the proposed model (curve) would be useful
Tribal knowledge of the organization
There are no known best practices - but I think these are pretty good
Each of these has two sides - what product can do, and what analytics can do
Story: Case study about - avoid presenting estimates as iron-clad facts
Your users need to understand the best case and the worst case in your forecasts
A/B tests are part of the industrial practice
In my experience, they are worth the investment
ZipRecruiter - system for launching, tracking, analyzing A/B tests has paid off a lot
New analysts often tend to give detailed technical reports - that goes in the appendix
No one cares about your damned P-values
This goes hand-in-hand with the last one - it’s about supporting decision making, not doing statistics for its own sake
Story: Scammer detection - initial improvement leading to bigger wins - easier to manage than a full overhaul involving so many people