Product Camp LA 2016 breakout session

1. I talked to 100 teams about data.
This is what I learned …
@johncutlefish
@johncutlefish

2. Why this talk?

3. Why
Trust
Learn
Ritual
Access
Action
Safety

4. WhyIf Google is doing it … then we should be doing it right?
The VP got us together one day and told us that we needed
to become more data-driven. Our features are going to be
measured now. It was pretty stressful!
I have no idea whether the work we are doing is actually
having an impact. Whether it really makes it difference. I’m
getting discouraged, actually. People say it does, but I don’t
see it. We are just throwing spaghetti at the wall right now!
We just have more information at our fingertips. We build
better products. It is just another skillset and tool
Obstacles
The Jones
“Alignment”
Tiebreaker
Catalysts
Customer outcomes
Sense of impact
Probe and listen

5. TrustI’m just not sure I trust the team to act on the data
appropriately. What about nuance? What about my
experience?
Just to be blunt, I don’t trust her. We talk about being a
team, but it is always about her ideas and her intuition. My
intuition matters too! Right?
I know we all have the best intentions. Of course I get
nervous, and sometimes out of my comfort zone, but I know
we’re together on this
Obstacles
The unknown
Cherry picking
Intuition sucks!
Catalysts
Just try!
Data quality
Full team activity

7. LearnJust between you and me, there aren’t really any open
questions at the moment. We have to execute, and we have
to move quickly. Yes … I’m giving them a solution
Do you think the code is a thing? That is has value, really? I
don’t think so. It is what this founding team has learned
over the last four, hard years. You can’t quantify it. We could
tear the stack down and start over
All I hear all day is “experiment” and “hypothesis” and
“assumption”. I think we’re bastardizing the words. This isn’t
science! This is shipping software
Obstacles
Relearn / rework
Zombie features
Perfect plan
Catalysts
Curiosity & stories
Great questions
We learned ...

9. RitualIt is just ad hoc queries for right now. When someone needs
something, they have run the reports. I’m always asking
engineers about SQL
We share our findings on Thursdays. Like a big show and
tell. It was rocky at first because some people were new to
doing this type of analysis. But we learned quickly. It makes
it more routine
I have no idea where to start to be honest. Each time feels
like a restart. New questions. New metrics. I get the idea,
but it just doesn’t seem feasible. I have a day job!
Obstacles
Scrumfall
“Listen for …”
Reporting
Catalysts
Show and tell
Cadence
Radiators

10. AccessSo we had to downsize a bit. Not cool. But because of that a
small group of us got together and started learning more.
The basics, really. We had relied on someone else
previously
The product team doesn’t want us to have a login. It is like
they want to own MixPanel. Only they can run reports
I’m just an intern! I’m doing an MBA, and I’m not a data
scientist. But for some reason they gave me the keys to all
of this data. I used some tools from [school] and figured out
some key predictors of churn and success. I feel kind of bad
for the people who never get that freedom
Obstacles
Tool festishism
Data hoarding
The “Data Scientist”
Catalysts
Study groups
Explore/exploit
Democratize data

11. ActionWe had so much learning … so many interviews and stuff.
And then some good data. We made the effort as a whole
team. But suddenly we stopped. Everyone was kind of
pissed. Why didn’t we act on it?
I feel like we are looking for a needle in a haystack. And
meanwhile we have everyday questions that need answers.
Why does it need to be this big deal?
Teams know that you’ll never get it right the first time. We
just accept that. The drive is to keep moving the needle, and
we mostly let people do that until it stops working, or never
works
Obstacles
Feature factory
“Owner”
Get ahead of it
Catalysts
Missions/outcomes
Explore/exploit
Design a dashboard

13. SafetyThere is a certain vulnerability to all this. That’s a good word
for it. It was kind of easier before when we’d build what
they asked for, and then they took responsibility for it. Now
it is on our watch! What if it fucks up?
The problem with experiments is that our experiments
would never fail. Never. Experiment was another word for
“do what I say!” Having the new [title] here made so much
difference. We’re not pushing hard enough unless we mess
some things up
I learned some stats. And I totally botched up a test.
Completely. But it was OK. The world didn’t end
Obstacles
Success theater
Penalty!
The inmates!
Catalysts
Celebrate learning
One team / all data
By example

14. Discuss