Thanks John Lynn, @jmlynn7 @startupinst for bringing us in to talk about leanstartup and the practical application for folks.
Folks at home - if you want to PRACTICE some of the lessons in this deck, we are giving away $100 bills at #sxswi for people to do customer development, state problem hypotheses, and run experiments.
5. what will I learn?
1. What is LS?
2. Why should you care?
3. How do you do it (if you decide to care)
4. Discussion of Topics:
• What is an MVP (what are some great examples?)
• What do I test, what do I measure?
• What do I do with my learnings?
5. Get INTO and THROUGH BML once!
(you are not practicing until you START
the second loop)
6. Jonathan Bertfield
• 20 Years in Product
Management
• Consulting clients: Pearson Inc.,
Disney, Associated Press, News
Corp. Sesame Workshop,
Business Insider, American
Press Institute
• Coach at General Assembly
Enterprise, Lean Startup Co
Faculty, NYU Stern
Adam Berk
• What problem are you solving? I
help teams pick their riskiest
assumptions and run
experiments.
• Currently at Pearson with >>
• Founded a sharing economy site
with the opposite result of AirBnb
7. we believe that you…
1. have heard the term LS before today
(95%)
2. have read “the book” (60%)
3. believe that LS is “at least worth
considering” (80%)
4. have run an experiment or thought about
it in your lifetime (maybe) (60%)
5. want (are willing) to run (more)
experiments with your (new/future) team
(80% now 95% end of preso)
9. what is a startup?
“A human institution designed to create a
new product or service under conditions of
extreme uncertainty. “
- Eric Ries
author, The Lean Startup
10. A method to systematically address
uncertainty through rapid iteration and market
learning.
what is the Lean Startup?
11. we WILL cover the following
buzzwords. MVP. CUSTDEV.
PIVOT...
• Entrepreneurs
• Startups
• Uncertainty (Product / Market / Model)
• Minimum Viable Product / Pre MVP
• Experiments
Assumptions*
Hypotheses*
(Vanity) metrics,
Customer development
• Validated Learning
• (how to do test and measure the right things)
• Pivots
17. Defining waste.
• Building things that no one wants
• GOING TOO SLOWLY (wasting time)
• NOT LEARNING FAST ENOUGH
• Doing things the “old way” because “that is
how they are done”
• Solutions looking for problems (most of the
time) ((almost ALL the time in early stage
startups))
18. why there’s waste.
• cognitive biases
• poor communication
• fear of penalty kicks down the
center
• sunk cost syndrome
• MORE?
28. Scientists do not coddle
ideas. They crash test
them. They run them
into a brick wall at sixty
miles per hour. If the
idea is sound the pieces
will be of the wall. 28
30. Minimum Viable Product (MVP)
Experiment that helps you validate (or
invalidate) hypotheses about the value or
growth potential for a new product
An MVP helps you answer a specific question
about one of your assumptions (as fast as
possible and helps you understand the WHY)
31. scientific method
1. Identify problems to generate ideas
2. Identify and prioritize the underlying
assumptions behind those ideas
3. Generate hypotheses and predictions
4. Define and deliver tests that expose
opportunities for learning
5. Use resulting data (learnings) to inform
activity
6. Repeat forever (remember the second loop
in the roller coaster)
35. components of an example hypothesis
We believe [doing this]
for [these people]
will [achieve this outcome]
We will know this to be true when
we see [this business outcome]
by [this specific timeframe]
37. components of an example hypothesis
We believe [building a network of independent drivers]
for [New York]
will [make it faster and easier to grab a cab a peak times]
We will know this to be true when
we see [20 drivers with 20 rides/day ]
by [1 month from now]
38. Devin’s hypothesis
We believe [building a network of independent drivers]
for [New York]
will [make it faster and easier to grab a cab a peak times]
We will know this to be true when
we see [20 drivers with 20 rides/day ]
by [1 month from now]
39. components of an example hypothesis
We believe [creating content based on journalists
passions]
for [online readers]
will [generate high levels of reader engagement]
We will know this to be true when
we see [6 posts per day from each editor ]
by [3 months from now]
40. MVP: expert tip
Not a static thing! Commitment to a process.
Experiment(s) that helps you validate (or
invalidate) hypotheses about the value or
growth potential for a new product over time.
(Credit to Eric Ries)
41. MVP: expert tip
● Isolate variables
● Make the call to action 3x harder than you think it should be
● Make the time to deliver 7x faster than you think it should
be
● WRITE DOWN THE EXPECTED OUTCOME
● Say it out loud (tweet it, blog about it)
● If not… email it to yourself or a co-founder, put a calendar
invite for one of your mentors
42. It is a BAD experiment
if...#badscience
Audience Participation
43. what kinds of uncertainty?
Technical / product
Can we build this?
Customer / market
If we build this, will people use/buy it?
Business model
Once we build this, can we find a way to make money from it?
44. ●Technical / product
○ TESLA BATTERY, SOLAR, GOGO?
○ SOME STUFF THAT AUBREY WILL TALK ABOUT IN
301
●Customer / market
○ YOUR IDEA (99% of the time)
●Business model
○ GROUPON, HOMEJOY, LYFT
what kinds of uncertainty?
45. ●Technical / product
○ usability tests, prototypes, wizard of oz
●Customer / market
○ problem interviews, landing pages,
●Business model
○ landing pages, crowdfunding,
testing different kinds of
uncertainty?
54. Minimum Viable Product
Translate your critical assumptions into an experiment:
1. Isolate critical assumptions for testing
2. Prioritize assumptions for testing
3. Draft your hypothesis to be tested
4. Execute an experiment (that includes MSC ahead of time)
5. Measure the results
6. Collect the data and learning in a systematic way and incorporate the
learnings into new experiments
7. IF you do not do #6, 1-5 are irrelevant
validated learning
using scientific principles
60. If you think your customer segment is
MILLENNIALS, YOUNG PROFESSIONALS,
EVERYONE, SMBs, MULTIPLE ACTORS, YOUR
MENTORS, PEOPLE WHO HAVE A VESTED
INTEREST IN BUILDING YOUR SOLUTION (added
after the webinar) FRIENDS, FAMILY,
you do not
UNDERSTAND IT
well enough.
<<< That guy
60
61. If you think the problem you are
solving is
LACK OF TIME, LACK OF
MONEY, LACK OF CUSTOMERS,
LACK OF YOUR SOLUTION,
you do
not
UNDERSTAND IT
well enough.
<<< That guy
61
74. measure results
What knowledge are you looking to gain?
LEARN or PROVE
What you are going to do with the knowledge
when you get it?
START THE LOOP AGAIN BASED ON NEW
EVIDENCE, NEW LEARNINGS, NEW
PROOF!!!
80. “The ability to LEARN faster
than your competition may be
the only sustainable
competitive advantage”.
Arie de Geus
81. “I was out of touch. But it
wasn't because I didn't
know enough. I just knew
too much.”
Gnarles Barkley
82. “Running down a dream
that never would come to me
workin on a mystery,
goin wherever it leads”
Tom Petty, Rock n Roller
83. The end (for real this time).
But really just the beginning for
you. Get out there and do not
be afraid to challenge the status
quo with HYPOTHESES and
DATA.
84. Don’t LOOK DOWN (notes, old slides, appendix,
crazy experimental shit below)
85. 1. Superforecasters - Podcast
2. The Experimenter - on Netflix
3. The ESPN experiment
4. LeanUX
86. 1. Can experiments themselves be a waste
2. What is a bad experiment
3. How do you set a baseline metric
4. What about a pure vision
5. What if the riskiest assumption is investment
(designing MVPs for investors)
88. Please use the COMMENTS to comment directly in
the deck on
● one slide, lesson, comment you loved,
● one that we could have improved or eliminated
●
We will publish the new and improved deck at the
end of the week with your feedback and our
learnings incorporated.
It was a privilege to speak with you all today, our
doors are always open!
@adamberk + @berters
Notas del editor
- Start the timer
single biggest source of waste is building something no one wants (cut the BS - the birdshit as someone said in Detroit)
single biggest source of waste is building something no one wants (cut the BS - the birdshit as someone said in Detroit)
Kidding of course that is the whole workshop… but that IS the whole process… Reduce waste achieve your vision by running experiments, learning, and iterating - OVER AND OVER AGAIN… sounds simple - it is not (which is why we have more slides)
“dont linger on the punchline” #custdev with Gabe:)
10:10 -1020 (WHAT IS LEAN STARTUP)
1020-1030 (WHY -- the philosophical why -- we will get into more WHY questions at the end in the discussion)
1030-1040 (HOW #custdev)
1040-1050 (HOW #other mvps, riskiest assumptions etc)
1050-11 (HOW in reality… how to do this with your team, how to actually get beyond the fear of running experiments and going fast, how to find what works for you)
11-1130 (ASK US ANYTHING… TALK ABOUT HOW EVOLUTION ISNT GOOD ENOUGH, FIELD THE STEVE JOBS / HENRY FORD QUESTION ETC)
JB and I will have a premade list of contingencies on how to make real time adjustments based on answers
Welcome -- we want to get you INTO THE LOOP - demystifying… not an exercise… the more you do it -- but you have to do it to do it so let's get to it and do it….
rapid learning - GO FAST THROUGH THE LOOP IS THE ULTIMATE GOAL
it is NOT a smaller version of a bigger company
it is also not just a higher risk bet on the future
think of the BML as simply the scientific method -
small tests/ small batches mentioned here
rapid learning & experimentation mentioned here
JB and I will have a premade list of contingencies on how to make real time adjustments based on answers
WHY chances are, as an entrepreneur about to deliver a new product or service.. "you are going the wrong way" - the trick is to stay true to your vision and not just listen to mentors like me for the sake of listening… but for the avoidable mistakes like in the video, correct those before you crash into a truck (run out of money/time/credibility)
JB and I will have a premade list of contingencies on how to make real time adjustments based on answers
GET SOME FROM AUDIENCE
WHY Because build it and they will come is a FLAWED strategy. especially for startups. especially these days. #1 on the waste list so it gets its own slide
ADAM - but the reality is this… and this is what we will use to to help you understand the process… (we can replace this image w empty stadium) -- we will give you the tools to AVOID THIS and to APPLY the loop to ANY risky assumption
WHY This is what you are building right now.
WHY (because you are building this and you don’t realize)
WHY - (IMPORTANT SEGUE SLIDE FROM WHY TO HOW) because customers have multiple problems and problems have multiple solutions -- who is your customer... not everyone... remember the butterfly effect
“So instead we ask….”
WHY (BUT ALSO HOW) Problems over solutions
WHY -- because in the real world getting things to line up all at once is like trying to hold a greased watermelon
HOW = RUN EXPERIMENTS “this is only a test” Here’s how.
Run experiments! Keep learning!
Run experiments! Keep learning! (thanks gabe for the lovelace reference)
will LIVE WHITEBOARD the spreadsheet I created if we have time
AIRBNB
FOOD ON THE TABLE
LINKEDIN
FACEBOOK
INSTAGRAM
cars.com
aardvark
flows perfectly… (tips on how) NOW the reason REASON we do this is to turn these into hypotheses and test them
HOW -- test the riskiest things first. If you have a 3 part plan, if part 1 falls apart parts 2 + 3 are irrelevant. The riskiest things (as we mentioned above, are that there is no problem, no customer, people will not pay etc - so a deeper dive in testing those things below)
go around the room, hearing ideas. speak with 5 each get 2 minutes. . Ask them to share 5 assumptions.
10-15 minutes open room - 4-10 ideas with a few assumptions each MAX 2 minutes per person but some people may be able to say an idea + 3 assumptions in 30 seconds… goal is to hear 10 ideas
split the room up into at least 3 teams will list ALL assumptions around their idea / business model (we can guide them) - even if it's a 30 person team
2 enterprise ideas (one searching one executing) and 1 startup idea
with one idea we have at least 10 assumptions prepared (either for vodaphone idea or another patsy)
10 minutes coming up with at least 30 dependencies
15 minutes ranking them visually
10 minute share out - 3 per team, how did you rank them
REMIX BONUS IF WE HAVE TIME - you can move someones but you have to explain why to the room
QUICKLY WRAP those 3 in the upper right corner and translate it to a hypothesis
THEN BRIEFLY introduce methods and custdev and learnings and loops and pivots
HOW (may kill slide)
assumption > prioritize > hypothesis > mvp
from the 3 riskiest on the boards - we whiteboard at least ONE HYPOTHESIS
assumption > prioritize > hypothesis > mvp
from the 3 riskiest on the boards - we whiteboard at least ONE HYPOTHESIS
We believe that…
building an app to hail taxis
for urban professionals
will massively disrupt public transportation
We will know this is true when
we see drivers in 3 cities
by 2011
We believe that…
building an app to hail taxis
for urban professionals
will massively disrupt public transportation
We will know this is true when
we see drivers in 3 cities
by 2011
We believe that…
building content based on journalist passions
for online readers
will generate high volumes of reader traffic
We will know this is true when
we see 6 posts per day from each editor
by 1 month
will LIVE WHITEBOARD the spreadsheet I created if we have time
AIRBNB
FOOD ON THE TABLE
LINKEDIN
FACEBOOK
INSTAGRAM
cars.com
aardvark
will LIVE WHITEBOARD the spreadsheet I created if we have time
AIRBNB
FOOD ON THE TABLE
LINKEDIN
FACEBOOK
INSTAGRAM
cars.com
aardvark
AB COVERING THIS
ADAM (read the notes) SEGUE TO THERE ARE DIFFERENT EXPERIMENTS TO TEST DIFFERENT THINGS (STARTING WITH THE PROBLEM) We are going to assume for 201 that customer market is your biggest risk and will use that - but in 301 its YOUR JOB as a scientist to know your true riskiest assumption it WILL Be TECHNICAL and it will BE BM in some cases
ADAM (read the notes) SEGUE TO THERE ARE DIFFERENT EXPERIMENTS TO TEST DIFFERENT THINGS (STARTING WITH THE PROBLEM) We are going to assume for 201 that customer market is your biggest risk and will use that - but in 301 its YOUR JOB as a scientist to know your true riskiest assumption it WILL Be TECHNICAL and it will BE BM in some cases
pull dont push- tug of war slide (no selling allowed slide too maybe- or can be it's own)
humans are terrible at predicting the future or saying what they WOULD DO... focus on the past... HAVE YOU, TELL ME ABOUT THE TIME...
HOW #custdev specific pull don't push- tug of war slide (no selling allowed slide too maybe- or can be it's own) hunting elephants
HOW #custdev specific pull don't push- tug of war slide (no selling allowed slide too maybe- or can be it's own) hunting elephants
cars.com / aardvark / dropbox /
See Innovators Method (page 7):
Airbnb’s team had a hunch that better photos would increase rentals
They tested the idea w/ concierge MVP, putting the least amount of effort possible
When the experiment showed good results, they build the necessary components and rolled it out to all customers.
keep raising the bar ‘ PROVE they want to get in (maybe show succession of high bar attempts or show high jump next to pole vault)
Sometimes not having it is even better... the further they get trying to look for it - the stronger the signal. Take everything you think you know about conventional marketing and do the opposite. You WANT it to be hard, so you know it's valuable and indicative.
HOW (quintessential how)
HOW (quintessential how)
If you THINK TEST PROVE. IF YOU TALK TO 5 customers. IF YOU optimize for learning. If you listen to customers.
way too SLOW -- its good in the sense that it is adaptive of course - and we are only trying to make a point here - but natural selection is TOO SLOW and it only solves for SURVIVAL -- to THRIVE and GROW we want to cheat… we want UNNATURAL SELECTION… GMO is bad for food but good for learning)
what is your goal - how will you know you have accomplished it - how do you avoid retrospective coherence…
what is your goal - how will you know you have accomplished it - how do you avoid retrospective coherence…
what is your goal - how will you know you have accomplished it - how do you avoid retrospective coherence…
you are maximizing for learning so you can be right on purpose YOU HAVE TO FIGURE THINGS OUT (LIKE WHERE THE SPOON IS) by running experiments and doing things that dont scale
you are maximizing for learning so you can be right on purpose
only when you get people through your funnel ıs ıt tıme to try and scale DO THINGS THAT DON'T SCALE AND RUN EXPERIMENTS TO FIGURE OUT WHAT THAT THING IS!
p
instead - focus on one value proposition - perfection is not when there is nothing left to add but when there is nothing left to take away - this applies to your soluton but also applies to your customer and problem statements in the beginning
The genesis of a company is timing and demand and ability to solve the problem really well
The genesis of a company is timing and demand and ability to solve the problem really well
vision without execution is hallucination… it is OK to have an opinion to FORM your vision and strategy and what you test first and how… but then the RESULTS must factor into your next round of decisions
you are maximizing for learning so you can be right on purpose
Detroit - best use of this slide… really is no way to teach the value of custdev without practice
this only workks if you turn your learnings into actions and actions into metrics
instead of being eaten by a coin flip - soar on the wings of dependent probability -- the ALTERNATIVE IS making adjustments each loop through
WHY instead of being eaten by a coin flip - soar on the wings of dependent probability -- the ALTERNATIVE IS making adjustments each loop through (this bombed last time - if it bombs again it gets cut - but we cant be scared to show it after ONE data point!)
We started about talking about the loop… it doesn't end when you “embrace lean” or identify assumptions or even when you run an experiment… or even when you have learnings.. the loop closes when you run your next experiment based on those learnings… we challenge you to close the loop on your team at least once!
If you THINK TEST PROVE. IF YOU TALK TO 5 customers. IF YOU optimize for learning. If you listen to customers.
If you THINK TEST PROVE. IF YOU TALK TO 5 customers. IF YOU optimize for learning. If you listen to customers.
If you THINK TEST PROVE. IF YOU TALK TO 5 customers. IF YOU optimize for learning. If you listen to customers.