SlideShare una empresa de Scribd logo
1 de 74
Customer, market and
business validation
For early stage startups
+ case study: TransferWise ~ the mission-driven startup
Hi, I’m Jeff!
Integral
Designs Bell Canada ING Bank Skype → Microsoft TransferWise
MSc -
Info
Science
Bachelors - Business
4 steps to world domination
Your
customer
Your
Business
Scale your
startup
Your
Market
Validate
Case Study
Grow
Validate
Your Customer
Proposition
How many startups START.
Intuitive understanding
of user need
Passion for particular
technology
Propose solution
= Successful startup?
How many startups STOP.
Intuitive understanding
of user need
Passion for particular
technology
Propose solution
= Successful startup?
← Not tested.
← Customer’s don’t care.
← Not based on actual.
customer needs.
← Not as likely.
Jobs to be done (JTBD)
Understand Customer +
Situation
“Your customers have a job and
they’re hiring your app to solve
it”
JTBD: weather apps, iPhone apps, SaaS/Services
Learn by focusing on JTBD
How valuable is getting that job
done for this type of person?
How many people have this job?
How often do they have this job?
How is this job currently being
done?
JTBD: more info/exercises
← Willingness to pay.
← Market size.
← Usage frequency/pattern.
← Actual competition..
(usually status quo).
Problem interviews
Your goal is to learn, not sell
Start before you build anything
15 mins per interview
Do 20-60 of them
Learn more: Jeff’s highlights, from this book
Demographics - who are they?
Tell a story - that covers several
problems they might have
Problem ranking - which problem is
most painful?
Explore problems - go in depth into
each problem they have
Hook & Ask - keep them engaged
Document - take 5 mins, write down
Structure: intro, demographics, story
Demo, as real as possible, iterate
Test price, don’t ask price
Raise friction, don’t lower it. Find
early adopters. Be the prize.
Solution interviews
Your startup goal is to build
the simplest product that.
solves a real problem.
Validate your understanding
of (1) customer, (2) product, (3)
market proposition
Visual demo of an MVP for
early adopters
20-60 interviews needed
The best blog post on pricing
...but...
You know what
customers need
It takes too long
It’s a waste of time
...but...
You know what
customers need
It takes too long
It’s a waste of time
← No you don’t, unless you’ve.
talked to enough of them, in a.
structured way.
← it’s nothing compared to.building.
something for 6 months and discover.
nobody will buy it.
← not if you learn what to build.
and who will buy it.
Validate
Your Business Idea
Learn More: Wiki, Book
Increase probability of success by understanding
your key risks
Start with the big boxes
Value proposition, customer
segments, key partners
Start with whatever is most risky,
and figure out what you need to
learn to remove that risk.
Ask the hardest questions
first
Who’s this for? Why do they
need it?
How much money do you
need to make to break even?
Will anyone actually pay for
that?
How much will it cost you?
✅ ✅ ✅
✅
❓ ❓
❓ ❓
Collect high quality data & validate it
Make it easy for potential
customers to tell you
what they want:
Currency
“wishlist”
Research & explore
Use data where you
can… but not as your
only source
Ask “so what” and find
the insights - who can
we help?
Validate
Your Market
Launching your MVP
Minimum Viable Product
Be an order of magnitude better
You need to be way better than
the alternative. Aim for 10X
10X faster
10X cheaper
10X more convenient
10X easier to use
or a combination
MVP Learning Cycle
MVP - does not mean
“minimum viable piece of crap”
You need to understand who
your customers are and
what they want.
Can your customers use
your product?
Can you do for the customer
what you actually
promise?
Is your MVP Successful?
You need to understand your
metrics and set realistic targets!
This will help you define success
and start iterating.
For instance:
New users
NPS
Cancellations
Conversion
See appendix for more KPI examples
Learn more about NPS, Product/Market fit
Achieving Market Fit
NPS! (Net Promoter Score)
Virality
Comments
Work on the detractors
Then neutrals
Then profit
Early startup KPIs
Do Not Focus On
Vanity metrics
i.e. total registered users, total volume transferred
Getting statistical significance,
split testing (A/B testing)
More than a handful of
metrics
Focus on
The 3 or 4 most important metrics
(KPIs) that indicate your progress.
Perhaps:
Growth & conversion
How you’re adding value
How much runway you have
Helpful to work backwards from
some target state (i.e. 10K new users)
and validate how you get there
Learn more about KPIs in the appendix
Incubation &
Achieving Market Fit
Listen to customers
They are usually your best
critics and advisors!
Surveys
Phonecalls
Meetings
The trick is not to give them
what they want but what they
need.
Iterate
Usually the one that iterates
fastest wins!
Prioritising is they key here
Make sure you know what
you effect and measure it!
A/B testing
Scaling Your Startup
Your startup
grows at.
the rate.
you learn.
What is scaling?
You have a mountain to climb to achieve
your vision - KPIs are your guideposts
Validation: It’s proof that you’re moving
forward
Investors/valuations
You have a hard job
You have a limited
opportunity to become
successful
Your success depends on how
quickly you learn.
Specifically, how quickly you
learn how to deliver:
A product customers value
A sustainable business at scale
Practical tips for scaling
Marketing: Find scalable channels
Always be testing
Massive differences
between channels
Stand for something
Keep your growth sustainable
Return the savings to
your customers or to
growth
Setup backups
Renegotiate as
you scale
Do less.
Time is your most limited resource.
Make the hard choices.
Get the basics
right
Full report here
Know your.Customers.
and focus on the.
customer job.
Don’t die.
Set up your team correctly.
(didn’t speak about but book recommendation .
at the end).
The mission-driven startup
What is TransferWise today?
TRANSFERWISE WAS STARTED.
TO SOLVE A PROBLEM.
WHY DO PEOPLE RECOMMEND
OUR PRODUCT?
RATIONAL EMOTIONAL
PRODUCT BRAND
TO ACHIEVE ADVOCACY YOUR PRODUCT.
NEEDS TO BE AN ORDER OF MAGNITUDE.
BETTER THAN THE ALTERNATIVE.
PRICE
SPEED
COVERAGE
CONVENIENCE
BRAND
A BANK COULD HAVE STARTED
TRANSFERWISE AT ANY POINT IN
THE LAST 100 YEARS
PRICE
SPEED
COVERAGE
CONVENIENCE
A MOVEMENT
PEOPLE DON’T TALK ABOUT
MONEY TRANSFER IN THE PUB
GIVE YOUR CUSTOMERS.
STORIES TO TELL.
PRICE
SPEED
COVERAGE
CONVENIENCE
OWNERSHIP
PRIDE
WHAT’S OUR MISSION?
EVANGELICAL CUSTOMERS.
INSPIRE PEOPLE WITH YOUR CAUSE
AND YOU WILL CREATE
A MOVEMENT.
BIGGER
THAN YOUR STARTUP
AND WHAT’S THE RESULT?
EXPLOSIVE GROWTH
DRIVEN BY
EVANGELICAL
CUSTOMERS
€1 BILLION IN TRANSFERS
PER MONTH
€45 MILLION OF CUSTOMERS
MONEY SAVED EVERY MONTH
TALLINN
LONDON
TAMPA
NEW YORK
REVOLUTION SUPPORTED FROM
BUDAPEST
SINGAPORE
TOKYO
CHERKASY
SYDNEY
join us
Wrap up
Recap
Startups = learning: your job is to learn
Focus on customer jobs to be done (JTBD)
Problem interview → solution interview → build MVP
De-risk with business model canvas
Be 10X better, focus on KPIs, and listen to your customers
From TransferWise:
Cultivate a mission bigger than yourself
Learn more
Get in touch
jeff.mcclelland@yahoo.ca
Skype: jrmcclel
LinkedIn
Useful books you haven’t read yet
Lean startup,
theoretical
Lean startup,
practical
Bootstrapping
(starting without
investors)
Founder equity
that makes sense
These slides:
Questions?
APPENDIX:
KPIs
A/B Testing
What are well-defined KPIs?
You’ll learn faster with good KPIs
Using well-defined KPI’s help
you learn faster:
Which products/features
customers value
How effectively you’re
scaling:
marketing channels
costs
operations
Which activities you should
focus on
5 principles of well-defined KPIs
Directional &
Not
Ambiguous
IntuitiveImportant
Reflects
reality
In control of
the group it’s
measured
against
if performance is good, it's
good for the user and for
the company
Can be explained why it's
important for the user and
to the company
Proportional to reality, dynamic
enough to raise attention (not flat
day-after-day), but not so jumpy
that you can’t discern signal from
the noise
higher or lower is always
better all other things
being equal
so that the team can commit
to and be accountable for
improvement (for targets in
particular)
The best metric/KPI
We did
another thing!
time
the
best
metric
We did
a thing!
Metric is stable and flat
when you’re not
influencing it
KPI Examples
Growth - Signups, Downloads, New users, Active Users, Revenue, Profit
Engagement/conversion - feature use, frequency of use, # features
Quality - Net Promoter Score (NPS), Customer Satisfaction (CSAT)
Profitability/Sustainability/cost per user (marketing and/or OPEX)
Churn - users stop using your product/service, downgrade
Virality/network effects - spreading the word, building the network, market
share
Speed/Efficiency/Backlog/Queues/Time to X - efficiency of your machine
Risk/Exposure - Leverage, FX Exposure, Value at Risk, SLA, cash/runway
Where does A/B testing fit in?
What are A/B tests?
Everyone believes they know
what’s right...but they’re often
don’t.
Everyone has biases
Scientific experiments
A/B Tests can help you learn
If done well they give you
unbiased “proof” of whether A is
better than B
When should you use A/B testing?
Work well for optimisation
problems - small iterative
learning
Product, pricing, messaging,
support all possible
But even if you could, you
sometimes shouldn’t as they can
be expensive. Learning is
expensive.
Get high volumes and be patient
waiting on results
Don’t corrupt with other
overlapping tests
Know basic statistics
AB testing 101
Are your results significant? tl;dr: no
It’s difficult to achieve statistical
significance
You need to:
Plan/design in advance
Pick the right KPI(s) & scope
Random or pseudo-random
allocation

Más contenido relacionado

La actualidad más candente

Digital marketing plan
Digital marketing plan Digital marketing plan
Digital marketing plan Vanessa Baires
 
Why Startups Suck at Marketing
Why Startups Suck at MarketingWhy Startups Suck at Marketing
Why Startups Suck at MarketingRand Fishkin
 
Mixpanel - Our pitch deck that we used to raise $65M
Mixpanel - Our pitch deck that we used to raise $65MMixpanel - Our pitch deck that we used to raise $65M
Mixpanel - Our pitch deck that we used to raise $65MSuhail Doshi
 
Vee Park Long Presentation
Vee Park Long PresentationVee Park Long Presentation
Vee Park Long PresentationEdward Irby
 
Building a Digital Marketing Plan
Building a Digital Marketing PlanBuilding a Digital Marketing Plan
Building a Digital Marketing PlanBehrouz Jafarnezhad
 
24 Design Tips from Real Designers
24 Design Tips from Real Designers24 Design Tips from Real Designers
24 Design Tips from Real DesignersEdahn Small
 
31 Best Growth Hacking Resources
31 Best Growth Hacking Resources31 Best Growth Hacking Resources
31 Best Growth Hacking ResourcesStephen Jeske
 
Value Proposition Canvas Explained
Value Proposition Canvas ExplainedValue Proposition Canvas Explained
Value Proposition Canvas Explainedazlaan
 
Mobile-First SEO - The Marketers Edition #3XEDigital
Mobile-First SEO - The Marketers Edition #3XEDigitalMobile-First SEO - The Marketers Edition #3XEDigital
Mobile-First SEO - The Marketers Edition #3XEDigitalAleyda Solís
 
5 Prover Customer Acquisition Strategies For B2C Startups
5 Prover Customer Acquisition Strategies For B2C Startups5 Prover Customer Acquisition Strategies For B2C Startups
5 Prover Customer Acquisition Strategies For B2C StartupsMarketing Masala
 
26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...
26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...
26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...European Innovation Academy
 
Building a Sales and Marketing Machine
Building a Sales and Marketing MachineBuilding a Sales and Marketing Machine
Building a Sales and Marketing MachineDavid Im
 
Linkedin Series B Pitch Deck
Linkedin Series B Pitch DeckLinkedin Series B Pitch Deck
Linkedin Series B Pitch DeckJoseph Hsieh
 
Foursquare's 1st Pitch Deck
Foursquare's 1st Pitch DeckFoursquare's 1st Pitch Deck
Foursquare's 1st Pitch DeckRami Al-Karmi
 
Training manual - customer development
Training manual - customer developmentTraining manual - customer development
Training manual - customer developmentStartup Braga
 
WeWork Series D Pitch Deck ($355M Raised)
WeWork Series D Pitch Deck ($355M Raised)WeWork Series D Pitch Deck ($355M Raised)
WeWork Series D Pitch Deck ($355M Raised)startuphome
 
SaaS Metrics: The Secret to Subscription Success
SaaS Metrics: The Secret to Subscription SuccessSaaS Metrics: The Secret to Subscription Success
SaaS Metrics: The Secret to Subscription SuccessEd Shelley
 

La actualidad más candente (20)

Uber pitch deck
Uber pitch deckUber pitch deck
Uber pitch deck
 
Digital marketing plan
Digital marketing plan Digital marketing plan
Digital marketing plan
 
Why Startups Suck at Marketing
Why Startups Suck at MarketingWhy Startups Suck at Marketing
Why Startups Suck at Marketing
 
Mixpanel - Our pitch deck that we used to raise $65M
Mixpanel - Our pitch deck that we used to raise $65MMixpanel - Our pitch deck that we used to raise $65M
Mixpanel - Our pitch deck that we used to raise $65M
 
Vee Park Long Presentation
Vee Park Long PresentationVee Park Long Presentation
Vee Park Long Presentation
 
Building a Digital Marketing Plan
Building a Digital Marketing PlanBuilding a Digital Marketing Plan
Building a Digital Marketing Plan
 
24 Design Tips from Real Designers
24 Design Tips from Real Designers24 Design Tips from Real Designers
24 Design Tips from Real Designers
 
31 Best Growth Hacking Resources
31 Best Growth Hacking Resources31 Best Growth Hacking Resources
31 Best Growth Hacking Resources
 
Value Proposition Canvas Explained
Value Proposition Canvas ExplainedValue Proposition Canvas Explained
Value Proposition Canvas Explained
 
Mobile-First SEO - The Marketers Edition #3XEDigital
Mobile-First SEO - The Marketers Edition #3XEDigitalMobile-First SEO - The Marketers Edition #3XEDigital
Mobile-First SEO - The Marketers Edition #3XEDigital
 
5 Prover Customer Acquisition Strategies For B2C Startups
5 Prover Customer Acquisition Strategies For B2C Startups5 Prover Customer Acquisition Strategies For B2C Startups
5 Prover Customer Acquisition Strategies For B2C Startups
 
26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...
26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...
26_07_Growth Mindset to Growth Hacking_ Tools to Win Your First Customers_Kri...
 
Building a Sales and Marketing Machine
Building a Sales and Marketing MachineBuilding a Sales and Marketing Machine
Building a Sales and Marketing Machine
 
Linkedin Series B Pitch Deck
Linkedin Series B Pitch DeckLinkedin Series B Pitch Deck
Linkedin Series B Pitch Deck
 
Foursquare's 1st Pitch Deck
Foursquare's 1st Pitch DeckFoursquare's 1st Pitch Deck
Foursquare's 1st Pitch Deck
 
Training manual - customer development
Training manual - customer developmentTraining manual - customer development
Training manual - customer development
 
WeWork Series D Pitch Deck ($355M Raised)
WeWork Series D Pitch Deck ($355M Raised)WeWork Series D Pitch Deck ($355M Raised)
WeWork Series D Pitch Deck ($355M Raised)
 
Build Features, Not Apps
Build Features, Not AppsBuild Features, Not Apps
Build Features, Not Apps
 
SaaS Metrics: The Secret to Subscription Success
SaaS Metrics: The Secret to Subscription SuccessSaaS Metrics: The Secret to Subscription Success
SaaS Metrics: The Secret to Subscription Success
 
StairWear Pitch Deck
StairWear Pitch DeckStairWear Pitch Deck
StairWear Pitch Deck
 

Destacado

The TransferWise Revolution - FinTech and the customer revolution in banking
The TransferWise Revolution - FinTech and the customer revolution in bankingThe TransferWise Revolution - FinTech and the customer revolution in banking
The TransferWise Revolution - FinTech and the customer revolution in bankingNilan Peiris
 
Startup KPIs and A/B Testing
Startup KPIs and A/B TestingStartup KPIs and A/B Testing
Startup KPIs and A/B TestingJeff McClelland
 
Scaling transferwise
Scaling transferwiseScaling transferwise
Scaling transferwiseNilan Peiris
 
Retail Banking Innovations N26-TransferWise
Retail Banking Innovations N26-TransferWiseRetail Banking Innovations N26-TransferWise
Retail Banking Innovations N26-TransferWiseJordan Valdma
 
Marketing in a mission driven startup
Marketing in a mission driven startupMarketing in a mission driven startup
Marketing in a mission driven startupNilan Peiris
 
Engineering at TransferWise
Engineering at TransferWiseEngineering at TransferWise
Engineering at TransferWiseErik Kaju
 
NPS Driven Growth or How to grow your startup w
NPS Driven Growth or How to grow your startup wNPS Driven Growth or How to grow your startup w
NPS Driven Growth or How to grow your startup wNilan Peiris
 
Product engineering @ TransferWise
Product engineering @ TransferWiseProduct engineering @ TransferWise
Product engineering @ TransferWiseAlvar Lumberg
 
Design at TransferWise
Design at TransferWiseDesign at TransferWise
Design at TransferWiseNilan Peiris
 
Marketing in FinTech
Marketing in FinTechMarketing in FinTech
Marketing in FinTechActa School
 
UX Camp 2015: Conversion Rate Optimierung
UX Camp 2015: Conversion Rate OptimierungUX Camp 2015: Conversion Rate Optimierung
UX Camp 2015: Conversion Rate OptimierungFerdinand v. Seggern
 
A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....
A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....
A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....etracker GmbH
 
Customer validation with Diary Studies
Customer validation with Diary StudiesCustomer validation with Diary Studies
Customer validation with Diary StudiesBoon Yew Chew
 
OneHourTranslation - AWS Cloud Case Study
OneHourTranslation - AWS Cloud Case StudyOneHourTranslation - AWS Cloud Case Study
OneHourTranslation - AWS Cloud Case StudyNewvewm
 
Coming to America: How to Set Up Shop in the U.S.
Coming to America: How to Set Up Shop in the U.S.Coming to America: How to Set Up Shop in the U.S.
Coming to America: How to Set Up Shop in the U.S.Ott Ilves
 
Building a Startup Team (ECUT 2014)
Building a Startup Team (ECUT 2014)Building a Startup Team (ECUT 2014)
Building a Startup Team (ECUT 2014)Alvar Lumberg
 
Continuous Delivery intro @ Devclub
Continuous Delivery intro @ DevclubContinuous Delivery intro @ Devclub
Continuous Delivery intro @ DevclubAlvar Lumberg
 

Destacado (20)

The TransferWise Revolution - FinTech and the customer revolution in banking
The TransferWise Revolution - FinTech and the customer revolution in bankingThe TransferWise Revolution - FinTech and the customer revolution in banking
The TransferWise Revolution - FinTech and the customer revolution in banking
 
Startup KPIs and A/B Testing
Startup KPIs and A/B TestingStartup KPIs and A/B Testing
Startup KPIs and A/B Testing
 
Scaling transferwise
Scaling transferwiseScaling transferwise
Scaling transferwise
 
Retail Banking Innovations N26-TransferWise
Retail Banking Innovations N26-TransferWiseRetail Banking Innovations N26-TransferWise
Retail Banking Innovations N26-TransferWise
 
Marketing in a mission driven startup
Marketing in a mission driven startupMarketing in a mission driven startup
Marketing in a mission driven startup
 
Engineering at TransferWise
Engineering at TransferWiseEngineering at TransferWise
Engineering at TransferWise
 
NPS Driven Growth or How to grow your startup w
NPS Driven Growth or How to grow your startup wNPS Driven Growth or How to grow your startup w
NPS Driven Growth or How to grow your startup w
 
Product engineering @ TransferWise
Product engineering @ TransferWiseProduct engineering @ TransferWise
Product engineering @ TransferWise
 
Design at TransferWise
Design at TransferWiseDesign at TransferWise
Design at TransferWise
 
Marketing in FinTech
Marketing in FinTechMarketing in FinTech
Marketing in FinTech
 
KPI driven growth
KPI driven growthKPI driven growth
KPI driven growth
 
UX Camp 2015: Conversion Rate Optimierung
UX Camp 2015: Conversion Rate OptimierungUX Camp 2015: Conversion Rate Optimierung
UX Camp 2015: Conversion Rate Optimierung
 
A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....
A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....
A/B-Testing - aber richtig. Mit gezieltem A/B-Testing zu mehr Website-Erfolg....
 
Customer validation with Diary Studies
Customer validation with Diary StudiesCustomer validation with Diary Studies
Customer validation with Diary Studies
 
OneHourTranslation - AWS Cloud Case Study
OneHourTranslation - AWS Cloud Case StudyOneHourTranslation - AWS Cloud Case Study
OneHourTranslation - AWS Cloud Case Study
 
Coming to America: How to Set Up Shop in the U.S.
Coming to America: How to Set Up Shop in the U.S.Coming to America: How to Set Up Shop in the U.S.
Coming to America: How to Set Up Shop in the U.S.
 
Building a Startup Team (ECUT 2014)
Building a Startup Team (ECUT 2014)Building a Startup Team (ECUT 2014)
Building a Startup Team (ECUT 2014)
 
Continuous Delivery intro @ Devclub
Continuous Delivery intro @ DevclubContinuous Delivery intro @ Devclub
Continuous Delivery intro @ Devclub
 
TransferWise
TransferWise TransferWise
TransferWise
 
Transfer
TransferTransfer
Transfer
 

Similar a Customer, market and business validation for early-stage startups

Mvc denver startup week 2014
Mvc denver startup week 2014Mvc denver startup week 2014
Mvc denver startup week 2014perryevans
 
Ryan's Lean Startup 3-Hour Workshop
Ryan's Lean Startup 3-Hour WorkshopRyan's Lean Startup 3-Hour Workshop
Ryan's Lean Startup 3-Hour WorkshopRyan D. Hatch
 
Find the Best Customer Success Software & Grow Lifetime Customers
Find the Best Customer Success Software & Grow Lifetime CustomersFind the Best Customer Success Software & Grow Lifetime Customers
Find the Best Customer Success Software & Grow Lifetime CustomersQuekelsBaro
 
Milestones That Get Investor Interest = Traction
Milestones That Get Investor Interest = TractionMilestones That Get Investor Interest = Traction
Milestones That Get Investor Interest = TractionKelly Schwedland
 
What comes after product market fit?
What comes after product market fit?What comes after product market fit?
What comes after product market fit?Intercom
 
So you want to be customer centric?
So you want to be customer centric?So you want to be customer centric?
So you want to be customer centric?Futurelab
 
Ryan's Lean Startup Introduction
Ryan's Lean Startup IntroductionRyan's Lean Startup Introduction
Ryan's Lean Startup IntroductionRyan D. Hatch
 
How to close your first 10 B2B Deals
How to close your first 10 B2B DealsHow to close your first 10 B2B Deals
How to close your first 10 B2B DealsAmbassify
 
From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...Marko Taipale
 
From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...Gosei Oy
 
Crafting Products from idea to an MVP
Crafting Products from idea to an MVPCrafting Products from idea to an MVP
Crafting Products from idea to an MVPAlexandr Gribenko
 
Message-Market Fit VWO Workshop by Daphne tideman
Message-Market Fit VWO Workshop by Daphne tidemanMessage-Market Fit VWO Workshop by Daphne tideman
Message-Market Fit VWO Workshop by Daphne tidemanVWO
 
Learn build measure building products customers love
Learn build measure building products customers loveLearn build measure building products customers love
Learn build measure building products customers loveRahul PruthI
 
Startup Business Models
Startup Business Models Startup Business Models
Startup Business Models Dan Nelson
 
Writing an effective brief - Panteion University, Ad & PR Lab
Writing an effective brief - Panteion University, Ad & PR LabWriting an effective brief - Panteion University, Ad & PR Lab
Writing an effective brief - Panteion University, Ad & PR LabLina Kiriakou
 
Eugene Crehan - Understanding Your Business Model
Eugene Crehan - Understanding Your Business ModelEugene Crehan - Understanding Your Business Model
Eugene Crehan - Understanding Your Business ModelBizcamp South East
 

Similar a Customer, market and business validation for early-stage startups (20)

Mvc denver startup week 2014
Mvc denver startup week 2014Mvc denver startup week 2014
Mvc denver startup week 2014
 
Ryan's Lean Startup 3-Hour Workshop
Ryan's Lean Startup 3-Hour WorkshopRyan's Lean Startup 3-Hour Workshop
Ryan's Lean Startup 3-Hour Workshop
 
Find the Best Customer Success Software & Grow Lifetime Customers
Find the Best Customer Success Software & Grow Lifetime CustomersFind the Best Customer Success Software & Grow Lifetime Customers
Find the Best Customer Success Software & Grow Lifetime Customers
 
Milestones That Get Investor Interest = Traction
Milestones That Get Investor Interest = TractionMilestones That Get Investor Interest = Traction
Milestones That Get Investor Interest = Traction
 
What comes after product market fit?
What comes after product market fit?What comes after product market fit?
What comes after product market fit?
 
So you want to be customer centric?
So you want to be customer centric?So you want to be customer centric?
So you want to be customer centric?
 
Basics of a pitch deck
Basics of a pitch deckBasics of a pitch deck
Basics of a pitch deck
 
Ryan's Lean Startup Introduction
Ryan's Lean Startup IntroductionRyan's Lean Startup Introduction
Ryan's Lean Startup Introduction
 
How to close your first 10 B2B Deals
How to close your first 10 B2B DealsHow to close your first 10 B2B Deals
How to close your first 10 B2B Deals
 
From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...
 
From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...From a concept to viable business — How do we know if we are building the rig...
From a concept to viable business — How do we know if we are building the rig...
 
Crafting Products from idea to an MVP
Crafting Products from idea to an MVPCrafting Products from idea to an MVP
Crafting Products from idea to an MVP
 
Better Content, Meaningful Reach: How to Grow Your B2B Efforts - Susan Wenogr...
Better Content, Meaningful Reach: How to Grow Your B2B Efforts - Susan Wenogr...Better Content, Meaningful Reach: How to Grow Your B2B Efforts - Susan Wenogr...
Better Content, Meaningful Reach: How to Grow Your B2B Efforts - Susan Wenogr...
 
Message-Market Fit VWO Workshop by Daphne tideman
Message-Market Fit VWO Workshop by Daphne tidemanMessage-Market Fit VWO Workshop by Daphne tideman
Message-Market Fit VWO Workshop by Daphne tideman
 
Startup in a nutshell
Startup in a nutshellStartup in a nutshell
Startup in a nutshell
 
Learn build measure building products customers love
Learn build measure building products customers loveLearn build measure building products customers love
Learn build measure building products customers love
 
Startup Business Models
Startup Business Models Startup Business Models
Startup Business Models
 
Writing an effective brief - Panteion University, Ad & PR Lab
Writing an effective brief - Panteion University, Ad & PR LabWriting an effective brief - Panteion University, Ad & PR Lab
Writing an effective brief - Panteion University, Ad & PR Lab
 
Eugene Crehan - Understanding Your Business Model
Eugene Crehan - Understanding Your Business ModelEugene Crehan - Understanding Your Business Model
Eugene Crehan - Understanding Your Business Model
 
FMG Life Risk Advisers Training
FMG Life Risk Advisers TrainingFMG Life Risk Advisers Training
FMG Life Risk Advisers Training
 

Último

Sangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Sangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceSangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Sangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDamini Dixit
 
EV Electric Vehicle Startup Pitch Deck- StartupSprouts.in
EV Electric Vehicle Startup Pitch Deck- StartupSprouts.inEV Electric Vehicle Startup Pitch Deck- StartupSprouts.in
EV Electric Vehicle Startup Pitch Deck- StartupSprouts.inStartupSprouts.in
 
Bangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Bangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceBangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Bangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDamini Dixit
 
Tirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Tirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceTirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Tirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDamini Dixit
 
Hyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Hyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceHyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Hyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDamini Dixit
 
Famedesired Project portfolio1 . Fullsail
Famedesired Project portfolio1 . FullsailFamedesired Project portfolio1 . Fullsail
Famedesired Project portfolio1 . Fullsailfergusonamani
 
Dàni Velvet Personal Brand Exploration (1).pptx
Dàni Velvet Personal Brand Exploration (1).pptxDàni Velvet Personal Brand Exploration (1).pptx
Dàni Velvet Personal Brand Exploration (1).pptxdmtillman
 
Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...
Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...
Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...StartupSprouts.in
 
Sohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort Service
Sohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort ServiceSohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort Service
Sohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort ServiceDamini Dixit
 
Call girls in Andheri with phone number 9892124323
Call girls in Andheri with phone number 9892124323Call girls in Andheri with phone number 9892124323
Call girls in Andheri with phone number 9892124323Pooja Nehwal
 
Connaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verified
Connaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verifiedConnaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verified
Connaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verifiedDelhi Call girls
 
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verifiedSector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verifiedDelhi Call girls
 
Lucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Lucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceLucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Lucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDamini Dixit
 
NEON LIGHT CITY pitch deck for the new PC game
NEON LIGHT CITY pitch deck for the new PC gameNEON LIGHT CITY pitch deck for the new PC game
NEON LIGHT CITY pitch deck for the new PC gametess51
 
Dehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Dehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Dehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDamini Dixit
 
Karol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verified
Karol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verifiedKarol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verified
Karol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verifiedDelhi Call girls
 

Último (16)

Sangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Sangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceSangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Sangareddy Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
 
EV Electric Vehicle Startup Pitch Deck- StartupSprouts.in
EV Electric Vehicle Startup Pitch Deck- StartupSprouts.inEV Electric Vehicle Startup Pitch Deck- StartupSprouts.in
EV Electric Vehicle Startup Pitch Deck- StartupSprouts.in
 
Bangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Bangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceBangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Bangalore Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
 
Tirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Tirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceTirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Tirupati Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
 
Hyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Hyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceHyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Hyderabad Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
 
Famedesired Project portfolio1 . Fullsail
Famedesired Project portfolio1 . FullsailFamedesired Project portfolio1 . Fullsail
Famedesired Project portfolio1 . Fullsail
 
Dàni Velvet Personal Brand Exploration (1).pptx
Dàni Velvet Personal Brand Exploration (1).pptxDàni Velvet Personal Brand Exploration (1).pptx
Dàni Velvet Personal Brand Exploration (1).pptx
 
Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...
Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...
Shareholders Agreement Template for Compulsorily Convertible Debt Funding- St...
 
Sohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort Service
Sohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort ServiceSohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort Service
Sohna Call Girls Service ☎ ️93326-06886 ❤️‍🔥 Enjoy 24/7 Escort Service
 
Call girls in Andheri with phone number 9892124323
Call girls in Andheri with phone number 9892124323Call girls in Andheri with phone number 9892124323
Call girls in Andheri with phone number 9892124323
 
Connaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verified
Connaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verifiedConnaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verified
Connaught Place, Delhi Call girls :8448380779 Model Escorts | 100% verified
 
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verifiedSector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
 
Lucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Lucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceLucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Lucknow Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
 
NEON LIGHT CITY pitch deck for the new PC game
NEON LIGHT CITY pitch deck for the new PC gameNEON LIGHT CITY pitch deck for the new PC game
NEON LIGHT CITY pitch deck for the new PC game
 
Dehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Dehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort ServiceDehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
Dehradun Call Girls Service ☎ ️82500–77686 ☎️ Enjoy 24/7 Escort Service
 
Karol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verified
Karol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verifiedKarol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verified
Karol Bagh, Delhi Call girls :8448380779 Model Escorts | 100% verified
 

Customer, market and business validation for early-stage startups

  • 1. Customer, market and business validation For early stage startups + case study: TransferWise ~ the mission-driven startup
  • 2. Hi, I’m Jeff! Integral Designs Bell Canada ING Bank Skype → Microsoft TransferWise MSc - Info Science Bachelors - Business
  • 3. 4 steps to world domination Your customer Your Business Scale your startup Your Market Validate Case Study Grow
  • 5. How many startups START. Intuitive understanding of user need Passion for particular technology Propose solution = Successful startup?
  • 6. How many startups STOP. Intuitive understanding of user need Passion for particular technology Propose solution = Successful startup? ← Not tested. ← Customer’s don’t care. ← Not based on actual. customer needs. ← Not as likely.
  • 7. Jobs to be done (JTBD) Understand Customer + Situation “Your customers have a job and they’re hiring your app to solve it” JTBD: weather apps, iPhone apps, SaaS/Services
  • 8. Learn by focusing on JTBD How valuable is getting that job done for this type of person? How many people have this job? How often do they have this job? How is this job currently being done? JTBD: more info/exercises ← Willingness to pay. ← Market size. ← Usage frequency/pattern. ← Actual competition.. (usually status quo).
  • 9. Problem interviews Your goal is to learn, not sell Start before you build anything 15 mins per interview Do 20-60 of them Learn more: Jeff’s highlights, from this book Demographics - who are they? Tell a story - that covers several problems they might have Problem ranking - which problem is most painful? Explore problems - go in depth into each problem they have Hook & Ask - keep them engaged Document - take 5 mins, write down
  • 10. Structure: intro, demographics, story Demo, as real as possible, iterate Test price, don’t ask price Raise friction, don’t lower it. Find early adopters. Be the prize. Solution interviews Your startup goal is to build the simplest product that. solves a real problem. Validate your understanding of (1) customer, (2) product, (3) market proposition Visual demo of an MVP for early adopters 20-60 interviews needed The best blog post on pricing
  • 11. ...but... You know what customers need It takes too long It’s a waste of time
  • 12. ...but... You know what customers need It takes too long It’s a waste of time ← No you don’t, unless you’ve. talked to enough of them, in a. structured way. ← it’s nothing compared to.building. something for 6 months and discover. nobody will buy it. ← not if you learn what to build. and who will buy it.
  • 15. Increase probability of success by understanding your key risks Start with the big boxes Value proposition, customer segments, key partners Start with whatever is most risky, and figure out what you need to learn to remove that risk. Ask the hardest questions first Who’s this for? Why do they need it? How much money do you need to make to break even? Will anyone actually pay for that? How much will it cost you?
  • 16. ✅ ✅ ✅ ✅ ❓ ❓ ❓ ❓
  • 17. Collect high quality data & validate it Make it easy for potential customers to tell you what they want: Currency “wishlist”
  • 18. Research & explore Use data where you can… but not as your only source Ask “so what” and find the insights - who can we help?
  • 20. Launching your MVP Minimum Viable Product
  • 21. Be an order of magnitude better You need to be way better than the alternative. Aim for 10X 10X faster 10X cheaper 10X more convenient 10X easier to use or a combination
  • 22. MVP Learning Cycle MVP - does not mean “minimum viable piece of crap” You need to understand who your customers are and what they want. Can your customers use your product? Can you do for the customer what you actually promise?
  • 23. Is your MVP Successful? You need to understand your metrics and set realistic targets! This will help you define success and start iterating. For instance: New users NPS Cancellations Conversion See appendix for more KPI examples
  • 24. Learn more about NPS, Product/Market fit Achieving Market Fit NPS! (Net Promoter Score) Virality Comments Work on the detractors Then neutrals Then profit
  • 25. Early startup KPIs Do Not Focus On Vanity metrics i.e. total registered users, total volume transferred Getting statistical significance, split testing (A/B testing) More than a handful of metrics Focus on The 3 or 4 most important metrics (KPIs) that indicate your progress. Perhaps: Growth & conversion How you’re adding value How much runway you have Helpful to work backwards from some target state (i.e. 10K new users) and validate how you get there Learn more about KPIs in the appendix
  • 27. Listen to customers They are usually your best critics and advisors! Surveys Phonecalls Meetings The trick is not to give them what they want but what they need.
  • 28. Iterate Usually the one that iterates fastest wins! Prioritising is they key here Make sure you know what you effect and measure it! A/B testing
  • 30. Your startup grows at. the rate. you learn.
  • 31. What is scaling? You have a mountain to climb to achieve your vision - KPIs are your guideposts Validation: It’s proof that you’re moving forward Investors/valuations
  • 32. You have a hard job You have a limited opportunity to become successful Your success depends on how quickly you learn. Specifically, how quickly you learn how to deliver: A product customers value A sustainable business at scale
  • 34. Marketing: Find scalable channels Always be testing Massive differences between channels Stand for something
  • 35. Keep your growth sustainable Return the savings to your customers or to growth Setup backups Renegotiate as you scale
  • 36. Do less. Time is your most limited resource. Make the hard choices.
  • 37. Get the basics right Full report here Know your.Customers. and focus on the. customer job. Don’t die. Set up your team correctly. (didn’t speak about but book recommendation . at the end).
  • 40. TRANSFERWISE WAS STARTED. TO SOLVE A PROBLEM.
  • 41. WHY DO PEOPLE RECOMMEND OUR PRODUCT?
  • 44. TO ACHIEVE ADVOCACY YOUR PRODUCT. NEEDS TO BE AN ORDER OF MAGNITUDE. BETTER THAN THE ALTERNATIVE.
  • 46. A BANK COULD HAVE STARTED TRANSFERWISE AT ANY POINT IN THE LAST 100 YEARS
  • 48. PEOPLE DON’T TALK ABOUT MONEY TRANSFER IN THE PUB
  • 52.
  • 54. INSPIRE PEOPLE WITH YOUR CAUSE AND YOU WILL CREATE A MOVEMENT. BIGGER THAN YOUR STARTUP
  • 55. AND WHAT’S THE RESULT?
  • 58. €1 BILLION IN TRANSFERS PER MONTH
  • 59. €45 MILLION OF CUSTOMERS MONEY SAVED EVERY MONTH
  • 60. TALLINN LONDON TAMPA NEW YORK REVOLUTION SUPPORTED FROM BUDAPEST SINGAPORE TOKYO CHERKASY SYDNEY join us
  • 62. Recap Startups = learning: your job is to learn Focus on customer jobs to be done (JTBD) Problem interview → solution interview → build MVP De-risk with business model canvas Be 10X better, focus on KPIs, and listen to your customers From TransferWise: Cultivate a mission bigger than yourself
  • 63. Learn more Get in touch jeff.mcclelland@yahoo.ca Skype: jrmcclel LinkedIn Useful books you haven’t read yet Lean startup, theoretical Lean startup, practical Bootstrapping (starting without investors) Founder equity that makes sense These slides:
  • 67. You’ll learn faster with good KPIs Using well-defined KPI’s help you learn faster: Which products/features customers value How effectively you’re scaling: marketing channels costs operations Which activities you should focus on
  • 68. 5 principles of well-defined KPIs Directional & Not Ambiguous IntuitiveImportant Reflects reality In control of the group it’s measured against if performance is good, it's good for the user and for the company Can be explained why it's important for the user and to the company Proportional to reality, dynamic enough to raise attention (not flat day-after-day), but not so jumpy that you can’t discern signal from the noise higher or lower is always better all other things being equal so that the team can commit to and be accountable for improvement (for targets in particular)
  • 69. The best metric/KPI We did another thing! time the best metric We did a thing! Metric is stable and flat when you’re not influencing it
  • 70. KPI Examples Growth - Signups, Downloads, New users, Active Users, Revenue, Profit Engagement/conversion - feature use, frequency of use, # features Quality - Net Promoter Score (NPS), Customer Satisfaction (CSAT) Profitability/Sustainability/cost per user (marketing and/or OPEX) Churn - users stop using your product/service, downgrade Virality/network effects - spreading the word, building the network, market share Speed/Efficiency/Backlog/Queues/Time to X - efficiency of your machine Risk/Exposure - Leverage, FX Exposure, Value at Risk, SLA, cash/runway
  • 71. Where does A/B testing fit in?
  • 72. What are A/B tests? Everyone believes they know what’s right...but they’re often don’t. Everyone has biases Scientific experiments A/B Tests can help you learn If done well they give you unbiased “proof” of whether A is better than B
  • 73. When should you use A/B testing? Work well for optimisation problems - small iterative learning Product, pricing, messaging, support all possible But even if you could, you sometimes shouldn’t as they can be expensive. Learning is expensive.
  • 74. Get high volumes and be patient waiting on results Don’t corrupt with other overlapping tests Know basic statistics AB testing 101 Are your results significant? tl;dr: no It’s difficult to achieve statistical significance You need to: Plan/design in advance Pick the right KPI(s) & scope Random or pseudo-random allocation