Main takeaways: - Start with hypotheses and measurable outcomes instead of features - Make Personas data driven - Celebrate the “landings," not just launches

1. www.productschool.com
Shifting to Hypothesis-Driven Dev at
Scale by Squarespace Sr PM

2. Join 55,000+
Product Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™
20 HOURS40 HOURS40 HOURS

5. Corporate
Training
Level up your team’s Product
Management skills

6. Jonathan
Hastings
T O N I G H T ’ S S P E A K E R

7. @jonathan_sqsp
linkedin.com/in/jonathanshastings
Jonathan Hastings
Group Product Manager, Growth

8. Squarespace makes beautiful products to help
people with creative ideas succeed.

9. Squarespace makes beautiful products to help
people with creative ideas succeed.
Our platform empowers millions of people – from
individuals and local artists to entrepreneurs shaping
the world's most iconic businesses – to share their
stories and create an impactful, stylish, and
easy-to-manage online presence.

10. Agenda
01 Context
02 HDD Approach & Process
03 Adapted Software Development Lifecycle
04 Roadmap
05 Application

11. But first,
a quick story…
Art and Fear:
Observations on the Perils (and Rewards) of Artmaking,
by David Bayles and Ted Orland

12. Two classes, otherwise
identical from which a
natural A/B test
emerged.
Art and Fear:
Observations on the Perils (and Rewards) of Artmaking,
by David Bayles and Ted Orland

13. 2 Ceramics
Classes
The first, judged on
the quality of a single
vase.

14. 2 Ceramics
Classes
The second, judged
on the quantity
produced.

15. 2 Ceramics
Classes
The semester proceeds similarly for both
classes, learning techniques and
methods for manipulating the clay.

16. 2 Ceramics
Classes
The semester proceeds similarly for both
classes, learning techniques and
methods for manipulating the clay.
But as the first class attempted to
apply those methods to their singular
vase, the second group incorporated
what they’ve learned into the next
vase they make.

17. 2 Ceramics
ClassesBy the end of the course, it
emerges that the second
class had produced pieces
of significantly higher
quality.

18. 2 Ceramics
Classes
“What you need to
know about the next
piece is contained in
the last piece.”
Art and Fear:
Observations on the Perils (and Rewards) of Artmaking,
by David Bayles and Ted Orland

19. “Teams often focus on the delivery of a thing, instead of
its business impact. This happens when we define
success by short timeframes, and performance relates
to output. We also tend to rely on our intuition or
acumen, without seeking any validation. Even with years
of experience in a domain, intuition can be flawed by our
biases. Left unchecked, we define and pursue solutions
before fully understanding the opportunity space.”
- Jonny Schneider
Principal, ThoughtWorks

20. Context
01 Context
02 HDD Approach & Process
03 Adapted Software Development Lifecycle
04 Roadmap
05 Application

21. HYPOTHESIS DRIVEN
DEVELOPMENT
21
2015 2017 2019

22. HYPOTHESIS DRIVEN
DEVELOPMENT
2
2
At Jet, our early strategy was a combination of
market-leading innovation and parity with Amazon
Membership Model
Price at Marginal Cost
Smart Cart Dynamic Discounts
Competitive assortment through
3P/marketplace coverage
“Brand friendly” policies
Closed Marketplace

23. HYPOTHESIS DRIVEN
DEVELOPMENT
23
But very quickly we learned that only some of the
initial assumptions were validated by the market
and by our users.
Membership Model
Price at Marginal Cost
Smart Cart Dynamic Discounts
Competitive assortment through
3P/marketplace coverage
“Brand friendly” policies
Closed Marketplace

24. HYPOTHESIS DRIVEN
DEVELOPMENT
24
We found product-market fit
(and gained traction) by
understanding user needs and
focusing on the parts of the
business that resonated with
shoppers.

25. HYPOTHESIS DRIVEN
DEVELOPMENT
25
At RTR, we were figuring out how to build an
experience that maximized value for users during a
time when our user base was diversifying.

26. HYPOTHESIS DRIVEN
DEVELOPMENT
26
We had new kinds of traffic coming into the app
and we needed to deliver relevant experiences to
them.
SECTION
SUBSECTION

27. HYPOTHESIS DRIVEN
DEVELOPMENT
27
We didn’t know exactly what to
build, so we deliberately designed
aspects of the product to identify
user needs, provide the business
with insights, and deliver increased
value from the service.

28. HYPOTHESIS DRIVEN
DEVELOPMENT
28
2015 2017 2019

29. HYPOTHESIS DRIVEN
DEVELOPMENT
29
We had two teams with overlapping areas of
development and somewhat vague objectives
• Template Store
• Linear Onboarding
• User Affinity Service
• Pendo Onboarding
• Onboarding Wizard
CONVERSION TRIAL EXPERIENCE

30. HYPOTHESIS DRIVEN
DEVELOPMENT
30
The teams had built several
features but few had the
impact we’d hoped for.

31. HYPOTHESIS DRIVEN
DEVELOPMENT
31
SECTION
SUBSECTION
The teams had built several
features but few had the
impact we’d hoped for.
Even though we had been
“using data!”

32. HYPOTHESIS DRIVEN
DEVELOPMENT
32
The teams had done A/B tests…
SECTION
SUBSECTION

33. HYPOTHESIS DRIVEN
DEVELOPMENT
33
The teams had done A/B tests…
SECTION
SUBSECTION
“Hermione”
Linear Onboarding
Pendo Onboarding
Intent Interstitial v1
Template Store Search
Layout filters
Frontsite Redesign
In-app KB Search
Customer Examples on Template Store
Template Store Reordering
Frontsite CTA copy changes
“Supervariant”
Template Store Search v2
Template Store Search v3
In app chat in checkout
Starter Page Layout Test
New Template Store Categories
Member Account Upgrade Button
Mobile Member Account-First
“Launch Pages”
Site Visibility

34. HYPOTHESIS DRIVEN
DEVELOPMENT
34
But most didn’t have a positive impact.
“Hermione”
Linear Onboarding
Pendo Onboarding
Intent Interstitial v1
Template Store Search
Layout filters
Frontsite Redesign
In-app KB Search
Customer Examples on Template Store
Template Store Reordering
Frontsite CTA copy changes
“Supervariant”
Template Store Search v2
Template Store Search v3
In app chat in checkout
Starter Page Layout Test
New Template Store Categories
Member Account Upgrade Button
Mobile Member Account-First
“Launch Pages”
Site Visibility

35. HYPOTHESIS DRIVEN
DEVELOPMENT
35
SECTION
SUBSECTION
“Hermione”
Linear Onboarding
Pendo Onboarding
Intent Interstitial v1
Template Store Search
Layout filters
Frontsite Redesign
In-app KB Search
Customer Examples on Template Store
Template Store Reordering
Frontsite CTA copy changes
“Supervariant”
Template Store Search v2
Template Store Search v3
In app chat in checkout
Starter Page Layout Test
New Template Store Categories
Member Account Upgrade Button
Mobile Member Account-First
“Launch Pages”
Site Visibility
And only one had a measurable effect on
the business.

36. HYPOTHESIS DRIVEN
DEVELOPMENT
36
We had (mostly)
disconnected tests that
didn’t systematically reveal
information about our users
or the market.

37. Approach
01 Context
02 HDD Approach & Process
03 Adapted Software Development Lifecycle
04 Roadmap
05 Application

38. New Development
Framework
Based on the British Design Council’s Double Diamond

39. “Double Diamond” Process
NEW FRAMEWORK
Divergent and convergent stages of a design process
DEFINE STRATEGY
Understand why and define how
EXECUTE SOLUTION
Create the desired outcome
UNDERSTAND DEFINE EXPLORE CREATE
learn
build
measure
trigger vision &
plan
solution

40. Understand
Starting with a trigger, like a new business objective, a competitor’s
actions, a shift in customer strategy, etc, we first understand the
current condition and explore. Opportunities are identified for further
consideration through observation and enquiry about customer
behavior and business drivers.
Round 1: Problem | Understand current condition
and search for questions
NEW FRAMEWORK

41. Define
With the requisite context from the “Understand” step, we start to
synthesize what we know into compelling opportunities to pursue.
This begins to define our plan to move towards a vision of the future.
Importantly, here we assess viability/impact of our plans and
determine how we’d measure success.
We do not define all details of a solution, but should focus on desired
outcomes and impact.
Round 1: Problem | Knowledge becomes insight
NEW FRAMEWORK

42. Explore
With a vision in place and a clear definition for what success would
look like, by exploring and validating options, we elucidate the paths
to success. Details and requirements have not been defined; the right
solution is discovered.
Round 2: Solution | From vision to possible solutions
NEW FRAMEWORK

43. Create
Now, the engineering work begins.
We deliver (what we believe to be) a working solution to market and
get real world feedback about the market, our users, and their
behavior. This causes our strategy to develop further and proposes
next steps.
Round 2: Solution | Build and optimize working
software
NEW FRAMEWORK

44. “Double Diamond” Process
NEW FRAMEWORK
Each step may inform or cause us to reevaluate a future or past step
DEFINE STRATEGY
Understand why and define how
EXECUTE SOLUTION
Create the desired outcome
UNDERSTAND DEFINE EXPLORE CREATE
learn
build
measure
trigger vision &
plan
solution

45. This framework should help
us make better decisions
about what to build.
CONFIDENTIAL
DO NOT DISTRIBUTE

46. Recent History
01 Context
02 HDD Approach & Process
03 Recent History
04 Roadmap
05 Application

47. Conversion

48. HYPOTHESIS DRIVEN
DEVELOPMENT
48
We focused our efforts in the template store to help
us understand who our users are and what they
want.
HYPOTHESIS 1
Users who select a template from a more curated list
(like within a category) are more likely to start a trial
and be successful in their trial.
Topic/Vertical is one of only several dimensions which
may be relevant to which template is best for the user.
HYPOTHESIS 2

49. HYPOTHESIS DRIVEN
DEVELOPMENT
49
So, we developed a framework for the dimensions
we believe are important to understand users and
can test to determine the relative importance of
each.
Content Readiness
Vertical/Topic
Ideation Narrative
Technical Sophistication
Available Time
eCommerce

50. HYPOTHESIS DRIVEN
DEVELOPMENT
50
Our first (Minimum Lovable Product) test was to
structure the data collection on the site-intent
interstitial.

51. HYPOTHESIS DRIVEN
DEVELOPMENT
51
Since launch, we’ve collected over 3m data points
from the new interstitial.

52. HYPOTHESIS DRIVEN
DEVELOPMENT
52
We’ve been able to confirm and quantify the
importance of certain segments.
Photography is our most popular topic for users starting a new site.

53. HYPOTHESIS DRIVEN
DEVELOPMENT
53
And determine some areas where we should be
focusing on creating a better user experience.
eCommerce intent is our most common use-case.

54. HYPOTHESIS DRIVEN
DEVELOPMENT
54
Or where we have a disconnect between topic and
intended use-case.
eCommerce-intending users are disproportionately likely to be interested in
fashion and art.

55. HYPOTHESIS DRIVEN
DEVELOPMENT
55
We can now see that not everyone who enters the
funnel may be immediately ready to create or
publish a site.
Users are varied in their ideation narrative.

56. HYPOTHESIS DRIVEN
DEVELOPMENT
56
Now, we can track changes in user composition
over time to identify trends.
For example, we see an uptick in “more ready” users after the first full week
of the year.

57. HYPOTHESIS DRIVEN
DEVELOPMENT
57
Now, we can track changes in user composition
over time to identify trends.
We also see an influx of users interested in selling products or services
related to “art”

58. HYPOTHESIS DRIVEN
DEVELOPMENT
58
Now, we can track changes in user composition
over time to identify trends.
We also see an influx of users interested in selling products or services
related to “art”

59. Trial Experience

60. HYPOTHESIS DRIVEN
DEVELOPMENT
60
Inside of trial, we built a tool to help us understand
which high-impact actions users wanted to take
early in their trial.
HYPOTHESIS 1
Users wanted prescriptive help on which actions they
ought to take and what order to do them in.
Most users follow relatively predictable and uniform
paths when getting started with Squarespace.
HYPOTHESIS 2

61. HYPOTHESIS DRIVEN
DEVELOPMENT
61
Our MLP to test those hypotheses was called the
“Trial Assistant.”

62. HYPOTHESIS DRIVEN
DEVELOPMENT
62
We could see which cards had high engagement, which
resulted in better outcomes, and which outcomes were
correlated with subscription or publishing a site.
SECTION
SUBSECTION
% expand
% check
% expand
% exp
<% e
% click % click
% check
% check
% check
% exp

63. HYPOTHESIS DRIVEN
DEVELOPMENT
And now we have a better sense for what types of help users
want and what types they don’t.
SECTION
SUBSECTION
For example, clicks out to the KB have very low engagement.

64. HYPOTHESIS DRIVEN
DEVELOPMENT
Users don’t need (or want) as much help with basic actions as
we thought.
SECTION
SUBSECTION
Users are more likely to check the box than expand it, but that does serve as
a useful intent signal.

65. HYPOTHESIS DRIVEN
DEVELOPMENT
65
SECTION
SUBSECTION
Based on these results,
we’re confident about what
to build next and its
likelihood of improving the
user experience.

66. Roadmaps
01 Context
02 HDD Approach & Process
03 Adapted Software Development Lifecycle
04 Roadmaps
05 Application

67. HYPOTHESIS DRIVEN
DEVELOPMENT
67
SECTION
SUBSECTION
For this style of development
to be successful, we need to
rethink how we talk about our
development plans.

68. AMPLIFY 2019
BACKSTAGE
GOALS > FEATURES
Projects
We got used to describing our “roadmaps” as a
semi-prioritized backlog of features we were planning to
build.

69. AMPLIFY 2019
BACKSTAGE
GOALS > FEATURES
But with HDD, a better representation is this nested
structure that both demonstrates a vision and preserves
freedom to explore approaches.

70. AMPLIFY 2019
BACKSTAGE
GOALS > FEATURES
Allow any user to find the right starting point in
Squarespace.
OBJECTIVE

71. AMPLIFY 2019
BACKSTAGE
GOALS > FEATURES
Allow any user to find the right starting point in
Squarespace.
OBJECTIVE
OUTCOME Users can quickly locate a template that meets
their needs and understand what is customizable.

72. AMPLIFY 2019
BACKSTAGE
GOALS > FEATURES
Allow any user to find the right starting point in
Squarespace.
OBJECTIVE
OUTCOME Users can quickly locate a template that meets
their needs and understand what is customizable.
HYPOTHESIS Users exposed to product education at the top of
funnel are more likely to engage with key features
of the CMS once they’ve started their trial.

73. AMPLIFY 2019
BACKSTAGE
GOALS > FEATURES
HYPOTHESIS Users exposed to product education at the top of
funnel are more likely to engage with key features
of the CMS once they’ve started their trial.
How do we make this measurable and
testable?

74. SECTION
CONFIDENTIAL
DO NOT DISTRIBUTE
PRE-MORTEM

75. SECTION
CONFIDENTIAL
DO NOT DISTRIBUTE
PRE-MORTEM
What could
happen?

76. SECTION
CONFIDENTIAL
DO NOT DISTRIBUTE
PRE-MORTEM
What could
happen?
How would we
know?

77. SECTION
CONFIDENTIAL
DO NOT DISTRIBUTE
PRE-MORTEM
What could
happen?
How would we
know?
What would we
do next?

78. HYPOTHESIS DRIVEN
DEVELOPMENT
78
Projects start with two artifacts
SECTION
SUBSECTION
ANALYTICS DASHBOARDPREMORTEM

79. Application
01 Context
02 HDD Approach & Process
03 Adapted Software Development Lifecycle
04 Roadmap
05 Application

80. Feature Launches

81. EMPHASIZE LANDINGS, NOT JUST
THE LAUNCH
Feature Launches
FEATURE ANNOUNCEMENT +
LEARNING PLAN
AMPLITUDE DASHBOARD +
IMPACT ANALYSIS
LEARNING DIGESTS

82. @jonathan_sqsp
linkedin.com/in/jonathanshastings
Jonathan Hastings
Group Product Manager, Growth
Thanks!

83. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training