1. Testing, measuring and
metrics
Soren Harner, Bigcommerce

2. About me
2012 VP Engineering Stage 2 Startup
Venture-backed
$35MM
Stage 1 Startup
2010 Founder & Datavore
Bookstrapped
2007 VP Engineering Stage 2 Startup
Stage 2 Startup
2000 Search & Analytics, Stage 3 Acquisition
Director Engineering Venture-backed > $80MM

3. Business Metrics
• Cost to acquire customers (CAC)
• Average Revenue per User (ARPU)
• Churn
• Avoidable & unavoidable
• Conversion (CVN)
• Compare Search, Display, Radio,
etc.
• Life time value (LTV)
• LTV = ARPU/Churn
• LTV / CAC > 1 is success!
• Monthly Recurring Revenue (MRR)

4. LTV / CAC
1.2
Model
“successful”
0.9
0.6
0.3
0
Q1-10 Q2-10 Q3-10 Q4-10 Q1-11 Q2-11 Q3-11 Q4-11
To IPO range is maybe 4 to 6 LTV / CAC today
Fine print: Like all metrics use to track, not as strategy

6. Split Testing Website
• Persona analysis and customer interviews
on new brand
• Of new traffic, 50% old, 50% new
• Measure effect on conversion: looked at
new trials after 50/50 coming from old and
new to determine lift with 95% confidence
interval of 20-30%

7. Product UX Testing
Workshops Concepting Qualitative Quantitative
Prototypes
(personas) (sketches) (Axure) (Loop11)
You can use an iPad at a Cafe and show keynote.
Key point is to test mockups on people first.

8. Continuous Delivery
• Feature flag everything and gently deploy
• Weekly deploys (should be daily)
• Read 12factor.net (dev, staging, production)
• Monitor (i.e. NewRelic)
• Practice BDD & continuous integration

9. Ask permission, get feedback

10. Product Metrics
• Customer success TTL TTFS
• Time to first sale Aug
• Time to launch July
• Trend monthly or June
weekly cohorts
May
• Engagement score
during trial 0 25 50 75 100

11. Feature KPI
• Before you build a • e.g. completing task
feature, define a success more quickly
KPI
• Add instrumentation
• e.g. Number of before and after
product reviews
completed • Deploy to subset of
customers to test
• e.g. 30% fewer
abandoned carts • Consider Mixpanel

12. Startup Friendly Stack
• Use Ruby Sinatra (or Clojure) +
CoffeeScript + Backbone + Handlebars
on Heroku
• Nesta for CMS and learn textmate,
Haml & Sass, see https://github.com/
sharner/Workshop-Site
• HTML5 mobile first, responsive design
• Use Mailchimp & google forms
• Log events to understand customer
success (see Heroku Plugins)

13. Peopleware
• Technical co-founder
challenges
• Don’t offshore
• Don’t churn people
• Find designer who codes
frontend and does user
testing