Patrick McKenzie, Software Developer at Kalzumeus Software You've run A/B tests. You're long past the point where "Guess Which Alternative Won" articles teach you something useful. - How do you get to the next level of conversion optimization? - How do you get it to be a repeatable team effort? - How do you retain organizational know-how about previous tests? - How do you track conversions over more involved funnels? - How do you keep the team onboard with testing for years at a time? - How do you manage the engineering aspects of changes which are more in--depth than changing the marketing site? This session delves into the "Here There Be Dragons" parts of the testing map. Patrick McKenzie, Software Developer at Kalzumeus Software and Bingo Card Creator, has helped a dozen companies through testing maps, and he presents some of the mistakes he's made so that you don't have to repeat them—and also a success or two.

1. Conversion
Op,miza,on:
The
World
Beyond
Headlines
and
Bu:on
Color
Patrick
McKenzie
Founder
Kalzumeus
So>ware
@pa,o11

2. Low
hanging
fruit
tastes
great,
but
the
rest
is
worth
the
work,
too.

3. Test
Idea
#1
Quotas
/
Tier
Levels
/
etc

5. @user.card_limit
=
ab_test("user_card_limit_May_2012",
[8,
15],
:conversion
=>
"purchase")

6. What
Did
We
Learn
From
That
Test?
• ~60%
increase
in
sales
going
from
15
-­‐>
8
cards
• Synergis,c
effects
can
occasionally
actually
happen.
– More
on
that
later.
• You
don’t
know
what
you
don’t
know
un,l
you
test
for
it.
This
post
on
HN
was
me
losing
$100,000:

7. Key
Points
With
TesPng
Quotas
• Grandfather
in
exis,ng
users.
• Set
quota
levels
intelligently.
– Quotas
should
segment
people
by
willingness
to
pay
– Break
out
a
histogram
of
exis,ng
accounts
• Be
ready
to
tolerate
small
up,ck
in
support
load.
• Your
product
is
s%ll
too
cheap.

8. Test
Idea
#2
Payment
Methods
/
Checkout
Flow

9. Threw
This
Together
In
An
Hour
(Oops)

11. What
We
Learned
From
This
Test
• A
single
A/B
test
can
help
your
boZom
line
more
than
a
product
launch.
• Dealing
with
synergis,c
effects
in
tes,ng
can
occasionally
be
maddening.
– First
a:empts
at
implemen,ng
on-­‐site
checkout
had
null
result
a>er
null
result
– Combina,on
of
on-­‐site
checkout
plus
redesign
plus
aforemen,oned
card
limit
test
plus
hillclimbing
won
big
• The
data
is
never
wrong,
but
some,mes
the
data
isn’t
telling
you
what
you
think
it
is.

12. Key
Points
With
TesPng
Checkout
Flow
• Make
absolutely
sure
you
don’t
break
this!
• Have
UX-­‐focused
team
(product
/
marke,ng
/
etc)
go
over
every
step
of
checkout
funnel
with
zero
tolerance
for
bad
UX.
• Checkout
flows
go
“stale”
at
many
successful
companies.
Schedule
a
deep
dive
on
them
twice
a
year,
at
least.
• Upsells
and
cross
selling
are
classics
for
a
reason!

13. Test
Idea
#3
Offline
Business
Processes

15. TesPng
degree
of
sales
which
maximizes
conversion
of
leads

16. What
We
Learned
From
This
Test
• Important
to
communicate
with
sales/CS.
– Allay
fears
about
being
replaced,
leads
creamed,
etc.
– Demonstrate
it
directly
makes
them
money.
• Lead
to
resource-­‐intensive
but
hugely
successful
lead-­‐scoring
efforts.
• Jus,fied
very
expensive
interven,ons
like
e.g.
onboarding
help
for
everybody.

17. Key
Points
With
TesPng
Offline
Processes
• Your
so>ware/workflow
has
to
close
the
loop
back
to
the
tes,ng
so>ware.
This
can
be
tricky.
• Virtually
no
CRM
or
A/B
tes,ng
so>ware
accurately
models
enterprise
decisionmaking
process.
You
might
be
forced
to
op,mize
for
proxy
conversions.
• Volumes
required
for
sta,s,cal
significance
are
really
punishing,
so
some,mes
we
“proxy
and
pray.”

18. Test
Idea
#4
Product
On-­‐boarding
/
FRE

19. What
Engineers
Will
Ship
If
You
Let
Them

20. Key
Goals
of
Onboarding
Process
• Get
user
to
a
concrete
success
as
soon
as
possible.
• For
more
complex
sales,
lay
groundwork
for
future
success
by
e.g.
invi,ng
in
team
or
gepng
permission
to
follow-­‐up.
• For
B2C,
especially
in
social,
you
live
and
die
on
virality
during
the
onboarding
experience.

23. What
We
Learned
From
This
Test
• Users
who
start
onboarding
tour
25%
to
100%+
more
likely
to
purchase
so>ware
• Users
some,mes
have
visceral
reac,on
to
elements
of
the
onboarding
experience.
– The
wise-­‐alecky
Kiwi
– “Beau,ful
So>ware
Lady
Susan”
• Engineers
prefer
in-­‐app
onboarding
but
it
is
expensive
as
heck
to
build
and
does
not
always
jus,fy
itself
versus
e.g.
a
Youtube
video.

24. Key
Points
With
TesPng
Offline
Processes
• Your
so>ware/workflow
has
to
close
the
loop
back
to
the
tes,ng
so>ware.
This
can
be
tricky.
• Virtually
no
CRM
or
A/B
tes,ng
so>ware
accurately
models
enterprise
decisionmaking
process.
You
might
be
forced
to
op,mize
for
proxy
conversions.
• Volumes
required
for
sta,s,cal
significance
are
really
punishing,
so
some,mes
we
“proxy
and
pray.”

25. Test
Idea
#5
Core
Product
InteracPons

26. Too
Complicated

27. 80%
of
Users
Use
20%
of
Features

28. Try
Radically
Simplifying

30. What
We
Learned
From
This
Test
• Just
hiding
the
advanced
features
increased
task
success
by
10%,
which
directly
improved
sales
• Further
jump
of
15%
when
we
did
the
redesign
– Absolutely
no
increase
in
sales…
– …
but
we
saw
in
the
funnel
that
this
was
solely
due
to
less
people
hipng
the
trial
limits
…
– …
which
we
were
able
to
profitably
exploit
with
the
quota
limit
test
described
earlier

31. Key
Points
With
TesPng
Core
InteracPons
• Test
the
en,re
funnel.
Some,mes
task
success
doesn’t
(unfortunately)
help
bo:om
line
metrics.
• A
good
understanding
of
where
people
spend
their
,me/energy
is
vital
–
test
the
90%
interac,on,
not
the
2%.

32. ROI
on
~$500
A
Month
TesPng
Budget
0.00%
0.50%
1.00%
1.50%
2.00%
2.50%
3.00%
3.50%
February,
2010
March,
2010
April,
2010
May,
2010
June,
2010
July,
2010
August,
2010
September,
2010
October,
2010
November,
2010
December,
2010
January,
2011
February,
2011
March,
2011
April,
2011
May,
2011
June,
2011
July,
2011
August,
2011
September,
2011
October,
2011
November,
2011
December,
2011
January,
2012
February,
2012
March,
2012
April,
2012
May,
2012
June,
2012
July,
2012
August,
2012
September,
2012
October,
2012
November,
2012
December,
2012
January,
2013
February,
2013
March,
2013
April,
2013
May,
2013
June,
2013
July,
2013
August,
2013
September,
2013
October,
2013
November,
2013
December,
2013
January,
2014
February,
2014

33. Let’s
Talk
TesPng
Challenges

34. A/B
TesPng
Across
System
Boundaries

35. Not
A
Hard
Problem
Our
Marke,ng
Site
A/B

36. Gets
A
Wee
Bit
Trickier
With
2
Sites
Our
Marke,ng
Site
A/B
The
Web
Applica,on

37. Total
Freaking
Madness
Our
Marke,ng
Site
A/B
The
Web
Applica,on
iPhone
app
Offline
processes
Android
app
Partner
site
Content
minisite
Offsite
checkout
Japanese
subsidiary

38. Sanity
Restored
Consumer
Applica,ons
Iden,ty
&
A/B
Manager

39. Lessons
Learned
With
This
• Think
about
strategy
for
transi,oning
people
between
systems
–
easier
to
build
into
design
earlier
than
later
• Cookies
with
unique
iden,,es
are
a
stopgap,
not
a
solu,on.
You
need
to
understand
the
sales
process.
• This
is
a
good
opportunity
to
standardize
on
an
A/B
tes,ng
tool
and
workflow…
or
you’ll
hate
your
life
• This
is
a
“high
class
problem”
which
can
be
deferred
un,l
$50M+
in
revenue,
but
when
you
solve
it,
it
is
going
to
cost
serious
,me/money.

40. Increasing
A/B
TesPng
AdopPon
In
Engineering
and
MarkePng
Teams

41. Single
Most
Important
Lesson:
One
Line
of
Code
@user.card_limit
=
ab_test("user_card_limit_May_2012",
[8,
15],
:conversion
=>
"purchase")

42. Other
Lessons
Learned
• Organiza,on
should
learn
rather
than
guess
be:er.
– Best
investment
ever:
organiza,on-­‐wide
wiki.
– Next
best:
weekly
“What
we
tried
this
week”
email.
• Engineers
love
A/B
tes,ng
when
you
describe
it
like
a
puzzle
or
a
video
game,
with
bragging
rights.
• Marketers
love
A/B
tes,ng
when
it’s
posi,oned
as
a
way
to
increase
their
status
in
the
organiza,on
rather
than
a
replacement
for
their
talents.

43. Things
To
Never
Be
Without
• CS
should
be
able
to
force-­‐override
a
user’s
par,cipa,on
in
an
A/B
test
(and,
relatedly,
“see
what
users
is
seeing”
is
great
if
you
can
get
it).
• A/B
tests
deeper
than
surface
level
need
unit/etc
tests.
Make
sure
your
framework
supports
it.
• Test
switcher
UI
in
development/staging
is
a
huge
win.
• URL
query
parameter
to
flip
tests
a
huge
win.

44. Thanks
for
Listening!
• I
love
talking
about
this
and
you
cannot
possibly
waste
my
,me:
@pa,o11
or
patrick@kalzumeus.com
• Free
email
mini-­‐course
on
CRO
in
so>ware
companies:
h:p://bit.ly/patrick-­‐op,mizely
• Liked
this
presenta,on?
I
deliver
something
like
this
twice
a
month,
free,
to
my
email
list.
h:p://training.kalzumeus.com
to
sign
up

45. Conversion
Op,miza,on:
The
World
Beyond
Headlines
and
Bu:on
Color
Patrick
McKenzie
Founder
Kalzumeus
So>ware
@pa,o11