Featuring speakers from Blu Dot, Village Roadshow, Clearhead Mike Wodtke, eCommerce Director, Blu Dot Jon Satterley, Group Director, Digital Development, Village Roadshow Matty Wishnow, CEO , Clearhead Building an organization around continuous optimization is a distinctly different task from running A/B tests. Getting from those first experiments as a "project" to a data driven team and process requires vision and a commitment to hypothesis driven decision-making. How do entrepreneurial leaders and companies incorporate testing as a KPI and use it to drive a broader product roadmap? In this panel discussion with Blu Dot, Village Roadshow, and Optimizely Solutions Partner, Clearhead, will share their experiences from driving optimization forward - and the 'X factors' that made it stick.

1. #opticon2015
Making it Stick
A conversation about adopting & sustaining optimization
Matty Wishnow
Founder & CEO,
Clearhead
@ClearheadMe
matt@clearhead.me
Mike Wodtke
Director of E-commerce,
Blu Dot
Jon Satterley
Group Director of Digital Development,
Village Roadshow
jon_satterley@vrl.comaumwodtke@bludot.com
@bludot

2. hi. i’m matty,
founder & ceo at
the optimization & validation agency
- Proud Optimization Agency for over two dozen clients
including: Adidas, Tory Burch, P&G, New Balance, Patagonia,
Fresh Direct and Keurig.
- Nominated for 5 Optie Awards including Solution Partner
of the Year
- Built & Optimized in Austin, TX

3. hi. i’m mike,
director of
e-commerce at
- Founded in 1996 in Minneapolis with a goal of making modern
design more accessible
- Stores in NYC, LA, San Francisco, Austin, Sydney and
Monterrey and over 100 independent retail partners
worldwide
- Winner of many design awards including a product design
finalist for the Cooper-Hewitt National Design Award

4. hi. i’m jon,
group director
of digital
development at
- Village Roadshow Limited is the parent corporation of numerous
divisions including Village Cinemas, Village Roadshow Theme
Parks and Village Roadshow Entertainment Group.
- Village Roadshow has produced scores of well known movies
including The Matrix, Sherlock Holmes and American Sniper.
- VRL owns and operate many of the world’s most spectacular
theme parks, including Sea World and Wet’n’Wild, and theaters
throughout Australia and other countries outside of the far Pacific.

5. how do you know?
making it stick.
align
validate
optimize
sustain
A B
C

6. here’s how you know.
making it stick.
1
2
3
goals
a
b
c
d
problems
1
2
3
4
hypotheses
prioritized by data
1
2
3
4
learnings
validated by data
key investments
UX
product
merchandising
content

7. let’s get real!
It’s so easy!
Build a data driven culture.
Everybody’s doing it!
Just get an executive sponsor.
You can personalize everything!

8. what was your business case?
from the beginning.
to
this
from
this

9. what was your business case?
from the beginning.
M
O
P
U
P
to
easure
ptimize
ersonalize
plift
rofits

10. the path to alignment
Goals Opportunities Problems Hypotheses Prioritization Go Time!
We’re all bought in!
Goals Opportunities Problems Hypotheses Prioritization Go Time!
Are we all bought in, now?

11. display color
options
experiment:
I believe that we can
increase "adds to cart" by
showing alternate color
options as distinct
products on search results
and category pages.
hypothesis:
variation display colors as textcontrol
160
conversion index
100
conversion index
+60.46%
lift

12. box office
design
experiment:
I believe that if we update the
VIP and Non-VIP box office
pages to present ticketing
options with a cleaner, more
simplified visual hierarchy,
we will make it easier for
customers to browse the
options and choose the right
ticket for them.
hypothesis:
variation new VIP & new box officecontrol
8.00%
conversion rate
7.48%
conversion rate
+6.90%
lift

13. where are
you on your
journey ?
You were curious
You got initial stakeholder buy in
You bought the tools
You assigned some resources
You developed an internal process
You tested some hypotheses
You scaled the program
Your hypotheses got smarter
You made it core to all investments
You slayed
the dragon.
It stuck!

14. personalization
run a simple a/b test
to everyone
a/b test
find the winner
you found winner!
segment data
…but it really only
won for new users
personalize & a/b test
test the winner vs a new
variation to new users
hey! you’re personalizing!
now you found yet another
winner for new users!
find the winner
segment data
…but really only for new
users on desktop.
ba
aa ab
a b
ba bb
it’s time to further personalize!

15. key
takeaways 1 Goals > Problems > Hypotheses
2 Reimagine your teams as hypothesis managers.
3 Time is always our enemy.
Resources are always limited.
4 100% confidence is expensive.
Prioritization is essential.
5 There’s risk in all of this.
But there’s bigger risk in the alternative.

16. Questions?

17. #opticon2015
Thank you!
Matty Wishnow
Founder & CEO,
Clearhead
@ClearheadMe
matt@clearhead.me
Mike Wodtke
Director of E-commerce,
Blu Dot
Jon Satterley
Group Director of Digital Development,
Village Roadshow
jon_satterley@vrl.comaumwodtke@bludot.com
@bludot