Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Building a culture of testing like lucid

347 visualizaciones

Publicado el

Watch the webinar recording at: http://grow.kissmetrics.com/webinar-195

Developing a culture of testing is not something that happens accidentally or overnight. It can initially be difficult to justify the added effort and time required to test new ideas and changes, especially in the fast-moving startup culture. However, at Lucid Software, we have found that investing in a culture of testing is key to our sustainable growth.

Learn how we:

Started testing ideas that scared us
Scaled testing to include ideas that we “knew” were winners
Included engineering infrastructure in our testing
Tried testing everything at once and deeply regretted it
It’s good to learn from your mistakes, but it’s even better to learn from someone else's mistakes. Learn from our missteps (and eventual successes) as we established a testing culture at Lucid Software.

Publicado en: Marketing
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Building a culture of testing like lucid

  1. 1. Building a Culture Of Testing SPENCER MANN - VP OF GROWTH AT LUCID SOFTWARE
  2. 2. @Kissmetrics #KissWebinar @thuelmadsen
  3. 3. Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a Kissmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras. THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen Spencer Mann is the VP of Growth at Lucid Software. Prior to working at Lucid Software, he worked in the chemical industry and with alternative sources of energy. He has a BS and MS in biological engineering and an MBA. Spencer is passionate about AB testing and spending time with his family (only occasionally do these passions mix). SPENCER MANN VP of Growth, Lucid Software @spencer_mann
  4. 4. @lucidchart #KissWebinar @spencer_mann
  5. 5. 1 THE GROWTH TEAM AT LUCID 2 HOW DID WE BUILD A CULTURE OF TESTING? 3 WHERE TO FIND GOOD IDEAS WHAT WE’LL COVER 4 HOW TO GET “BUY IN” FOR BIG TESTS
  6. 6. The Growth Team at Lucid
  7. 7. THE GROWTH TEAM AT LUCID Payments! Registrations! Payment Quality! Retention! Team Growth! Payments
 40% Registrations
 20% Payment Quality
 18% Retention
 18% Team
 Growth
 18% EXPERIMENT FOCUS AREAS Over 200 experiments in 2016… around 35% success rate. NOTABLE TESTS Credit card required for trial! Multiple paywalls! Several price increases! Onboarding! Personalization! Registration experience!
  8. 8. How Did We Build a Culture of Testing?
  9. 9. SOME EXPERIENCES THAT DROVE OUR CULTURE OF TESTING Four experiences… §  The first time data suggested we do something everyone hated §  Why would we test something we already know is a good idea? §  Few things are worse than a test that can’t fail §  “Let’s save time by just doing this all at once.”
  10. 10. THE FIRST TIME DATA SUGGESTED WE DO SOMETHING EVERYONE HATED The Hypothesis “If we shorten our trial length from 2 weeks to 1 week we will significantly increase the number of users who experience that high conversion event.” The Problem Many of the biggest opportunities involved changing something put in place by a founder. §  Good leadership is more interested in being successful than being right. §  Most scary tests can be de-risked by testing a low value portion of the population, or only testing a component of the principle. §  Data is the best source for high value test ideas. Learnings TotalPayments
  11. 11. WHY WOULD WE TEST SOMETHING WE ALREADY KNOW IS A GOOD IDEA? The Hypothesis “If we increase our price by 20% today, tomorrow we will make 20% more money.” The Problem In the early days of a startup you are forced to make gut decisions that keep you moving fast. Transitioning from that to a culture of deliberate testing and data based decisions can be uncomfortable at first. §  People are wrong a lot. §  Transitioning to a culture of testing is painful, but obvious in retrospect. §  AB tests will provide the “what” but not always the “why” Learnings
  12. 12. FEW THINGS ARE WORSE THAN A TEST THAT CAN’T FAIL The Hypothesis “Updating our engineering architecture will improve site performance, which will drive user engagement and payments.” The Problem This was not a traditional A/B test. Setting up the experiment was very expensive so casually failing it was not an option. Additionally, this was a strategic initiative for the company. §  Ask yourself, “What will we do if this doesn’t win?” §  Testing engineering architecture is a great way to diagnose. §  Analyzing another team’s experiences can be “your Vietnam.” Learnings Browser – OS Drop in Engagement Chrome - Linux 34% Firefox - Mac 23% Firefox - Linux -2% Chrome - Mac 2%
  13. 13. LET’S SAVE TIME BY JUST DOING ALL THIS AT ONCE . . .
  14. 14. LET’S SAVE TIME BY JUST DOING ALL THIS AT ONCE . . . Flight to moon took many iterative launches. The Hypothesis “We can save time when testing a change to our editor by changing both the design and the underlying architecture at the same time.” The Problem Hundreds of things changed at once, and it was nearly impossible to identify the problematic elements. §  Simplicity is better than complexity in testing. §  Test components independently if possible. §  It’s extremely unlikely that a whole-scale change will beat a multi-year, optimized experience on it’s first try. §  Long running complex tests can be a “tax” on engineering and QA resources, sometimes you need to call it and move on. Learnings
  15. 15. Where to Find Good Ideas
  16. 16. WHERE TO FIND GOOD IDEAS §  Mapping out flows and conversion rates §  Focus on areas with high volume and high potential for improvement §  Take “growth hacks” with a high degree of skepticism §  Don’t just “throw things against the wall” §  Discuss successes and failures with other credible growth leaders §  Bi-weekly brainstorming with entire growth team, with focus area provided §  Talk with customer facing groups to identify pain points §  Learn from you failures and iterate 30% of our wins started as failed experiments
  17. 17. How To Get “Buy In” for Big Tests
  18. 18. HOW DO I GET “BUY IN” FOR BIG TESTS? §  Credibility is a core to success §  Visually diagram proposed test flow §  Come prepared with all the relevant data §  Describe risk and upside potential §  Calculate test length §  Break the test into smaller low risk components §  Call out your own failures and NEVER bias your data §  Benchmark against other similar companies §  Provide updates on tests early, but be cautious about calling wins §  Balance low test investments with not screwing it up Pro Tips
  19. 19. SOME TOOLS WE HAVE FOUND USEFUL §  Google Sheets for tracking all tests and pre-test work §  Flow chart of test implementation §  AB Test configuration tool §  Kissmetrics Access our AB testing template at: http://bit.ly/2qjEGv1
  20. 20. HOW WE STRUCTURED OUR GROWTH TEAM Growth Manager Senior Engineer 2 additional engineers •  Data-focused •  Curious •  Challenges assumptions •  Not afraid to fail •  Expert on payment system •  Business minded •  Enjoys experiments •  High learning agility •  Balanced customer/ business mindset Analyst •  Curious •  Disciplined •  Communicates well
  21. 21. HOW TO SET YOUR GROWTH TEAM UP TO FAIL §  No Executive buy-in. Many ideas you test will challenge long-held beliefs §  No Independence. Continuous optimization requires freedom to be innovative §  No Engineers. “Borrowing” engineering time is frustrating and doomed to fail §  No Data. Without good data you can’t identify success or focus areas Beware these mistakes which can set your growth team up to fail…
  22. 22. SPENCER MANN VP of Growth, Lucid Software @spencer_mann THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen Questions? (Illustrations) Libby Thomas, Creative Content Specialist Lucid Software

×