SlideShare una empresa de Scribd logo
1 de 50
Descargar para leer sin conexión
Testing & Optimization:
    A Deeper Look
           January 16, 2013
            Brendan Regan
    Marketing Optimization Specialist
             Analytics Pros
BestPractices – Seattle, WA – 12.6.12
Late last year, I presented “Building a Successful
Optimization Program.”
TO GET THE SLIDES:
http://www.slideshare.net/analyticspros/best-
practices-optimizationpresentationbrendanregan
TODAY, we’re focusing in for a deeper look…
        People | Process | Tools
                 Process | Tools
Why is Optimization
“process” worth a
     webinar?
     I’m glad you asked.
“Process is boring.”
Process is the stuff of superheroes!
Process lets you…
1. Achieve valid, effective tests
2. Take full advantage of business
   opportunities
Without Process                              With Process
Unclear, misleading results              Statistically valid results 


Results that don’t add $                 Results that add to the bottom line 


High effort, low return                  Reasonable effort, huge return 


Opinion-based testing                    Data-driven testing 


“Inconclusive” results (a costly tie)    Clear winners and losers 
Take advantage of opportunities
1. Time to market
  – How fast can you get
    improvements in place?
2. Test the right stuff, in the right
   order
  – Low Investment, high Return
Maximize Return Over Time
Today, You Will…
1. Learn the Scientific Method…for
   Marketers
2. Learn a proven Optimization
   process
3. Discover 4 hidden BestPractices
   nuggets 
The Scientific Method…for Marketers
1.   Define the problem
2.   Research/observe the problem
3.   Form a hypothesis
4.   Conduct experiment
5.   Analyze experiment results     Look familiar?

6.   Form a conclusion
7.   Socialize results
Optimization Process




               Optimization
     Science     Process    Marketing
1) Business Goals




  Image credit: Square2Marketing.com
1) Business Goals
• In science-speak, this is “define
  the problem”
• Resist temptation to:
  – skip it
  – assume
• Multiple goals OK
  – prioritize
1) Business Goals
• Example:
 – Conversion Rate = primary business goal
 – Forward to a friend engagement =
   secondary
 – Test for Conversion Rate
 – Analyze/monitor F2F engagement
Hidden BestPractice Nugget #1:

Test as “close to” revenue as you can.
Testing further upstream can be
misleading.
2) Understand Target Audience




  Image credit: Closed Loop Marketing
2) Understand Target Audience
• In science-speak, this is “research the
  problem”
• Generally qualitative data (“market
  research”)
  – personas
  –   feedback forms
  –   focus groups
  –   social media listening
  –   chat transcripts
  –   call center monitoring
  –   demographics/psychographics
3) Web Data Analysis
3) Data Analysis (Quantitative)
• In science-speak, this is
  “research the problem”
• Web Analytics
  – High traffic points
  – Decision/conversion points
  – Persuasion points
  – High abandonment points
• Survey Data
4) Heuristic Analysis




  Image credit: Printed Circuit University
4) Heuristic Analysis
• In science-speak, this is
  “research the problem”
• Are “best practices” (rules of thumb)
  applied?
• Are you at parity with competitors
  and industry leaders?
5) Capture Challenges (roadblocks)


                            No, not
                            this kind of
                            Roadblock.


  Image credit: YoJoe.com
5) Capture Challenges (roadblocks)




  Image credit: ArtsJournal.com
5) Capture Challenges (roadblocks)
• Challenges presented by your
  site/marketing to your
  prospects
• Ex: “Landing page lacks ‘social
  proof’ content.”
6) Challenges » Hypotheses
6) Challenges » Hypotheses
• Every challenge/roadblock has
  potential solution(s)
• Best guess, best solution =
  hypothesis
• Ex: “Adding ‘social proof’
  content will increase conversion
  rate.”
Hidden BestPractice Nugget #2:
• Hypothesis: “Adding ‘social proof’
  content will increase conversion rate.”
• Research Question: “Which version of
  social proof content will perform
  best?”
7) Design Experiment
7) Design Experiment
• Formal documentation
• Capture:
  – Goal(s)
 – Problem/challenge
 – Quantitative/Qualitative proof
 – Hypothesis
 – Specifications
Hidden BestPractice Nugget #3:
“Confidence level” - the statistical probability
that your results are not due to chance.

Own your confidence level. Most tools are 95%.
Get buy-in on tool’s default or other confidence
level.
8) Design Variations
8) Design Variations
• Sketch/wireframe
• Copy deck
• Designs/mockups
• Approval process
  –Per experiment design
  –Marketing/Legal blessing
9) Build Variations
9) Build Variations
• You may have to write code 
• You may “throw away” code
• Page load speed needs to be
  consistent!
10) Implement Experiment
10) Implement Experiment
• Integrate w/ testing tool
• Test before you test
  – X-browser
  – Acceptance testing?

• Launch & Monitor
  – 7 days minimum to mitigate “day-of-week”
  – Patience, please 
11) Analyze Results


                      No, not
                      this kind of
                      analysis.
11) Analyze Results
11) Analyze Results
• Google Analytics integrations
 – Optimizely, Visual Website Optimizer, etc.
11) Analyze Results
• What does the result mean
  about the target audience?
• Are the learnings applicable
  elsewhere?
 – Other sites?
 – Other campaigns?
 – Other channels?
12) Report Findings
12) Report Findings
• If +, tie it to revenue
  – “estimated”
12) Report Findings
• If +, tie it to revenue
  – “estimated”

• If -, what did you learn?
13) Next Steps
• Install winning version
   – You don’t make $ until you install
    the winner!
   – Production-ready code, SEO, etc.
• Follow-up test ideas
• Add to Marketing knowledge
  base
Image credit: http://rahuleink.wordpress.com/2010/04/23/what-is-six-sigma/
Hidden
BestPractice
Nugget #4:
Optimize your Optimization. Stay lean,
learn as you go, smooth out the process
using Six Sigma or other methodologies.
Key Takeaways
1. Optimization Process is your friend
2. Optimization Process maps to science…
  – …but bows to Marketing
3. Optimize your own Optimization efforts
  – “…the best is the enemy of the good.”


                                            Voltaire (1694–1778)
Questions?

Más contenido relacionado

La actualidad más candente

How Will Your ML Project Fail
How Will Your ML Project FailHow Will Your ML Project Fail
How Will Your ML Project FailElena Samuylova
 
CRISP-DM: a data science project methodology
CRISP-DM: a data science project methodologyCRISP-DM: a data science project methodology
CRISP-DM: a data science project methodologySergey Shelpuk
 
Customer Discovery at Venture Out Moldova, Fall 2013
Customer Discovery at Venture Out Moldova, Fall 2013Customer Discovery at Venture Out Moldova, Fall 2013
Customer Discovery at Venture Out Moldova, Fall 2013David Kirsch
 
The 6 Jobs of Highly Effective Problem Solvers
The 6 Jobs of Highly Effective Problem SolversThe 6 Jobs of Highly Effective Problem Solvers
The 6 Jobs of Highly Effective Problem SolversRod King, Ph.D.
 
Ria Sankar on Building AI Products
Ria Sankar on Building AI ProductsRia Sankar on Building AI Products
Ria Sankar on Building AI ProductsRia Sankar
 
Искусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья РыбникИскусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья РыбникSigma Software
 
UX STRAT USA 2021: Jane Davis, Zoom
UX STRAT USA 2021: Jane Davis, ZoomUX STRAT USA 2021: Jane Davis, Zoom
UX STRAT USA 2021: Jane Davis, ZoomUX STRAT
 
A basic course on analytical thinking
A basic course on analytical thinking A basic course on analytical thinking
A basic course on analytical thinking Vincent G Valentyn PhD
 
MEMSI January 2018: Primary Market Research Workshops
MEMSI January 2018: Primary Market Research WorkshopsMEMSI January 2018: Primary Market Research Workshops
MEMSI January 2018: Primary Market Research WorkshopsElaine Chen
 
What Is Six Sigma? An Introduction for Technical Writers
What Is Six Sigma? An Introduction for Technical WritersWhat Is Six Sigma? An Introduction for Technical Writers
What Is Six Sigma? An Introduction for Technical WritersSTC-Philadelphia Metro Chapter
 
[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...
[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...
[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...CXL
 
How To Make Better Product Decisions
How To Make Better Product DecisionsHow To Make Better Product Decisions
How To Make Better Product DecisionsStartup Product
 
Product Management for AI/ML
Product Management for AI/MLProduct Management for AI/ML
Product Management for AI/MLJeremy Horn
 
Audience Research on a Dime - Nonprofit of Influence
Audience Research on a Dime - Nonprofit of InfluenceAudience Research on a Dime - Nonprofit of Influence
Audience Research on a Dime - Nonprofit of InfluenceCourtney Clark
 
Shannon Kelly: #MOC with VWO
Shannon Kelly: #MOC with VWOShannon Kelly: #MOC with VWO
Shannon Kelly: #MOC with VWOVWO
 
A Primer on Primary Market Research
A Primer on Primary Market ResearchA Primer on Primary Market Research
A Primer on Primary Market ResearchElaine Chen
 
Product School - AI Funding / Trends & Product Management
Product School - AI Funding / Trends & Product ManagementProduct School - AI Funding / Trends & Product Management
Product School - AI Funding / Trends & Product ManagementAarthi Srinivasan
 
10 Tips for women to build a career in data science
10 Tips for women to build a career in data science10 Tips for women to build a career in data science
10 Tips for women to build a career in data scienceCarol Hargreaves
 

La actualidad más candente (20)

How Will Your ML Project Fail
How Will Your ML Project FailHow Will Your ML Project Fail
How Will Your ML Project Fail
 
CRISP-DM: a data science project methodology
CRISP-DM: a data science project methodologyCRISP-DM: a data science project methodology
CRISP-DM: a data science project methodology
 
Data Analyst Role
Data Analyst RoleData Analyst Role
Data Analyst Role
 
Customer Discovery at Venture Out Moldova, Fall 2013
Customer Discovery at Venture Out Moldova, Fall 2013Customer Discovery at Venture Out Moldova, Fall 2013
Customer Discovery at Venture Out Moldova, Fall 2013
 
The 6 Jobs of Highly Effective Problem Solvers
The 6 Jobs of Highly Effective Problem SolversThe 6 Jobs of Highly Effective Problem Solvers
The 6 Jobs of Highly Effective Problem Solvers
 
Ria Sankar on Building AI Products
Ria Sankar on Building AI ProductsRia Sankar on Building AI Products
Ria Sankar on Building AI Products
 
Искусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья РыбникИскусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья Рыбник
 
UX STRAT USA 2021: Jane Davis, Zoom
UX STRAT USA 2021: Jane Davis, ZoomUX STRAT USA 2021: Jane Davis, Zoom
UX STRAT USA 2021: Jane Davis, Zoom
 
A basic course on analytical thinking
A basic course on analytical thinking A basic course on analytical thinking
A basic course on analytical thinking
 
MEMSI January 2018: Primary Market Research Workshops
MEMSI January 2018: Primary Market Research WorkshopsMEMSI January 2018: Primary Market Research Workshops
MEMSI January 2018: Primary Market Research Workshops
 
What Is Six Sigma? An Introduction for Technical Writers
What Is Six Sigma? An Introduction for Technical WritersWhat Is Six Sigma? An Introduction for Technical Writers
What Is Six Sigma? An Introduction for Technical Writers
 
[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...
[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...
[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...
 
How To Make Better Product Decisions
How To Make Better Product DecisionsHow To Make Better Product Decisions
How To Make Better Product Decisions
 
Product Management for AI/ML
Product Management for AI/MLProduct Management for AI/ML
Product Management for AI/ML
 
Audience Research on a Dime - Nonprofit of Influence
Audience Research on a Dime - Nonprofit of InfluenceAudience Research on a Dime - Nonprofit of Influence
Audience Research on a Dime - Nonprofit of Influence
 
Shannon Kelly: #MOC with VWO
Shannon Kelly: #MOC with VWOShannon Kelly: #MOC with VWO
Shannon Kelly: #MOC with VWO
 
A Primer on Primary Market Research
A Primer on Primary Market ResearchA Primer on Primary Market Research
A Primer on Primary Market Research
 
Product School - AI Funding / Trends & Product Management
Product School - AI Funding / Trends & Product ManagementProduct School - AI Funding / Trends & Product Management
Product School - AI Funding / Trends & Product Management
 
10 Tips for women to build a career in data science
10 Tips for women to build a career in data science10 Tips for women to build a career in data science
10 Tips for women to build a career in data science
 
Analytical skills
Analytical skillsAnalytical skills
Analytical skills
 

Similar a Testing & Optimization - A Deeper Look

BEST Practices - Testing & Optimization | Bredan Rendan
BEST Practices - Testing & Optimization | Bredan RendanBEST Practices - Testing & Optimization | Bredan Rendan
BEST Practices - Testing & Optimization | Bredan RendanCaleb Whitmore
 
Data Analytics: Better Decision, Better Business
Data Analytics: Better Decision, Better BusinessData Analytics: Better Decision, Better Business
Data Analytics: Better Decision, Better BusinessMcKonly & Asbury, LLP
 
Nancy's webinar
Nancy's webinarNancy's webinar
Nancy's webinarVWO
 
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick SoChris Goward
 
Product Management in the Era of Data Science
Product Management in the Era of Data ScienceProduct Management in the Era of Data Science
Product Management in the Era of Data ScienceMandar Parikh
 
Testing the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesisTesting the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesisArdita Karaj
 
Giving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business QuestionsGiving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business QuestionsOReillyStrata
 
Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Ron Lazaro
 
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...BrittanyRubinstein
 
How to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessHow to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessKissmetrics on SlideShare
 
Webinar: Finding Your Fit in Product by Planoly Head of Product
Webinar: Finding Your Fit in Product by Planoly Head of ProductWebinar: Finding Your Fit in Product by Planoly Head of Product
Webinar: Finding Your Fit in Product by Planoly Head of ProductProduct School
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteWebanalisten .nl
 
Data That Sparks Joy by Heap Analytics Sr Product Manager
Data That Sparks Joy by Heap Analytics Sr Product ManagerData That Sparks Joy by Heap Analytics Sr Product Manager
Data That Sparks Joy by Heap Analytics Sr Product ManagerProduct School
 
20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid themCraig Sullivan
 
Cox Automotive: Testing Across Multiple Brands
Cox Automotive: Testing Across Multiple BrandsCox Automotive: Testing Across Multiple Brands
Cox Automotive: Testing Across Multiple BrandsOptimizely
 
Converge 2014: Digital Analytics - Getting Leadership Buy-in - Thayer
Converge 2014: Digital Analytics - Getting Leadership Buy-in - ThayerConverge 2014: Digital Analytics - Getting Leadership Buy-in - Thayer
Converge 2014: Digital Analytics - Getting Leadership Buy-in - ThayerConverge Consulting
 
Exaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilitiesExaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilitiesExaudio
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion ConferenceCraig Sullivan
 
Startup Marketing: A Systems Approach
Startup Marketing: A Systems ApproachStartup Marketing: A Systems Approach
Startup Marketing: A Systems ApproachApril Dunford
 

Similar a Testing & Optimization - A Deeper Look (20)

BEST Practices - Testing & Optimization | Bredan Rendan
BEST Practices - Testing & Optimization | Bredan RendanBEST Practices - Testing & Optimization | Bredan Rendan
BEST Practices - Testing & Optimization | Bredan Rendan
 
Data Analytics: Better Decision, Better Business
Data Analytics: Better Decision, Better BusinessData Analytics: Better Decision, Better Business
Data Analytics: Better Decision, Better Business
 
Nancy's webinar
Nancy's webinarNancy's webinar
Nancy's webinar
 
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
 
Product Management in the Era of Data Science
Product Management in the Era of Data ScienceProduct Management in the Era of Data Science
Product Management in the Era of Data Science
 
Testing the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesisTesting the unknown: the art and science of working with hypothesis
Testing the unknown: the art and science of working with hypothesis
 
Giving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business QuestionsGiving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business Questions
 
Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)
 
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
 
How to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessHow to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your Business
 
Webinar: Finding Your Fit in Product by Planoly Head of Product
Webinar: Finding Your Fit in Product by Planoly Head of ProductWebinar: Finding Your Fit in Product by Planoly Head of Product
Webinar: Finding Your Fit in Product by Planoly Head of Product
 
Mike Roe Creating Client Value
Mike Roe Creating Client ValueMike Roe Creating Client Value
Mike Roe Creating Client Value
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynote
 
Data That Sparks Joy by Heap Analytics Sr Product Manager
Data That Sparks Joy by Heap Analytics Sr Product ManagerData That Sparks Joy by Heap Analytics Sr Product Manager
Data That Sparks Joy by Heap Analytics Sr Product Manager
 
20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them
 
Cox Automotive: Testing Across Multiple Brands
Cox Automotive: Testing Across Multiple BrandsCox Automotive: Testing Across Multiple Brands
Cox Automotive: Testing Across Multiple Brands
 
Converge 2014: Digital Analytics - Getting Leadership Buy-in - Thayer
Converge 2014: Digital Analytics - Getting Leadership Buy-in - ThayerConverge 2014: Digital Analytics - Getting Leadership Buy-in - Thayer
Converge 2014: Digital Analytics - Getting Leadership Buy-in - Thayer
 
Exaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilitiesExaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilities
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference
 
Startup Marketing: A Systems Approach
Startup Marketing: A Systems ApproachStartup Marketing: A Systems Approach
Startup Marketing: A Systems Approach
 

Último

OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimizationarrow10202532yuvraj
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementNuwan Dias
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5DianaGray10
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideHironori Washizaki
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 

Último (20)

OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API Management
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 

Testing & Optimization - A Deeper Look

  • 1. Testing & Optimization: A Deeper Look January 16, 2013 Brendan Regan Marketing Optimization Specialist Analytics Pros
  • 2. BestPractices – Seattle, WA – 12.6.12 Late last year, I presented “Building a Successful Optimization Program.”
  • 3. TO GET THE SLIDES: http://www.slideshare.net/analyticspros/best- practices-optimizationpresentationbrendanregan
  • 4. TODAY, we’re focusing in for a deeper look… People | Process | Tools Process | Tools
  • 5. Why is Optimization “process” worth a webinar? I’m glad you asked.
  • 7. Process is the stuff of superheroes!
  • 8. Process lets you… 1. Achieve valid, effective tests 2. Take full advantage of business opportunities
  • 9. Without Process With Process Unclear, misleading results  Statistically valid results  Results that don’t add $  Results that add to the bottom line  High effort, low return  Reasonable effort, huge return  Opinion-based testing  Data-driven testing  “Inconclusive” results (a costly tie)  Clear winners and losers 
  • 10. Take advantage of opportunities 1. Time to market – How fast can you get improvements in place? 2. Test the right stuff, in the right order – Low Investment, high Return
  • 12. Today, You Will… 1. Learn the Scientific Method…for Marketers 2. Learn a proven Optimization process 3. Discover 4 hidden BestPractices nuggets 
  • 13. The Scientific Method…for Marketers 1. Define the problem 2. Research/observe the problem 3. Form a hypothesis 4. Conduct experiment 5. Analyze experiment results Look familiar? 6. Form a conclusion 7. Socialize results
  • 14. Optimization Process Optimization Science Process Marketing
  • 15. 1) Business Goals Image credit: Square2Marketing.com
  • 16. 1) Business Goals • In science-speak, this is “define the problem” • Resist temptation to: – skip it – assume • Multiple goals OK – prioritize
  • 17. 1) Business Goals • Example: – Conversion Rate = primary business goal – Forward to a friend engagement = secondary – Test for Conversion Rate – Analyze/monitor F2F engagement
  • 18. Hidden BestPractice Nugget #1: Test as “close to” revenue as you can. Testing further upstream can be misleading.
  • 19. 2) Understand Target Audience Image credit: Closed Loop Marketing
  • 20. 2) Understand Target Audience • In science-speak, this is “research the problem” • Generally qualitative data (“market research”) – personas – feedback forms – focus groups – social media listening – chat transcripts – call center monitoring – demographics/psychographics
  • 21. 3) Web Data Analysis
  • 22. 3) Data Analysis (Quantitative) • In science-speak, this is “research the problem” • Web Analytics – High traffic points – Decision/conversion points – Persuasion points – High abandonment points • Survey Data
  • 23. 4) Heuristic Analysis Image credit: Printed Circuit University
  • 24. 4) Heuristic Analysis • In science-speak, this is “research the problem” • Are “best practices” (rules of thumb) applied? • Are you at parity with competitors and industry leaders?
  • 25. 5) Capture Challenges (roadblocks) No, not this kind of Roadblock. Image credit: YoJoe.com
  • 26. 5) Capture Challenges (roadblocks) Image credit: ArtsJournal.com
  • 27. 5) Capture Challenges (roadblocks) • Challenges presented by your site/marketing to your prospects • Ex: “Landing page lacks ‘social proof’ content.”
  • 28. 6) Challenges » Hypotheses
  • 29. 6) Challenges » Hypotheses • Every challenge/roadblock has potential solution(s) • Best guess, best solution = hypothesis • Ex: “Adding ‘social proof’ content will increase conversion rate.”
  • 30. Hidden BestPractice Nugget #2: • Hypothesis: “Adding ‘social proof’ content will increase conversion rate.” • Research Question: “Which version of social proof content will perform best?”
  • 32. 7) Design Experiment • Formal documentation • Capture: – Goal(s) – Problem/challenge – Quantitative/Qualitative proof – Hypothesis – Specifications
  • 33. Hidden BestPractice Nugget #3: “Confidence level” - the statistical probability that your results are not due to chance. Own your confidence level. Most tools are 95%. Get buy-in on tool’s default or other confidence level.
  • 35. 8) Design Variations • Sketch/wireframe • Copy deck • Designs/mockups • Approval process –Per experiment design –Marketing/Legal blessing
  • 37. 9) Build Variations • You may have to write code  • You may “throw away” code • Page load speed needs to be consistent!
  • 39. 10) Implement Experiment • Integrate w/ testing tool • Test before you test – X-browser – Acceptance testing? • Launch & Monitor – 7 days minimum to mitigate “day-of-week” – Patience, please 
  • 40. 11) Analyze Results No, not this kind of analysis.
  • 42. 11) Analyze Results • Google Analytics integrations – Optimizely, Visual Website Optimizer, etc.
  • 43. 11) Analyze Results • What does the result mean about the target audience? • Are the learnings applicable elsewhere? – Other sites? – Other campaigns? – Other channels?
  • 45. 12) Report Findings • If +, tie it to revenue – “estimated”
  • 46. 12) Report Findings • If +, tie it to revenue – “estimated” • If -, what did you learn?
  • 47. 13) Next Steps • Install winning version – You don’t make $ until you install the winner! – Production-ready code, SEO, etc. • Follow-up test ideas • Add to Marketing knowledge base
  • 48. Image credit: http://rahuleink.wordpress.com/2010/04/23/what-is-six-sigma/ Hidden BestPractice Nugget #4: Optimize your Optimization. Stay lean, learn as you go, smooth out the process using Six Sigma or other methodologies.
  • 49. Key Takeaways 1. Optimization Process is your friend 2. Optimization Process maps to science… – …but bows to Marketing 3. Optimize your own Optimization efforts – “…the best is the enemy of the good.” Voltaire (1694–1778)