Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Automated Analytics
Testing
MARCUS MERRELL, RETAILMENOT, INC
@MMERRELL
What We’re
Talking about
 Conservancy
 Overview of Web/User
Analytics
 Explanation of A/B
Testing
 Why this matters to...
Who Is Conservancy?
 Software Freedom Conservancy
 Protects the branding, copyrights, and licenses for over
40 free and ...
Overview of User Analytics
 Primarily B2C: Banks, shopping, high-volume apps
 Applies to all platforms, including video ...
User Analytics
Hits
Sessions
Users
“If you are not paying for it… you are the product being sold”
—Andrew Lewis (blue_beet...
A/B Testing
 Don’t change everything at once
 Major source of regressions in analytics:
 Analytics will need to change ...
Real-life RMN Example
...&b21=17&…
...&b21=18&…
...&b21=0&…
(wait, shouldn’t that say 18?)
How Is This Related?
 Analytics have historically not been included in test
design
 Major source of regression defects
...
Real-world Examples
 Shopping cart—shipping & tax calculation
 Suggesting products and content based on cookie, not
login
Real-world Examples
 RetailMeNot’s email program
 Category and Merchant “affinity”
Classic Approach
Marketing
Business
Intelligence
Product
Manager
EngineeringQA
(Usually the newest
person on the team)
Why You Should Care
 What if the analytics contain the wrong information?
 But furthermore…
 This is everywhere
 It is...
Why You Should Care
Why You Should Really Care
 As a tester:
 There is a team of people working on this
 It gets worked into features as th...
Fortunately, It’s Easy
 Usually one extra HTTP request, made during a
navigation event
 Intercept this request, then ver...
Process
 Break down into key-value pairs
 Build a Map of “expected results”
 Assert on expected vs actual results
seque...
Test Cycle
@Test
Launch
Browser
Navigate
to Position
Start Proxy
Perform
Main
Action
Stop Proxy
Clean-up
Test
Execution
 Send JSON to the “expectation” end-point
 Includes a “timer” and an alert email
 Later (asynchronously), mor...
Reporting
 Report to a dashboard
 Indicates “PASS”, “FAIL”, and “Staleness”
Alternatives
 Anand Bagmar
 https://github.com/anandbagmar/WAAT
 Tool for testing Omniture Analytics
 Uses Omniture’s ...
Conclusion
 User Analytics are your CEO’s favorite subject!
 Deliver real value—million-dollar decisions are made with
t...
Questions?
 Once again, thank you so very much
 Twitter: @mmerrell
 mmerrell@gmail.com
 I’m also on LinkedIn
Automated Analytics Testing
Próxima SlideShare
Cargando en…5
×

Automated Analytics Testing

2.113 visualizaciones

Publicado el

Selenium Committer Day 2017
Marcus Merrell さんによる講演資料です。
ご本人の好意により公開許可を頂きましたので共有いたします。

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Automated Analytics Testing

  1. 1. Automated Analytics Testing MARCUS MERRELL, RETAILMENOT, INC @MMERRELL
  2. 2. What We’re Talking about  Conservancy  Overview of Web/User Analytics  Explanation of A/B Testing  Why this matters to QA personnel  Examples  How to get started http://bit.ly/2u6tJMa http://bit.ly/2urc80O
  3. 3. Who Is Conservancy?  Software Freedom Conservancy  Protects the branding, copyrights, and licenses for over 40 free and open source projects  Includes Selenium, git, Samba, PyPy, and many others  www.sfconservancy.org  Non-profit organization  The reason Jim, Manoj, and I are here
  4. 4. Overview of User Analytics  Primarily B2C: Banks, shopping, high-volume apps  Applies to all platforms, including video game consoles  Tells the ”story” of user interaction with a product
  5. 5. User Analytics Hits Sessions Users “If you are not paying for it… you are the product being sold” —Andrew Lewis (blue_beetle)
  6. 6. A/B Testing  Don’t change everything at once  Major source of regressions in analytics:  Analytics will need to change as a result of a test, but sometimes those changes are forgotten (  We used to just release changes after basic QA  “The Highball Incident” http://bit.ly/2ttj1RK
  7. 7. Real-life RMN Example ...&b21=17&… ...&b21=18&… ...&b21=0&… (wait, shouldn’t that say 18?)
  8. 8. How Is This Related?  Analytics have historically not been included in test design  Major source of regression defects  In this case, no A/B testing was done  Data loss was permanent, irrecoverable
  9. 9. Real-world Examples  Shopping cart—shipping & tax calculation  Suggesting products and content based on cookie, not login
  10. 10. Real-world Examples  RetailMeNot’s email program  Category and Merchant “affinity”
  11. 11. Classic Approach Marketing Business Intelligence Product Manager EngineeringQA (Usually the newest person on the team)
  12. 12. Why You Should Care  What if the analytics contain the wrong information?  But furthermore…  This is everywhere  It is only growing  Companies are becoming smart  (Really really smart)  QA is being left behind
  13. 13. Why You Should Care
  14. 14. Why You Should Really Care  As a tester:  There is a team of people working on this  It gets worked into features as they are developed  It is rarely called out separately in a scheduled task  It rarely receives QA outside of the Product Management and Business Intelligence people who really care about it http://bit.ly/2u6j4AW
  15. 15. Fortunately, It’s Easy  Usually one extra HTTP request, made during a navigation event  Intercept this request, then verify the data within it
  16. 16. Process  Break down into key-value pairs  Build a Map of “expected results”  Assert on expected vs actual results sequence=12 sequence=12 category=food category=food type=code type=sale
  17. 17. Test Cycle @Test Launch Browser Navigate to Position Start Proxy Perform Main Action Stop Proxy Clean-up Test
  18. 18. Execution  Send JSON to the “expectation” end-point  Includes a “timer” and an alert email  Later (asynchronously), more JSON is sent to the “result”  If there’s no match before the timer runs out, ALERT Expectation Action Result
  19. 19. Reporting  Report to a dashboard  Indicates “PASS”, “FAIL”, and “Staleness”
  20. 20. Alternatives  Anand Bagmar  https://github.com/anandbagmar/WAAT  Tool for testing Omniture Analytics  Uses Omniture’s built-in web interface to examine attributes emitted by page, rather than using a proxy
  21. 21. Conclusion  User Analytics are your CEO’s favorite subject!  Deliver real value—million-dollar decisions are made with this data  Can be implemented with just as many bugs as any other kind of software
  22. 22. Questions?  Once again, thank you so very much  Twitter: @mmerrell  mmerrell@gmail.com  I’m also on LinkedIn

×