SlideShare una empresa de Scribd logo
1 de 27
Descargar para leer sin conexión
AB Testing
Revolution through constsant
evolution
AB Testing at Expedia
Expedia SF
114 Sansome
www.expedia.com
@expediaeng
Work with us:
letstalk@expedia.com
Paul Lucas
Sr Director, Technology
Want to visit next? Greece
Jeff Madynski
Director, Technology
Want to visit next? Croatia
Anuj Gupta
Sr Software Dev Engineer
Want to visit next? Peru
Revolution through constant
evolution
AB Testing at Expedia
Technology Evolution
V0 – batch processing from abacus
exposure logs, Omniture, and booking
datamart. Tableau visualization
V1 - Storm, Kestrel, DynamoDB /
Postgresql reading UIS messages and
client log data. (Nov 2014 - Dec 2015)
V2 - Introduce Kafka and Cassandra (May
2016)
TNL – original solution
• Batch processing
• Tableau visualization
• Merged data from OMS/omniture
• Problems:
– 1-2d feedback loop – what if we had mistakes
in test implementation(bucketing not what
anticipated)?
– In order to fix data import errors - start over
again
TNL Dashboard v0
Omniture
click data
Booking
datamart
Abacus
exposures
Tableau
Hadoop ETL
TNL v0 -> v1
Begin Jeff
delete this page
AB Testing at Expedia
TNL v1 Problems
• Database size 420GB, queries took 3-5
minutes
• Data drop (kestrel)
• Increase in data (multi-brand,
+customers)
TNL v1->v1.1, v2
• Fighting fires, borrowing more time
• POC next
Fighting fires – borrowing more time
User Interaction Service(UIS) Traffic
Scaling messaging system
Kafka
• Publish-subscribe based
messaging system
• Distributed and reliable
• Longer retention and
persistence
• Monitoring dashboard
and alerts
• Buffer for system
downtime
Kestrel limitation
• Message durability is not
available
• Reaching potential
scalability issues
• In-active open source
project
Scaling database performance
• Database views for caching
–Views created every 6 hours
–UI only loads data from views
–Read-only replicas for select queries
• Archive data
–Moved old and completed experiment data to
separate tables
–DB cleanup using vacuum and re-indexing
TNL Dashboard v2
Product Demo
Streaming
•Column-oriented, time series schema
•Time-to-live(TTL) on data
•Only store most popular aggregates
v1 VS v2
•New Architecture
–More scalable
–More responsive
–Less prone to data loss
• Lessons learnt
–System is as fast as the slowest component
–Fault-tolerance and resilience
–Partition data
–Pre-production environment
Questions/discussion
APPENDIX
27
Apply statistical power to test results results
Using 90% confidence level, 1 out of 10
tests will be false positive or negative
Heads Tails
Right hand 51 49
Left hand 49 51
Right hand is superior at
getting heads!
Do’s and Don’ts when concluding tests
Don’t call test too early;
this increases false
positives or negatives
Don’t call tests as soon
as you see positive
results because test
result frequently goes
up and down
To claim a test Winner/Loser, the
positive/negative effect has to stay for
at least 5 consecutive days and the
trend is stable
Please note this type of chart is not currently available in the Test and Learn dashboard or SiteSpect UI; The shape of Confidence Interval lines varies test by test
Define one success metric and run
tests for a pre-determined duration;
(For hotel/flight tests in the US,
suggest running until confidence
interval of conversion change is
within +/- 1%); tests should run at
least 10 days
Don’t assume the midpoint (observed
% change during the test period) will
hold true after the feature is rolled
out: a 4.0% +/- 4.0% test may have
zero impact and may not be much
better than a 1.0% +/- 1.0% test
Don’t call an inconclusive test
“trending positive” or “trending
negative” as test result fluctuates
Contact ARM testing
team for questions
gaotest@expedia..com
Using 90% confidence level
Winner: Lower bound of % change >= 0 (or
probability of test being positive >= 95%);
Loser: Higher bound of % change <= 0 (or
probability of test being negative >= 95%)
Else: Inconclusive or Neutral

Más contenido relacionado

La actualidad más candente

Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) :: 한국 카오스엔지니어링 밋업
Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) ::  한국 카오스엔지니어링 밋업Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) ::  한국 카오스엔지니어링 밋업
Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) :: 한국 카오스엔지니어링 밋업Channy Yun
 
Experimentation Platform at Netflix
Experimentation Platform at NetflixExperimentation Platform at Netflix
Experimentation Platform at NetflixSteve Urban
 
Software testing performance testing
Software testing  performance testingSoftware testing  performance testing
Software testing performance testingGaneshKumarKanthiah
 
Matching the Scale at Tinder with Kafka
Matching the Scale at Tinder with Kafka Matching the Scale at Tinder with Kafka
Matching the Scale at Tinder with Kafka confluent
 
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platformApache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platformrhatr
 
Sparklens: Understanding the Scalability Limits of Spark Applications with R...
 Sparklens: Understanding the Scalability Limits of Spark Applications with R... Sparklens: Understanding the Scalability Limits of Spark Applications with R...
Sparklens: Understanding the Scalability Limits of Spark Applications with R...Databricks
 
AWS Aurora 운영사례 (by 배은미)
AWS Aurora 운영사례 (by 배은미)AWS Aurora 운영사례 (by 배은미)
AWS Aurora 운영사례 (by 배은미)I Goo Lee.
 
Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...
Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...
Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...Databricks
 
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenApache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenDatabricks
 
Performance testing jmeter
Performance testing jmeterPerformance testing jmeter
Performance testing jmeterBhojan Rajan
 
Building Event Streaming Architectures on Scylla and Kafka
Building Event Streaming Architectures on Scylla and KafkaBuilding Event Streaming Architectures on Scylla and Kafka
Building Event Streaming Architectures on Scylla and KafkaScyllaDB
 
Spark Tuning for Enterprise System Administrators By Anya Bida
Spark Tuning for Enterprise System Administrators By Anya BidaSpark Tuning for Enterprise System Administrators By Anya Bida
Spark Tuning for Enterprise System Administrators By Anya BidaSpark Summit
 
Scala Data Pipelines @ Spotify
Scala Data Pipelines @ SpotifyScala Data Pipelines @ Spotify
Scala Data Pipelines @ SpotifyNeville Li
 
Grafana and AWS - Implementation and Usage
Grafana and AWS - Implementation and UsageGrafana and AWS - Implementation and Usage
Grafana and AWS - Implementation and UsageManish Chopra
 
Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017
Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017
Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017Amazon Web Services Korea
 
[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자Oracle Korea
 
Troubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolutionTroubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolutionJoel Koshy
 
Apache Hadoop YARN: Understanding the Data Operating System of Hadoop
Apache Hadoop YARN: Understanding the Data Operating System of HadoopApache Hadoop YARN: Understanding the Data Operating System of Hadoop
Apache Hadoop YARN: Understanding the Data Operating System of HadoopHortonworks
 

La actualidad más candente (20)

Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) :: 한국 카오스엔지니어링 밋업
Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) ::  한국 카오스엔지니어링 밋업Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) ::  한국 카오스엔지니어링 밋업
Chaos Engineering 시작하기 - 윤석찬 (AWS 테크에반젤리스트) :: 한국 카오스엔지니어링 밋업
 
Experimentation Platform at Netflix
Experimentation Platform at NetflixExperimentation Platform at Netflix
Experimentation Platform at Netflix
 
Software testing performance testing
Software testing  performance testingSoftware testing  performance testing
Software testing performance testing
 
Matching the Scale at Tinder with Kafka
Matching the Scale at Tinder with Kafka Matching the Scale at Tinder with Kafka
Matching the Scale at Tinder with Kafka
 
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platformApache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platform
 
Sparklens: Understanding the Scalability Limits of Spark Applications with R...
 Sparklens: Understanding the Scalability Limits of Spark Applications with R... Sparklens: Understanding the Scalability Limits of Spark Applications with R...
Sparklens: Understanding the Scalability Limits of Spark Applications with R...
 
AWS Aurora 운영사례 (by 배은미)
AWS Aurora 운영사례 (by 배은미)AWS Aurora 운영사례 (by 배은미)
AWS Aurora 운영사례 (by 배은미)
 
Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...
Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...
Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...
 
HDFS: Optimization, Stabilization and Supportability
HDFS: Optimization, Stabilization and SupportabilityHDFS: Optimization, Stabilization and Supportability
HDFS: Optimization, Stabilization and Supportability
 
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenApache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
 
Performance testing jmeter
Performance testing jmeterPerformance testing jmeter
Performance testing jmeter
 
Building Event Streaming Architectures on Scylla and Kafka
Building Event Streaming Architectures on Scylla and KafkaBuilding Event Streaming Architectures on Scylla and Kafka
Building Event Streaming Architectures on Scylla and Kafka
 
Spark Tuning for Enterprise System Administrators By Anya Bida
Spark Tuning for Enterprise System Administrators By Anya BidaSpark Tuning for Enterprise System Administrators By Anya Bida
Spark Tuning for Enterprise System Administrators By Anya Bida
 
Scala Data Pipelines @ Spotify
Scala Data Pipelines @ SpotifyScala Data Pipelines @ Spotify
Scala Data Pipelines @ Spotify
 
Unified Stream and Batch Processing with Apache Flink
Unified Stream and Batch Processing with Apache FlinkUnified Stream and Batch Processing with Apache Flink
Unified Stream and Batch Processing with Apache Flink
 
Grafana and AWS - Implementation and Usage
Grafana and AWS - Implementation and UsageGrafana and AWS - Implementation and Usage
Grafana and AWS - Implementation and Usage
 
Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017
Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017
Amazon Aurora 성능 향상 및 마이그레이션 모범 사례 - AWS Summit Seoul 2017
 
[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자
 
Troubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolutionTroubleshooting Kafka's socket server: from incident to resolution
Troubleshooting Kafka's socket server: from incident to resolution
 
Apache Hadoop YARN: Understanding the Data Operating System of Hadoop
Apache Hadoop YARN: Understanding the Data Operating System of HadoopApache Hadoop YARN: Understanding the Data Operating System of Hadoop
Apache Hadoop YARN: Understanding the Data Operating System of Hadoop
 

Destacado

Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanUXPA UK
 
AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale
AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at ScaleAppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale
AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at ScaleAppDynamics
 
楽天市場を取り巻く状況と開発
楽天市場を取り巻く状況と開発楽天市場を取り巻く状況と開発
楽天市場を取り巻く状況と開発Rakuten Group, Inc.
 
ヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのか
ヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのかヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのか
ヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのかYahoo!デベロッパーネットワーク
 
Expedia at London Technology Week 2015
Expedia at London Technology Week 2015Expedia at London Technology Week 2015
Expedia at London Technology Week 2015Open Destinations
 
A Millennial’s Magnetic Influence on LinkedIn
A Millennial’s Magnetic Influence on LinkedInA Millennial’s Magnetic Influence on LinkedIn
A Millennial’s Magnetic Influence on LinkedInLinkedIn
 
デブサミ2013 【15-A-1】「爆速」を支えるテクノロジー
デブサミ2013 【15-A-1】「爆速」を支えるテクノロジーデブサミ2013 【15-A-1】「爆速」を支えるテクノロジー
デブサミ2013 【15-A-1】「爆速」を支えるテクノロジーDevelopers Summit
 
Brand Management Study of Amazon
Brand Management Study of Amazon Brand Management Study of Amazon
Brand Management Study of Amazon Nikhil Narayan
 
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )Amazon Web Services
 
ライブショッピングモール
ライブショッピングモールライブショッピングモール
ライブショッピングモールYouhei Kanbe
 
【トピカル】風雲急を告げるEC業界を展望する
【トピカル】風雲急を告げるEC業界を展望する 【トピカル】風雲急を告げるEC業界を展望する
【トピカル】風雲急を告げるEC業界を展望する Yuya Sakai
 
OpenStack Summitの歩き方
OpenStack Summitの歩き方OpenStack Summitの歩き方
OpenStack Summitの歩き方Hideki Saito
 

Destacado (16)

Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig Sullivan
 
ACS DataMart_ppt
ACS DataMart_pptACS DataMart_ppt
ACS DataMart_ppt
 
ACS DataMart_ppt
ACS DataMart_pptACS DataMart_ppt
ACS DataMart_ppt
 
AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale
AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at ScaleAppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale
AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale
 
楽天市場を取り巻く状況と開発
楽天市場を取り巻く状況と開発楽天市場を取り巻く状況と開発
楽天市場を取り巻く状況と開発
 
Case study on Travel
Case study on TravelCase study on Travel
Case study on Travel
 
ヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのか
ヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのかヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのか
ヤフー株式会社はアクセシビリティ対応を
なぜ始めたのか、どう進めているのか
 
Expedia at London Technology Week 2015
Expedia at London Technology Week 2015Expedia at London Technology Week 2015
Expedia at London Technology Week 2015
 
A Millennial’s Magnetic Influence on LinkedIn
A Millennial’s Magnetic Influence on LinkedInA Millennial’s Magnetic Influence on LinkedIn
A Millennial’s Magnetic Influence on LinkedIn
 
デブサミ2013 【15-A-1】「爆速」を支えるテクノロジー
デブサミ2013 【15-A-1】「爆速」を支えるテクノロジーデブサミ2013 【15-A-1】「爆速」を支えるテクノロジー
デブサミ2013 【15-A-1】「爆速」を支えるテクノロジー
 
Expedia case study
Expedia case studyExpedia case study
Expedia case study
 
Brand Management Study of Amazon
Brand Management Study of Amazon Brand Management Study of Amazon
Brand Management Study of Amazon
 
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )
 
ライブショッピングモール
ライブショッピングモールライブショッピングモール
ライブショッピングモール
 
【トピカル】風雲急を告げるEC業界を展望する
【トピカル】風雲急を告げるEC業界を展望する 【トピカル】風雲急を告げるEC業界を展望する
【トピカル】風雲急を告げるEC業界を展望する
 
OpenStack Summitの歩き方
OpenStack Summitの歩き方OpenStack Summitの歩き方
OpenStack Summitの歩き方
 

Similar a AB Testing at Expedia

End-to-End Testing with the Heuristic Software Test Model
End-to-End Testing with the Heuristic Software Test ModelEnd-to-End Testing with the Heuristic Software Test Model
End-to-End Testing with the Heuristic Software Test ModelTechWell
 
Hypothesis-Driven Development & How to Fail-Fast Hacking Growth
Hypothesis-Driven Development & How to Fail-Fast Hacking GrowthHypothesis-Driven Development & How to Fail-Fast Hacking Growth
Hypothesis-Driven Development & How to Fail-Fast Hacking GrowthPrabhat Gupta
 
Delivering @ Warp Speed
Delivering @ Warp SpeedDelivering @ Warp Speed
Delivering @ Warp Speedandremadeira
 
Leverage alter table conversions methodology to alter your upgrade downtime
Leverage alter table conversions methodology to alter your upgrade downtime Leverage alter table conversions methodology to alter your upgrade downtime
Leverage alter table conversions methodology to alter your upgrade downtime KPIT
 
Sangram Nayak_22Jan15
Sangram Nayak_22Jan15Sangram Nayak_22Jan15
Sangram Nayak_22Jan15Sangram Nayak
 
Continuous Performance Testing: The New Standard
Continuous Performance Testing: The New StandardContinuous Performance Testing: The New Standard
Continuous Performance Testing: The New StandardTechWell
 
Resume - Santi Gong__
Resume - Santi Gong__Resume - Santi Gong__
Resume - Santi Gong__Zhenglei Gong
 
DMM9 - Data Migration Testing
DMM9 - Data Migration TestingDMM9 - Data Migration Testing
DMM9 - Data Migration TestingNick van Beest
 
Cmsc 1215 ensuring a quality order management (3)
Cmsc 1215  ensuring a quality order management  (3)Cmsc 1215  ensuring a quality order management  (3)
Cmsc 1215 ensuring a quality order management (3)Perficient, Inc.
 
Ensuring a Quality Order Management Implementation
Ensuring a Quality Order Management ImplementationEnsuring a Quality Order Management Implementation
Ensuring a Quality Order Management ImplementationPerficient, Inc.
 
Agile Test Management and Reporting—Even in a Non-Agile Project
Agile Test Management and Reporting—Even in a Non-Agile ProjectAgile Test Management and Reporting—Even in a Non-Agile Project
Agile Test Management and Reporting—Even in a Non-Agile ProjectTechWell
 
Tech Talk: The New CA Application Performance Management Team Center—Faster T...
Tech Talk: The New CA Application Performance Management Team Center—Faster T...Tech Talk: The New CA Application Performance Management Team Center—Faster T...
Tech Talk: The New CA Application Performance Management Team Center—Faster T...CA Technologies
 
Testing Attributes
Testing AttributesTesting Attributes
Testing AttributesAbiha Naqvi
 
Understanding System Performance
Understanding System PerformanceUnderstanding System Performance
Understanding System PerformanceTeradata
 
ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2Yogindernath Gupta
 
Whitepaper: How to perform better test on SAP PI/PO
Whitepaper: How to perform better test on SAP PI/POWhitepaper: How to perform better test on SAP PI/PO
Whitepaper: How to perform better test on SAP PI/PODaniel Graversen
 
Enhancing Quality and Test in Medical Device Design - Part 2.pdf
Enhancing Quality and Test in Medical Device Design - Part 2.pdfEnhancing Quality and Test in Medical Device Design - Part 2.pdf
Enhancing Quality and Test in Medical Device Design - Part 2.pdfICS
 
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...Tieturi Oy
 
Introduction to testing.
Introduction to testing.Introduction to testing.
Introduction to testing.Jithinctzz
 

Similar a AB Testing at Expedia (20)

End-to-End Testing with the Heuristic Software Test Model
End-to-End Testing with the Heuristic Software Test ModelEnd-to-End Testing with the Heuristic Software Test Model
End-to-End Testing with the Heuristic Software Test Model
 
Hypothesis-Driven Development & How to Fail-Fast Hacking Growth
Hypothesis-Driven Development & How to Fail-Fast Hacking GrowthHypothesis-Driven Development & How to Fail-Fast Hacking Growth
Hypothesis-Driven Development & How to Fail-Fast Hacking Growth
 
Delivering @ Warp Speed
Delivering @ Warp SpeedDelivering @ Warp Speed
Delivering @ Warp Speed
 
Leverage alter table conversions methodology to alter your upgrade downtime
Leverage alter table conversions methodology to alter your upgrade downtime Leverage alter table conversions methodology to alter your upgrade downtime
Leverage alter table conversions methodology to alter your upgrade downtime
 
Sangram Nayak_22Jan15
Sangram Nayak_22Jan15Sangram Nayak_22Jan15
Sangram Nayak_22Jan15
 
Continuous Performance Testing: The New Standard
Continuous Performance Testing: The New StandardContinuous Performance Testing: The New Standard
Continuous Performance Testing: The New Standard
 
Resume - Santi Gong__
Resume - Santi Gong__Resume - Santi Gong__
Resume - Santi Gong__
 
DMM9 - Data Migration Testing
DMM9 - Data Migration TestingDMM9 - Data Migration Testing
DMM9 - Data Migration Testing
 
Cmsc 1215 ensuring a quality order management (3)
Cmsc 1215  ensuring a quality order management  (3)Cmsc 1215  ensuring a quality order management  (3)
Cmsc 1215 ensuring a quality order management (3)
 
Ensuring a Quality Order Management Implementation
Ensuring a Quality Order Management ImplementationEnsuring a Quality Order Management Implementation
Ensuring a Quality Order Management Implementation
 
Raghav_QA
Raghav_QARaghav_QA
Raghav_QA
 
Agile Test Management and Reporting—Even in a Non-Agile Project
Agile Test Management and Reporting—Even in a Non-Agile ProjectAgile Test Management and Reporting—Even in a Non-Agile Project
Agile Test Management and Reporting—Even in a Non-Agile Project
 
Tech Talk: The New CA Application Performance Management Team Center—Faster T...
Tech Talk: The New CA Application Performance Management Team Center—Faster T...Tech Talk: The New CA Application Performance Management Team Center—Faster T...
Tech Talk: The New CA Application Performance Management Team Center—Faster T...
 
Testing Attributes
Testing AttributesTesting Attributes
Testing Attributes
 
Understanding System Performance
Understanding System PerformanceUnderstanding System Performance
Understanding System Performance
 
ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2
 
Whitepaper: How to perform better test on SAP PI/PO
Whitepaper: How to perform better test on SAP PI/POWhitepaper: How to perform better test on SAP PI/PO
Whitepaper: How to perform better test on SAP PI/PO
 
Enhancing Quality and Test in Medical Device Design - Part 2.pdf
Enhancing Quality and Test in Medical Device Design - Part 2.pdfEnhancing Quality and Test in Medical Device Design - Part 2.pdf
Enhancing Quality and Test in Medical Device Design - Part 2.pdf
 
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
 
Introduction to testing.
Introduction to testing.Introduction to testing.
Introduction to testing.
 

AB Testing at Expedia

  • 1. AB Testing Revolution through constsant evolution
  • 4. Paul Lucas Sr Director, Technology Want to visit next? Greece Jeff Madynski Director, Technology Want to visit next? Croatia Anuj Gupta Sr Software Dev Engineer Want to visit next? Peru
  • 7. Technology Evolution V0 – batch processing from abacus exposure logs, Omniture, and booking datamart. Tableau visualization V1 - Storm, Kestrel, DynamoDB / Postgresql reading UIS messages and client log data. (Nov 2014 - Dec 2015) V2 - Introduce Kafka and Cassandra (May 2016)
  • 8. TNL – original solution • Batch processing • Tableau visualization • Merged data from OMS/omniture • Problems: – 1-2d feedback loop – what if we had mistakes in test implementation(bucketing not what anticipated)? – In order to fix data import errors - start over again
  • 9. TNL Dashboard v0 Omniture click data Booking datamart Abacus exposures Tableau Hadoop ETL
  • 10. TNL v0 -> v1
  • 13. TNL v1 Problems • Database size 420GB, queries took 3-5 minutes • Data drop (kestrel) • Increase in data (multi-brand, +customers)
  • 14. TNL v1->v1.1, v2 • Fighting fires, borrowing more time • POC next
  • 15. Fighting fires – borrowing more time
  • 17. Scaling messaging system Kafka • Publish-subscribe based messaging system • Distributed and reliable • Longer retention and persistence • Monitoring dashboard and alerts • Buffer for system downtime Kestrel limitation • Message durability is not available • Reaching potential scalability issues • In-active open source project
  • 18. Scaling database performance • Database views for caching –Views created every 6 hours –UI only loads data from views –Read-only replicas for select queries • Archive data –Moved old and completed experiment data to separate tables –DB cleanup using vacuum and re-indexing
  • 22. •Column-oriented, time series schema •Time-to-live(TTL) on data •Only store most popular aggregates
  • 23. v1 VS v2 •New Architecture –More scalable –More responsive –Less prone to data loss • Lessons learnt –System is as fast as the slowest component –Fault-tolerance and resilience –Partition data –Pre-production environment
  • 26. 27 Apply statistical power to test results results Using 90% confidence level, 1 out of 10 tests will be false positive or negative Heads Tails Right hand 51 49 Left hand 49 51 Right hand is superior at getting heads!
  • 27. Do’s and Don’ts when concluding tests Don’t call test too early; this increases false positives or negatives Don’t call tests as soon as you see positive results because test result frequently goes up and down To claim a test Winner/Loser, the positive/negative effect has to stay for at least 5 consecutive days and the trend is stable Please note this type of chart is not currently available in the Test and Learn dashboard or SiteSpect UI; The shape of Confidence Interval lines varies test by test Define one success metric and run tests for a pre-determined duration; (For hotel/flight tests in the US, suggest running until confidence interval of conversion change is within +/- 1%); tests should run at least 10 days Don’t assume the midpoint (observed % change during the test period) will hold true after the feature is rolled out: a 4.0% +/- 4.0% test may have zero impact and may not be much better than a 1.0% +/- 1.0% test Don’t call an inconclusive test “trending positive” or “trending negative” as test result fluctuates Contact ARM testing team for questions gaotest@expedia..com Using 90% confidence level Winner: Lower bound of % change >= 0 (or probability of test being positive >= 95%); Loser: Higher bound of % change <= 0 (or probability of test being negative >= 95%) Else: Inconclusive or Neutral