Expedia built a self-service analytics platform on AWS to optimize their payment processing and track costs. They used Tableau for reporting and implemented loosely coupled components including data collection from various sources, transformation and storage using Hive on EMR and Redshift, and orchestration. Their goal was to provide a familiar and usable interface while embracing cloud impermanence and scalability.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Self-Service Analytics with AWS
Big Data and Tableau
T A D B U H M A N - E X P E D I A
A N N A T E R P - E X P E D I A
N o v e m b e r 2 9 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AGENDA
WHO AND WHY
PLATFORM
PRODUCT
SELF-SERVICE ANALYTICS

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHO AND WHY?
Expedia is a global leader in
online travel
Expedia does business in 200 countries
$2.96B revenue/millions of transactions Q3 2017
Optimize payment processing
Track high cost of credit card processing
Transaction systems are not built for reporting

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STRATEGIC TOPICS
Think about guiding principles
Embrace the impermanence of the cloud
End product is usable and familiar

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GUIDING PRINCIPLES
Transaction level data not aggregated
Cloud-ready technology
Standard, loosely coupled interfaces
Horizontally scalable
Self-Service analytics

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PRODUCT AND PLATFORM
HIGH LEVEL OVERVIEW

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ENGINEERED FACT TABLES AND
ASSOCIATED VIEWS
MANAGED METRICS MANAGED DATA SOURCES
CORE PRODUCT – ENABLES SELF-SERVICE
orig_payment_instrument_
type
orig_paymen
t_transaction
_code
Success Fail
CreditCard AUTH X X
CreditCard BASV X
GiftCard AUTH X X
InternetBankPayment CAPTURE X X
InternetBankPayment VOID X
PayPal REDIRECT X
PayPal AUTH X X
Points AUTH X X
ELV CAPTURE X X

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PARAMETERIZED REPORTS PIVOT/CHART BUILDERS “START FROM SCRATCH”
EDITABLE TEMPLATES
SELF-SERVICE REPORTING

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PLATFORM ARCHITECTURE
Collection Transformation & Data Store
Orchestration
Information Delivery

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PLATFORM – BUILD STRATEGY
Platform components
Data sources
Transformation/data store
Orchestration
Information delivery
Loosely coupled interfaces
Prototype
Justify complexity
#1 Risk and Priority
#2 Risk and Priority

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
COLLECTION
• Loosely coupled interface
• JSON data
• Physical files
• Wide support
• Self describing
• S3 Landing Zone
• Central location

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TRANSFORMATION – DATA STORE
• Hive on EMR
• OnDemand
• HiveQL
• MySQL Hive Metastore
• Persist schema
• S3 Data Store
• Persist data

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service Interaction Data lineage & Workflow management
ORCHESTRATION
DynamoDBData Pipeline

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
INFORMATION DELIVERY
Redshift
• Fast analytic queries
• OnDemand reports
• SQL
• Client and Reporting tool support
• Scalable
Data abstraction

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
JUSTIFY COMPLEXITY COMPONENT INTERFACES – IMPERMANENCE
PLATFORM – STRATEGY REVIEW
Amazon S3
Data Lake
Amazon Redshift
Spectrum
Amazon Athena
Amazon EMR

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TEST DATA VOLUME AGGREGATE WHERE NEEDEDUNDERSTAND TABLE USAGE &
BUILD ACCORDINGLY
PLATFORM MEETS PRESENTATION

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SELF-SERVICE ANALYTICS
HIGH LEVEL OVERVIEW

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PRESENTATION LAYER IMPLEMENTATION
How we evaluated products in the Cloud
How we maintained familiar concepts
Our Self-Service methodology

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PRESENTATION LAYER CHOICE
Evaluation Best Practices
Test Evaluate
Load Software
Drop Instance
Repeat
Create EC2 Instance

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FAMILIAR CONCEPTS
• Consistent, familiar
names
• Numbering for sorting
• Folders
• Time and effort on
calculated measures
Cube – Legacy Tableau Data Source
CALCULATED MEASURES

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FAMILIAR CONCEPTS – FOLDERS

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SELF-SERVICE ANALYTICS
Parameterized Reports
Pivot Builder
Editable Templates

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TABLEAU
Best Practices
Lessons Learned
Optimization

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TABLEAU – BEST PRACTICES
• Abstract tables using Views
• Naming: Generic, descriptive, readable

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TABLEAU – LESSONS LEARNED
• Don’t try to extract 500M rows
• Consider performance – always
• Edit XML to change environment

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TABLEAU – EDIT DATA SOURCE

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TABLEAU – OPTIMIZATION
• Aggregation
• Assume Referential Integrity
• Consider performance when writing reports
• Data Source Optimization
• Create Calculations
• Create Hierarchies
• Hide fields – hide anything empty or not used by anyone, for
improved extract generation performance
• Set default properties (number format, comments)
• Assign Data Types and Geographic Roles

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CONCLUSION

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TAKEAWAYS
Think about guiding principles
Embrace the impermanence of the cloud
Development doesn’t have to happen in sequence
End product is usable and familiar

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
QUESTIONS