The cloud is now the first choice for large-scale analytics, but organizations that have sunk investment into Hadoop on-premises are also challenged with maintaining operations. This can make a move to modern analytics platforms like Spark difficult or impossible. Learn about innovations for large-scale migration that can take full advantage of cloud-based analytics without disrupting operations.

1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics

2. Omar Hommos & Rodel van Rooijen, Adyen
Building a Modern
FinTech Big Data
Infrastructure
#UnifiedDataAnalytics #SparkAISummit

3. Adyen
Payments Processor
Tech company w/ banking license
International customers
Omnichannel

4. Example payment

5. Example payment

6. Example payment

7. Example payment

8. Example payment

9. Adyen

10. Decision Points
Update account?
Fraud?
How to retry?
When to retry?
Do I know him? Weird behaviour?
How to validate card?
Strong authentication?

11. Update account?
Fraud?
How to retry?
When to retry?
Do I know him? Weird behaviour?
How to validate card?
Strong authentication?
Data-Driven Products and Services
RevenueProtect ShopperDNA RevenueAccelerate MonitoringTokenization
* few examples of many web services

12. Live Platform
RevenueProtect ShopperDNA RevenueAccelerate MonitoringTokenization

13. Live Platform
RevenueProtect ShopperDNA RevenueAccelerate MonitoringTokenization
Highly Available Low Latency Sensitive Data

14. Enter Data Platform
Live
Platform
Data
Platform

15. Live Platform <> Data Platform
ETL ML in production
Magic ✨
Products 💸

16. Live Platform <> Data Platform
ETL ML in production
Magic ✨
Products 💸

17. ETL Pipelines
Avro
Filebeat
Financial StreamAnalytics Stream
Avro/Parquet
Derived tables

18. Live Platform <> Data Platform
ETL ML in production
Magic ✨
Products 💸

19. ML in Production
BeamPush
Push configurations
Alfred
Serve models

20. ML in Production: Alfred
Alfred
Serve models
Unified Framework
• Build models
• Track experiments
• Re-train
• Rollout
• Serve
• …

21. Build/Experiment (Re)-training
ML in Production: Alfred
Score Serve,
Rollout
Score
Score

22. ML in Production: Alfred Rollout
Registered Ghost Test Live Retired Failed
Model available
in live platform
Score and log,
Async
Requests to
new model,
Specified %
Requests to
new model,
100%
Trash, failed AT,
Not to be used
Fallback
Replace by
newer model
Performance Tracking
(PT)
PT PT

23. Scalable Time Series
Forecasting and Monitoring
using Apache Spark and
ElasticSearch at Adyen
Today
17h20
G102

24. We are hiring!
www.adyen.com/careers

25. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT