AWS User Group: Building Cloud Analytics Solution with AWS
•
0 recomendaciones•207 vistas
Abebooks is one of Amazon Subsidiary and it treats data as an asset. It always looks the way to improve existing analytics solution and extract information from terabytes of data. One of the recent initiatives was the migration from legacy DW platform to the AWS Redshift. During this journey, our data engineers met lots of challenges and sometimes tried to reinvent the wheel. This talk will cover Abebooks journey towards Cloud DW. Moreover, we will cover the ETL tool selection process for the Cloud as well as the adoption process for the end users. This talk will help you understand the potential of the modern cloud DW and learn about our use case and save time for the future projects.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a AWS User Group: Building Cloud Analytics Solution with AWS
Similar a AWS User Group: Building Cloud Analytics Solution with AWS (20)
Más de Dmitry Anoshin
Más de Dmitry Anoshin (20)
Último
Último (20)
AWS User Group: Building Cloud Analytics Solution with AWS
- 1. select outline from victoria_aws_user_group where topic_name = ‘Building Modern Analytics Solutions‘ and solution_type = ’Cloud’ and solution_provider = ‘AWS’ and presenter = ‘Dmitry Anoshin’ And contact_info = ’Dmitry.Anoshin@gmail.com’
- 2. --------------------------------------------------------------- Outline --------------------------------------------------------------- About Myself Role of Analytics About Abebooks About Amazon Innovation and Data Abebooks DW modernization ETL Tool selection for the Cloud AWS Analytics Solution Prices example Some Free Learning resources
- 3. --------------------------------------------------------------- Disclaimer --------------------------------------------------------------- The content of this presentation doesn’t represent Abebooks, Amazon, and AWS. This information is based on my knowledge, experience and doesn’t have any sensitive or confidential data. All information and pictures are available online.
- 4. About Myself • Work с Business Intelligence c 2007 • Canada since 09/2015 • Pacific Ocean since 05/2016
- 6. Technical Skills Matrix 2015 2010 2007 Data Warehouse (Oracle, Teradata, Snowflake , Redshift) ETL (Pentaho DI, Informati ca, Matillion ETL, AWS Glue, Azure Data Factory) BI (SAP BusinessO bjects, Tableau, Microstra tegy, Pentaho BI, SAS BI) Bigdata (Hadoop, Data Lake) Digital Marketing Data Analytics 2018
- 7. Other Activities Tableau Cookbook 2019.X • BI Tech Talk (100+ BI teams globally) • Amazon Tableau User Group (2000+ users) • Conferences (EDW 2018, 2019) • Amazon internal conferences
- 9. Business Value Stakeholders Employees Customers Value ”The goal of any organization is to generate Value” The Future of Competition. https://www.amazon.com/Future-Competition-Co-Creating-Unique-Customers/dp/1578519535
- 10. BI Value Chain Stakeholders Employees Customers Value Decisions Data Value creation based on effective decisions Effective decisions based on accurate information
- 12. About Abebooks • Online marketplace for books, art & collectibles. • Amazon subsidiary since 2008 we are a marketplace for used books and increasingly non-book-collectibles • 350 Mlns listings • 3 in ‘Data Engineering Team’ • 2 locations: Victoria, BC and Dusseldorf
- 13. --------------------------------------------------------------- Outline --------------------------------------------------------------- A bit about Amazon Culture
- 15. Amazon Fly Wheel http://www.samseely.com/blog/2016/5/2/the-amazon-flywheel-part-1 In 90 Jeff Bezos shared Amazon business model: Amazon Fly Wheel = Price!=>Customers"=>Sellers"
- 16. Leadership Principles • Customer Obsession • Ownership • Invent and Simplify • Are Right, A Lot • Learn and Be Curious • Hire and Develop the Best • Insist on the Highest Standards • Think Big • Bias for Action • Frugality • Earn Trust • Dive Deep • Have Backbone; Disagree and Commit • Deliver Results https://www.amazon.jobs/en/principles
- 17. Leadership Principles (for me) • Customer Obsession • Ownership • Invent and Simplify • Are Right, A Lot • Learn and Be Curious • Hire and Develop the Best • Insist on the Highest Standards • Think Big • Bias for Action • Frugality • Earn Trust • Dive Deep • Have Backbone; Disagree and Commit • Deliver Results https://www.amazon.jobs/en/principles
- 19. Press Release, Narratives, FAQ
- 21. It is still Day One
- 23. History of Innovation Industrial Revolution 2nd Industrial Revolution Digital Revolution ??
- 33. We are here https://www.weforum.org/about/the-fourth-industrial-revolution-by-klaus-schwab Industrial Revolution 2nd Industrial Revolution Digital Revolution ??
- 38. AWS Rapid Pace of Innovation Chart: #AWS services https://www.slideshare.net/AmazonWebServices/a-culture-of-innovation-powered-by-aws
- 40. For Data to be a differentiator, customers need to be able to… • Capture and store new non- relational data at PB-EB scale in real time • Discover value in a new type of analytics that go beyond batch reporting to incorporate real-time, predictive, voice, and image recognition • Democratize access to data in a secure and governed way New types of analytics Dashboards Predictive Image Recognition VoiceReal-time New types of data
- 41. OLTP ERP CRM LOB Data Warehouse Business Intelligence § Relational data sources § TBs–PBs scale § Schema defined prior to data load § Operational reporting and ad hoc Traditionally, analytics used to look like this
- 42. Data & analytics partners extend the traditional approach Data Warehouse Business Intelligence OLTP ERP CRM LOB Devices Web Sensors Social Big Data processing, real-time, Machine Learning Data Lake § Relational and non-relational data § TBs–EBs scale § Diverse analytical engines § Low-cost storage & analytics
- 44. Abebooks BI/DW (before) Storage LayerSource Layer Ad-hoc SQL SFTP Data Warehouse ETL (PL/SQL) Files Inventory Sales Access Layer
- 45. BI/DW Survey
- 46. Cloud Migration Strategy Lift & Shift • Typical Approach • Move all-at-once • Target platform then evolve • Approach gets you to the cloud quickly • Relatively small barrier to learning new technology since it tends to be a close fit Split & Flip • Split application into logical functional data layers • Match the data functionality with the right technology • Leverage the wide selection of tools on AWS to best fit the need • Move data in phases — prototype, learn and perfect
- 48. Choosing ETL Tool for Cloud Use Cases • OLTP to S3 • S3 to Redshift • SFTP/API to Redshift • Data Transformation • Dimensional Modelling Tools • Pentaho DI • Informatica • AWD Data Pipeline/Glue • Talend • Matillion
- 49. ETL Criteria High: • Support native Redshift driver • Easily capture from relational db, CDC • Ease of Use for BI/DW • Cover use cases • On-Premise Medium: • Support NoSQL • Company “Winner” • Deployment/Architecture • Encryption • Ease of Use for non BI/DW • Data Transformations • Management • Pricing • Performance Low: • Version Control • Linux OS • ETL Monitoring • Logging • R/Pyhton
- 50. Why We Picked Matillion • specific redshift support, built around Redshift platform • speed of ETL operations • speed of development • wide range of data sources supported • ease of use outside of DE/DBA expertise • Native with AWS • $$$ • The biggest risk – putting our eggs in the Matillion future, betting on a small and new player.
- 51. Source Layer OLTP External API SFTP Ad-hock SQL Business Layer Amazon Redshift Amazon Elastic Load Balance SQS SNSAmazon Chime AWS Virtual Private Cloud Matillion Amazon EC2 instance Event/Notification Services Tableau Web Tableau Desktop Abebooks DW Modernization (after) ETL ELT Tableau Server Amazon EC2 instance
- 52. Abebooks DW extending Source Layer DynamoDB Amazon RDS External API SFTP APPs Ad-hock SQL Business Layer Amazon Athena Amazon EMR Amazon Redshift Amazon Redshift Spectrum Amazon Elastic Load Balance SQS SNS Amazon Chime Tableau Server AWS Virtual Private Cloud Matillion Amazon EC2 instance Event/Notification Services Amazon S3 Data Lake Tableau Web Tableau Desktop Apache Spark Big Data Processing
- 53. How does it look and feel?
- 55. X-Small Package Medium Package • BI Server** (EC2) (16vCPU/64RAM) – 585$ • ETL Server (EC2) (8 RAM) – 73$ • Data Lake (S3) 50 TB – 1200$ • Big Data Processing (EMR) 3node • DW (Redshift) 10 TB – 2500$ • 3rd Party ETL (Matillion) 1780$ • Redshift Spectrum ~500$ • Support – 460$ Total: ~7098*$ Example of Monthly Prices in US$ * you might get significant discount for Yearly Reserved Instances ** not include BI tool license cost https://calculator.s3.amazonaws.com/index.html • BI Server (EC2) – 146$ • ETL Server (EC2) – 31$ • DW (Redshift) 2 TB – 622$ • S3 Storage (50Gb) – 1.15 • 3rd Party ETL (Matillion) 986$ • Support – 123$ Total: ~2348$*
- 57. Coursera: • Data Warehouse for Business Intelligence Specialization • Data Engineering on Google Cloud Platform • Architecting with Google Cloud Platform AWS Tutorials: • Getting Started with Amazon Redshift • Sizing Amazon Redshift • Getting Started with Amazon Spectrum, Athena, Glue, EMR • AWS Free Tier (for example 2 months of Redshift) AWS Trainings: • AWS Technical Essentials Other: Google Machine Learning Crash Course (Deep Learning with TensorFlow) http://hackerrank.com Sql-ex.ru