Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Data Warehousing Trends, Best Practices, and Future Outlook

Ad

Data Warehousing Trends,
Best Practices, and Future
Outlook
James Serra
Data Platform Architecture Lead
EY
JamesSerra3@gma...

Ad

About Me
 EY, Data Platform Architecture Lead
 Was previously a Data & AI Architect at Microsoft for seven years
 In IT...

Ad

Agenda
 Why Data Warehouse
 Data Warehouse vs Data Mart
 Kimball vs Inmon
 Data Vault
 Data Lake
 Why Data Warehouse...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Cargando en…3
×

Eche un vistazo a continuación

1 de 36 Anuncio
1 de 36 Anuncio

Data Warehousing Trends, Best Practices, and Future Outlook

Descargar para leer sin conexión

Over the last decade, the 3Vs of data - Volume, Velocity & Variety has grown massively. The Big Data revolution has completely changed the way companies collect, analyze & store data. Advancements in cloud-based data warehousing technologies have empowered companies to fully leverage big data without heavy investments both in terms of time and resources. But, that doesn’t mean building and managing a cloud data warehouse isn’t accompanied by any challenges. From deciding on a service provider to the design architecture, deploying a data warehouse tailored to your business needs is a strenuous undertaking. Looking to deploy a data warehouse to scale your company’s data infrastructure or still on the fence? In this presentation you will gain insights into the current Data Warehousing trends, best practices, and future outlook. Learn how to build your data warehouse with the help of real-life use-cases and discussion on commonly faced challenges. In this session you will learn:

- Choosing the best solution - Data Lake vs. Data Warehouse vs. Data Mart
- Choosing the best Data Warehouse design methodologies: Data Vault vs. Kimball vs. Inmon
- Step by step approach to building an effective data warehouse architecture
- Common reasons for the failure of data warehouse implementations and how to avoid them

Over the last decade, the 3Vs of data - Volume, Velocity & Variety has grown massively. The Big Data revolution has completely changed the way companies collect, analyze & store data. Advancements in cloud-based data warehousing technologies have empowered companies to fully leverage big data without heavy investments both in terms of time and resources. But, that doesn’t mean building and managing a cloud data warehouse isn’t accompanied by any challenges. From deciding on a service provider to the design architecture, deploying a data warehouse tailored to your business needs is a strenuous undertaking. Looking to deploy a data warehouse to scale your company’s data infrastructure or still on the fence? In this presentation you will gain insights into the current Data Warehousing trends, best practices, and future outlook. Learn how to build your data warehouse with the help of real-life use-cases and discussion on commonly faced challenges. In this session you will learn:

- Choosing the best solution - Data Lake vs. Data Warehouse vs. Data Mart
- Choosing the best Data Warehouse design methodologies: Data Vault vs. Kimball vs. Inmon
- Step by step approach to building an effective data warehouse architecture
- Common reasons for the failure of data warehouse implementations and how to avoid them

Anuncio
Anuncio

Más Contenido Relacionado

Similares a Data Warehousing Trends, Best Practices, and Future Outlook (20)

Anuncio
Anuncio

Data Warehousing Trends, Best Practices, and Future Outlook

  1. 1. Data Warehousing Trends, Best Practices, and Future Outlook James Serra Data Platform Architecture Lead EY JamesSerra3@gmail.com Blog: JamesSerra.com
  2. 2. About Me  EY, Data Platform Architecture Lead  Was previously a Data & AI Architect at Microsoft for seven years  In IT for 35 years, worked on many BI and DW projects  Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM architect, PDW/APS developer  Been perm employee, contractor, consultant, business owner  Presenter at PASS Summit, SQLBits, Enterprise Data World conference, Big Data Conference Europe, SQL Saturdays  Blog at JamesSerra.com  Former SQL Server MVP  Author of book “Reporting with Microsoft SQL Server 2012”
  3. 3. Agenda  Why Data Warehouse  Data Warehouse vs Data Mart  Kimball vs Inmon  Data Vault  Data Lake  Why Data Warehouses fail  How Data Warehouses succeed  Data Warehouse concepts  Modern Data Warehouse  Data Fabric  Data Lakehouse  Data Mesh
  4. 4. I tried to build a data warehouse on my own… And ended up passed-out drunk in a Denny’s parking lot Let’s prevent that from happening…
  5. 5. What a Data Warehouse is not • A data warehouse is not a copy of a source database with the name prefixed with “DW” • It is not a copy of multiple tables (i.e. customer) from various sources systems unioned together in a view • It is not a dumping ground for tables from various sources with not much design put into it
  6. 6. What is a Data Warehouse and why use one? A data warehouse is where you store data from multiple data sources to be used for historical and trend analysis reporting. It acts as a central repository for many subject areas and contains the "single version of truth". It is NOT to be used for OLTP applications. Reasons for a data warehouse:  Reduce stress on production system  Optimized for read access, sequential disk scans  Integrate many sources of data  Keep historical records (no need to save hardcopy reports)  Restructure/rename tables and fields, model data  Protect against source system upgrades  Use Master Data Management, including hierarchies  No IT involvement needed for users to create reports  Improve data quality and plugs holes in source systems  One version of the truth  Easy to create BI solutions on top of it (i.e. Azure Analysis Services Cubes)
  7. 7. Why use a Data Warehouse? Legacy applications + databases = chaos Production Control MRP Inventory Control Parts Management Logistics Shipping Raw Goods Order Control Purchasing Marketing Finance Sales Accounting Management Reporting Engineering Actuarial Human Resources Continuity Consolidation Control Compliance Collaboration Enterprise data warehouse = order Single version of the truth Enterprise Data Warehouse Every question = decision Two purposes of data warehouse: 1) save time building reports; 2) slice in dice in ways you could not do before
  8. 8. Enterprise Data Maturity Stages Structured data is transacted and locally managed. Data used reactively STAGE 2: Informative STAGE 1: Reactive Structured data is managed and analyzed centrally and informs the business Data capture is comprehensive and scalable and leads business decisions based on advanced analytics STAGE 4: Transformative STAGE 3: Predictive Data transforms business to drive desired outcomes. Real-time intelligence Rear-view mirror Any data, any source, anywhere at scale
  9. 9. Data Warehouse vs Data Mart  Data Warehouse: A single organizational repository of enterprise wide data across many or all subject areas  Holds multiple subject areas  Holds very detailed information  Works to integrate all data sources  Feeds data mart  Data Mart: Subset of the data warehouse that is usually oriented to specific subject (finance, marketing, sales) • The logical combination of all the data marts is a data warehouse In short, a data warehouse as contains many subject areas, and a data mart contains just one of those subject areas
  10. 10. Kimball and Inmon Methodologies Two approaches for building data warehouses
  11. 11. Kimball and Inmon Myths  Myth: Kimball is a bottom-up approach without enterprise focus  Really top-down: BUS matrix (choose business processes/data sources), conformed dimensions, MDM  Myth: Inmon requires a ton of up-front design that takes a long time  Inmon says to build DW iteratively, not big bang approach (p. 91 BDW, p. 21 Imhoff)  Myth: Star schema data marts are not allowed in Inmon’s model  Inmon says they are good for direct end-user access of data (p. 365 BDW), good for data marts (p. 12 TTA)  Myth: Very few companies use the Inmon method  Survey had 39% Inmon vs 26% Kimball. Many have a EDW  Myth: The Kimball and Inmon architectures are incompatible  They can work together to provide a better solution
  12. 12. Kimball and Inmon Methodologies  Relational (Inmon) vs Dimensional (Kimball)  Relational Modeling:  Entity-Relationship (ER) model  Normalization rules  Many tables using joins  History tables, natural keys  Good for indirect end-user access of data  Dimensional Modeling:  Facts and dimensions, star schema  Less tables but have duplicate data (de-normalized)  Easier for user to understand (but strange for IT people used to relational)  Slowly changing dimensions, surrogate keys  Good for direct end-user access of data
  13. 13. Relational Model vs Dimensional Model Relational Model Dimensional Model If you are a business user, which model is easier to use?
  14. 14. Kimball and Inmon Methodologies • Kimball: • Logical data warehouse (BUS), made up of subject areas (data marts) • Business driven, users have active participation • Decentralized data marts (not required to be a separate physical data store) • Independent dimensional data marts optimized for reporting/analytics • Integrated via Conformed Dimensions (provides consistency across data sources) • 2-tier (data mart, cube), less ETL, no data duplication • Inmon: • Enterprise data model (CIF) that is a enterprise data warehouse (EDW) • IT Driven, users have passive participation • Centralized atomic normalized tables (off limit to end users) • Later create dependent data marts that are separate physical subsets of data and can be used for multiple purposes • Integration via enterprise data model • 3-tier (data warehouse, data mart, cube), duplication of data
  15. 15. Kimball Model Why staging: Limit source contention (ELT), Recoverability, Backup, Auditing
  16. 16. Inmon Model
  17. 17. Reasons to add a Enterprise Data Warehouse  Single version of the truth  May make building dimensions easier using lightly denormalized tables in EDW instead of going directly from the OLTP source  Normalized EDW results in enterprise-wide consistency which makes it easier to spawn-off the data marts at the expense of duplicated data  Less daily ETL refresh and reconciliation if have many sources and many data marts in multiple databases  One place to control data (no duplication of effort and data)  Reason not to: If have a few sources that need reporting quickly
  18. 18. Which model to use?  Models are not that different, having become similar over the years, and can compliment each other  Boils down to Inmon creates a normalized DW before creating a dimensional data mart and Kimball skips the normalized DW  With tweaks to each model, they look very similar (adding a normalized EDW to Kimball, dimensionally structured data marts to Inmon)  Bottom line: Understand both approaches and pick parts from both for your situation – no need to just choose just one approach  BUT, no solution will be effective unless you possess soft skills (leadership, communication, planning, and interpersonal relationships)
  19. 19. Hybrid Model Advice: Use SQL Server Views to interface between each level in the model (default values, computations, rename fields, etc) In the DW Bus Architecture, each data mart could be a schema (broken out by business process subject areas), all in one database. Another option is to have each data mart in its own database with all databases on one server or spread among multiple servers. Also, the staging areas, CIF, and DW Bus can all be on the same powerful server (MPP)
  20. 20. Kimball Methodology From: Kimball’s The Microsoft Data Warehouse Toolkit Kimball defines a development lifecycle, where Inmon is just about the data warehouse (not “how” used)
  21. 21. Data Vault • Daniel Linstedt, author of Building Scalable Data Warehouse with Data Vault 2.0 • Not just a modeling technique, but entire methodology for DW projects • Layer between Star Schema and Staging (use in combination with Kimball model, modeled somewhere between the 3NF and star schema) • Improves audit and historical tracking • Hubs (business entities) • Links (relationships) • Satellites (descriptive attributes) https://www.indellient.com/blog/data-vault-what-is-it-and-when-should-it-be-used/
  22. 22. Observation Pattern Theory Hypothesis What will happen? How can we make it happen? Predictive Analytics Prescriptive Analytics What happened? Why did it happen? Descriptive Analytics Diagnostic Analytics Confirmation Theory Hypothesis Observation Two Approaches to getting value out of data: Top-Down + Bottoms-Up
  23. 23. Implement Data Warehouse Physical Design ETL Development Reporting & Analytics Development Install and Tune Reporting & Analytics Design Dimension Modelling ETL Design Setup Infrastructure Understand Corporate Strategy Data Warehousing Uses A Top-Down Approach Data sources Gather Requirements Business Requirements Technical Requirements
  24. 24. The “data lake” Uses A Bottoms-Up Approach Ingest all data regardless of requirements Store all data in native format without schema definition Do analysis Using analytic engines like Hadoop Interactive queries Batch queries Machine Learning Data warehouse Real-time analytics Devices
  25. 25. Data Lake + Data Warehouse Better Together Data sources What happened? Descriptive Analytics Diagnostic Analytics Why did it happen? What will happen? Predictive Analytics Prescriptive Analytics How can we make it happen?
  26. 26. Exactly what is a data lake? A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed. Reasons to use a data lake: • Inexpensively store unlimited data • Centralized place for multiple subjects (single version of the truth) • Collect all data “just in case” (data hoarding). The data lake is a good place for data that you “might” use down the road • Easy integration of differently-structured data • Store data with no modeling – “Schema on read” • Complements enterprise data warehouse (EDW) • Frees up expensive EDW resources for queries instead of using EDW resources for transformations (avoiding user contention) • Wanting to use technologies/tools (i.e Databricks) to refine/filter data that do the refinement quicker/better than your EDW • Quick user access to data for power users/data scientists (allowing for faster ROI) • Data exploration to see if data valuable before writing ETL and schema for relational database, or use for one-time report • Allows use of Hadoop tools such as ETL and extreme analytics • Place to land IoT streaming data • On-line archive or backup for data warehouse data • With Hadoop/ADLS, high availability and disaster recovery built in • It can ingest large files quickly and provide data redundancy • ELT jobs on EDW are taking too long because of increasing data volumes and increasing rate of ingesting (velocity), so offload some of them to the Hadoop data lake • Have a backup of the raw data in case you need to load it again due to an ETL error (and not have to go back to the source). You can keep a long history of raw data • Allows for data to be used many times for different analytic needs and use cases • Cost savings and faster transformations: storage tiers with lifecycle management; separation of storage and compute resources allowing multiple instances of different sizes working with the same data simultaneously vs scaling data warehouse; low-cost storage for raw data saving space on the EDW • Extreme performance for transformations by having multiple compute options each accessing different folders containing data • The ability for an end-user or product to easily access the data from any location
  27. 27. Data Warehouse Serving, Security & Compliance • Business people • Low latency • Complex joins • Interactive ad-hoc query • High number of users • Additional security • Large support for tools • Dashboards • Easily create reports (Self-service BI) • Know questions
  28. 28. Why Data Warehouses fail • 70-80% corporate BI projects fail (Gartner) • The executive who thinks data warehousing is “easy” • Starting with an end date and working backwards • Lack of user involvement • Insufficient business requirements or too much • Insufficient architect/design or too much • Not enough planning for data governance • Chose the wrong technology
  29. 29. Why Data Warehouses fail • Poor communication between IT and the business • Not setting up time to “validate the numbers” • Performance issues (slow response times) • Inexperience and lack of technical knowledge • Modeling the data warehouse to reflect the source data rather than the business
  30. 30. Why Data Warehouses fail • The wrong consulting company is hired • Lack of experience • Using offshore developers • Owning the whole project • No mentoring or knowledge transfer • Budget runs out • Developers • Testers • DBA • PM • Cloud costs • Software licenses • Training
  31. 31. How DW project succeed • Choosing the right technology • Understand all aspects of data warehouse architectures • Kimball vs Inmon • Star Schema vs Relational • Modern data warehouse vs data fabric vs data lakehouse vs data mesh • Find a project champion/sponsor • Have DW director who keeps everyone running at 80% efficiency • Interative/Agile approach to delivery with frequent releases • Break project into phases • Start with a high-priority subject but one that is small and easy • Get a quick win that shows lot’s of benefits and market it • Build a solid foundation with a repeatable process • Get people excited about DW thru training • User Power BI to prototype/requirements
  32. 32. Modern Data Warehouse Data Fabric adds: data access, data policies, data catalog, MDM, data virtualization, data scientist tools, APIs, building blocks, products
  33. 33. Data Lakehouse
  34. 34. Data Mesh
  35. 35. Data Mesh Credit to Zhamak Dehghani
  36. 36. Q & A ? James Serra, Big Data Evangelist Email me at: JamesSerra3@gmail.com Follow me at: @JamesSerra Link to me at: www.linkedin.com/in/JamesSerra Visit my blog at: JamesSerra.com

×