9. Data Warehouse Architecture Data Warehouse Engine Optimized Loader Extraction Cleansing Analyze Query Metadata Repository Relational Databases Legacy Data Purchased Data ERP Systems
14. Data Integration Across Sources Trust Credit card Savings Loans Same data different name Different data Same name Data found here nowhere else Different keys same data
15. Data Transformation Example Data Warehouse encoding unit field appl A - balance appl B - bal appl C - currbal appl D - balcurr appl A - pipeline - cm appl B - pipeline - in appl C - pipeline - feet appl D - pipeline - yds appl A - m,f appl B - 1,0 appl C - x,y appl D - male, female
37. Vertical Partitioning Frequently accessed Rarely accessed Smaller table and so less I/O Acct. No Name Balance Date Opened Interest Rate Address Acct. No Balance Acct. No Name Date Opened Interest Rate Address
55. From the Data Warehouse to Data Marts Data Information Departmentally Structured Individually Structured Data Warehouse Organizationally Structured Less More History Normalized Detailed
56.
57. Data Warehouse and Data Marts OLAP Data Mart Lightly summarized Departmentally structured Organizationally structured Atomic Detailed Data Warehouse Data
69. Bitmap Index Customer Query : select * from customer where gender = ‘F’ and vote = ‘Y’ gender (f) vote (y) result vote gender 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 M F F F F M Y Y Y N N N
87. A Visual Operation: Pivot (Rotate) 10 47 30 12 Juice Cola Milk Cream NY LA SF 3/1 3/2 3/3 3/4 Month Region Product
88. “ Slicing and Dicing” Product Sales Channel Regions Retail Direct Special Household Telecomm Video Audio India Far East Europe The Telecomm Slice
89.
90.
91.
92.
93.
94.
95.
96. Relational OLAP: 3 Tier DSS Store atomic data in industry standard RDBMS. Generate SQL execution plans in the ROLAP engine to obtain OLAP functionality. Obtain multi-dimensional reports from the DSS Client. Data Warehouse ROLAP Engine Decision Support Client Database Layer Application Logic Layer Presentation Layer
97.
98.
99. MD-OLAP: 2 Tier DSS MDDB Engine MDDB Engine Decision Support Client Database Layer Application Logic Layer Presentation Layer Store atomic data in a proprietary data structure (MDDB), pre-calculate as many outcomes as possible, obtain OLAP functionality via proprietary algorithms running against this data. Obtain multi-dimensional reports from the DSS Client.
110. References/External Links (1) Data Warehousing & Data Mining S. Sudarshan Krithi Ramamritham IIT Bombay (2) Data Warehousing Hu Yan e-mail: [email_address] (3) Data Warehosing Concept: MOLAP, ROLAP and HOLAP http://www.1keydata.com/datawarehousing/molap-rolap.html
111. Thank you for your attention! [email_address] www.blueballgroup.com