Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (10)
Similar a SNS practice: Generating ETL
Similar a SNS practice: Generating ETL (20)
Último
Último (20)
SNS practice: Generating ETL
- 1. SNS practice: Generating ETL Maarten Ketelaars Data Architect
- 2. 2 Agenda Introduction SNS Bank ICT strategy DWH framework & tooling Data VaultTM modeling ETL-procedure Modeling deliverables & procedure Advantages Further developments
- 3. 3 Introduction SNS Bank Create financial resources for individuals and companies Focus on innovative products and services Mortgages Asset accrual and protection Key figures 2003 Employee 3400 Profit E 129 Million Balance sheet E 38,5 Billion
- 4. 4 Introduction SNS Bank Create financial resources for individuals and companies Focus on innovative products and services Mortgages Asset accrual and protection Key figures 2003 Employee 3400 Profit E 129 Million Balance sheet E 38,5 Billion
- 5. 5 ICT strategy Standardisation Development environments Infrastructure components Application-integration Solutions Re-use -- Buy -- Build Open Systemarchitecture / Open standards No vendor lock-in Business-ICT alignment (developments in ICT & Business translate to changes in standards & directions)
- 6. 6 DWH framework Data exploitationData storageData extraction Data Staging Interface Data Mart 1 Operational Data Store Analysis Standard Report Data Warehouse Ad-hoc questions Interface Source System Data Mart 2 Data Mart 3 Data Mart 4 Analysis Report Ad-hoc questions Operational TacticalStrategical Metadata management Maintenance Infrastructure Organisation Source System Source System External Source Operational
- 7. 7 Tooling Data exploitationData storageData extraction Data Staging Interface Data Mart 1 Operational Data Store Analysis Standard Report Data Warehouse Ad-hoc questions Interface Source System Data Mart 2 Data Mart 3 Data Mart 4 Analysis Report Ad-hoc questions Operational TacticalStrategical Metadata management Maintenance Infrastructure Organisation Source System Source System External Source Operational
- 8. 8 Tooling Data exploitationData storageData extraction Data Staging Interface Data Mart 1 Operational Data Store Analysis Standard Report Data Warehouse Ad-hoc questions Interface Source System Data Mart 2 Data Mart 3 Data Mart 4 Analysis Report Ad-hoc questions Operational TacticalStrategical Metadata management Maintenance Infrastructure Organisation Source System Source System External Source Operational
- 9. 9 Tooling Data exploitationData storageData extraction Data Staging Interface Data Mart 1 Operational Data Store Analysis Standard Report Data Warehouse Ad-hoc questions Interface Source System Data Mart 2 Data Mart 3 Data Mart 4 Analysis Report Ad-hoc questions Operational TacticalStrategical Metadata management Maintenance Infrastructure Organisation Source System Source System External Source Operational
- 10. 10 Data Vault™ DWH model (1) • 3 NF+Timestamps (80’s) • Dimensional modelling (90’s) • Data Vault TM (since 2000) detail oriented historical tracking uniquely linked set of normalized tables Hybrid approach encompassing the best of breed between 3NF and dimensional modelling. Dan Linstedt : www.danlinstedt.com
- 11. 11 Data Vault™ DWH model (2) Hub Table carrying a unique list of business keys, representing a business component Link Represents the relation or transaction between two or more business components Satellite Context information about Hubs or Links. Information is subject to change over time
- 12. 12 Data Vault™ DWH model (3) Complementary choices SNS Satellites State satellite. Validation of hubs & links Information satellite. Specific properties of hubs and links, stored in attributes
- 13. 13 Data Vault™ DWH model (4) Rekening kantoor van opening status Rekening status Rekening kantoor van inschrijving status Rekening rekeningsoort status Rekeningsoort ID Rekeningsoort Rekeningsoortnummer Banknummer Bank INTEGER INTEGER INTEGER Link Rekeningsoort Rekening ID Rekeningsoort ID Rekening INTEGER INTEGER Rekening ID Rekening Rekeningnummer INTEGER DECIMAL(10) Link rekening kantoor van inschrijving ID Rekening ID Kantoor INTEGER INTEGER Rekeningsoort Status ID Rekeningsoort Rekeningsoort indicator Begindatum rekeningsoort Einddatum rekeningsoort INTEGER SMALLINT INTEGER INTEGER Rekeningsoort Informatie ID Rekeningsoort Begindatum rekeningsoort Einddatum rekeningsoort Indicator zakelijk product Indicator lening Methode maximum debetstand WTK product codering Naam rekeningsoort Korte naam rekeningsoort INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40) Rekening compensatielimiet ID Rekening Agenderingsnummer Begindatum rekening Einddatum rekening Compensatielimietbedrag INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) Rekening betaalruime ID Rekening Agenderingsnummer Begindatum rekening Einddatum rekening Betaalruimte INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) Rekening saldo ID Rekening Begindatum rekening Einddatum rekening Saldo rekening INTEGER INTEGER INTEGER DECIMAL(11,2) Rekening informatie ID Rekening Begindatum rekening Einddatum rekening Bedrag Kredietlimiet Saldocompensatie Blokkeringsindicator WTK codering Datum opening rekening Datum afbetaald Datum herleefd INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER Kantoren ID Kantoor Banknummer Bank Kantoornummer INTEGER INTEGER INTEGER Link rekening kantoor van opening ID Rekening ID Kantoor INTEGER INTEGER
- 14. 14 Data Vault™ DWH model (5) DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40)
- 15. 15 DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40) Data Vault™ DWH model (6) DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40)
- 16. 16 DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40) Data Vault™ DWH model (7) DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER
- 17. 17 DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40) Data Vault™ DWH model (8) DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2)
- 18. 18 DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40) Data Vault™ DWH model (9) DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2)
- 19. 19 DWS_KNO_STS DWS_KNI_STS DWS_REK_STS DWS_REK_RST_STS DWS_RST_STS RST_ID RST_IND RST_BEG_DAT RST_END_DAT INTEGER SMALLINT INTEGER INTEGER DWH_KAN KAN_ID BNK_NR KAN_NR INTEGER INTEGER INTEGER DWL_KNO REK_ID KAN_ID INTEGER INTEGER DWL_KNI REK_ID KAN_ID INTEGER INTEGER DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_SLD REK_ID REK_BEG_DAT REK_END_DAT REK_SLD_BDR INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_INA REK_ID REK_BEG_DAT REK_END_DAT REK_KLM_BDR REK_SCP_IND REK_BLK_IND REK_WTK_COD REK_OPR_DAT REK_AFB_DAT REK_HRL_DAT INTEGER INTEGER INTEGER DECIMAL(11,2) SMALLINT SMALLINT SMALLINT INTEGER INTEGER INTEGER DWS_REK_CLM REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_CLM_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2) DWL_REK_RST RST_ID REK_ID INTEGER INTEGER DWH_RST RST_ID RST_NR BNK_NR INTEGER INTEGER INTEGER DWS_RST_INA RST_ID RST_BEG_DAT RST_END_DAT RST_ZAK_IND RST_LEN_IND RST_MMD_NCD RST_WTK_NCD RST_NAA RST_NAK INTEGER INTEGER INTEGER SMALLINT SMALLINT SMALLINT SMALLINT VARCHAR(40) VARCHAR(40) Data Vault™ DWH model (10) DWH_REK REK_ID REK_NR INTEGER DECIMAL(10) DWS_REK_BTR REK_ID REK_ANR REK_BEG_DAT REK_END_DAT REK_BTR_BDR INTEGER INTEGER INTEGER INTEGER DECIMAL(11,2)
- 20. 20 Data Vault™ DWH model (11) Hubs & Links: Type 0 Satellites: Type 2 Begindate Enddate 99991231 Transaction-satellite: Type 0
- 26. 26 Modeling deliverables DV DWH Create table scripts Proces Database create P_table scripts create view scripts create macro scripts Sources & Targets in PowerCenter (Metadata Exchange)
- 27. 27 Modeling procedure 2b SNS DV-repository Database script2a 2c DWH create Proces P-tabel create Proces View create Proces Macro create 3b Sources Targets PowerCenter 3a PDM in repository 1b PDM1a CDM1a CDM 1b PDM Database script2a 2b SNS DV-repository 2c DWH create Proces P-tabel create Proces View create Proces Macro create 3a PDM in repository
- 28. 28 Advantages DWH modeling Standardized Flexible model Very well suited for incremental development ETL development Less development and test effort Database dependancy is isolated Performance Using capacity Teradata in ETL-proces
- 29. 29 Further developments 1. Generating PowerCenter workflows/mappings 2. Adding Data Mart functionality 1. Type 1 functionality 2. Type 3 functionality 3. Generating DV DWH model direct from source system specification
- 30. 30 For more information SNS Juan-Jose.vanderLinden@sns.nl Maarten.Ketelaars@sns.nl Data VaultTM www.danlinstedt.com