Isbank initiated a DB2 for z/OS project in 2007 with two System z9 EC machines running z/OS 1.7 and DB2 V8. They installed DB2 V8 with Turkish codepage support, enabled one-way and two-way data sharing, attended training, and explored DB2 functionality. They developed a test environment with 5 data sharing groups and 4 members each and a production environment with 1 data sharing group and 4 members. They implemented a new core banking Java application using DB2 and explored performance monitoring and tuning techniques.

1. Lessons Learned from Isbank –
A Story of a DB2 for z/OS initiative
Cüneyt Göksu
IBM DB2 Gold Consultant
VBT
Session Code: F13
Wednesday, 10 November 2010 | Platform: z/OS
1

2. Due to Isbank’s confidentiality regulations some of
the performance numbers, naming conventions and
figures are removed, rearranged or renamed.
2

3. Agenda
• How did we start?
• Installation, Parallel Sysplex and More
• Brand New Corebank Application and Java
• IMS & DB2 Coexistence
• Unicode, SQL and Application Tuning Experiences
• Moving Windows ODS Application to z/OS
• V9 Migration
• Thinking V10
3

4. How did we start?
• In 2007
• 2 x System z9 EC
• z/OS 1.7
• DB2 V8
• Omegamon for DB2 V4.1.0
• DB2 Admin Tool from IBM
• No Application!
4

5. Installations, Parallel Sysplex and More
• Sandbox LPAR, V8 Installation and Naming Conventions.
• Install DB2 V8 NFM with Codepage 1026 (Turkish)
• One-Way Data Sharing Enabled
• Two-Way Data Sharing...
• CF Sizing and DataSharing Failure Test Scenarios
• CF350 - DB2 for z/OS and Data Sharing Implementation
Workshop
• CF83 - DB2 for z/OS Database Admin Workshop
• CV721 - DB2 9 for z/OS Application Programming
Workshop
• CE130 – Advanced SQL Workshop
5

6. Test Environment
• 4 LPARs
• 5 DataSharing Groups
• 4 Members / Group WHY?!?!?!
DEV3 DEV4
DEV1 DEV2
TEST3 TEST4
TEST1 TEST2
PAT3 PAT4
PAT1 PAT2
UAT3 UAT4
UAT1 UAT2
PERF3 PERF4
PERF1 PERF2
LPAR1 LPAR1 LPAR1
LPAR1
6

7. Production Environment
• 4 LPARs
• 1 DataSharing Group
• 4 Members / Group
DEV1 Prod 2 Prod 3 Prod 4
Prod 1
TEST1
PAT1
UAT1
PERF1
LPAR1 LPAR1 LPAR1
LPAR1
7

8. Explore the DB2
• IpPlex / VIPA
• RTS and Utility Automation
• Dynamic SortWork Allocation for DB2 Utilities
PK45916: REMOVE THE NEED TO SPECIFY
SORTNUM ON UTILITY STATEMENTS FOR
DYNAMICALLY ALLOCATED SORT WORK DATA
SETS
• UTSORTAL = YES
IGNSORTN = YES
MAX_UTIL_PARTS  PK51853
8
• SORTWKnn, SW01WKnn, DATAWKnn, STATWKnn, ST01WKnn

9. Explore the DB2
Monitor and Action against critical DB2 messages such as
• DSNJ111E (OUT OF SPACE IN ACTIVE LOG DATA SETS)
• DSNI014I(Broken Page)
• DSNP007I(EXTEND FAILED)
SMFACCT=(1,2,3,7,8,10)
SMFSTAT=(1,3,4,5,6,8)
ACCUMACC = NO  Development-Performance
ACCUMACC = 10  Prod
SYSIBM.SYSPACKAGE WHERE OPERATIVE <> 'Y' OR
VALID <> 'Y'
PARMLIB(CSVLLAxx)  ....SDSNEXIT
/F LLA,UPDATE=xx
9

10. Explore the DB2
• “Automated Space Management” or “sliding secondary extent”
MGEXTSZ=YES, PriQTY = -1, SecQTY = -1
• CLI Packages (SYSxynzz)
x: L or S : package size
y: H or N : hold or nohold
n: 1,2,3,4 : 1=UR, 2=CS, 3=RS, 4=RR
SYSLH3xx, SYSLH4xx, SYSSH3xx, SYSSH4xx  DISABLED
odbc.ini (WithHold=1)  1: With HOLD, 0: No HOLD
10

11. Corebank Application & JAVA
• Insert ONLY Banking Application.
DELETEs and UPDATEs are logical.
• Member Cluster
• Trackmod NO
• FREEPAGE 0 & PCTFREE 10
• DB2 for z/OS
Optimizing INSERT Performance
John Campbell
Insert Performance in DB2 UDB for z/OS V7/V8
Akira Shibamiya
11

12. Corebank Application & JAVA
• Optimistic Locking
1. Read row
2. Modify data
3. Try to write back data
4. Before updating the row check whether it has been
changed in the meantime; if so return with error
• Lock Avoidance
Dirty Read on package level or statement level
12

13. Corebank Application & JAVA
• Date, Time, TimeStamp, Type Conversion Costs
• Timestamp  STRING Datatype in Java
• SELECT ..., CHAR (AAPCR_START_DATE),
CHAR (DIP_EFCTV_DATE), ..., CHAR (DIL_HOST_TS),
CHAR (DIL_LOCAL_TS), DIL_WS_ID, DIL_USER_ID,
RULE_SYSTEM_ID, LDBID
FROM DRVD_INTRST_LINE D
• 10,000 rows in table
13

14. Corebank Application & JAVA
• Avg CPU Cost of retrieving a row %1,3
with SQL Function is 0,000107026
without SQL Function is 0,000105629
107,500000000
107,025927708
107,000000000
106,500000000
CPU Usage with SQL Func
106,000000000
CPU Usage w/o SQL Func
105,628945059
105,500000000
105,000000000
104,500000000
1000000
14

15. Corebank Application & JAVA
15

16. Corebank Application & JAVA
• Beware!  PreTest Connection Properties
16

17. Corebank Application & JAVA
• SELECT ....FROM ...WHERE ? BETWEEN C1 AND C2
Non-Indexable
• SELECT ....FROM ...WHERE C1 <= ? AND C2 >=?
Indexable
17

18. Corebank Application and Package Versioning
2800+ Program
30,000 Package with all versions!
Impacts on SPT01!
+-------------------------------------------------------------------------------------------------------------+
| COLLECTION | PACKAGE | VERSION | CREATETIME |
+-------------------------------------------------------------------------------------------------------------+
1_| CRBFID0 | AA01MP | CJ45_8210_TDV | 2008-06-06-09.03.57.265416 |
2_| CRBFID0 | AA01MP | isb45_8210_1 | 2008-10-30-10.44.13.749868 |
3_| CRBFID0 | AA01MP | isb45_8210fp3_Hotfix001.081021 | 2009-01-14-09.31.34.361108 |
4_| CRBFID0 | AA01MP | isb45_8210fp5_Hotfix001_090216 | 2009-02-27-11.27.59.529836 |
5_| CRBFID0 | AA01MP | isb45_8210fp7_Hotfix001_090310 | 2009-04-30-15.53.47.888161 |
6_| CRBFID0 | AA01MP | isb45_8210fp8_Hotfix001_090609 | 2009-06-16-14.27.03.542562 |
7_| CRBFID0 | AA01MP | isb45_8210fp8_Hotfix001_090615_WAS6.1 | 2009-06-17-18.24.05.582171 |
8_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090722 | 2009-07-27-15.53.44.351321 |
9_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090730_WAS6.1 | 2009-08-04-09.48.02.599017 |
10_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090819 | 2009-08-20-17.37.08.907330 |
11_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090821_WAS6.1 | 2009-08-21-16.21.56.794659 |
+-------------------------------------------------------------------------------------------------------------+
18

19. Corebank Application and Package Versioning
• Keep the last 5 versions
SELECT DISTINCT SUBSTR(SYSPACK.VERSION,1,46) AS VERSION,
DATE(TIMESTAMP) AS TARIH FROM SYSIBM.SYSPACKAGE SYSPACK WHERE
SYSPACK.COLLID LIKE 'CRB%‘ ORDER BY TARIH DESC;
+-------------------------------------------------------------+
! VERSION ! TARIH !
+-------------------------------------------------------------+
1_! isb50_9440fp5_DEV_100701 ! 2010-07-12 !
2_! isb50_9440fp4_DEV_100416 ! 2010-05-06 !
3_! isb50_9440fp3_DEV_100402 ! 2010-04-09 !
4_! isb50_9440fp3_DEV_100305 ! 2010-03-18 !
5_! isb50_9440fp3_DEV_100305 ! 2010-03-17 !
6_! isb45_8210fp10_Hotfix001_091109_zOS ! 2009-11-12 !
7_! isb45_8210fp10_Hotfix001_091023_WAS61 ! 2009-10-23 !
+-------------------------------------------------------------+ 19

20. Corebank Application and Package Versioning
• DISABLE 4 versions except with the highest CREATETIME
REBIND PACKAGE (CRBMIG0.*.(isb50_9440fp4_DEV_100416)) FLAG(E) -
DISABLE(BATCH,CICS,DB2CALL,DLIBATCH,IMSBMP,IMSMPP,REMOTE,
RRSAF) )
DSNT408I SQLCODE = -807, ERROR: ACCESS DENIED: PACKAGE PGM01
IS NOT ENABLED FOR ACCESS FROM BATCH
20

21. Corebank Application and Package Versioning
• Free the rest of the packages
FREE PACKAGE (CRBDEV0.*.(isb45_8210fp10_Hotfix001_091109_zOS)) )
21

22. IMS & DB2
IMS DB2 Co-existence
• All DB2s in Data Sharing z-OS
LPAR
LPAR
COBOL LPAR LPAR
Application
IMS DB DB2 DB2
SQLs
Data
Sharing
AIX
API
• Proven Architecture
JVM
• High Availability, Scalability, Flexibility with DB2
Data Sharing and IPPlex
• Additional SQL Workload for IMS LPAR
22

23. Cobol & Unicode
• CREATE TABLE .... CCSID UNICODE
• Unicode Support in Enterprise COBOL
• Compiler Option : NSYMBOL(NATIONAL)
BIND Option : ENCODING(UNICODE)
• DATE Datatype
77 W-TARIH-YYAG PIC N(10).
DISPLAY FUNCTION DISPLAY-OF (W-TARIH-YYAG,
1026).
23

24. Cobol & Unicode
• IDTYP_VALUE GRAPHIC(4)
• 77 V-IDTYP-MUSNO PIC N(4).
...
MOVE N'csid' TO V-IDTYP-MUSNO.
SELECT COUNT(*) FROM CRBFID0.IP_ID
WHERE IDTYP_VALUE = :V-IDTYP-MUSNO
24

25. Unicode & Unload
• UNLOAD TABLESPACE DMIGCUST.SIP
PUNCHDDN TEMP_2 UNLDDN TEMP_1
FROM TABLE CRBMIG0.IP WHEN
(IP_USER_ID<>'DMUSER')
• IP_USER_ID GRAPHIC(6)
• IP_USER_ID<>x'444D55534552‘
• MR1030081910  Not working for V8, V9, V10
25

26. Unicode & Load
• All tables have GRAPHIC • Lot’s of external data in
datatype such as; Delimeted Format such as;
IFS_IDFR DECIMAL(15, 0) 11.,”9310”,2010-01-01
CARDT_VALUE GRAPHIC(4) 22.,”9343”,2010-01-02
CARD_EXPR_DATE
LOAD DATA INDDN CXZ837AQ LOG NO REPLACE
EBCDIC CCSID(01026,01208,01200)
GRAPHIC (4)  CHAR(4)
FORMAT DELIMITED INTO TABLE
ÜCRBMIG0Ü. ÜFNRATLÜ
( ÜIFS_IDFRÜ POSITION(*) DECIMAL
, ÜCARDT_VALUEÜ POSITION(*) CHAR(004)
, ÜCARD_EXPR_DATEÜ POSITION(*) DATE
EXTERNAL )
26

27. IDTHTOIN
• IDTHTOIN : Idle Thread Timeout in seconds
IDTHTOIN = 0 ; Disables Timeout!
• IDTHTOIN = 900 or 3600 or else... 
• MR1122081733 : Different IDTHTOIN settings for different
cases!
• DB2 V10 : Profile Tables 
27

28. IDTHTOIN
• DB2 V10 GA 
• For monitoring threads and connections
• SYSIBM.DSN_PROFILE_TABLE
PROFILEID : unique identifier for the profile
AUTHID, PLANNAME, COLLID, PKGNAME, IPADDR : Monitored
Items
• SYSIBM.DSN_PROFILE_ATTRIBUTES
PROFILEID : unique identifier for the profile
KEYWORDS, ATTRIBUTE1, ATTRIBUTE2 =
MONITOR IDLE THREADS, WARNING or EXCEPTION, threshold
value 28

29. INCORROUT
• SELECT *
FROM CRBFID0.ISB_PROVISION_INPUT T1
WHERE T1.PRV_IDFR = 134000001420080
AND T1.PRVIN_HOST_TS = (SELECT MAX (PRVIN_HOST_TS)
FROM CRBFID0.ISB_PROVISION_INPUT
WHERE PRV_IDFR = T1.PRV_IDFR
AND PRVIT_TYPE = T1.PRVIT_TYPE)
• SELECT * FROM CRBFID0.ISB_PROVISION_INPUT T1, ( SELECT MAX (PRVIN_HOST_TS) PRVIN_HOST_TS, PRVIT_TYPE
FROM CRBFID0.ISB_PROVISION_INPUT WHERE PRV_IDFR = 134000001420080
GROUP BY PRVIT_TYPE ) AS T2 WHERE T1.PRV_IDFR = 134000001420080
AND T1.PRVIN_HOST_TS = T2.PRVIN_HOST_TS
• Tested with DB2 for z V8, V9, DB2 for LUW & Oracle
• PMR - 20549,001,862 = Apar Fix: AM05402
29

30. Tuning Experiences
CLASS 2 TIME DISTRIBUTION
-------------------------------------
CPU !==> 4%
SECPU !
NOTACC !========================> 48%
SUSP !========================> 48%  Accounting Reports
AVERAGE APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 2:06:13.02 1:46:04.64
SQL and Index Design
Buffer Pools
Physical Design, Partitioning...
30

31. Tuning Experiences
DASD AVG RESP TIME : .927
CPU Activity:
LPAR Busy: 40.04  RMF Reports
MVS BUsy: 100.0
Distribution of address spaces in IN
READY QUEUE <=N: 3.2 %
Physical CPU was “saturated”
Little Paging as well.
31

32. Tuning Experiences
AVERAGE APPL(CL.1) DB2 (CL.2)
------------
ELAPSED TIME
----------
20:13.1970
----------
17:41.2783
 Accounting Reports
Almost %500 Better Performance !
32

33. Tuning Experiences
Batch Cobol
Will Update 100M Records with nice Commit freq. 1-2
Million Updates in 15-20 seconds
Then slows down significiantly
+ Waits Count Total
+ --------------------------------- ---------- ------------
+ Synchronous I/O Wait 19834 00:00:21.688
+ Asynchronous Read I/O Wait 34 00:00:00.074
+ Log Write I/O Wait 360252 00:07:48.482
+ Sync EX Unit Sw-com/abort/dealloc 22 00:00:00.032
+ ------------------------------------------------------------
+ Total Class 3 Wait Time 00:08:27.970
Check RMF Reports!
CPU Cap after 10-20 sec.
With enough CPU  40,000 Update/sec
33

34. ODS Application Migration
ODS Application
Near 7/24 Application
Business Logic : Cobol Stored Procedure
Data Migration  DRDA & XLoader & Parallelism
34

35. ODS Application Migration
DSNL030I -PR4B DSNLILNR DDF PROCESSING FAILURE FOR
LUWID=GA521E67.E4ED.000000000000
AUTHID=-N/A-, REASON=00D31034
The MAX REMOTE CONNECTED limit was reached
ETL Jobs
Feeding ODS from OLTP, DW and more!...
Idle Thread  3600 !
CondBat (max remote connected) is increased!
35

36. V9 Migration Update
RSU0908 Applied to V8
Migration/fallback : II14401, II14464, II14441
Development and Prod DB2 in Sandbox Lpar
V9 Order Arrived
V9 CM Install to Sandbox Lpar
Fallback, Remigrate, ENFM, NFM
Test CM, Prod CM, Test NFM, Prod NFM
36
36

37. V9 Migration Gotchas!
Before Plan Stability usage: SPT01 is enlarged 3
times more
GRAPHIC was seen as VARGRAPHIC and vice versa
in IBM Admin Tool 7.2.0
APAR PM17236  PMR 23637,001,862
DSNJ111E -AT3B OUT OF SPACE IN ACTIVE LOG DATA SETS
DSNJ115I -AT3B OFFLOAD FAILED, COULD NOT ALLOCATE AN ARCHIVE DATA SET
AT3BMSTR(REGION=32K  REGION=0M)
DB2 Crash during OMPE Start  Addresed in
PK68377
37
37

38. Thinking V10
GA Since Oct 22nd
DB2 10 is optimized on zEnterprise
CPU savings … right out of the box
zEnterprise Upgrade to
Hardware DB2 10
100 19% less CPU usage
80
60
40
20
0
DB2 9.1 DB2 10 38

39. Thinking V10
• Performance improvement Categories
1. Improvements that you receive by simply migrating to
Version 10
2. Improvements that are implemented by DBA-level
changes, without requiring application changes
3. Improvements that require application changes
4. Improvements from scalability enhancements
39

40. Thinking V10
1. Improvements that you receive by simply migrating to V10
• Out-of-the box savings in NFM
• Improved index matching for OR and IN predicates
• Parallelism enhancements such as for multi-row fetch
• Collection of autonomic statistics. Stored procedures will
determine if statistics need to be collected then DB2
performs the collection.
40

41. Thinking V10
2. Improvements that are implemented by DBA-level
changes, without requiring application changes
• New Access Path: Hash Access
WHO can get benefit ?
1-Queries uses equal predicates to access a single row
2-Tables of a predictable and reasonably static size.
41

42. Thinking V10
3. Improvements that require application changes
• Access to currently committed data it minimizes
transaction suspension
HOW ?
1- a new bind option
CONCURRENTACCESSRESOLUTION
2- two new PREPARE statement clauses,
USE CURRENTLY COMMITTED and WAIT FOR
OUTCOME, which you can specify in the attribute-string.
42

43. Thinking V10
4. Improvements from scalability enhancements
• The scalability enhancements contributes to improved performance
Reductions in log latch contention
Most cases, DB2 holds the log latch for less time when creating log
records, because of the increased use of compare and swap logic.
Enhancement does NOT require configuration or application changes.
Provides reductions in transaction and CPU time when logging rates
are high.
43

44. Thinking V10
MORE...
• 10x more concurrent users, up to 20,000 / subsystem
(MAXDBAT zparm)
• Temporal Data
• Thread Management thru Profile Tables
44

45. Cuneyt Goksu
VBT
IBM DB2 Gold Consultant
cuneyt.goksu@vbt.com.tr
F13
Lessons Learned from Isbank – A Story of a DB2 for
z/OS initiative
45