CMG 2012 - Tuning where it matters - Gerry Tuddenham
•
0 recomendaciones•1,634 vistas
The document summarizes a case study on tuning software costs for a large international bank. It describes how the consultant analyzed the client's workload profile, identified areas of high resource consumption like a CICS region with high RACF overhead, and made recommendations such as reducing security checks and spreading batch jobs to off-peak hours to lower software bills. Modeling showed how rescheduling backups could save an additional 20% of batch costs.
Recomendados
Más contenido relacionado
Destacado
Destacado (11)
Más de Rodrigo Campos
Más de Rodrigo Campos (20)
Último
Último (20)
CMG 2012 - Tuning where it matters - Gerry Tuddenham
- 1. CMG Brazil Conference 2012 Tuning where it matters – a case study Prepared by CPT Global Limited Presenter Gerry Tuddenham Telephone: +61 418 372 999 Email: gtuddenham@cptglobal.com Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 2. PRESENTATION CONTENT • Set the scene – the Client, the issue • How we targeted the reduction efforts • Approach taken – top down • Examples of things we looked for / found • Examples of changes recommended • Where we are going next • Results •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 3. SETTING THE SCENE • Client is a large International bank • Completely different culture • Very risk-averse, and change averse • Very untrusting of non-French speaking “experts” •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 4. THE ISSUE • Growing monthly SCRT based software cost • Approaching the point of requiring a large upgrade •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 5. TECHNICAL ENVIRONMENT • Client had recently centralised a number of region data centres onto 2 CECs in 2 data centres in a central location • Consolidation had just been physical, so LPARs were still regional, though mainly ran the same applications with regional flavours • Throughout the project the client has been migrating workloads and moving to a non-regional configuration • Applications mainly CICS / COBOL / DB2, though some CICS / VSAM •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 6. TUNING FOCUS Typical Challenge Approach • Capacity Issue • Move workloads to quieter times Run own • Tune what is left systems • Reduce software bill • Need to focus on the peak 4HRA and smoothing the workload • Billed based on peak hour • Move workloads to quieter times • Make outsourcer move housekeeping Outsourced • Tune what is left • Billed based on total • Tune consumption • Customer running out of • Move workloads to quieter times capacity • Tune what is left Outsourcer • Customer pays for application • Only tune what is not billed, don’t tune address spaces only system-wide parameters •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 7. TOP DOWN, FOCUSED APPROACH • Understand the goals of tuning • Understand the workload profile (RMF70, RMF72) • Decide on area of focus • Identify the big burners (SMF30) • Use a sampling tool to understand what they are doing • Build and implement recommendations •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 8. FOCUS WHERE IT MATTERS Data for CEC « B », Batch the other CEC has a Peak monthly 4HRA driven TSO similar profile Online by online morning peak STC 6000 System Uncaptured 5500 4HRA 5000 4500 4000 3500 3000 MIPS 2500 2000 1500 1000 500 0 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h Fri 0h Thu 0h Mon 0h Tue 0h Wed 0h Hourly intervals for week starting 2 December 2008 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 9. FOCUS WHERE IT MATTERS Peak monthly 4HRA driven Area of focus by online morning peak 4HRA 6000 5500 5000 4500 4000 3500 MIPS 3000 2500 2000 1500 1000 500 0 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h Thu 0h Mon 0h Fri 0h Tue 0h Wed 0h Hourly intervals for week starting 2 December 2008 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 10. FOCUS WHERE IT MATTERS Batch 7000 Batch Large online peak Large midday TSO Online 6500 throughout day batch suite STC System 6000 Uncaptured 5500 4HRA 5000 4500 4000 3500 MIPS 3000 2500 2000 1500 1000 500 0 08h00 08h30 09h00 09h30 10h00 10h30 11h00 11h30 12h00 12h30 13h00 13h30 14h00 15 minute intervals for Tuesday 3 December 2008 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 11. INITIAL SAMPLE – LARGEST CICS REGION High RACF overhead MODULE SECTION PROCEDURE/FUNCTION CPU TIME PERCENT NAME NAME NAME TOTAL .SUPERVS IGC0013{ IRRRFC23 SUPERVISOR SERVICES 17.78 .NUCLEUS IEAVESL SUSPEND LOCK SERVICE 7.36 .DB2 DSNK2DM DSNKNXT2 FETCH NEXT TYPE 2 INDEX 4.64 .DB2 DSNK2DM DSNKFTCH FETCH TYPE 2 IDX ENTRY 2.61 .CICS DFHSIP DFHTRPX TR DOM-TRC PUT-FAST PATH 2.51 .DB2 DSNBBM DSNB1GET RETRIEVE REQUESTED PAGE 1.38 .LELIB CEEPLPKA CEEV#GTS GET HEAP STORAGE 2.00 .DB2 DSNIDM DSNIRNXT FETCH NEXT ROW TO PROG 1.37 .DB2 DSNIDM DSNIONX2 NEXT ON CUB ON IXED FAN 1.02 CICS Trace active Lots of DB2 activity LE Storage Parameters •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 12. REDUCING THE RACF OVERHEAD • SIT security parameters • Check the Xxxx parameters in SIT • XPPT and XFCT are very expensive • If they are set, ask why • MRO session security • With ATTACHSEC = IDENTIFY on a CICS MRO link, CICS will perform 2 RACF checks at transaction attach: one for the incoming userid, another for the session userid • Letting the SESSION userid default to the region default userid reduces this to 1 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 13. FOCUS WHERE IT MATTERS - 2 6000 Peak now overnight Area of focus batch 4HRA in December 2008 5500 4HRA 5000 4500 4000 3500 3000 MIPS 2500 2000 1500 1000 500 0 Thu 0h 4h 8h 4h 8h 4h 8h 4h 8h 4h 8h 12h 16h 20h 12h 16h 20h 12h 16h 20h 12h 16h 20h 12h 16h 20h Tue 0h Wed 0h Fri 0h Mon 0h Hourly intervals for week starting 2 March 2008 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 14. DEALING WITH BATCH • Spread what you can into white space • Tune or squeeze the rest • The difficult bit is convincing the operations area •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 15. MODELING CHANGES TO BACKUPS BATMD load shown 8500 Other Batch Medium over both CECs 8000 Consumption of ADRDSSU backups 7500 7000 6500 Batch all kicks off at 19h00 6000 Backups 5500 around 20% of 5000 4500 batch 4000 MIPS 3500 3000 2500 2000 1500 1000 500 0 Thu 0h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h Tue 0h Fri 0h Wed 0h Mon 0h Hourly intervals for week starting 30 March 2009 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 16. MODELING CHANGES TO BACKUPS 8500 Original position 8000 Other Batch Medium (modeled) 7500 Modeled consumption of rescheduled backups 80% of backups now 7000 delayed until 02h00 6500 6000 Saving in hour 5500 5000 4500 4000 MIPS 3500 3000 2500 2000 1500 1000 500 0 Thu 0h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h Tue 0h Fri 0h Wed 0h Mon 0h Hourly intervals for week starting 30 March 2009 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 17. MODELING CHANGES TO BACKUPS 8500 Original position 8000 Other Batch Medium (modeled) 7500 Modeled rescheduled & uncompressed backups Assume 50% 7000 compression removed 6500 6000 5500 5000 4500 4000 MIPS 3500 3000 2500 2000 1500 1000 500 0 Thu 0h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h Tue 0h Fri 0h Wed 0h Mon 0h Hourly intervals for week starting 30 March 2009 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 18. MODELING CHANGES TO BACKUPS 8500 Original position 8000 Unsqueezed Batch Medium (modeled) 7500 Squeezed Batch Medium (modeled) BATMD squeezed to Modeled rescheduled & uncompressed backups 7000 5000 MIPS 6500 6000 5500 5000 4500 4000 MIPS 3500 3000 2500 2000 1500 1000 500 0 Thu 0h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h 4h 8h 12h 16h 20h Tue 0h Fri 0h Wed 0h Mon 0h Hourly intervals for week starting 30 March 2009 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 19. SYSTEM SETUP - MONITORS & TRACES • The way to reduce CICS Trace overhead is to turn off the all options you can • Default for standard trace in CICS is level 1 for all components (STNTRxx = 1 in SIT) • Trace points from most of these domains are of no use in debugging application problems • For most sites application problems can be diagnosed with trace running for application (AP), EXEC Interface (EI), and File Control (FC) domains • TMONCICS turns on some trace points (EI, DS, MN, and ISC). •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 20. SYSTEM SETUP - MONITORS & TRACES • The only thing you can go with monitors is try to switch off as many options as you can get away with TXSTGCOL means that TMON645 < 4720 .02 . TMON912 > 968 .17 . TMON GLOBAL TRACE TMON924 > 5128 .21 . is monitoring transaction TM2N950 > 11288 .19 . TM2N952 > 24032 .02 . storage utilisation. Most TM2N958 > 8032 .10 . sites won’t be interested TM2N960 TM2N960 > 13808 3.00 .*** in this, and the default is TM2N969 > 2336 .08 . TM2N973 > 4032 .36 . ON TX2EIIN TXEIIN > 7930 2.28 .** TX2EIOUT TXEIOUT 68038 1.20 .* TX2EIOUT TXSTGCOL 856 2.17 .** TX2EIOUT TXUCFAT 2288 .34 . ----- TX2EIOUT TOTALS > 71182 3.71 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 21. INVESTIGATING HIGH DB2 ACTIVITY IN CICS • Use your sampling tool to identify high burning DBRMs ** SQL CPU USAGE SUMMARY ** SQL SQL STMT EXECUTION TIME/COUNT % CPU TIME TYPE NAME CNT AVG-TIME TOTAL …… DBRM NK1K600T 30,306 .0000 09:06:59 21/01/2005 .11 . DBRM NK1NT0AT 194 .0035 16:31:23 13/04/2005 .12 . DBRM NK1NT0IT 45 .0043 08:36:15 06/08/2007 .05 . DBRM NK1NT0NT 18,860 .0027 07:16:23 21/07/2008 8.94 .*************************++++ DBRM NK1NT0OT 17,312 .0027 13:57:17 15/04/2005 11.00 .****************************++++++ DBRM NK1NT00T 0 .0000 08:36:59 06/08/2007 .02 . DBRM NK1NT03T 131 .0163 15:50:43 09/12/2004 .58 .* DBRM NK1NT14T 2 .4020 15:07:11 02/06/2006 .22 . DBRM NK1U30ET 7,147 .0001 15:39:28 27/02/2008 .06 . … • Use DB2 Explain to review, and if possible improve the access path •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 22. FOCUS ON APPLICATION CODE • Cover simple code changes • Look for tight loops, definition of index variables MODULE SECTION PROCEDURE/FUNCTION STARTING PROCEDURE CPU TIME NAME NAME NAME LOCATION LENGTH TOTAL .LELIB CEEBINIT CEEVGTSI GET A STACK INCREMENT 13.36 .COBLIB IGZCPAC IGZCFPC COBOL LIBRARY SUBROUTINE 1.97 .COBLIB IGZCPAC IGZCNMV NUMVAL/NUMVAL-C FUNCTION 1.93 .COBLIB IGZCPAC IGZCFCC DYNAMIC CALL MANAGER 1.88 .COBLIB IGZCPAC IGZCFPCX COBOL LIBRARY SUBROUTINE 1.74 RK3DJAMV RK3DJAMV 0292A8 2 1.10 RKJB20 RKJB20 0806EC 2 1.10 RKJB20 RKJB20 0807BC 2 1.06 RK3DJAMV RK3DJAMV 0292D0 2 1.01 RKJB20 RKJB20 019BEC 2 .92 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 23. FOCUS ON APPLICATION CODE • Offset points to area of code doing basic index function IF WSOL-NUCTE-SR > 900 AND < 961 MOVE WPRO TO WPRO5 END-IF. IF ( WSOL-NUCTE-SR > 200 AND < 225 ) OR WSOL-NUCTE-SR = 298 MOVE WSCL TO WSCL5 END-IF. • Variables are declared as characters 20 WSOL-NUCTE PIC 9(10) VALUE 0. 20 FILLER REDEFINES WSOL-NUCTE. 25 WSOL-NUCTE-SR PIC 9(3). 25 WSOL-NUCTE-RG PIC 9(7). •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 24. LINGERING RACF OVERHEAD • RACF overhead is still high • Noticed odd user sign-on behaviour in CICS logs • Investigated further with Type 80 RACF records • Spotted sign-on trend • Piece together user behavior •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 25. ACCESSING CICS FROM BRANCH DESKTOP Vtam / GOR Many AORs TELNET Application on desktop TOR GOR CTGs WOR Websphere on AIX •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 26. LOGON VIA GOR Vtam / GOR Many AORs Connects to TELNET mainframe Autoinstall Signon IPADDR : Port NETNAME TERMID ACEE TXN1 No TERMID Ship Signon Single TERMID business transaction TERMID ACEE TXN1 TXNn TXNn CESF Breaks Delete Signoff connection •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 27. LOGON VIA GOR Vtam / GOR Many AORs TELNET TERMID ACEE IPADDR : New = NEW = NEW = NEW Port number NETNAME TERMID ACEE OVERHEAD TXN1 No TERMID Next Ship Business TERMID Transaction = NEW = NEW TXN1 TERMID ACEE TXNn TXNn CESF Breaks Delete Signoff connection •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 28. IDEAL LOGON VIA GOR Vtam / GOR Many AORs TELNET First NETNAME TERMID ACEE business transaction TXN1 Ship TERMID TERMID ACEE TXN1 TXNn TXNn Single TXNn Business Day TXNn Subsequent TXNn business TXNn transactions TXNn TXNn TXNn •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 29. MEASURING RESULTS ON A REGION BASIS • A key success point for early opportunities was building confidence within the technical and operational areas • Opportunies were measured at a very granular level to prove the effectiveness •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 30. MEASURING SYSTEM WIDE RESULTS Transactions per hour •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 31. SUMMARY OF OPPORTUNITIES RAISED Number of Savings to 4HRA Savings to 4HRA Opportunities at midday in batch peak Move 6 200 MIPS 890 MIPS workloads System 43 1,940 MIPS 210 MIPS settings Application 29 244 MIPS 330 MIPS changes Access path 34 1,010 MIPS 110 MIPS improvements •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 32. RESULTS OVER THE PROJECT LIFETIME CICS CPU hours consumed in the month Millions of transactions in the month Start of tuning implementations + 35% - 13% •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 33. CONTINUE TO FOCUS WHERE IT MATTERS Less batch Peak load around 800 Large midday Batch 7000 TSO throughout MIPS lower running 38% batch suite Online 6500 the day more transactions now at 13h00 STC System 6000 Uncaptured 5500 4HRA 5000 4500 4000 3500 MIPS 3000 2500 2000 1500 1000 500 0 08h00 08h30 09h00 09h30 10h00 10h30 11h00 11h30 12h00 12h30 13h00 13h30 14h00 15 minute intervals for Tuesday 2 March 2010 •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 34. CURRENT POSITION •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 35. SUMMARY OF OPPORTUNITIES RAISED Number of Savings to 4HRA Savings to 4HRA Opportunities at midday in batch peak Move 6 200 MIPS 890 MIPS workloads System 43 1,940 MIPS 210 MIPS settings Application 29 244 MIPS 330 MIPS changes Access path 34 1,010 MIPS 110 MIPS improvements •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 36. A KEY TO SUCCESS - AREAS REVIEWED We covered everything that runs on Other vendors concentrate on application the system code base only •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 37. SUMMARY • You need to establish the business reason for tuning • Focus where it will achieve the impact you want • Start with big initiatives to gain momentum • Take an iterative approach • Track & publish results • Keep reminding yourself what the goal is •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 38. WHAT BENEFIT WILL CPT ENGAGEMENT DELIVER? • Your organization will make more profit • Your organization will be able to grow without acquiring more hardware • The service that you provide to your customers will be in more scalable, reliable and cost effective way •Proibida cópia ou divulgação sem permissão escrita do CMG Brasil.
- 39. Q&A Gerry Tuddenham - CPT Global gtuddenham@cptglobal.com www.CPTglobal.com