This presentation titled "Achieving highest availability and best performance with CICS TG on z/OS HA Architecture – User Experience" was presented at IMPACT 2013.
The Ultimate Guide to Choosing WordPress Pros and Cons
Ibm session tac 2104 - ctg presentation for impact 2013 final
1. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Achieving highest availability and best performance
with CICS TG on z/OS HA Architecture – User Experience
Elena Nanos
Lead Systems Architect
Health Care Service Corporation
IBM Certified Infrastructure Systems Architect
TOGAF Certified Architect - The Open Group Architecture Framework V8.1.1
IBM Certified Solution Expert - Dynamic Infrastructure Technical Leader
IBM Certified System Administrator - WebSphere Application Server ND V7.0
IBM Certified Solution Expert - CICS Web Enablement
IBM Certified System Specialist - WebSphere MQSeries
Email – ELENA_NANOS@BCBSIL.COM
Session TAC-2104
1
2. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
2
CICS TG Product Positioning
Presented by Catherine Moxey, IBM STSM, Email: moxeyc@uk.ibm.com
3. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Health Care Service Corporation (HCSC) is the fourth largest health
insurance company in the nation.
Largest customer-owned health insurer in the U.S., founded in 1936,
now with more than 13 million members, HCSC operates Blue Cross
and Blue Shield Plans in Illinois, New Mexico, Oklahoma and Texas.
We're greater than 18,000 employees strong with 60 local offices and
state-of-the-art technology, including two Tier IV data centers — the
industry's highest reliability level – that provide the speed and data
security to meet our customers' current and future business needs.
Who Are We? About Health Care Service Corporation
3
4. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Common Architectures and Standard Used with CICS
Connectivity
option
Required
middleware
Main capabilities Usage notes
Web services CICS-only solution • Inbound and outbound
• Low coupling
• Synchronous (HTTP)
• Asynchronous (WMQ)
• QoS based on transport type
• Support for some WS-* standards
Should be first consideration for service
enabling CICS applications, particularly
when you need to support multiple service
requester types or need bi-directional
support
JCA with
CICS TG
CICS TG
(normally WebSphere
Application Server)
• Inbound to CICS
• Medium coupling
• Synchronous
• High qualities of service (QoS)
Most appropriate solution when service
requester is JEE component and when high
QoS required
JCA with
WOLA
WebSphere
Application Server
for z/OS
• Inbound and outbound
• Tight coupling
• Synchronous
• High QoS
Particularly useful for JCA access to and
from CICS, and for very high throughput
and performance requirements.
CICS web
support
CICS-only solution • Inbound and outbound
• Medium coupling
• Synchronous
• Medium QoS
Use with web services, RESTful services,
and Atom feeds, or when remote
client/server only supports HTTP
WebSphere
MQ for z/OS
CICS-only solution • Inbound and outbound
• Medium coupling
• Asynchronous, with almost-
synchronous capabilities
• Assured delivery
Exploit WMQ for basic messaging and
flowing web services
4
5. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
HCSC CICS TG Topology
• Topology 1 was our main WASd/CICS TG configuration.
• Topology 2 was proposed configuration moving CICS TG off distributed platform into
z/OS, using CICS TG on z/OS stand-alone (Remote mode). This is our current
topology.
• Topology 3 is also used with WAS on z/OS and CICS TG configuration, using CICS TG
in Local mode. Note: Local mode on z/OS can use either EXCI (which is MRO cross memory), or
IPIC (which is still sockets based, but will be optimized in TCPIP to Fast Local Sockets - effectively cross
memory - when co-located within the same LPAR as CICS).
5
6. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Current Configuration
6
Previous HCSC Distributed Configuration
7. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
7
HCSC CICS TG HA Architecture
8. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V7.0
WLM Health reporting for storm drain avoidance that required use of the
SHAREPORTWLM keyword (a variant of SHAREPORT)
WLM Feedback support – Storm drain avoidance
Feedback of CICS region status to IP load balancing mechanism via WLM
Allows more intelligent autonomic decisions to be performed by IP load balancers
Pushes CICS health out to the IP load balancer
CICS TG V7.2
Sysplex-wide support for XA with HA which used Sysplex Distributor.
Port-sharing across Sysplex but this could only be combined with DSS for 1-phase commit
transactions and synconreturn.
Support for XA transactions in High Availability topologies using TCPIP load balancing in a z/OS
Sysplex
All CICS TG versions have ability to use SHAREPORT on the PORT statement in the TCP/IP profile
Enables a group of cloned servers to listen on the same port
New connections are distributed across available servers using a weighted round-robin
8
High Availability and XA Transactions
History of Enhancements
9. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V8.0
Provides additional High Availability capabilities, allowing dynamic selection of CICS server
within the Gateway daemon
Provides a working, configurable CICS request exit (including source) for performing
failover or round robin
Support for cross-sysplex port sharing XA/2PC requests in combination with DSS
CICS TG V8.1
Introduced the "out-of-the-box" configurable DSSPOLICY as an alternative to CREXIT, for
those wanting a basic roundrobin or failover implementation. CREXIT still provides maximum
flexibility when it comes to custom algorithms.
CICS TG V9.0
Routing or filtering by client IP address
Full request details for XA transactions
New Request Monitoring exit point “RequestDetails”
Invoked just before calling CICS, after Dynamic
Server Selection
9
High Availability and XA Transactions
History of Enhancements
10. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Health reporting is effective only in TCP/IP load balancing topologies, with the CICS Transaction
Gateway running in remote mode.
At intervals specified by the Health interval, the Gateway daemon monitors error codes like
ECI_ERR_NO_CICS, ECI_ERR_RESOURCE_SHORTAGE, ECI_ERR_SYSTEM_ERROR
to determine the health of communications with the CICS subsystem. The load balancer can then
use this health to set priorities when creating new incoming IP socket connections to cloned
Gateway daemons.
MVS console command can be used to display Gateway health status –
F <JOB>,APPL=STATS,GS=GD_CHEALTH
10
WLM Feedback
11. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
TCP/IP load balancing
• TCP/IP load balancing provides high availability of the Gateway daemon by distributing
connections from clients across multiple Gateway daemons. The efficiency of load balancing
can also be increased by taking into account CICS server status provided by dynamic feedback to
the IBM z/OS Workload Manager (WLM).
Sysplex Distributor
• Sysplex Distributor provides TCP/IP load balancing across multiple LPARs.
Port sharing
• TCP/IP port sharing enables requests for work to be shared between several
Gateway daemons through a single TCP/IP port. Port sharing provides TCP/IP
load balancing in a single LPAR. When connections are established between a Client application
and a Gateway daemon, the TCP/IP port sharing component of the z/OS TCP/IP. Communications
subsystem distributes requests across multiple Gateway daemons. When the connection is
established, subsequent requests from the Client application continue to use the same connection.
Health monitoring
• A TCP/IP load balancer that is allocating a connection to the Gateway daemon detects whether or
not a CICS server is available. The Gateway daemon reports the health of its CICS server
connections to the TCP/IP load balancer.
Support Pack CH51 – CICS TG Simple Request Monitor Utility
Support Pack CA1T – CICS TG High Availability User Exits
MICS report using SMF 111 records
CICS Explorer
11
Technology Utilized in HCSC CICS TG HA Architecture
12. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Two different workload balancing polices (round robin or fail over) are supplied – both of
which use server rules from the HA configuration file to remap and retry failed requests.
For example, the following server rule defines the server alias SERVER1 as mapping
to 4 actual CICS servers CICSA, CICSB, CICSC & CICSD:
SERVER1=CICSA,CICSB,CICSC,CICSD
The round robin policy will distribute requests in a round robin fashion, so that
requests are evenly distributed across servers CICSA, CICSB, CICSC & CICSD.
The fail over policy will distribute requests in a hierarchical order so that all requests
are sent to CICSA, if this is unavailable requests will be sent to CICSB, if CICSB is
unavailable requests will be sent to CICSC, and if CICSC is unavailable requests will
be sent to CICSD.
http://www-01.ibm.com/support/docview.wss?uid=swg24021484
12
SupportPack CA1T – CICS TG High Availability User Exits
13. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Three different CICS request exit implementations are provided as follows:
1. FailOver – uses a list of CICS servers to define how server names are remapped.
Requests will be retried using a static list of servers, each server being used in turn from
the list until the list is exhausted or the request succeeds.
2. RoundRobin – distributes ECI requests around a list of CICS servers in round-robin
fashion. Requests will be distributed evenly across the servers, and retried until the
request succeeds or all serves have been used.
3. RequestValidation – validates the transaction ID, program and user ID parameters in
the ECI request against a defined list. No server name remapping is performed.
Latest enhancements
CA1T V3.0 support pac has been recently updated and gives ability to set new
environment variable CTG_HAVALIDATION=EXCLUSIVE, then any request
which matches one of the USERID, TRANSID and PROGRAM validation rules will
be rejected with ECI_ERR_INVALID_CALL_TYPE (-14).
Note: Configuration is based on a flexible set of rules defined in a configuration file. On z/OS, the configuration file can reside either on HFS or
in an MVS dataset. Rules for workload balancing can be generic per Gateway, based on server aliases, or on the payload type (for example,
channel or commarea). Rules for request validation can be specified based on user IDs, transaction IDs, or program names, and operate as
either a white list (INCLUSIVE) or black list (EXCLUSIVE). All rules can be dynamically updated while the Gateway daemon is running. A
configurable retry interval is provided to improve the efficiency of request distribution when using workload balancing.
http://www-01.ibm.com/support/docview.wss?uid=swg24021484
ftp://public.dhe.ibm.com/software/htp/cics/support/supportpacs/individual/ca1t_v110.pdf 13
(cont.)
SupportPack CA1T V3.0 – CICS TG High Availability User Exits
15. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Set at CTGSTART_OPTS=
-j-Dcom.ibm.CTG.samples.requestexit.out=/usr/cicstg/CH51LOGS/CTGname01.log
-j-Dcom.ibm.CTG.samples.requestexit.log.number=5 -j-Dcom.
ibm.CTG.samples.haexit.out=//DD:HALOG -j-Dcom.ibm.CTG.samples.haexit.ca1t=1
-statsport 3101 -applid=CTGname -j-Xms256M -j-Xmx512M - j-
Xdump:system:dsn=yourprefix.CTGname.JAVA.DUMP
Logging levels available – set at -j-Dcom.ibm.CTG.samples.haexit.ca1t
15
Support Pack CH51: CICS TG
Simple Request Monitor Utility Setup
Can also issue console command. For example to enable level 3 logging issue the following
command - /F <JOB>,APPL=CREXIT,CMD=CA1T_3
16. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
This SupportPac allows integration test tools to intercept ECI calls to CICS
TG from Java™ applications before they are processed by CICS TG.
Allows integration test tools ( for example Rational Test Workbench / Rational
Integration Tester V8.0.1.1) to intercept External Call Interface (ECI) calls to
IBM CICS TG, originating from Java Standard Edition (Java SE) and Java
Enterprise Edition (JEE) applications. This, therefore, allows such applications
to be tested without requiring a running CICS TG or CICS Transaction
Server.
This SupportPac provides updated versions of the CICS TG V8.0 Java base classes and non-extended
architecture (non-XA) ECI resource adapter. These updated files support the use of a Gateway intercept plug-
in that can intercept application calls made through the Java Application Programming Interfaces (APIs).
Documentation for the APIs is also provided by this SupportPac.
When enabled, a Gateway intercept plug-in is notified before a connection to a Gateway daemon is opened or
closed, and before a request is sent to the Gateway daemon. Before a connection is opened, the plug-in can
change properties of the connection object, and can prevent the connection from opening. Before a request is sent
over a connection, the plug-in can change properties of the request object, and can cause the request to be
returned to the application without being sent. Before a connection is closed, the plug-in can prevent the
connection from closing. This allows a variety of scenarios to be simulated for application testing.
Can be downloaded from – http://www.ibm.com/support/docview.wss?uid=swg24033968
16
Support Pack CA5F: IBM CICS TG
'gateway' Intercept Plug-in Support
17. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
( set in Runtime env CICSCLI)
SECTION IPICSERVER = SYS1CIC1
description = IPIC connection to CICS region #1
hostname = your.host.net
port = 3101
tcpkeepalive = Y
srvidletimeout = 60
srvretryinterval = 0
connecttimeout = 10
sendsessions = 200
ENDSECTION
SECTION IPICSERVER = SYS1CIC2
description = IPIC connection to CICS region #2
hostname = your.host.net
port = 3102
tcpkeepalive = Y
srvidletimeout = 60
srvretryinterval = 0
connecttimeout = 10
sendsessions = 200
ENDSECTION
SECTION IPICSERVER = SYS2CIC3
description = IPIC connection to CICS region #3
hostname = your.host.net
port = 3103
tcpkeepalive = Y
srvidletimeout = 60
srvretryinterval = 0
connecttimeout = 10
sendsessions = 200
ENDSECTION
SECTION IPICSERVER = SYS2CIC4
description = IPIC connection to CICS region #4
hostname = your.host.net
port = 3104
tcpkeepalive = Y
srvidletimeout = 60
srvretryinterval = 0
connecttimeout = 10
sendsessions = 200
ENDSECTION
On the Gateway you can set - CONNECTTIMEOUT, which is a numeric value between 0 and 3600, specifying the maximum
time in seconds that establishing a connection is allowed to take. The default value of 0 means that no limit is set. For
more details see - http://publib.boulder.ibm.com/infocenter/txformp/v7r1/index.jsp?topic=/com.ibm.cics.tx.doc/tasks/t_config_ipic.html
17
Sample CICS TG CONFIG File
18. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V8.1 and WebSphere V6, V7 & V8 - http://www-01.ibm.com/support/docview.wss?uid=swg21577454
Runtime Environments for Java and JEE Applications
18
CICS TG Compatibility with WebSphere
19. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS TG V8.0 or prior has two CICS Resource Adapters – XA and non-XA.
CICS TG V8.1 has support for Java Enterprise Edition 6
The CICS connectors included with CICS TG V8.1 exploit JEE 6 / JCA 1.6
JEE 6 certified application servers (e.g. WAS 8.0) benefit from:
A single ECI resource adapter (rather than two) including support for both XA (two-phase
commit) and non-XA transactions
A more flexible approach to setting the transactional capabilities of the CICS ECI resource
adapter
Lower administrative and maintenance overheads
Resource adapters compatible with JEE 5 (or earlier) application servers are available
use with CICS TG V8.1 in remote mode only from Support Pac CC03
19
CICS JCA Resource Adapters
20. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
The CICS JCA resource adapters are provided, in order to support
connectivity from JEE application servers to CICS using the facilities
of the CICS Transaction Gateway.
The latest service levels of the CICS Transaction Gateway Java
Enterprise Edition Connector Architecture (JCA) resource adapters for
CICS TG are available for download for deployment into supported
Java Enterprise Edition (JEE) application servers. These resource adapters can be
used to provide JCA connectivity into CICS using the facilities of a licensed CICS TG installation, and are
designed around the JCA specification as supported by JEE application servers such as WebSphere Application
Server and Oracle Weblogic Server.
This Support Pack includes the CICS JCA resource adapters from the
following CICS Transaction Gateway for Multiplatforms releases:
V7.0.1.1, V7.1.0.4, V7.2.0.4, V8.0.0.4, V8.1.0.1, V9.0.0.0
http://www-1.ibm.com/support/docview.wss?uid=swg24008817&myns=swgother&mynp=OCSSGMJ2&mync=E
20
Support Pack CC03: CICS JCA Resource
Adapters for Use With JEE Application Servers
21. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
J2C connection factories > CICS Connection Factory > Connection pools
Maximum connections
Specifies the maximum number of physical connections that you can create in this pool. These are
the physical connections to the backend resource. Once this number is reached, no new physical
connections are created and the requester waits until a physical connection that is currently in use
returns to the pool, or a onnectionWaitTimeoutException is thrown. Ensure that there are no waits
for connections and that CICS can handle concurrent volume set here.
Minimum connections
Specifies the minimum number of physical connections to maintain. If the size of the connection pool
is at or below the minimum connection pool size, the Unused Timeout thread does not discard
physical connections. Best to set to zero, to allow stale connections to be discarded.
Reap time
Specifies the interval, in seconds, between runs of the pool maintenance thread. Do not set too
low, as it will increase the overhead.
Unused timeout
Specifies the interval in seconds after which an unused or idle connection is discarded. Set the
Unused Timeout value higher than the Reap Timeout value for optimal performance.
Aged timeout
Specifies the interval in seconds before a physical connection is discarded. Setting Aged Timeout
to 0 supports active physical connections remaining in the pool indefinitely. Do not set it at
zero to avoid connection retries on stale connections.
21
Connection Pool Tuning in WAS
22. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
The graph shows the difference in processor usage cost between EXCI and IPIC.
The CICS address space costs were less when using EXCI, because of the efficiency of using
cross-memory MRO for communication between the gateway address space and the CICS
address space.
Athough the overall cost to use IPIC was greater, the amount of processing offloaded to a
zAAP was also greater, which made the use of general-purpose CPs in the gateway address
space less critical for IPIC.
22
EXCI vs. IPIC CPU Usage
23. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Data from redpaper IBM CICS Performance Series: A Processor Usage Study of Ways into CICS
at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4906.pdf
The graph shows the total microseconds of processor usage per transaction to get
32 KB of data into CICS and back to the client using the access methods tested
Environment
Hardware
The following hardware was used:
• IBM zEnterprise® 196 (Z196) 2817-779 model M80
• LPAR with three dedicated central processors (CPs)
and one dedicated IBM System z®
• (zAAP) on zIIP)
• DASD DS8800
• Internal coupling facility peer (ICP)
• 1 GB OSA-Express3
• Crypto Express3 coprocessor
• CP Assist for Cryptographic Function feature (enabled)
Software
The following software was used:
• IBM z/OS® Version 1, Release 13
• CICS Transaction Server Version 4, Release 2
• IBM WebSphere MQ Version 7, Release 1
• Communication Server IP Sockets for CICS
• CICS Transaction Gateway Version 8, Release 1
23
CPU Cost Getting CICS Data
24. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
24
POC Testing – High zAAP Offload Rate
25. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
25
CA Wily Introscope CICS TG Metrics
26. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
26
Wily Introscope ECI Pool Size Metrics
27. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Compare to week prior
27
Wily Introscope Response Time Metrics
28. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
28
Wily Introscope CICS TG/JCA Metrics
29. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS Transaction Gateway - Activity Summary
_____________________________________________
0STTG0001 Printed at 11:29:15 3/01/2013 Data from 08:00:00 2/21/2013 to 08:00:00 2/21/2013 Page 1
CICS TG GATEWAY 02/21/13 REPORT
Gateway ID Start time Up time Number of Number of Peak Peak TPS time Connect Worker
Intervals requests TPS Timeouts Timeouts
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 7days 17.34.09 95 1643676 13 2013-02-21-17.15.00 0 0
ZOSCGRP0.CICS TG2P004 2013-02-14 06.12 7days 17.32.46 95 1656635 15 2013-02-21-18.00.00 0 0
ZOSCGRP0.CICS TG1P001 2013-02-14 06.14 7days 17.30.41 95 1686357 25 2013-02-21-18.00.00 0 0
ZOSCGRP0.CICS TG1P002 2013-02-14 06.09 7days 17.35.47 95 1695259 26 2013-02-21-17.15.00 0 0
Gateway ID Start time Collection time Region Java heap Clients Workers EXCI pipes IPIC sessions
Size/Max Size/Max Peak/Max Peak/Max Num/Max Num/Avail
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 2013-02-21-16.00.00 875M/1000M 71M/512M 19/500 3/800 0/0 0/400
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 2013-02-21-16.15.00 875M/1000M 74M/512M 28/500 6/800 0/0 3/400
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 2013-02-21-16.30.00 875M/1000M 76M/512M 27/500 7/800 0/0 0/400
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 2013-02-21-16.45.00 875M/1000M 76M/512M 23/500 6/800 0/0 0/400
CICS Performance Analyzer V5.1 now includes two CICS TG batch reports, "Activity Summary", and
"Usage and Capacity" which can be run against SMF 111 records from CICS TG for z/OS.
29
CICS Performance Analyzer for z/OS - Sample Reports
30. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CICS Transaction Gateway plug-in for the CICS Explorer V2.0.0.0 is now available for CICS Explorer V5.1, easily
installed through the common V5.1 update site which is pre-configured in the CICS Explorer "Install new software"
facility.
30
CICS Transaction Gateway Plug-in For The CICS Explorer
31. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
CSSYS1CIC1_SIPADDR=yourSYS1.network.net (CICS server
TCP/IP address)
CSSYS1CIC1_SIPPORT=3168 (CICS server TCP/IP port)
CSSYS1CIC1_LALLREQ=4549989 (Number of requests
processed)
CSSYS1CIC1_CWAITING=-2 (Number of requests waiting on a
response)
CSSYS1CIC1_SSESSMAX=200 (Number of requested IPIC
sessions)
CSSYS1CIC1_CSESSMAX=200 (Number of negotiated IPIC
sessions)
CSSYS1CIC1_CSESSCURR=0 (Number of IPIC sessions in use)
CSSYS1CIC1_LCOMMSFAIL=49 (Number of CICS
communication failures)
CSSYS1CIC1_LCONNFAIL=4484 (Number of connection failures)
CSSYS1CIC1_LLOSTCONN=32 (Number of lost connections)
CSSYS1CIC1_LSESSFAIL=0 (Number of IPIC session failures)
CSSYS1CIC1_LIDLETIMEOUT=0 (Number of timed out
connections)
CSSYS1CIC1_LREQDATA=28649890161 (Amount of CICS
request data)
CSSYS1CIC1_LRESPDATA=29043330152 (Amount of CICS
response data)
CSSYS1CIC1_LAVRESP=64 (Average CICS response time)
CSSYS1CIC1_CAPPLIDQ=ILA (APPLID qualifier of connected
CICS)
CSSYS1CIC1_CAPPLID=YourCICSApplid (APPLID of connected
CICS)
CSSYS1CIC1_ICOMMSFAIL=0 (Number of CICS communication
failures)
CSSYS1CIC1_ICONNFAIL=0 (Number of connection failures)
GD_IAVRESP=42 (Average Gateway daemon response time)
GD_LAVRESPIO=109 (Average Gateway daemon response time with
I/O)
GD_IAVRESPIO=98 (Average Gateway daemon response time with
I/O)
GD_SNAME=CICS TGPRD1 (Gateway daemon name)
GD_CHEALTH=100 (Gateway daemon health)
GD_LHAEXIT=4687171 (Number of CICS request exit calls)
GD_IHAEXIT=642 (Number of CICS request exit calls)
GD_SDFLTSRV= (Default server name)
GD_LRUNTIME=1758839 (Gateway daemon running time)
GD_SSTATINT=001500 (Length of the statistics interval HHMMSS)
GD_SSTATEOD=000000 (Logical End Of Day time HHMMSS)
GD_CNEXTRESET=150000 (End of interval time HHMMSS)
GD_IRUNTIME=199 (Interval running time)
WT - Worker thread
WT_SMAX=800 (Maximum number of worker threads)
WT_SINIT=200 (Initial number of worker threads)
WT_CCURR=200 (Current number of worker threads)
WT_CALLOC=0 (Currently allocated worker threads)
WT_LTIMEOUTS=0 (Number of times worker time out reached)
WT_ITIMEOUTS=0 (Number of times worker time out reached)
With CICS TG on z/OS stand alone we can use MVS console commands to display statistical
information about the CICS TG, or can obtain statistics using the Statistics API interface.
/F CICS TGname,APPL=STATS,GS
31
MVS Console Commands
32. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
If you would like to analyze CICS TG SMF 111 records, 1st you would need to extract these
records in a separate dataset by the IFASMFDP utility. Here is a sample JCL -
//* COPY SMF RECORDS FROM DAILY TAPES
//* THIS PARTICULAR JOB COPIES SMF 111 RECORDS FOR CTG
//* BEFORE RUNNING, SET DATE AND LPAR
// SET SMFHLQ=TSMF <<== HLQ OF SMF
// SET SMFTYPE=CTG
// SET LPAR=A <<== LPAR <- set LPAR name
// SET DATE=D032013 <- set date
//STEP2 EXEC PGM=IFASMFDP
//SYSPRINT DD SYSOUT=*
//SMFIN DD DISP=SHR,DSN=&SMFHLQ..SMF.&LPAR.NOSSDAY.&DATE
//DUMPOUT DD DSN=&SMFHLQ..SMF.&LPAR..&SMFTYPE..&DATE,
// DISP=(NEW,CATLG),UNIT=3490E,RETPD=9
INDD(SMFIN,OPTIONS(DUMP))
OUTDD(DUMPOUT,TYPE(111))
You can collect and report on the following statistics -
CICS Server Instance (CSx) SMF statistics for EXCI
CICS Server Instance (CSx) SMF statistics for IPIC
Gateway daemon (GD) statistics SMF data
Protocol handler (PH) statistics SMF data
System environment (SE) statistics SMF data
Worker thread (WT) statistics SMF data
For more details please see 'CICS Transaction Gateway Version 8 or 9 z/OS Administration' – section 'Displaying statistics‘.
32
CICS TG SMF 111 Records ReportingCICS TG SMF 111 Records Reporting
34. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
34
Other Monitoring Tools Available
IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0
35. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
35
Performance Tuning
Helps to prevent CICS becoming overloaded or underutilized
Allows for optimization of workflow through CICS TG and onto CICS
Alerts users to changes in behavior when abnormal workloads hit CICS TG
Provides users with an account of transactions per minute and CPU
consumption
Availability
Alerts users to communication problems between CICS TG and CICS
Managablity
Helps to determine where to control resources in CICS TG or CICS
Monitoring
Transaction Monitoring in Gateway daemon and WebShere Application
Server on z/OS
Transaction Response Time Analysis, Payload and Wiresize per
transaction
End To End Response Time Analysis
Problem Determination
Identifies the bottlenecks within CICS TG daemon
Identifies Abended transactions
Correlates all application Flows within a Transaction
Provides Transaction Flow Trace
Capacity Planning
Helps to plan for workload growth, perform trend analysis, detect system
anomalies and to identify performance bottlenecks using the historical data
via the Tivoli Data Warehouse repository
Total Cost of Ownership
Opens up the “black-box” of CICS TG thread management and helps to
save precious resources in both CICS TG and CICS
Other Monitoring Tools Available
IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0
36. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Issue
We implemented new Application using CICS TG on z/OS, which had invalid Userid set on
Connection Factory. This caused ID to get suspended and Application was redriving error routine
to restablish CICS TG connection, after each failure. This caused connections not getting
released, causing all connections to be in use across all 4 CICS TGs, resulting in CICS TG -16
return (ECI_ERR_RESOURCE_SHORTAGE).
Solution
Setup automation to issue CICS TG console command –
/F CICS TGname,APPL=STATS,GS every 5 minutes and send alert if field CS_CSESSCURR
(total # of IPIC sessions in use with all CICS servers) is at 80% or higher from CS_CSESSMAX (#
of IPIC sessions negotiated with CICS servers). Can also have automation (OPS/MVS for
example) check the values of CM_CALLOC and WT_ALLOC and alert if the threshold set
has been passed.
CICS TG on z/OS V8.0
PM38908: ECI_ERR_RESOURCE_SHORTAGE reported when sending extended mode ECI
requests over an IPIC connection to a CICS server
Error description
When an extended mode ECI request does not complete successfully (possibly due to
security reasons) before the CICS mirror transaction is attached, the IPIC session is not
freed.
http://www-01.ibm.com/support/docview.wss?uid=swg1PM38908
36
Issues To Watch Out For
37. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
GD_IAVRESP - The average time taken in milliseconds for the Gateway daemon to respond to API (ECI, ESI, EPI)
and XA requests from remote clients. Successful and failed requests are included. This value is inclusive of the CICS
response time, as provided by the corresponding CS_IAVRESP statistic.
GD_IAVRESPIO - The average time in milliseconds for the Gateway daemon to respond to API (ECI) and XA
requests from remote clients including network I/O time. Successful and failed requests are included. This value is
inclusive of the Gateway response time, as provided by the corresponding GD_IAVRESP statistic.
The latest versions of the CICS TG plug-in for CICS Explorer V1.1 for CICS Explorer V1.1 and V2.0 for CICS Explorer
V5.1 now shows Network Latency data. You can get lifetime average value for network latency, under the column
"GwyRespTimeIO", and is included in the pre-defined Column layout "Requests", of the Gateway daemons view.
37
CICS TG V8.1 New SMF Network Latency Data
38. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Limitations of 31-bit Gateway daemon
31-bit Gateway daemon, 375MB heap, REGION 0M (1632MB)
1350 Connection Mangers
999 Worker threads
1 IPIC connection with 999 sessions
32KB payload
64-bit Gateway daemon scalability
Note: Data above is a rough guide, your ‘milage’ may vary …
38
CICS TG Scalability
39. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
For best performance with 64-bit HWM, use compressed references
(-j-Xcompressedrefs)
Migration to 64-bit can reduce number of address spaces
Fully exploit IPIC and Dynamic Server Selection after migrating from the
constraint of 250 EXCI pipes
Rule of thumb for Gateway MEMLIMT
MEMLIMIT should be at least
128MB (core) + Heap size + (#CM+WT threads) * 3MB
39
Tuning Suggestions
40. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
GA December 14, 2012
Please do not miss IMPACT session TAC-3022 ‘Connecting to CICS with CICS
Transaction Gateway V9’ scheduled for 3:45pm on May 1st, presented by Andrew Smithson.
40
CICS Transaction Gateway V9.0
41. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Routing or filtering by client IP address
CICS Request Exit now provides the TCPIP address of the application
originating the request
InetAddress map object “ClientLocation”
Allows routing decisions based upon client location (e.g. subnet)
Allows rejection of requests from unauthorised locations
Full request details for XA transactions
CICS Request Exit allows custom Dynamic Server Selection
CICS TG V8 has limited request meta-data for XA transactions
CICS TG V9.0 provides all available request meta-data for both XA and non-
XA ECI requests
Allows customized routing logic for High Availability solutions with full
distributed two-phase commit
New Request Monitoring exit point “RequestDetails”
Invoked just before calling CICS, after Dynamic Server Selection
Allows clear visibility of target CICS server for HA solutions
Very useful for time outs, or hung request scenarios
41
CICS TG V9 HA Enhancements
42. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Cross component trace (XCT) contexts
Available on requests originating from WebSphere
Application Server V8.5, with XCT support enabled
Enabled/disabled
• Dynamically through the WAS admin console run time options
• Statically through the WAS admin console configuration
Uses High Performance Extensible Logging (HPEL)
XCT contexts
XCT context Begin and End demarcate component boundaries
• Contexts are hierarchical
A thread of execution can have upto 3 XCT contexts at any one time:
• Root – initial context (Request ID) of the component at the point of entry
• Parent – context of the calling component
• Current – context of the current component
42
CICS TG V9 Cross Component Trace
43. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
43
CICS TG V9 Enhancements To IPIC Connectivity
44. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
RFE ID Headline
17173 Asynchronous call support for ECI Version 2 applications
17258 CICS Gateway Support for OEL 5u4
19319 CICS TG Explorer Plugin - Logical grouping of Gateway daemons
21081 Transaction tracking from WAS into CICS through CICS TG
21082 Support for Gateway daemon running in a 64-bit JVM
21085 Additional exit point for Request Monitoring
21633 64-bit compile and run time support for ECI Version 2
21642 Remote Mode SSL for IPIC Connections
21828 Bi-directional language support
21871 Make program and transaction available to the CICS Request Exit for XA transactions
21872 ECI_TIMEOUT default value for IPIC connections
21875 64-bit C module compilation support for ECIv2 on AIX (IBM POWER) and Red Hat (x86)
22036 CICS Request Exit to provide client IP address
22049 Assembler DSECTs for mapping the SMF 111 records
22081 Sample SMF formatter CICS TGSMFRD support for mixed-release installations
22088 Information about channels in request monitoring exits
22108 Asserted identity support for Multiplatforms C applications making ECI requests
See details at http://www-01.ibm.com/support/docview.wss?uid=swg27037911
44
Customer Requirements Answered in CICS TG V9.0
45. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
45
Questions
46. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
We love your Feedback!
Don’t forget to submit your Impact session and speaker
feedback!
• Your feedback is very important to us – we use it to improve
next year’s conference
• Go to the Impact 2013 SmartSite (http://impactsmartsite/com):
– Use the session ID number to locate the session
– Click the “Take Survey” link
– Submit your feedback
46
48. Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.
48