Más contenido relacionado Similar a Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne (19) Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne1. IMS in a Sysplex Environment:
Challenges and Solutions
© 2014 IBM Corporation
Nick Griffin
World Wide IMS Software Group Sales Advisor
IBM Silicon Valley Lab
3. IMS Sysplex Manager Highlights
– Real-time management of the IMS Sysplex Environment
• Single system image thru local and aggregate view of data
• Simplified User Interface (TSO/ISPF)
• Structured displays of IMS resources and CF structures
• Global Type-1 command, OM Type-2 and IMS SPOC
• Basic z/OS performance information and SVC dump capture
• Statistics for CSL (OM, RM and SCI), IRLM, VSO, and CQS
• Dashboard with key system indicators and threshold monitoring
• Intercept System exceptions and generate Console alerts
© 2014 IBM Corporation
2014
2
• Intercept System exceptions and generate Console alerts
• Produce real-time IRLM Long Lock Report
• Browse, delete and recover messages on Shared Queues
• Delete RM resource structure entries
• Assign affinity for transactions in Shared Queues environment
• Protect against buffer overflow in Shared Queues environment
– Support IMS DB/TM, DBCTL, and DCCTL for IMS v11 and later
4. MVS1
MVS4MVS3
MVS2
CQS1 CQS2
SCI1 RM1 OM1 SCI2 RM2 OM2
IMS Sysplex Manager Sample Configuration
IMSSM
DC
IMSSM
DC
IMS1 IMS2
IMS Connect IMS Connect
FDR1
© 2014 IBM Corporation
2014
3
IMSSM
UIS
IMSSM
UIS
FDR1 FDR2
6. IMS Sysplex Review - Components
V13
V13
© 2014 IBM Corporation
2014
5
Source: IMS Redbook
Source: IMS Redbook (SG24-6908-00)
IRLM
IRLM Lock
V13
8. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
7
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
9. IMS Sysplex Challenges and Solutions
Shared Queues
–Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
8
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
10. Transaction affinity – The Challenges
Force local affinity for efficiency
Assign affinity to specific IMS due to resource availability
Reduce false-scheduling overhead
Reduce contention
© 2014 IBM Corporation
2014
9
Reduce contention
11. Sysplex Manager Tool - Transaction Affinity Highlights
Finer control of transaction scheduling
Non-invasive to existing definition and operation
– No omission of transaction definitions in sysgen
– No stopping of transactions
– No re-classing of dependent regions
– No operational impact for loss of a system
© 2014 IBM Corporation
2014
10
User defined affinity to route transaction messages
– Any IMS in the shared queues group
– Any subset of IMS systems
– Equal or weighted distribution
17. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
–CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
16
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
18. IMS Sysplex Review – IMS CF structures and log streams
Log streams
(CQS, OM,
© 2014 IBM Corporation
2014
17
Source: IMS Redbook (SG24-6908-00)
(CQS, OM,
RS)
19. IMS CF Structures and log streams – The Challenges
Growing use of Coupling Facility Structures
– Data Sharing, Shared Message Queues, Resource Manager, CQS log streams
– No single source for list of in use structures and details
Check log stream attributes
© 2014 IBM Corporation
2014
18
Monitor offload activities
20. SM Solution - IMS CF Structures and log streams
Real-time display of structure list
Statistics, Connections, Coupling Facility information
Structure rebuild and checkpoint stats
© 2014 IBM Corporation
2014
19
Log streams attributes and offload monitoring
28. CQS and System Logger
© 2014 IBM Corporation
2014
27
Source: IMS Redbook (SG24-6908-00)
29. SM Solution - CQS Log Stream info
© 2014 IBM Corporation
2014
28
31. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
–Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
30
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
33. Shared Queue Management – The Challenges
Visualize queue content
Browse, Delete, and Recover messages
View CQS processing counters
© 2014 IBM Corporation
2014
32
34. SM Solution - Shared Queue Management
Filter based on Destination name or ID
High level queue depth information, Drill down to levels of detail
Browse, Delete and Recover messages from Cold queue
© 2014 IBM Corporation
2014
33
CQS processing counters
40. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
–Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
39
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
41. Full Queue Protection Challenges
Local Queue
– Local buffer overflow can abend or stall IMS control region
Shared queue structures
– Full structures will stall IMS
© 2014 IBM Corporation
2014
40
42. SM Solution - Full Queue Protection
Local Queue
– Real-time local queue buffer counts and stats
– Reject new messages by employing the Queue Space Notification Exit
(DFSQSSP0)
– Automatic actions for warning, critical or Severe thresholds
– Identify programs using large number of buffers
© 2014 IBM Corporation
2014
41
Shared queue structures
– Alert structure utilization exceeding predefined thresholds
– Unable to reject new messages (a possible future enhancement)
43. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
–Long lock detection
© 2014 IBM Corporation
2014
42
–Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
45. Diagnosing Long Locks - The Challenges
DB Lockouts by applications holding IRLM locks for an inordinate amount of
time
– Could go unrecognized until it becomes critical
– Lack of supported tools to assist in recognition and identification of problem
– Manual intervention required to resolve
© 2014 IBM Corporation
2014
44
46. SM Solution - Data Sharing Long Locks
Automatic real-time recognition when IRLM detects
Information consolidated, analyzed for top blocker, and presented
Information recorded in exceptions file and sent to z/OS console
Messages can be sent to z/OS console using user exit so that
automated operations can resolve
© 2014 IBM Corporation
2014
45
automated operations can resolve
Problem quickly resolved with automation
49. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
48
– Long lock detection
–Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
50. Real-time IRLM / PI Locking Display – The Challenges
Identify resource contention
– Which IMS, dependent region, application is holding resources
– Untangle queue of holder/waiters
– Determine resources with contention
© 2014 IBM Corporation
2014
49
51. SM Solution - Real-time IRLM / PI Locking Display
Display of resources with contention
– Select by dependent region and drill down to resource
– Select by resource and drill down to dependent region
© 2014 IBM Corporation
2014
50
55. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
54
– Long lock detection
– Investigate lock holders and waiters
–IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
56. IRLM Statistics – The challenges
Managing the well being of IRLM(s)
– Deadlocks, false contentions, storage utilization?
– Multiple IRLMs to check
© 2014 IBM Corporation
2014
55
57. SM Solution - Aggregated IRLM Statistics
Information gathered from IRLMs across Sysplex
– Aggregated into single system image
– Drill down for information from individual IRLMs
© 2014 IBM Corporation
2014
56
59. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
58
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
–VSO stats
Shared Resources
– RM structure content, status, ownership, and removal
62. IMS Sysplex Challenges and Solutions
Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
© 2014 IBM Corporation
2014
61
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
–RM structure entries’ status, ownership, and removal
66. Sysplex Serialized Program Management – The Challenges
The IMS who scheduled the serial program abnormally terminated and not
coming back for a while, the scheduled serial program is locked and no
other IMS in the Sysplex can use it.
© 2014 IBM Corporation
2014
65
67. SM Solution - Sysplex Serialized Program Management
Delete the locked RM entry to allow the serial program to be
scheduled by other IMS via ISPF
© 2014 IBM Corporation
2014
66
73. Summary
Challenges Sysplex Manager
Transaction Affinity management Seamless, persistent, dynamic update of
transaction affinity
CF Structures monitoring All IMS CF info in one place including VSO
Current utilization and checkpoint stats
Log streams offload monitoring Alert of too many offloads
Shared Queues Management Queue depths for destinations and message details
© 2014 IBM Corporation
2014
72
Long Lock Detection Real time WTO alerts
Lock Contention report Untangle Holder/Waiter queue for locks
Dependent Region activities
IRLM Statistics report Deadlock and false contention stats
VSO stats Buffer hit stats for shared and non-shared VSO
RM structure management Check ownership and removal of troubled entries
74. Q & A
© 2014 IBM Corporation
2014
73
76. Scenarios
IMS
– Scenario 1 – Taking Inventory and capture diagnostic data
– Scenario 2 – Managing IMS System Parameters
– Scenario 3 – Verifying IMS Resource Definitions
– Scenario 4 – Issuing IMS Commands
– Scenario 5 – Maintaining Command Audit Trail
– Scenario 6 – Managing Dependent Regions
– Scenario 7 – Viewing IMS CF Structures
IMSplex
© 2014 IBM Corporation
2014
75
IMSplex
– Scenario 9 – Viewing Aggregate SCI, RM, OM Statistics
Shared Queue and CQS
– Scenario 10 - View local buffer utilization
– Scenario 11 – Managing Destination Queue Depth
– Scenario 12 – Viewing CQS Structures
Dashboard
– Scenario 13 – Dashboard and Thresholds
77. Scenario 1 – Taking Inventory and Capture Diagnostics
Many address spaces – IMS Control Region, IMS DLI/SAS, IMS
DBRC, IRLM, CQS, RM, OM, SCI, etc..
– How do you identify related IMS components across the Sysplex?
– What is the status of these components?
– What version of IMS components are involved?
– How much resource are they using from z/OS perspective?
– How do you collect diagnostic data to debug sysplex problem?
© 2014 IBM Corporation
2014
76
IMS Sysplex Manger structured TSO/ISPF interface
– Guided display of IMS components
– Provides component id, task or job name, version, status and basic z/OS
information such as CPU time and EXCP counts
– Drill-down to detailed component information
– Easily capture console dumps for IMS components across the plex
– Check DBRC RECON datasets placement and VSAM stats
83. DBRC RECON datasets and VSAM stats
Show RECON datasets placement (COPY1, COPY2, SPARE) for all
IMS systems
Built-in LISTCAT for RECONs when drill down
© 2014 IBM Corporation
2014
82
85. Scenario 2 – Managing IMS System Parameters
Many system run-time parameters
– Sources: DFSPBxxx, overrides via Control Region PARM=
– Which ones are being used?
– Are the parameters the same across the Sysplex?
System parameter display
© 2014 IBM Corporation
2014
84
System parameter display
– Real-time scrollable display of “resolved” values
– Parameter values across all IMS systems for easy comparison
– New – System Parameter Tutor for instant description
90. Scenario 3 – Verifying IMS Resource Definitions
Resource definitions
– Transactions, Programs, Data Bases, Nodes, LTERMS, etc.
– Are the definitions the same across the Sysplex?
– What is the resource status across the Sysplex?
– How do you alter the status across the Sysplex?
Resource definition display
– Query resource by status or attribute
– Real-time scrollable display of resources
© 2014 IBM Corporation
2014
89
– Real-time scrollable display of resources
– Resource attributes and status across all IMS systems for easy comparison
– Integrated IMS Type 1 / Type 2 command interface to alter resource status
– Ability to drill-down to related resource (from transaction to PSB, from PSB to
databases… )
97. Scenario 4 – Issuing IMS Commands
Issue Type 1 command to all IMS systems
– Display same resource type across Sysplex
– Alter resource status across Sysplex
Command issued to each IMS system
– Output recorded to command file and logged history file
© 2014 IBM Corporation
2014
96
– Output recorded to command file and logged history file
– ISPF Browse(view) used to display result
– Scrollable, Primary and line commands (find, exclude, etc.)
– Retrievable list of previously entered commands
101. Scenario 5 – Maintaining command Audit Trail
Keeping track of operations of your IMS systems
– Need for audit trail?
– Easy way of viewing IMS commands & system messages?
– Automatic archive of audit log?
Sysplex Manager with its history datasets
– Log most type-1 IMS commands, command responses and MTO messages
from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI
© 2014 IBM Corporation
2014
100
from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI
programs using CMD or ICMD interface)
– Log OM processed command and output
– Searchable, filterable online viewing of log data using ISPF dialog
– Automatic archive for full history dataset
– Archive Directory to assist locating archived data
– Issue type-1 and type-2 IMS commands while viewing log data (similar to z/OS
SDSF)
– Check SM History datasets status
104. SM History database status and usage
Up to 8 VSAM Linear datasets make up SM history database
Used in a circular fashion similar to IMS OLDS
Automatic archive process to save full datasets to user-defined output location
Directory to help users keep track of archive output
Real-time information on status of these history datasets
© 2014 IBM Corporation
2014
103
105. View and retrieve archive datasets
Locate which archive dataset has the info you need
Browse the archive dataset instantly
© 2014 IBM Corporation
2014
104
106. Scenario 6 – Managing Dependent Regions
Transaction workload back-logged
– What dependent regions are available?
– What transaction classes do they handle?
– Are they occupied with work?
– What are they doing?
© 2014 IBM Corporation
2014
105
Scrollable list of all dependent regions across Sysplex
– Enhanced IMS /DISPLAY ACTIVE REG
– Displays overall DL/I DB/TM call counts
– Sort the lock held column to see which reg holds the most locks
– Resequenced by primary or secondary classes
– Drill down to lower level of detail
113. Scenarios
IMS
– Scenario 1 – Taking Inventory and capture diagnostic data
– Scenario 2 – Managing IMS System Parameters
– Scenario 3 – Verifying IMS Resource Definitions
– Scenario 4– Issuing IMS Commands
– Scenario 5 – Maintaining Command Audit Trail
– Scenario 6 – Managing Dependent Regions
– Scenario 7 – Viewing IMS CF Structures
IMSplex
© 2014 IBM Corporation
2014
112
IMSplex
– Scenario 8 – Viewing Aggregate SCI, RM, OM Statistics
Shared Queue and CQS
– Scenario 9 - View local buffer utilization
– Scenario 10 – Managing Destination Queue Depth
– Scenario 11 – Viewing CQS Structures
Dashboard
– Scenario 12 – Dashboard and Thresholds
114. Scenario 8 – Aggregated SCI, RM, OM Statistics
Managing the well being of Common Service Layer (CSL) address
spaces
– Lack of tools to obtain CSL statistical information
– Multiple instances to check
Information gathered from CSL address spaces across Sysplex
© 2014 IBM Corporation
2014
113
Information gathered from CSL address spaces across Sysplex
– Aggregated into single system image
– Drill down for information from individual address space
118. Scenarios
IMS
– Scenario 1 – Taking Inventory and capture diagnostic data
– Scenario 2 – Managing IMS System Parameters
– Scenario 3 – Verifying IMS Resource Definitions
– Scenario 4– Issuing IMS Commands
– Scenario 5 – Maintaining Command Audit Trail
– Scenario 6 – Managing Dependent Regions
– Scenario 7 – Viewing IMS CF Structures
IMSplex
© 2014 IBM Corporation
2014
117
IMSplex
– Scenario 8 – Viewing Aggregate SCI, RM, OM Statistics
Shared Queue and CQS
– Scenario 9 - View local buffer utilization
– Scenario 10 – Managing Destination Queue Depth
– Scenario 11 – Viewing CQS Structures
Dashboard
– Scenario 12 – Dashboard and Thresholds
119. Scenario 9 - SQ local buffer utilization
© 2014 IBM Corporation
2014
118
120. Scenarios
IMS
– Scenario 1 – Taking Inventory and capture diagnostic data
– Scenario 2 – Managing IMS System Parameters
– Scenario 3 – Verifying IMS Resource Definitions
– Scenario 4– Issuing IMS Commands
– Scenario 5 – Maintaining Command Audit Trail
– Scenario 6 – Managing Dependent Regions
– Scenario 7 – Viewing IMS CF Structures
IMSplex
© 2014 IBM Corporation
2014
119
IMSplex
– Scenario 8 – Viewing Aggregate SCI, RM, OM Statistics
Shared Queue and CQS
– Scenario 9 - View local buffer utilization
– Scenario 10 – Managing Destination Queue Depth
– Scenario 11 – Viewing CQS Structures
Dashboard
– Scenario 12 – Dashboard and Thresholds
121. Scenario 12 – Dashboard Sysplex at a Glance
Managing the IMS Sysplex through key system indicators
– Determine overall Sysplex health at a Glance
– Alert when indicators violate predefined thresholds
– Set multiple monitoring thresholds depending on workloads, time of day…
Critical data from key areas summarized on one or more screens
© 2014 IBM Corporation
2014
120
Critical data from key areas summarized on one or more screens
– User customized content, positioning, threshold value
– Threshold exceptions are logged for future review
– One or more per user, default supplied
– Automatic screen refresh capability for operational ease
– Drill down for additional detail
124. Low and High Thresholds
Define threshold values
© 2014 IBM Corporation
2014
123
128. References
For publications and demo, visit IMS Sysplex Manager for z/OS at
– http://www-306.ibm.com/software/data/db2imstools/imstools-library.html
© 2014 IBM Corporation
2014
127