2. Increasing complexity: Although the declining cost of storage per megabyte makes it
attractive to add additional disks, the increasing complexity of managing this storage
results in overused staff and under-used IT resources. Combining this with the shortage of
skilled storage administrators, it is possible to add significant cost and introduce risk to
storage management.
Maintaining availability: The added complexity of 24x7 environments significantly
reduces, for example, the efficiency of conducting routine maintenance, scheduling
backups, migrating data, and introducing new software and hardware. This problem is
compounded by the fact that as availability increases, so does the cost inherent with
making it so.
These challenges still exist, although large SANs do offer desirable and tangible benefits, for
example, better connectivity, improved performance, distance flexibility, and scalability. Yet
even these benefits might be outweighed by the added complexity that they introduce.
As an example, large enterprise SANs often contain different types of storage devices. These
differences could be in the types of disk deployed, their level of performance, or the
functionality provided, such as RAID or mirroring. Often customers have different vendor
storage devices as the result of mergers or consolidations. The result, however, is that
storage and SAN administrators need to configure storage to servers, and then keep track of
which servers own or have access to that storage. The storage administrative tasks can
become daunting as the SAN grows and as the storage administrators manually attempt to
manage the SAN.
Furthermore, the complexity of different file systems in the same SAN requires that storage
administrators know how to administer each client operating system (OS) platform. The
management interfaces for each might be different, because there is no common standard to
which all vendors adhere. Lastly, because the file systems are tied to each of the servers,
storage management functions potentially have to be run on hundreds of servers. It is easy to
see why manageability and interoperability are the top areas for concern, especially in a SAN
where the number of possible storage and OS platform permutations are considerable.
These challenges are at odds with the commonly held belief that storage is decreasing in cost
per megabyte. It is clear that the cost of managing storage is greater than the initial purchase
price. A strategy is needed to address storage manageability, while at the same time,
addressing the need for interoperability. This strategy is the IBM TotalStorage Open Software
Family.
This strategy represents the next stage in the evolution of storage networking. It affords you
the opportunity to fundamentally improve your company’s effectiveness and efficiency in
managing its storage resources. With the IBM SAN virtualization products, you are witnessing
IBM deliver on its continued promise to provide superior on demand solutions that will assist
in driving down costs and reducing TCO.
In this IBM Redpaper, we cover the interaction of two products and their components.
IBM TotalStorage SAN Volume Controller
IBM TotalStorage SAN Volume Controller (SVC) is a virtualization appliance solution that
maps virtualized volumes visible to hosts and applications to physical volumes on storage
devices. Each server within the SAN has its own set of virtual storage addresses that are
mapped to a physical address. If the physical addresses change, the server continues
running using the same virtual addresses it had before. This means that volumes or storage
can be added or moved while the server is still running. The IBM virtualization technology
2 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
3. improves management of information at the “block” level in a network, enabling applications
and servers to share storage devices on a network.
For more information, see the IBM Redbook, IBM TotalStorage SAN Volume Controller,
SG24-6423.
IBM TotalStorage Productivity Center
The IBM TotalStorage Productivity Center is an open storage infrastructure management
solution designed to help reduce the effort of managing complex storage infrastructures to
help improve storage capacity utilization and to help increase administrative efficiency. It is
designed to enable the storage infrastructure to have the ability to respond to “on demand”
storage needs.
IBM TotalStorage Productivity Center consists of the following four products.
IBM TotalStorage Productivity Center for Data (formerly IBM Tivoli® Storage Resource
Manager) can provide more than 300 enterprise-wide reports, monitoring and alerts,
policy-based action, and file system capacity automation in a heterogeneous environment.
IBM TotalStorage Productivity Center for Fabric (formerly IBM Tivoli Storage Area
Network Manager) can provide automated device discovery, topology rendering, error
detection fault isolation, SAN error predictor, zone control, real-time monitoring and alerts,
and event management for heterogeneous enterprise SAN environments.
IBM TotalStorage Productivity Center for Disk (formerly IBM TotalStorage Multiple
Device Manager – Performance Manager feature) can enable device configuration and
management of supported SAN-attached devices from a single console. It can discover
storage and provides configuration capabilities for IBM TotalStorage Enterprise Storage
Server® (ESS), IBM TotalStorage SAN Volume Controller (2145), and the IBM
TotalStorage DS4000 (formerly Fibre Array Storage Technology, FAStT) storage. IBM
TotalStorage Productivity Center for Disk manages the performance for the ESS and SAN
Volume Controller (SVC).
IBM TotalStorage Productivity Center for Replication (formerly IBM TotalStorage Multiple
Device Manager – Replication Manager feature) can provide copy services management
for the ESS. IBM TotalStorage Productivity Center for Replication provides configuration
and management of the point-in-time copy (FlashCopy®) and Metro Mirror (synchronous
point-to-point remote copy) capabilities of the ESS in supported configurations.
In this Redpaper, we focus on IBM TotalStorage Productivity Center for Disk.
IBM TotalStorage Productivity Center for Disk
IBM TotalStorage Productivity Center for Disk is designed to centralize management of
networked storage devices that implement the Storage Management Initiative Specification
(SMI-S) established by the Storage Networking Industry Association (SNIA), including the
IBM TotalStorage Enterprise Storage Server, IBM TotalStorage DS4000 series family
(formerly the FAStT family), and IBM TotalStorage SAN Volume Controller, and the devices
that they manage. The IBM TotalStorage Productivity Center for Disk is designed to:
Help reduce storage management complexity and costs while improving data availability
Centralize management of storage devices through open standards (SMI-S)
Enhance storage administrator productivity
Improve storage resource utilization
Offer proactive management of storage devices
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 3
4. Discovery of IBM storage devices that are SMI-S enabled
Centralized access to storage devices information, information concerning the system
attributes of connected storage devices, is available from the IBM TotalStorage Productivity
Center for Disk console.
Centralized management of storage devices
The device configuration and manager console for the SMI-S-enabled IBM storage devices
can be launched from the IBM TotalStorage Productivity Center for Disk console.
Device management
IBM TotalStorage Productivity Center for Disk can provide access to single-device and
cross-device configuration functionality. It enables the user to view important information
about the storage devices that are discovered by IBM TotalStorage Productivity Center for
Disk, examine the relationships between those devices, or change their configurations. IBM
TotalStorage Productivity Center for Disk supports the discovery and logical unit number
(LUN) provisioning of IBM TotalStorage DS4000 series storage systems, IBM TotalStorage
SAN Volume Controller, and IBM TotalStorage ESS. The user can view essential information
about the storage, view the associations of the storage to other devices, and change the
storage configuration. IBM TotalStorage DS4000 and ESS, attached to the SAN or attached
behind the SAN Volume Controller, can be managed by IBM TotalStorage Productivity Center
for Disk. IBM TotalStorage Productivity Center for Disk and IBM TotalStorage Productivity
Center for Replication also include the IBM TotalStorage Productivity Center for Fabric Bonus
Kit, which supports up to 64 Fibre Channel ports. With this function, the storage administrator
can view Fibre Channel as well as SMI-S enabled storage devices, automate zoning of
clients, LUNs, and storage devices for the TotalStorage SAN Volume Controller environment,
and drill-down from storage devices to the underlying infrastructure. If more than 64 SAN
ports are required in an enterprise, you can obtain additional capacity by ordering IBM
TotalStorage Productivity Center for Fabric.
Performance monitoring and management
IBM TotalStorage Productivity Center for Disk can provide performance monitoring of ESS
and SVC storage devices, customization of thresholds based on your storage environment,
and generation of events if thresholds are exceeded. In addition, IBM TotalStorage
Productivity Center for Disk is designed to help IT administrators select the LUN for better
performance. IBM TotalStorage Productivity Center for Disk enables an IT administrator to
specify both when and how often the data should be collected. IBM TotalStorage Productivity
Center for Disk is also designed to help support high availability or critical applications by
providing customization of threshold settings and generating alerts when these thresholds are
exceeded. Plus, it can provide gauges to track real-time performance. IBM TotalStorage
Productivity Center for Disk is designed to enable the IT administrator to:
Monitor performance metrics across storage subsystems from a single console
Receive timely alerts to enable event action based on customer policies
Focus on storage optimization through the identification of the best LUN
For more information, see the IBM Redbook, IBM TotalStorage Productivity Center: Getting
Started, SG24-6490.
In the topics that follow, we show how IBM TotalStorage Productivity Center for Disk can be
used to monitor the performance of an SVC and assist in making the correct decisions if the
back-end storage systems become bottlenecked.
4 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
5. Note: It must be noted that the environment was deliberately set-up so that the back-end
storage system would become a bottleneck. In a production environment, we would not
recommend the use RAID 5 arrays consisting of only three disk drives. In addition, the
workloads that were generated consisted of random reads. This workload was chosen so
that the SVC cache would be bypassed, making the back-end storage a bottleneck.
The test environment
As shown in Figure 1, our environment consisted of:
A pair of SVC nodes
SVC Master Console
One Microsoft® Windows® 2000 host, x345
One Linux® host, x345
Two Brocade 16 port switches
IBM TotalStorage Productivity Center server, x345
IBM TotalStorage Productivity Center client, x345
SAN Volume Controller - Connections
Windows 2000 Linux host
host
4 ports
4 ports
DS4300
Figure 1 Equipment schematic
The zoning was the standard SVC zoning and is shown in Figure 2.
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 5
6. SAN Volume Controller - Zoning
Windows host Linux host
Host zones
Switches SVC
SAN Volume Controller
node pair
Storage zone
Storage
Figure 2 SVC zoning
Scenario
We followed this outline scenario:
1. We configured multiple RAID 5 arrays on the FAStT600 (DS4300). Each array consisted of
three disk drives.
2. Multiple LUNs were created on each array so that the whole of the usable space was
occupied.
3. Each of the LUNs was “imported” into the SVC as a managed disk. The LUNs for each
RAID array were put into their own managed disk group (MDG).
4. Virtual disks (VDisks) were created from one of the managed disk groups and presented
to the two attached hosts (Windows and Linux).
5. A workload was run on the Windows host (generated by Iometer) and a steady state
condition was reached. In order to stress the back-end storage system, the workload
created consisted of a high proportion (70%) of random reads, thus nullifying the effects of
the SVC cache.
6. Performance gauges were created on IBM TotalStorage Productivity Center for Disk and
these were used to take measurements of the steady state performance.
7. A workload was generated on the Linux host. This workload was designed so that it would
stress the VDisks. Because the VDisks attached to the Linux host were created from the
same MDG as the VDisks attached to the Windows host, the back-end storage array from
which the MDG was created was stressed.
8. We used IBM TotalStorage Productivity Center for Disk to take performance
measurements. These measurements indicated that the MDG was becoming
bottlenecked.
6 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
7. 9. The effect of this bottleneck was to cause a drop off in performance of the Iometer
workload on the Windows host.
10.Further performance measurements were taken using IBM TotalStorage Productivity
Center for Disk gauges.
11.The gauges indicated that the VDisk performance on the Windows host was being
affected by the bottleneck at the managed disk (MDisk) level.
12.As we knew that the performance on the Windows host was being affected by poor MDisk
performance, we migrated the VDisks attached to the Windows host to another MDG.
13.The Iometer performance on the Windows host returned to its previous steady state.
14.Performance measurement showed that the input/output (I/O) bottleneck had been
removed and that the system was running correctly.
Details
The following figures showing IBM TotalStorage Productivity Center for Disk take you through
the process that we followed.
Note: These steps assume that you have already started collecting performance statistics
from the SVC. The procedure for doing this is described in the IBM Redbook IBM
TotalStorage Productivity Center: Getting Started, SG24-6490.
Important: If for some reason the Multiple Device Manager (MDM) performance statistics
task is stopped, you need to abort the older process from the SVC Master Console. Log on
to the SVC Master Console and go to Manage Clusters. You can then choose to stop data
collection. You will then be able to restart gathering statistics using IBM TotalStorage
Productivity Center for Disk.
The following steps describe our process:
1. Figure 3 shows the initial IBM TotalStorage productivity Center welcome window. Click
Manage Disk and Performance Replication at the top of the window.
Figure 3 Initial IBM TotalStorage Productivity Center welcome window
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 7
8. 2. Enter your password in the Multiple Device Manager (MDM) password window, as shown
in Figure 4.
Figure 4 MDM password window
3. Click Next in the IBM Director Event Action Plan wizard, as shown in Figure 5.
Figure 5 Event Action Plan wizard window
8 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
9. 4. Expand Multiple Device Manager in the Tasks pane on the right, as shown in Figure 6.
Figure 6 MDM front window
Figure 7 shows the expanded MDM tasks.
Figure 7 Multiple Device Manager expanded
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 9
10. 5. Then, click the Gauges icon in the Tasks pane, as shown in Figure 8.
Figure 8 Gauges icon highlighted
While pressing and holding the left mouse button, move the “hand” so that it is over the
SVC icon in the Group Contents pane, as shown in Figure 9.
Figure 9 “Hand” over SVC icon
10 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
11. A Job in Progress window opens, as shown in Figure 10.
Figure 10 Job in progress
Next, the Performance gauges window opens, as shown in Figure 11. Because at this
stage no gauges have been created, the only buttons available for use are the Create and
Help buttons.
Figure 11 Performance gauges window
6. Click Create, and the Create performance gauge window opens, as shown in Figure 12. It
is from this window that you can create the gauges that you need. The type of gauges that
you can create depends on what you are monitoring, VDisks or MDisks:
– Virtual disk (VDisk): For a single Vdisk or all Vdisks combined
• Total and average number of reads, writes
• Number of 512-bytes blocks read, writes
– Managed disk (MDisk): For a single MDisk or per MDisk group
• Total and average number of reads, writes
• Number of 512-bytes block read, writes
• Read and write transfer rates
• Total, average, minimum, maximum response time
7. In Figure 12, we selected All VDisks and the Total I/O’s/sec. This will be measured on
MDG1, DS4300-RAID5_1. You have some more choices to make:
– Name of the gauge.
– Whether to select the number of data points to show or the date range. We selected
the last 5 data points.
– Whether to display the gauge now or store it for late use. We display the gauge now.
After making your selections, click OK, and your gauge will be created.
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 11
12. Figure 12 Create performance gauge window
We made an error with the name of our gauge, VDisk total I/O / sec, because blanks are
not allowed in a gauge name. Therefore, the error message shown in Figure 13 appears.
Figure 13 Error message when blanks are used in a gauge name
12 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
13. We clicked OK, and the Create performance gauge window opens again, this time with all
our entries, as shown in Figure 14.
Figure 14 Create group VDisk performance gauge
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 13
14. We corrected the gauge name, replacing the blanks with underscores (_) so that the name
was VDisk_total_I/O_/_sec. We then clicked OK. The gauge was created and displayed,
as shown in Figure 15. As you can see, the gauge shows the I/O ramping up and reaching
a steady state. The only I/O activity taking place was that generated by Iometer on the
Windows host, and this gives us our baseline figure.
Figure 15 VDisk I/O per second gauge
14 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
15. 8. We then created a gauge for the single VDisk that the Windows host was using. Figure 16
shows the creation process. We used similar parameters as for the gauge created in
Figure 14 on page 13, the difference being that now we chose a single VDisk, barrytest1,
rather than all VDisks.
Figure 16 Single VDisk gauge create window
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 15
16. Because the I/O was only taking place to a single VDisk in the I/O group, the results
displayed in Figure 17 are the same as for the group that we saw in Figure 15 on page 14.
Figure 17 Single VDisk I/O gauge for Windows host VDisk
16 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
17. 9. We then created a single VDisk gauge for the Linux host and started the workload on this
host. Figure 18 shows how the I/O increased on the Linux host’s VDisk.
Figure 18 Linux host VDisk I/O gauge
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 17
18. As we describe in “Scenario” on page 6, we deliberately created the VDisks used by the
Windows and Linux hosts so that contention would occur on the storage system RAID
array, which would affect I/O performance. The effect of this is clearly shown in Figure 19,
which is a graph of the I/O performance of the Windows host’s VDisk. As the Linux
workload increases, the MDisks that support the VDisks become staturated. The result of
this is that I/O performance of the Windows host dramatically decreases.
Figure 19 Windows host’s VDisk gauge
18 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
19. Although the I/O rate of the Windows host decreased, the overall I/O on the back-end
storage system increased. Figure 20 shows this in a gauge of the MDG from which the
VDisks were created.
Figure 20 MDG I/O rate
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 19
20. 10.We then created a gauge to measure the I/O performance of the second MDG. Figure 21
shows this process.
Figure 21 MDG 2 I/O performance gauge creation
20 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
21. Because no I/O was taking place on this MDG, the resulting gauge, shown in Figure 22, is
rather flat.
Figure 22 MDG 2 I/O performance: No VDisks allocated
At this stage, we analyzed the results of our findings so far. We came to the conclusion that
the drop off in performance of our steady state Windows workload, our main production
server, was caused by the additional load on the back-end storage system caused by the
Linux host. In a non-virtualized world, we would be faced with the difficult and time-consuming
task of stopping production, copying the data to a different set of disks, and then resuming
production. Here, we have the benefit of virtualized storage. Using the SVC migrate VDisk
functionality, we migrated the offending VDisk attached to the Linux host to our second,
currently inactive MDG, DS-4300_RAID5_2.
As we migrated the VDisk, we monitored the performance of the VDisk attached to the
Windows host. Figure 23 shows the results.
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 21
22. Figure 23 Windows VDisk performance during data migration
Summary
This paper shows how IBM TotalStorage Productivity Center for Disk can be used to monitor
VDisk and MDisk performance on the SVC. If VDisk performance analysis is detected, the
supporting MDG can be easily analyzed to determine the root cause. After you determine the
offending storage components, you can use the VDisk migration functions of the SVC to
non-disruptively correct the problem.
The team that wrote this Redpaper
This Redpaper was produced by a team of specialists from around the world working at the
International Technical Support Organization, San Jose Center.
Barry Mellish is a certified I/T Specialist and is currently working as a Storage Software and
Virtualization Specialist in North Region, EMEA. Previous to this, he was a Project Leader at
the International Technical Support Organization (ITSO), San Jose Center, for four years. He
has coauthored 15 Redbooks™ and has taught many classes worldwide about storage
subsystems. He joined IBM United Kingdom 20 years ago. Before joining the ITSO, he
worked as a Senior Storage Specialist on the Disk Expert Team in EMEA.
Rob Nicholson is based in Hursley, United Kingdom, and is an IBM Senior Technical Staff
Member. He currently working on the software architecture for SAN Volume Controller and
has had various roles in development and testing for IBM storage products for more than 13
years.
22 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
23. Steve Hurley is currently Program Manager for open systems disaster recovery software
working in Tucson, Arizona. Prior to this role, he worked from Hursley, England, designing
storage virtualization solutions for customers throughout Europe.
Thanks to the following people for their contributions to this project:
Elizabeth Barnes
Editor, International Technical Support Organization, Austin Center
Emma Jacobs
Graphics Designer, International Technical Support Organization, San Jose Center
Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 23
24. 24 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC
26. Send us your comments in one of the following ways:
Use the online Contact us review redbook form found at:
ibm.com/redbooks ®
Send your comments in an email to:
redbook@us.ibm.com
Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. QXXE Building 80-E2
650 Harry Road
San Jose, California 95120-6099 U.S.A.
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Enterprise Storage Server® IBM® Tivoli®
FlashCopy® Redbooks (logo) ™ TotalStorage®
ibm.com® Redbooks™
The following terms are trademarks of other companies:
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
26 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC