IBM XIV Gen3 Storage with 3TB Drives:120,000 Mailbox Resiliency for MS Exchange 2010

®
IBM XIV Gen3 Storage System
Model 2810/114 120,000 Mailbox
Resiliency Exchange 2010
Storage Solution

Tested with: ESRP – Storage Version 3.0
Tested Date: Aug 31, 2011
Authors: Aviad Offeri, David Hartmanii, Betti Poratiii
Document version: 1.0

Content
Overview ...................................................................................................... 3
Features ....................................................................................................... 3
Solution Description ....................................................................................... 4
XIV Gen3 Storage System Model 2810 Type 114, with 3TB Drives .................................... 4
IBM System x3650 Servers ............................................................................... 10
Solution Description ..................................................................................... 11
Targeted Customer Profile .......................................................................... 16
Tested Deployment ................................................................................... 17
Simulated Exchange Configuration: .......................................................... 17
Storage Hardware .................................................................................. 18
Storage Software ................................................................................... 19
Storage Disk Configuration (Mailbox Store Disks) ....................................... 19
Storage Disk Configuration (Transactional Log Disks) ................................. 19
.
Best Practices .............................................................................................. 20
Storage Mailbox Resiliency ....................................................................... 20
Backup strategy ........................................................................................ 21
Contact for Additional Information ................................................................. 21
.
Test Result Summary ................................................................................... 22
Reliability ................................................................................................. 22
Storage Performance Results ...................................................................... 22
Database Backup/Recovery Performance ..................................................... 27
.
Database Readonly Performance ............................................................. 27
Transaction Log Recovery/Replay Performance .......................................... 27
.
Conclusion .................................................................................................. 27
Appendix A. Test Results .............................................................................. 28
.
Stress Testing .......................................................................................... 28
.
Performance Testing .................................................................................. 32
Streaming Backup Testing .......................................................................... 37
Soft Recovery Testing ................................................................................ 41

IBM XIV® Storage System Model
2810 60,000 Mailbox Resiliency Aviad Offer, David Hartman,
Exchange 2010 Storage
Page 2 Betti Porat, Betti Porat
©
Solution Copyright IBM Corp.20110831

Overview
This document provides information on the IBM XIV Gen3 2810 Storage System
storage solution for Microsoft Exchange Server, based the Microsoft Exchange
Solution Reviewed Program (ESRP) – Storage program*. For any questions or
comments regarding the contents of this document, please see the section titled
Contact for Additional Information.

*The ESRP – Storage program was developed by Microsoft Corporation to provide a
common storage testing framework for vendors to provide information on their
storage solutions for Microsoft Exchange Server software. For additional information
on the Microsoft ESRP – Storage program, please click:

http://technet.microsoft.com/enus/exchange/ff182054.aspx

Disclaimer
This document has been produced independently of Microsoft Corporation. Microsoft
Corporation expressly disclaims responsibility for, and makes no warranty, express
or implied, with respect to, the accuracy of the contents of this document.

Please refer to the IBM disclaimer, located at the end of this document for
additional information.

Features

This solution implements the new Mailbox Resiliency features in Exchange Server
2010; specifically the database availability group feature. A database availability
group (DAG) is a set of up to 16 Microsoft Exchange Server 2010 Mailbox servers
that work together to provide automatic databaselevel recovery from a database,
server, or network failure. Mailbox servers in a DAG monitor each other for failures.
Solutions that implement this feature can deliver a very robust Exchange
environment.

More information on this feature can be found by visiting:

http://technet.microsoft.com/enus/library/dd351172%28EXCHG.140%29.aspx

A two DAG solution comprising 24 mailbox servers was created that supported a
total of 120,000 mailboxes with a mailbox size of 1.5GB. Each server hosted 5,000
users, and had 8 active databases, with 625 users per database. Within each DAG,
there were two copies of every database; one local, and one on another server
connected to a second XIV Gen3 storage array. This configuration can provide for
both highavailability, and disasterrecovery scenarios. Database replication is
handled by the mailbox servers over Ethernet. Two XIV Gen3 frames were used,
and the databases and copies were equally distributed across them. A failure of any
©

database will cause the database to mount and become active on whichever server
is hosting the database copy, in a manner transparent to the end users.
The solution in this paper features the IBM XIV Gen3 Storage System together with
IBM System x3650 Intel based servers.

The following sections describe the actual hardware components used in this
solution.

Solution Description
XIV Gen3 Storage System Model 2810 Type 114, with 3TB Drives
As one of the newer members of IBM’s family of storage systems, the XIV Gen3
Storage System is an enterpriseclass array offering superior performance, solid
protection, and incredible ease of manageability. This solution leverages the new
XIV Gen3 system with 3TB drives with a usable capacity of 243TB per frame.

Exclusive to this storage system is the inherent selfoptimizing gridbased
architecture which provides massive parallelism. Massive parallelism design
distributes data in parallel across all available storage resources, by harnessing all
the processing power available to the system. Due to this design, hot spots (heavily
saturated storage components) normally associated with poor storage configuration
and/or workload variances, is eliminated. The XIV Gen3 architecture delivers
continuous optimal performance and eliminates the need for dubious tuning
activities on the storage.

Built using industrystandard Intel/Linux components a fully equipped XIV Gen3
Model 2810/114 Storage System is supplied with fifteen data modules, 6 out of
those are interface modules. Each module consists of one Intel Westmere® Quad
core 64bit CPU, 24 GB fully buffered RAM, and twelve 3TB SAS 7,200 RPM hard
disks.

In comparison to FibreChannel based disks, the XIV Gen3 has more disk density
allowing for a smaller physical storage footprint, which in turn translates to
significantly lower power & cooling requirements. The unique design of the
architecture can also yield similar or better performance.

©

Hardware Highlights
Total Storage Central Processing Units 15
Combined Physical Memory 360GB
Raw Storage 480TB
CPU to Disk Bandwidth 3000GBps
Internal Switching Capacity 600Gbps
Fibre Channel Ports (Speed: 2 Gbs, 4 Gbs & 24 x 8Gbs
8Gbs )
iSCSI Gigabit Ethernet Ports 22
UPS 3
Infiniband Switch 2
Disk Hot Spares 3 (9TB)
Data Module Hot Spare 1 (36TB)
Max Snapshots 12,000

Figure 1 below represents a fully configured XIV Gen3 Storage System:

Figure 1) XIV Gen3 front and rear view (full rack)

©

Reliability/Protection
By design, the XIV Gen3 Storage System architecture is selfhealing as data in the
background is automatically checked for integrity and mirrored to provide data
redundancy. Should the XIV Gen3 lose a 3TB disk, the system triggers an event
alerting IBM support while rebuilding the faulty disk. The disk rebuild process is
transparently handled without user intervention and can finish in as few as 80
minutes when the XIV Gen3’s capacity is 100% allocated and written.

Consistent performance is maintained even during the loss of a hard disk. The
rebuild time of losing one disk does not degrade performance, unlike other
competing storage systems which exhibit a significant degradation in performance.

Should a storage subcomponent fail on monolithic storage systems, about 50%
(CPU and read/write cache) of performance is lost. This not only impacts
productivity, but could have serious implications affecting application performance.
Losing an entire data module1 on the XIV Gen3 Storage System would temporarily
account for only 1/15th reduced performance.

To maintain data redundancy, all data on the storage and memory cache is
mirrored. The cache data is never stored in the same data module. This not only
provides data cache protection, but eliminates loss of data in case a data module’s
power is lost or interrupted.

Each data module is supplied with two field replaceable hotswappable power
supplies, and three UPS devices provide interim power in the event of interrupted
or lost main power. Enough power is provided to allow safe cache destaging and a
graceful shutdown of the XIV Gen3 Storage System.

The XIV Gen3 Storage System also supports synchronous and asynchronous
Remote Mirroring, allowing for replication of data to a remote XIV Gen3 Storage
System. Additionally, XIV Gen3 now supports seamless integration with Windows
Server 2008 R2 clustering, which enables intelligent, automatic failover/failback
using arraybased replication, all within the familiar clustering MMC.

Thin Provisioning
This feature permits the logical allocation of storage to appear larger than what is
physically committed. This provides the ability to resize storage pools and volumes
even after defining initial storage sizes.

This feature cannot only reduce upfront storage costs by avoiding purchasing
needless unused storage, but reduces the hardware and the associated

1
Loss of a data module compared to a data interface module has lesser impact in performance.
©

environmental footprint. A larger footprint is typically more expensive because it
requires more hard disks, electrical power, cooling requirements, disk expansions
enclosures, floor space, and management overhead.

Thin provisioning can reduce or eliminate the impact of future storage capacity
issues when current storage limits have been reached. This occurs when storage
has quickly outgrown original capacities. It can be a difficult task to predict how
data will grow as storage usage is often highly dynamic in nature. In an effort to
circumvent future capacity issues, storage can be appropriately sized beforehand or
resized later.

Figure 2 below depicts how to easily resize an XIV Gen3 volume.

Figure 2) Resizing an XIV Gen3 volume.
Important: While thin provisioning enables allocated storage to appear larger than
what is physically committed, constant monitoring of current available physical
storage space is vital to prevent capacity issues. Storage writes destined to logical
capacity not allocated with enough physical storage can hinder performance, delete
snapshots, and cause storage operations to fail due to lack of space. Prior to
reaching this limit, storage threshold alerts should be configured and enabled to
warn administrators of the condition. Regular monitoring of storage events and
proactive action can prevent these events from occurring.

Another innovative feature is the ability to consume disk space only when actual
data is written to the volume. This means that while a LUN may be allocated 100GB
of capacity, if only 10GB is in actual use, only 10GB is of physically allocated &
reported. This concept is referred to as a volume’s hard or physical utilization. This
can result in a significant savings in both space and cost. This differs from
traditional storage systems, in which the entire preallocated disk space is flagged
as 0’s and marked as used storage. Traditional storage systems quickly outgrow
space by immediately preallocating storage due to the inefficient method of
marking free space as reserved. Whereas on the XIV Gen3 Storage System it is

©

tracked by the system as free space, and can be used for other hosts, thus
maximizing overall storage utilization.

Note: It is important to realize that this concept differs from that of thin
provisioning. Also, this feature does not apply to free space within the EDB file(s).

Ease of Management
Managing the XIV Gen3 Storage System compared to other storage is markedly
different. Typical storage management tasks are eliminated and handled
automatically by the XIV Gen3 Storage System including automatic provisioning,
selftuning of data placement, read/write cache tuning, selfhealing, automatic
rebuilding of failed disk drives, and automatic phasingout or fencing of failed
components. The modular, redundant architecture dramatically reduces impact to
service levels from such events.
In general, storage management can be a daunting task, requiring extensive
storage experience and specialized training. These barriers have been removed, as
the XIV Gen3 Storage System provides a highly intuitive GUI called XIV Gen3
Storage Management, with a near zero learning curve.

A less experienced storage administrator with no XIV Gen3 Storage System
experience can allocate storage and map volumes to hosts in few minutes. Storage
management tasks take less time, allowing storage to be utilized faster.


We assume the host attached to the IBM XIV Gen3 is already configured with the vendor specific SAN
drivers and is properly zoned in the SAN fabric or a point-to-point connection is used.
©

Figure 3 below is a sample screen shot of the XIV Gen3 Storage Management
interface.

Figure 3) The highly intuitive XIV Gen3 Storage Management interface

A fullfeatured commandline management interface, XCLI is also provided for
scripting tasks, management, configuration, and monitoring.

Lower Power Consumption
The IBM XIV Gen3 Storage System is equipped with highcapacity SAS disks. SAS
disk drives provide significantly better GB/$ value than FibreChannel (FC) disks.
SAS disks also require less power consumption than FC disks. Since SAS disk drives
are generally much higher in capacity than FC disks – this translates to requiring
less hard drives, cooling, and power for a given capacity.

©

Snapshots
A key component of the XIV Gen3 Storage System is the inherent snapshot
capabilities which deliver nearinstantaneous volume replicas**. A total of 12,000
snapshots are supported. A snapshot is simply a pointintime image of data used
for quick data retrieval purposes. Transportable snapshots are often use for
offloaded backups of Exchange databases, which remove the backup load from the
production servers.

The snapshot method employed by the XIV Gen3 Storage System uses an efficient
method called RedirectonWrite. The RedirectonWrite snapshotbased method
addresses the typical performance overhead associated with the CopyonWrite or
COW snapshotbased method. For example, during a CopyonWrite snapshot
operation, new writes destined to target storage must wait until the original data is
copied to the snapshot area. This step is eliminated using RedirectonWrite as the
new writes are written immediately to the snapshot area resulting in faster
snapshots. The RedirectonWrite snapshot method does not impact the source
volumes as do CopyonWrite based snapshots.

For more information on the XIV Gen3 Storage System visit the IBM website at:
http://www.xivstorage.com
http://publib.boulder.ibm.com/infocenter/ibmxiv/r2/index.jsp

IBM System x3650 Servers
The System x3650 is a rackmount server designed for critical and high
performance computing.

For this particular solution, twelve System x3650 model 7979AC1 systems
containing 2 quad core e5450 3.0 GHz Intel Xeon CPUs were used. Each System
x3650 was equipped with 24 GB of internal memory.

Figure 4 below is a view of the System x3650

**
Actual snapshot operation times are measured on the IBM XIV. Monitoring of an ad-hoc or scheduled
snapshot operation from the TSM or IBM Data Protection interface known as the VSS requester may
appear longer as there is additional communication occurring between the requester client and server.
©

Figure 4) System x3650 server

Product specifications below list the actual configuration used in this solution.

Product Description
Description: IBM System x3650
Form Factor/Height Rack/2U
Machine Type/Model: 3650
Part Number: 7979MC1
BIOS Build ID: IBM GGE141AUS
Released: 10/8/2008
Revision: 1.12
Processor Dual 3.0 GHz Intel Xeon QuadCore
processors
Physical Memory 24 GB
Onboard Network Broadcom Gigabit Ethernet:
Model: BCM5708C NetXtreme II GigE
Driver Version: 4.4.15.0
Fibre Channel adapter Qlogic® 8Gb dual port Fiber HBA
Expansion (PCIe):
HBA Driver Version:
Storport 9.1.9.25
HBA Bios: 2.02
HBA Firmware Version: 4.03.01
Part Number: QLE2562

For more information on the current System x3650 M3 offering, please visit the
following IBM website:

http://www03.ibm.com/systems/x/hardware/rack/x3650m3/index.html

Solution Description
This solution demonstrates the performance capabilities utilizing two XIV Gen3
Storage System arrays and 24 System x3650 servers supporting a total of 120,000
mailboxes using a target 0.11 IOP profile (0.11 IOP with added 20% headroom –
0.13 as tested). The mailbox size is 1.5 GB.

©

Important: Though this solution was tested using 120,000 mailboxes, additional
storage to accommodate snapshots or frequent mailbox move operations may be
required for larger transaction log volumes. Carefully review these parameters
before finalizing your final target limits. The astested design allows for a single
mailbox server or storage array failure.

Storage hardware:
 2 x XIV Gen3 Storage Systems Model 2810 Type 114
 XIV Gen3 Storage System Software Version: 11.0.0
 24 GB physical memory/data module
 Twelve 3TB drives per data module
 180 SAS drives (3TB, 7,200 RPM) per array
 Form Factor: Rackmount
 Rack Height: 42u

Exchange servers:
 24 x System x3650 servers:
o 2 quadcore e5450 3Ghz Intel Xeon CPUs
o 24 GB RAM/Server
 Windows Server 2008 (x64) – Enterprise Edition, R2, Service Pack 1
o .NET Framework 3.5
o Microsoft Multipath I/O
 Qlogic 8Gb dualport Fibre HBA Expansion (PCIe):
o HBA Driver Version Storport 9.1.9.25
o HBA BIOS: 2.02
o HBA Firmware Version: 4.03.01
 XIV Storage System Host Attachment Package for Windows (x64) – Version 1.6

FibreChannel network:
 2 –8 Gb Brocade Model BR48000 Fibre Channel switches
 8 –FibreChannel Ports (Tested Speed: 8 Gb)
 16 FC LC Fibre Channel 50/125 micrometer shortwave cables
 16 Multimode cables (20 meters)

©

Figure 5 below depicts a highlevel representation of this solution.

Note: The FC switches are not diagrammed to keep the overall diagram simplified.
The brackets are to imply each FC switch as described in the section above.

Figure 5) ESRP physical configuration.
©

SAN Connectivity
As mentioned in the previous sections, the XIV Gen3 Storage System provides SAN
connectivity via 24 FC ports or 22 iSCSI Ethernet ports. For this solution, only the
Fibre Channel ports are utilized.

The rear of the XIV Gen3 Storage System’s patch panel reveals access to all six
host interface modules, each of which contains FC port designations 1, 2, 3, and 4.
For each Exchange server, a minimum of 6 paths were used to provide SAN
connectivity to the XIV Gen3 Storage System.

Two FC switches are used to establish 12 physical connections to the XIV Gen3
Storage System. Each switch has six FC ports connected. Each FC switch port is
connected to each interface module’s FC port #3 or FC port #1. This provides an FC
connection to all FC Interface Modules on the XIV Gen3 Storage System.

This configuration provides multiple redundant paths to the XIV Gen3 Storage
System and guarantees each host has a path to each host interface module. This
allows the hosts to evenly distribute I/O workload across all host interface modules
to obtain the best possible performance.

For each host to achieve optimal performance, each host FC adapter’s max queue
depth was set to 128.

Each FC port on each interface module concurrently supports up to 1,400 IOPs.
There are a total of 4 FC ports on each interface module. Two of the four are
available for target FC connections. The remaining two are reserved for remote
replication.

To calculate the preferred max queue depth for each host, divide total number of
hosts defined in your zone by 1,400.

Zoning Layout
The FC ports on each host are connected via both Fibre channel switches.

To help provide a starting reference for this test environment refer to how XIV
Gen3 host interface modules are connected in each SAN A, and B

©

Note: Zone XIV_SERENITY contains all XIV Gen3 host interface modules.

Zone Name Switch A Zone Switch B – Zone
Members Members
Serenity_XIV_HOST_20 XIV20   XIV20
    HBA 1 Port 0     HBA 1 Port 1
XIV_Serenity (port 3)    XIV_Serenity (port 1)
XIV_Serenity (port 3)   XIV_Serenity a (port 1)
Table 1) FC zoning configuration

©

Important: Before deploying FC switches into production carefully run 24 hour
tests and closely monitor the FC activity for errors. If errors are observed, take
action by ensuring that zoning parameters are correctly defined. Not following these
guidelines may not guarantee reliable operation of your storage environment and
could result in data loss.

The XIV Gen3 2810 System Storage array can be found in the Windows Server
Catalog here:

http://www.windowsservercatalog.com/item.aspx?idItem=8b0f7671f7d32391
327cf8d211ac6a29&bCatID=1282

The ESRPStorage program focuses on storage solution testing to address
performance and reliability issues with storage design. However, storage is not
the only factor to take into consideration when designing a scaleup Exchange
solution. Other factors which affect the server scalability are: server processor
utilization, server physical and virtual memory limitations, resource requirements
for other applications, directory and network service latencies, network
infrastructure limitations, replication and recovery requirements, and client usage
profiles. All these factors are beyond the scope for ESRPStorage. Therefore, the
number of mailboxes hosted per server as part of the tested configuration may
not necessarily be viable for some customer deployments.

For more information on identifying and addressing performance bottlenecks in an
Exchange system, please refer to Microsoft's Troubleshooting Microsoft Exchange
Server Performance, available at:

http://go.microsoft.com/fwlink/?LinkId=23454.

Targeted Customer Profile
This overall solution was designed toward medium to large organizations that
require basic highavailability and disasterrecovery features inherent in a Mailbox
Resiliency configuration, while minimizing administration complexity. The System
x3650 and XIV Gen3 Storage System offer scaleout flexibility to expand the
solution as the Exchange requirements increase in number of users, mailbox size,
or server failure tolerance. The target environment profile for this solution includes:

 24 Exchange 2010 servers (12 tested)
 2 backend storage subsystems (1 tested)
 2 DAGs
 2 copies/database
 120,000 mailboxes
 0.13 IOPS user profile (0.11 + 20% headroom)
©

 1.5GB mailbox quota
 8 Databases per Server
 1.7TB maximum database size tested
 Mailbox Resiliency provides high availability and is the primary data
protection mechanism. XIV Gen3 RAID provides for data protection against
physical loss/failure
 24x7 Background Database Maintenance is utilized

Tested Deployment
The following tables summarize the testing environment:

Simulated Exchange Configuration:
Number of Exchange mailboxes 120,000
simulated
Number of Database Availability 2
Groups (DAGs)
Number of servers/DAG 12
Number of active mailboxes/server 5000
Number of databases/host 8
Number of copies/database 2
Number of mailboxes/database 625
Simulated profile: I/O’s per second 0.13 (0.11 with 20% headroom
per mailbox (IOPS, include 20% tested)
headroom)
Database LUN size 2TB
Log LUN size 300GB
Total database size for performance 188TB
testing (per XIV Gen3 frame)
% storage capacity used by 78%
Exchange database**

**Storage performance characteristics change based upon the utilization
percentage of the individual disks. Tests that use a small percentage of the storage
(~25%) may exhibit reduced throughput if the storage capacity utilization is
significantly increased beyond what is tested in this paper. Actual (hard) capacity
used will vary due to the way the XIV Gen3 Storage System physically allocates
disk space. See the previous section for further information on how XIV Gen3
allocates & tracks physical disk space.

©

Storage Hardware
Storage Connectivity (Fiber Channel, SAS
SAS, SATA, iSCSI)
Storage model and OS/firmware XIV Gen3 Storage System
revision Model:2810 Machine type 114
XIV Gen3 Storage System Code
Version 11.0.0
Storage cache 360GB
Number of storage controllers 6
Number of storage ports 24
Maximum bandwidth of storage 96Gbit (12 X 8Gb)
connectivity to host
Switch type/model/firmware revision 2 Brocade 48000 (64 Ports) (8Gb)
Switch Firmware Version: 6.1.0
HBA model and firmware Storage:
Emulex 8GB FC PCI Express Adapter:
Model: LPE12002 Fibre
Channel
Operation Speed: 2, 4 &
         8Gbps)
HBA Firmware Version:
Firmware image's version:
         U3D200X7r3 (2/3/2011)

Host:
Qlogic QLE2562 8Gb dualport FC
HBA (PCIe)
Number of HBA’s/host 2
Host server type 24 x IBM System x3650 servers each
with two quadcore 3.0 GHz Intel
Xeon processors
Total number of disks tested in 180
solution
Maximum number of spindles can be 180
hosted in the storage

©

Storage Software
HBA driver Qlogic Storport 9.1.9.25
HBA QueueTarget Setting Default
HBA QueueDepth Setting 128
MultiPathing Microsoft Multipath I/O (Native to
Windows Server 2008), round robin
mode
Host OS Windows Server 2008 R2 –
Enterprise Edition SP1
(6.1.7601.65536)
ESE.dll file version 14.00.0639.019
Replication solution name/version N/A

Storage Disk Configuration (Mailbox Store Disks)
Disk type, speed and firmware SAS, 7,200 RPM
revision
Raw capacity per disk (GB) 3.0TB
Number of physical disks in test 180
Total raw storage capacity (GB) Actual: 480TB Net: 243TB
Disk slice size (GB) N/A
Number of slices per LUN or number 180
of disks per LUN
Raid level RAID X (IBM XIV Gen3 RAID)
Total formatted capacity 188TB
Storage capacity utilization 78%
Database capacity utilization 75%

Storage Disk Configuration (Transactional Log Disks)2
Disk type, speed and firmware SAS, 7,200 RPM
revision
Raw capacity per disk (GB) 3.0TB
Number of Spindles in test 180
Total raw storage capacity (GB) Actual: 480TB Net: 243TB
Disk slice size (GB) N/A
Number of slices per LUN or number 180
of disks per LUN
Raid level RAID X (IBM XIV Gen3 RAID)

2
Utilizes same disks as Mailbox Store Databases
©

Total formatted capacity 188TB

Best Practices
Exchange server is a diskintensive application. Based on the testing run using the
ESRP framework, we recommend the following to improve the storage performance.

For Exchange 2010 guidance on storage design, please visit:

http://technet.microsoft.com/enus/library/dd346703.aspx

Storage - Mailbox Resiliency
1. To accurately size final formatted volume size on the XIV Gen3 Storage
System, stage the target volume size prior to implementing. By default, the
minimum volume extent size is created in blocks of 17 GB. To get close to
target volume size, divide the target size by extent size of 17 GB. For
example, if the target volume size is 650 GB, divide 650/17 = 38 Extents
(equals to about 646 GB). After partitioning, and formatting volume, the final
size will be reduced even more. Factor in another 17 GB, to compensate for
the filesystem overhead. 646 GB + 17 GB will provide 663 GB unformatted
drive space. Format the newly created volume and check the available or
free space to determine if this meets target needs. Otherwise, increase or
reduce the volume to get a more accurate target size.

Note: "* 1GB=1000MB, 1MB=1000KB, 1KB=1000 bytes"

663GB translated is actually 663x1000x1000x1000/1024/1024/1024 = 617GB
on windows.

2. For optimal performance, format the database volumes using a 64k Windows
allocation unit size. The volumes used to store log data can be formatted
using default allocation unit size, as there are no significant gains increasing
this from the default value.
3. Increasing the Queue Depth setting on the HBA Fibre Controller may increase
overall IOPS. Refer to your manufacturer’s HBA documentation for detailed
support on how this value can be modified.
4. Increasing the number of FC HBA’s can provide additional IOPS and resiliency
options. Make sure to not define more than 24 paths to the storage, as this
the current limit at this time.
5. We strongly recommend implementing Mailbox Resiliency. This allows for
using a fewer number of larger LUNS for your databases and yields
significantly increased storage performance. Designing your Exchange XIV
Gen3 storage configuration with fewer and larger LUNs is often preferable,
and we have observed performance and capacity increases on the order of
25% when configured in this manner.
©

6. Troubleshooting any latency on the XIV Gen3 Storage System can be
monitored via standard tools such as Microsoft Performance Monitor or the
XIV Gen3 Storage System monitoring tools. These tools can help determine
where and when the heaviest disk I/O patterns are occurring, and what the
data transfer rates are, which helps identify and isolate potential I/O
bottlenecks.
7. Assuming all storage components are functioning, balanced, and configured
correctly, most Exchange Server 2010 storage latency issues are caused by
an insufficient number of disks for a given I/O load. The number of IOPS
hitting a given drive can be measured using the Performance Monitor Logical
Disk object, Disk Transfers/sec counter. The average disk latency should be
less than 20ms, with the maximum value no higher than 100ms. Disk latency
can be measured using the Performance Monitor Logical Disk object, Avg
Disk sec/Read. Since the XIV Gen3 Storage System uses all 180 disks for
I/O, and is dynamically selftuning, this concern is all but eliminated versus
conventional storage systems.
8. During our testing, the storage subsystems were dedicated to the Exchange
Server workload. Sharing the storage with other applications may negatively
impact Exchange I/O performance.
9. This solution used two database copies, which afford a mailbox resiliency
level of one server failure. We generally recommend no more than two copies,
given the modular, redundant design of the XIV Gen3 Storage System
coupled with RAIDX protection.
10.To obviate the need for a lagged database copy, consider deploying a
solution that leverages XIV Gen3’s instantaneous hardwarebased FlashCopy
capabilities, such as Tivoli Storage FlashCopy Manager.
11.For performance tips and bestpractice guidelines on the XIV Gen3 Storage
System, please visit IBM Techdocs.
12.For detailed technical information on the XIV Gen3 Storage System, please
visit IBM Redbooks.

Planning and monitoring your Exchange storage solution based on these guidelines
will help ensure optimal client response times, allow room for growth, and avoid
potential performance issues.

Backup strategy
N/A

Contact for Additional Information
For additional information regarding the IBM storage components used in this
solution, please visit:

https://www912.ibm.com/ssg/tsfeedback.nsf/feedback?OpenForm

©

Test Result Summary
This section provides a high level summary of the test data from the ESRP test runs
and the links to the detailed html reports which are generated by the ESRP testing
framework. Please click on the underlined headings below to view the html report
for each test.

Reliability
A number of tests in the framework are designed to check storage system reliability;
these tests run for 24 hours. The goal is to verify that the storage can handle high
Exchange I/O load for an extended period of time. Both the log and database files
are analyzed for integrity after the stress test to ensure that no database and/or log
corruption has occurred.

The following list provides an overview: (clicking on the underlined word will display
the html report generated after the reliability test runs)

 There were no errors reported in the saved eventlog file.
 There were no errors reported during the database and log checksum process.

Storage Performance Results
The primary storage performance testing is designed to exercise the storage with
the maximum sustainable Exchange type of I/O for 2 hours. The test is designed to
show how long it takes for the storage to respond to an I/O under load. The data
below is the sum of all of the logical disk I/O’s and average of all the logical disks
I/O latency in the 2 hour test duration. Each server is listed separately and the
aggregate numbers across all servers are listed as well.

Individual Server Metrics:
The sum of I/O’s across Storage Groups and the average latency across all Storage
Groups on a per server basis.

©

Server 1: XIV23
Database I/O
Database Disks Transfers/sec 1296.098
Database Disks Reads/sec 800.943
Database Disks Writes/sec 495.154
Average Database Disk Read Latency 17.754875
(ms)
Average Database Disk Write Latency 2.205
(ms)
Transaction Log I/O
Log Disks Writes/sec 461.318
Average Log Disk Write Latency (ms) 0.5435

Server 2: XIV25
Database I/O
(ms)
(ms)
Transaction Log I/O

Server 3: XIV20
Database I/O
(ms)
(ms)
Transaction Log I/O

©

Server 4: XIV121
Database I/O
(ms)
(ms)
Transaction Log I/O

Server 5: XIV122
Database I/O
(ms)
(ms)
Transaction Log I/O

Server 6: XIV30
Database I/O
(ms)
(ms)
Transaction Log I/O

Server 7: XIV31
Database I/O
©

(ms)
(ms)
Transaction Log I/O

Server 8: XIV32
Database I/O
(ms)
(ms)
Transaction Log I/O

Server 9: XIV21
Database I/O
(ms)
(ms)
Transaction Log I/O

Server 10: XIV33
Database I/O
(ms)
(ms)
©

Transaction Log I/O

Server 11: XIV27
Database I/O
(ms)
(ms)
Transaction Log I/O
Log Disks Writes/sec 469

Server 12: XIV24
Database I/O
(ms)
(ms)
Transaction Log I/O

Aggregate Performance across all servers Metrics:
The sum of I/O’s across servers in solution and the average latency across all
servers in solution.

Database I/O
(ms)
(ms)
Transaction Log I/O
©


Database Backup/Recovery Performance
There are two tests reports in this section. The first one is to measure the
sequential read rate of the database files, and the second is to measure the
recovery/replay performance (playing transaction logs into the database).

Database Read-only Performance
The test is to measure the maximum rate at which databases could be backed up
via VSS. The following table shows the average rate for a single database file.

MB read/sec per database 66.20625
MB read/sec total per server 529.65

Transaction Log Recovery/Replay Performance
The test is to measure the maximum rate at which the log files can be played
against the databases. The following table shows the average rate for 500 log files
played in a single storage group. Each log file is 1 MB in size.

Average time to play one Log file (sec) 1.560762

Conclusion
This testing validates this Exchange ESRP – Storage submission for the IBM XIV
Gen3 Storage System under the stated configuration parameters, and
demonstrates that this storage array is an ideal repository for Exchange Server
2010 data.

The new availability of 3TB SAS drives within the XIV Gen3 platform aligns well with
Microsoft’s desire for larger mailbox capacities. These drives also allow for a smaller
environmental footprint, increasing the storage density envelope without create
sprawl. The larger storage capacities tested here also highlight the scalable
performance of XIV Gen3’s modular design & controller architecture. Testing also
confirmed the enormous benefits of the new cache size support of 240GB

The incredible ease of management, advanced data distribution technology, self
tuning capability, and automatic (and extremely quick) drive rebuild capability offer
Exchange administrators and storage architects new tools to deliver a robust, yet
easily managed Exchange storage solution. Advanced features such as copy
services, thin provisioning, full MPIO & VSS support offer additional flexibility and
support in terms of performance, availability, quick backup/recovery, and ensure
optimal, efficient usage of the storage subsystem with minimal fuss.

©

This document was developed by storage solution providers, and reviewed by the
Microsoft Exchange Product team. The test results/data presented in this document
are based on the tests introduced in the ESRP test framework. Customers should
not quote the data directly for his/her predeployment verification. It is still
necessary to go through the requisite exercises to validate the storage design for a
specific customer environment.

ESRP program is not designed to be a benchmarking program; tests are not
designed to obtain the maximum throughput for a giving solution. Rather, it is
focused on producing recommendations from vendors for the Exchange application.
So the data presented in this document should not be used for direct comparisons
among the differing vendor solutions.

Appendix A. Test Results
This section contains Jetstress reports for Stress, Performance, Streaming Backup,
and Soft Recovery. All server test results were reviewed by Microsoft, and had
similar performance results. We include the results from only one of the servers
here, to make this section more readable.

Stress Testing
Server 1: XIV24
Stress Test Result Report
Test Summary
Overall Test Result Pass
Machine Name XIV24
Test Description
Test Start Time   8/27/2011 6:57:47 PM
Test End Time 8/28/2011 7:01:05 PM
Collection Start Time   8/27/2011 7:01:02 PM
Collection End Time 8/28/2011 7:00:57 PM
Jetstress Version 14.01.0225.017
Ese Version 14.00.0639.019
Operating System Windows Server 2008 R2 Enterprise Service Pack 1
(6.1.7601.65536)
Performance Log C:Program FilesExchange
JetstressGen3ESRP3TBPerformance_2011_8_27_18_58_5.blg

Database Sizing and Throughput
Achieved Transactional I/O per Second 1308.736
Target Transactional I/O per Second 1000
Initial Database Size (bytes) 7868707766272

©

Final Database Size (bytes) 7916816433152
Database Files (Count) 8

Jetstress System Parameters
Thread Count 7 (per database)
Minimum Database Cache 256.0 MB
Maximum Database Cache 2048.0 MB
Insert Operations 40%
Delete Operations 20%
Replace Operations 5%
Read Operations 35%
Lazy Commits 70%
Run Background Database Maintenance True
Number of Copies per Database 2

Database Configuration
Instance1208.1 Log Path: L:1
Database: F:1Jetstress001001.edb


Database: G:1Jetstress003001.edb


Database: H:1Jetstress005001.edb


Database: I:1Jetstress007001.edb


Transactional I/O Performance

MSExcha I/O I/O I/O I/O I/O I/O I/O I/O I/O I/O I/O I/O
©

nge Datab Datab Datab Datab Datab Datab Log Log Log Log Log Log
Database ase ase ase ase ase ase Read Writ Reads Writes Read Write
==> Read Write Reads Writes Reads Write s es /sec /sec s s
Instances s s /sec /sec Avera s Aver Aver Aver Avera
Avera Avera ge Avera age age age ge
ge ge Bytes ge Late Late Byte Bytes
Laten Laten Bytes ncy ncy s
cy cy (mse (mse
(mse (mse c) c)
c) c)
Instance1 17.92 99.97 63.71 33876 3561 54.87 4598.
208.1   2 2.557 1 4 .015 6.62 0 0.51 0 8 0 335
Instance1 18.00 99.95 63.72 33830 3562 55.02 4600.
208.2   5 2.558 4 5 .059 8.27 0 0.51 0 7 0 956
Instance1 18.02 99.52 63.39 33847 3561 54.77 4606.
208.3   8 2.238 5 3 .292 5.8 0 0.51 0 6 0 01
Instance1 18.02 99.97 63.72 33826 3562 55.03 4601.
208.4   2 2.238 1 1 .793 7.13 0 0.51 0 2 0 905
Instance1 18.06 100.0 63.80 33823 3561 4595.
208.5   9 1.835 51 9 .398 5.44 0 0.51 0 55.08 0 61
Instance1 18.02 63.63 33833 3561 54.87 4597.
208.6   8 1.835 99.84 8 .656 3.88 0 0.51 0 2 0 904
Instance1 18.04 99.95 63.71 33844 3562 0.51 54.99 4592.
208.7   4 1.67 1 4 .596 6.63 0 1 0 9 0 4
Instance1 100.0 63.74 33859 3562 0.51 55.00 4601.
208.8   18.02 1.675 17 1 .572 2.17 0 1 0 9 0 569

Background Database Maintenance I/O Performance
MSExchange Database ==> Database Maintenance IO Database Maintenance IO Reads
Instances Reads/sec Average Bytes
Instance1208.1   31.399 261723.73
Instance1208.2   31.418 261733.75
Instance1208.3   31.401 261733.377
Instance1208.4   31.406 261743.944
Instance1208.5   31.41 261737.446
Instance1208.6   31.403 261744.104
Instance1208.7   31.404 261727.131
Instance1208.8   31.404 261729.266

Log Replication I/O Performance
MSExchange Database ==> Instances I/O Log Reads/sec I/O Log Reads Average Bytes
Instance1208.1   1.022 232393.35
Instance1208.2   1.026 232489.395
Instance1208.3   1.022 232522.197
©

Instance1208.4   1.026 232438.479
Instance1208.5   1.025 232496.338
Instance1208.6   1.022 232355.443
Instance1208.7   1.024 232365.829
Instance1208.8   1.026 232344.251

Total I/O Performance
nge Data Data Datab Datab Datab Datab Log Log Log Log Log Log
Database base base ase ase ase ase Read Writ Reads Writes Read Write
Instances s s /sec /sec Avera s Aver Aver Avera Avera
Avera Avera ge Avera age age ge ge
ge ge Bytes ge Late Late Bytes Bytes
Laten Laten Bytes ncy ncy
cy cy (mse (mse
(mse (mse c) c)
c) c)
Instance 17.92 131.3 63.71 88334 3561 11.7 54.87 2323 4598.
1208.1   2 2.557 71 4 .824 6.62 15 0.51 1.022 8 93.4 335
Instance 18.00 131.3 63.72 88334 3562 11.9 55.02 2324 4600.
1208.2   5 2.558 72 5 .432 8.27 19 0.51 1.026 7 89.4 956
Instance 18.02 130.9 63.39 88503 3561 11.6 54.77 2325 4606.
1208.3   8 2.238 26 3 .126 5.8 9 0.51 1.022 6 22.2 01
Instance 18.02 131.3 63.72 88311 3562 12.0 55.03 2324 4601.
1208.4   2 2.238 77 1 .122 7.13 27 0.51 1.026 2 38.5 905
Instance 18.06 131.4 63.80 88278 3561 12.1 2324 4595.
1208.5   9 1.835 61 9 .137 5.44 74 0.51 1.025 55.08 96.3 61
Instance 18.02 131.2 63.63 88365 3561 11.9 54.87 2323 4597.
1208.6   8 1.835 43 8 .979 3.88 58 0.51 1.022 2 55.4 904
Instance 18.04 131.3 63.71 88326 3562 12.0 0.51 54.99 2323 4592.
1208.7   4 1.67 55 4 .263 6.63 86 1 1.024 9 65.8 4
Instance 131.4 63.74 88310 3562 11.8 0.51 55.00 2323 4601.
1208.8   18.02 1.675 21 1 .854 2.17 09 1 1.026 9 44.3 569

Host System Performance
Counter Average Minimum Maximum
% Processor Time 10.786 6.542 13.071
Available MBytes 20617.206 20595 20707
Free System Page Table Entries 33555320.8 33555309 33555826
Transition Pages RePurposed/sec 0.000 0.000 0.000
Pool Nonpaged Bytes 56594561.6 56279040 56836096
Pool Paged Bytes 100133109 97992704 110641152

©

Database Page Fault Stalls/sec 0.000 0.000 0.000

Test Log
8/27/2011 6:57:47 PM Jetstress testing begins ...
8/27/2011 6:57:47 PM Preparing for testing ...
8/27/2011 6:57:56 PM Attaching databases ...
8/27/2011 6:57:56 PM Preparations for testing are complete.
8/27/2011 6:57:56 PM Starting transaction dispatch ..
8/27/2011 6:57:56 PM Database cache settings: (minimum: 256.0 MB, maximum: 2.0 GB)
8/27/2011 6:57:56 PM Database flush thresholds: (start: 20.5 MB, stop: 40.9 MB)
8/27/2011 6:58:05 PM Database read latency thresholds: (average: 20 msec/read, maximum: 100
msec/read).
8/27/2011 6:58:05 PM Log write latency thresholds: (average: 10 msec/write, maximum: 100
msec/write).
8/27/2011 6:58:14 PM Operation mix: Sessions 7, Inserts 40%, Deletes 20%, Replaces 5%, Reads
35%, Lazy Commits 70%.
8/27/2011 6:58:14 PM Performance logging started (interval: 15000 ms).
8/27/2011 6:58:14 PM Attaining prerequisites:
8/27/2011 7:01:02 PM MSExchange Database(JetstressWin)Database Cache Size, Last:
1944084000.0 (lower bound: 1932735000.0, upper bound: none)
8/28/2011 7:01:03 PM Performance logging has ended.
8/28/2011 7:01:03 PM JetInterop batch transaction stats: 424066, 424858, 423094, 425001,
425237, 423053, 424278 and 424054.
8/28/2011 7:01:04 PM Dispatching transactions ends.
8/28/2011 7:01:04 PM Shutting down databases ...
8/28/2011 7:01:05 PM Instance1208.1 (complete), Instance1208.2 (complete), Instance1208.3
(complete), Instance1208.4 (complete), Instance1208.5 (complete), Instance1208.6 (complete),
Instance1208.7 (complete) and Instance1208.8 (complete)
8/28/2011 7:01:05 PM C:Program FilesExchange
JetstressGen3ESRP3TBPerformance_2011_8_27_18_58_5.blg has 5762 samples.
8/28/2011 7:01:05 PM Creating test report ...
8/28/2011 7:01:45 PM Instance1208.1 has 17.9 for I/O Database Reads Average Latency.
8/28/2011 7:01:45 PM Instance1208.1 has 0.5 for I/O Log Writes Average Latency.
8/28/2011 7:01:45 PM Instance1208.1 has 0.5 for I/O Log Reads Average Latency.
©

8/28/2011 7:01:45 PM Test has 0 Maximum Database Page Fault Stalls/sec.
8/28/2011 7:01:45 PM The test has 0 Database Page Fault Stalls/sec samples higher than 0.
8/28/2011 7:01:45 PM C:Program FilesExchange
JetstressGen3ESRP3TBPerformance_2011_8_27_18_58_5.xml has 5750 samples queried.

Performance Testing
Server 1: XIV24
Performance Test Result Report
Test Summary
Overall Test Result Pass
Machine Name XIV24
Test Description
Test Start Time   8/27/2011 9:12:11 AM
Test End Time 8/27/2011 11:16:05 AM
Collection Start Time   8/27/2011 9:16:02 AM
Collection End Time 8/27/2011 11:15:49 AM
Jetstress Version 14.01.0225.017
Ese Version 14.00.0639.019
Operating System Windows Server 2008 R2 Enterprise Service Pack 1
(6.1.7601.65536)
Performance Log C:Program FilesExchange
JetstressGen3ESRP3TBPerformance_2011_8_27_9_12_29.blg

Database Sizing and Throughput
Achieved Transactional I/O per Second 1323.702
Target Transactional I/O per Second 1000
Initial Database Size (bytes) 7864370855936
Final Database Size (bytes) 7868707766272
Database Files (Count) 8

Jetstress System Parameters
Thread Count 7 (per database)
Minimum Database Cache 256.0 MB
Maximum Database Cache 2048.0 MB
Insert Operations 40%
Delete Operations 20%
Replace Operations 5%
Read Operations 35%
Lazy Commits 70%
Run Background Database Maintenance True
Number of Copies per Database 2

©

Database Configuration








Transactional I/O Performance
nge Datab Datab Datab Datab Datab Datab Log Log Log Log Log Log
Database ase ase ase ase ase ase Read Writ Reads Writes Read Write
Instances s s /sec /sec Avera s Aver Aver Aver Avera
Avera Avera ge Avera age age age ge
ge ge Bytes ge Late Late Byte Bytes
Laten Laten Bytes ncy ncy s
cy cy (mse (mse
(mse (mse c) c)
c) c)
Instance1 17.54 102.2 63.25 33753 3705 0.51 58.75 4631.
208.1   5 2.594 65 7 .996 7.21 0 3 0 2 0 439
Instance1 17.60 102.7 63.52 33843 3705 0.51 59.33 4627.
208.2   7 2.579 07 5 .157 6.74 0 7 0 1 0 456
Instance1 17.62 102.5 63.49 33814 3710 0.51 59.15 4653.
208.3   9 2.248 99 9 .582 2.18 0 6 0 1 0 879
Instance1 17.63 2.26 102.6 63.54 33921 3708 0 0.51 0 58.98 0 4666.
©

IBM XIV Gen3 Storage with 3TB Drives:120,000 Mailbox Resiliency for MS Exchange 2010

IBM XIV Gen3 Storage with 3TB Drives:120,000 Mailbox Resiliency for MS Exchange 2010

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (15)

Similar a IBM XIV Gen3 Storage with 3TB Drives:120,000 Mailbox Resiliency for MS Exchange 2010

Similar a IBM XIV Gen3 Storage with 3TB Drives:120,000 Mailbox Resiliency for MS Exchange 2010 (20)

Más de IBM India Smarter Computing

Más de IBM India Smarter Computing (20)

Último

Último (20)

IBM XIV Gen3 Storage with 3TB Drives:120,000 Mailbox Resiliency for MS Exchange 2010