This document provides an overview of the Symmetrix 8000 Enterprise Plus storage systems. It describes the system architecture including components like channel directors, disk directors, cache memory and Enginuity software. It also outlines various data protection options available on the systems like mirroring, RAID parity and remote data replication using Symmetrix Remote Data Facility. Dynamic sparing is also mentioned for improving storage utilization and availability.
5. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Chapter 1
Introduction
Overview This technical overview provides information on the EMC Symmetrix® 8000 Enterprise Plus
Storage systems, including product descriptions and details of key features and operations.
This overview also describes EMC’s Symmetrix underlying storage system architectural
philosophy. The objective is to provide IT management and staff with a thorough technical
understanding of Symmetrix Enterprise Plus Storage systems.
EMC Enterprise Plus The Symmetrix architecture is designed to deliver industry-leading capabilities for customers who
Differentiated Platform have requirements beyond what industry standard storage delivers. Symmetrix goes beyond
Capabilities delivering just high performance to delivering optimized performance across hundreds of
applications with various workload characteristics. Symmetrix is also designed for customers who
require not just server or storage consolidation but hyper-consolidation of everything in the data
center from open systems, to mainframe and AS/400, to everything else.
Hyper-consolidation also dictates that the architecture be able to scale to terabytes of
information and support petabytes of information as a single managed infrastructure. And as
the number of applications grows and the amount of information increases, the need to
automate common management tasks becomes critical. But the most critical component of an
Enterprise Plus storage system is the ability to deliver true fault tolerance and non-disruptive
business continuity. All this and more is capable with the Symmetrix 8000 Enterprise Plus
storage systems.
Optimized Performance Symmetrix systems use a global memory and one hundred percent cache fast writes to ensure the
highest possible performance when writing data. EMC proprietary caching algorithms
dramatically increase the probability for “cache hits” when reading data. Symmetrix systems can
determine data access patterns in real time and intelligently optimize themselves for the best
performance, independent of the host processor, operating system, and application. Symmetrix
8000 series systems incorporate evolutionary improvements of Symmetrix cache with multiple
memory regions for increased concurrency of memory operations and provide the highest system-
level performance in the industry.
Also, with the introduction of Symmetrix 8000, EMC has incorporated more powerful
microprocessors, introduced faster memory, and doubled the number of internal data buses.
The result of these evolutionary enhancements is an enterprise storage system that operates at
peak efficiency, adapts to a constantly changing business climate, and easily accommodates
Internet-driven growth.
Hyper-Consolidation The Symmetrix 8000 series supports every major connectivity interface in the industry,
including mainframe connections through ESCON and FICON, as well as connections to open
UNIX, Windows, and AS/400 systems with connectivity to SCSI and Fibre Channels. Adding
Symmetrix Enterprise Storage Platform (ESP) software to Symmetrix 8000 systems enables
simultaneous support of mainframe and open systems connections, a capability unmatched in
the industry. This level of Symmetrix connectivity enables simultaneous support of multiple
hosts and multiple host types for greater configuration flexibility and the fulfillment of EMC’s
differentiated platforms philosophy.
4
6. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Beyond just connectivity, Symmetrix also delivers infinite scalability. Symmetrix 8000 systems
enable consolidated storage strategies by providing scalable storage in a common family.
System capacities scale from 72GB to tens of terabytes of fully protected storage. Symmetrix
offers new ways to manage change and growth in applications, databases, servers, and overall
business requirements.
Ensure Information Protection Symmetrix provides a variety of hardware information protection features as well as optional
software applications. The Symmetrix 8000 architecture offers a choice of data protection at
the disk level: Mirroring, the optimal Redundant Array of Independent Disks (RAID) level for
both performance and availability; EMC’s enhanced parity protection; Symmetrix Remote
Data Facility (SRDF™); and Dynamic Sparing.
These basic data protection schemes are supported by full redundancy of data paths, Disk and
Channel Directors, and redundant power supplies with full battery backup to provide
protection against loss of data access due to component failure or power loss. All Symmetrix
8000 components are capable of non-disruptive replacement in case of a failure, enabling
Symmetrix 8000 systems to remain online and operational during component repair, with full
data availability.
Provide System Intelligence Traditional systems have placed the bulk of storage management decisions and overhead on
the operating system and host processor. Through its operating system-independent
technology, Symmetrix 8000 enables customers to consolidate storage from multiple
heterogeneous hosts. And since Symmetrix does not require specialized host device drivers,
customers can add new versions of operating systems and platforms while minimizing
operational impact. Since these capabilities are not tied to specific operating systems or
versions of operating systems, they can be exploited and do not require time-consuming and
costly software upgrades. These capabilities are used for virtually all major mainframe, UNIX,
Windows, PC LAN, and AS/400 systems without incurring host processor overhead.
The Challenge of Businesses today run at the speed of their information. Access to timely, robust information is
Differentiated Platforms a powerful asset that can fuel new ideas, boost revenues, build competitive advantage, and
enhance customer service. Yet in order to derive maximum business value from information,
companies must first unlock it from behind specific applications and processors across the
enterprise. No one can take full advantage of information that is isolated by different operating
systems and platform-specific data formats.
To drive better business results with technology, many companies are now consolidating their
information. Servers are being moved into the data center. Mainframes are being blended into
client/server environments. IT managers are acknowledging the wasted resources, expense, and
negative business impact of managing information across multiple operating environments
without a common management framework for the enterprise.
The Solution: EMC Symmetrix To realize an organizational vision of enterprise information, more and more IT departments
Enterprise Plus Storage are rejecting the notion of storage as an isolated CPU add-on or peripheral and searching for a
higher category of storage. They want storage that acts as a strategic element of an IT structure,
bridging the gaps between disparate platforms, so they can use their information in powerful
new ways. Beyond simply holding information, this storage must allow companies to manage,
protect, provide access to, and efficiently plan the growth of enormous amounts of information
previously dispersed on multiple servers and mainframes.
5
7. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
EMC Enterprise Plus Storage is answering the demand for enterprise information. Organized
through a suite of intelligent software capabilities, EMC Enterprise Storage™ is becoming a
fundamental technology enabler-as fundamental as networks, servers, and databases.
Symmetrix 8000 Series The Symmetrix 8000-series Enterprise Plus Storage systems provide a shared repository for a
Systems company’s most valuable resource—its information. Symmetrix 8000 systems provide the
industry’s highest performance, availability, and scalable capacity with unique information
protection, sharing, and management capabilities for all major open systems, mainframe, and
other environments.
There are currently three models in the Symmetrix 8000 family—the Symmetrix 8230, 8530,
and 8830. They form scalable families with leadership performance and capabilities in each of
their respective capacity classes. Additionally, Symmetrix Enterprise Plus systems deliver a
flexible and continuously upgradeable information infrastructure. Symmetrix Enterprise
Storage systems deliver the performance, capacity, and availability required to compete in
today’s information-centric marketplace.
Symmetrix 8830 Symmetrix 8530 Symmetrix 8230
* Up to 69.5TB of storage with the * Up to 17.4TB of storage with Up to 4.3TB of storage with full
throughput, capacity, and con- increased capacity and perfor- Symmetrix functionality in the
nectivity to support the largest mance for multiple applications smallest footprint ever
data center consolidations and
* 8-96 disk drives * 4-48 disk drives
information infrastructures
* Up to 64GB of cache * Up to 32GB of cache
* 32-384 disk drives
* Up to 64GB of cache
As a result companies can:
• Connect to heterogeneous environments, facilitating the storage and retrieval from all major
computing platforms, including mainframe and open systems environments
• Create a competitive advantage by leveraging large amounts of information
• Provide high-level performance, capacity, and availability
• Ensure business continuity in the event of a disaster
• Deliver rapid and non-disruptive data migration from one system to another
6
8. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Chapter 2
Symmetrix 8000 Enterprise Plus Storage Product Overview
EMC’s Architecture for EMC revolutionized storage in the mainframe environment with the introduction of the first
Enterprise Storage: Symmetrix in 1990. EMC became the first company to provide intelligent storage systems based
MOSAIC on redundant arrays of small, independent hard disk drives for the mainframe market. As a result,
businesses were able to access information more rapidly and reliably than ever before, and they
quickly began to view the strategic use of information as a competitive advantage. Today,
redundant array of independent disks (RAID) technology is widely accepted as the industry
standard for storage systems. In 1994, EMC extended Symmetrix technology to create the first-
ever platform-independent storage system, capable of simultaneously supporting all major
computer operating systems. Since the introduction of Symmetrix, more than 60,000 systems have
been shipped to customers around the world. In October 1999, Fortune magazine named EMC
one of the top-three “World’s Most Admired Companies” in its annual executive survey of product
quality and services.
Symmetrix is based on MOSAIC architecture, which is the field-proven time-tested foundation
for Symmetrix Enterprise Storage Plus functionality. The modular hardware architecture,
developed by EMC in the early 1990s, has enabled EMC to rapidly deploy the most advanced
technology, features and functionalities on high-performance Symmetrix platforms for a decade.
When advances in hardware, software, connectivity, or disk technology offer enhanced
capabilities, they are easily and economically integrated into Symmetrix family systems. The
basic system architecture can be continually enhanced as individual elements are added or
replaced. Designed-in investment protection is a hallmark of all EMC storage systems. As a
direct result of MOSAIC, EMC continues to introduce advanced technology and features into
the Symmetrix family, maintaining EMC’s lead in performance, data availability and
protection, mainframe and client/server integration, and many other customer requirements.
Customer Cache
Support Disk Scrubbing
Cache Center Scrubbing
Management
Configuration
Management Cache
Continuous
Traffic
Power
Management
Disk
Channel SCSI Interface
Adapters
Disk
Adapters PC Interface
Remote
Interface
Service
Expert Processor
Application Systems
Module
7
9. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Symmetrix System Operation Basic operations in the Symmetrix 8000 systems involve Channel Directors, Global
Memory Directors, Disk Directors, Disks, and the flow of data among these components, as
illustrated in the following architectural diagrams.
Symmetrix 8230 Architecture
8
10. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Symmetrix 8530 Architecture
Symmetrix 8830 Architecture
Channel Connectivity Symmetrix systems can be integrated easily and quickly with all major enterprise servers and
and Host Integration mainframes systems. Symmetrix 8230, 8530, and 8830 systems support connectivity to
mainframe and/or open systems hosts. Open systems platforms connect through SCSI and Fibre
Channel interfaces. Mainframe connectivity is supported through ESCON and FICON channels.
All Symmetrix systems are operating-system independent. The Enginuity™ Storage Operating
Environment is self-managed, and Symmetrix 8000 systems do not depend on host cache
commands to receive the benefits of read and write caching. This means that the Enginuity
Storage Operating Environment provides simultaneous connections for mainframes (IBM
OS/390 and zSeries), UNIX, Linux, Windows, and AS/400 (IBM iSeries) systems.
9
11. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
This specialized Storage Operating Environment enables combinations of ESCON Channel
Directors, FICON Channel Directors, Ultra SCSI Channel Directors, and Fibre Channel
Directors on the same Symmetrix system. For configuration flexibility, these Directors can be
installed in combination in the Symmetrix systems, facilitating the concurrent storage of
mainframe and open systems data in the same system.
EMC Symmetrix systems support connectivity options to a vast majority of host environments
that include all major open systems and mainframes hosts. For details of specific server models
and supported operating system versions and interface technologies, see the EMC Support
Matrix at www.emc.com/horizontal/interoperability/interop_support_matrices.jsp, or contact
your EMC sales representative.
Host Channel Connection All Symmetrix 8000 systems provide exceptional host channel connectivity through combinations
of Channel Directors. Each Channel Director supplies multiple independent data paths to global
memory, then to disk, from the host system. Channel Directors are installed in pairs, providing
redundancy and continuous availability in the event of repair or replacement to any one Channel
Director. These include ESCON channels, FICON channels, SCSI and Fibre Channels, and
Remote Link Directors.
Open Systems The Symmetrix 8000 systems support open UNIX systems, Linux, Windows NT systems,
Channel Directors TRU64, and AS/400 connectivity through Symmetrix Fibre Channel and SCSI Channel
Directors. Each SCSI Channel Director is a single board with four host connections. Fibre
Channel Directors have two to twelve connections per Director, and depending upon the
Symmetrix 8000 model, there are from two to eight Channel Directors per system.
Mainframe Channel Directors The Symmetrix 8000 systems support mainframe connectivity through ESCON Channel
Directors and FICON Channel directors. Each ESCON Channel Director supports four
ESCON channel connections, and each FICON Channel Director supports two FICON
channels.
Remote Link Directors The EMC Remote Link Director (RLD) facilitates the direct connection between two
Symmetrix systems in a Symmetrix Remote Data Facility (SRDF) or Symmetrix Data
Migration Services (SDMS) configuration. SRDF and SDMS mainframe implementations
require a minimum of two, and support a maximum of four RLDs in each connected system.
SRDF implementations can be either ESCON or Fibre Channel. SDMS implementations are
ESCON only. For open systems, SRDF over Fibre Channel implementations use Remote Fibre
Directors (RFD) for connecting Symmetrix systems using high-speed Fibre Channel links.
Disk Directors The Disk Directors manage the interface to the physical disks and are responsible for data
movement between the disks and global memory over the Symmetrix 8000’s four-bus memory
architecture. Symmetrix 8000 models have up to eight Disk Directors per system, each with
two advanced microprocessors. Each Disk Director is connected to two memory buses to
maximize data throughput and performance. Each logical data volume is connected to two of
the Symmetrix 8000’s Disk Directors to provide a redundant, or alternate, data path. Disks are
connected to Disk Directors through industry-standard SCSI interfaces. This allows rapid
introduction of the latest disk drive technology into Symmetrix systems.
Disk Drives Symmetrix systems use industry-standard SCSI disk drives for physical disks, allowing EMC to
keep pace with customer needs as technology enables increased capacities and improved
performance. Each hard disk drive is configured with its own controller consisting of control
logic, a microprocessor, and a device-level cache, designed to enable high-speed transfer
between the buffer on the hard disk drive and the Disk Director.
10
12. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Every disk drive contains its own microprocessor that has the capability of self-management.
This gives Symmetrix the ability to perform parallel tasks such as diagnosis and simultaneous
transfers, and further enhances performance.
Symmetrix 8000-series systems support mixed configurations of 36GB and 73GB (10K rpm),
and 181GB (7200 rpm) disks drives. This breadth of scalable capacity and configuration
choices allows Symmetrix systems to adapt to virtually any enterprise storage requirement.
Any combination of disk drives can be deployed in Symmetrix 8000 systems to provide the
exact combination of performance and capacity required.
Disk Scrubbing During idle time, the disks are read (“disk scrubbing”), looking for any type of error. Disk
scrubbing is accomplished in a manner similar to cache scrubbing, as described later.
Upon sensing a correctable error, the error is corrected and then rewritten. The block of data is
read again to verify that it was a permanent correction. If it is correctable, the pertinent
information is logged and scrubbing continues. If the error is not permanently corrected, the
process is repeated until it is either corrected or the error recovery routines determine that a
skip defect must be executed. If the skip defect must be executed, it is done via Symmetrix
Enginuity. When the skip defect is complete, notification is given, and the scrubbing process
continues. Should a threshold number of skip defects occur on a track that would make an
alternate track assignment necessary, that too is accomplished through Symmetrix Enginuity
and is transparent to the user.
Hyper-Volume Extension Symmetrix enhances disk system functionality by supporting up to 128 logical volumes on one
physical device. Logical volumes are the actual volumes with which a host communicates. The
hyper-volumes are configured upon initial Symmetrix setup. Additional hyper-volumes can be
dynamically added as the customer requires more capacity. Up to a maximum of 8,000 logical
volumes are supported on a Symmetrix system.
For mainframe customers, the standard IBM device types are supported, including all 3380 D,
E, and K’s and 3390 models 1, 2, 3, 9, and 27. Non-standard hyper-volumes can also be
defined for customers who desire them.
For the customer using Symmetrix in an open systems, UNIX, NT, or Linux environment,
hyper-volumes can be created as large as 15GB in size. For those customers needing larger
volume sizes than 15GB, EMC offers meta volume addressing.
Meta Volume Addressing Symmetrix also enhances disk system functionality in Windows NT and open systems UNIX
and Linux environments through meta volume addressing capability. A meta volume is a group
of logically connected hyper-volumes that creates a single logical view to a host. Symmetrix
supports up to 255 logically connected logical volumes. These logically connected hyper-
volumes are not required to be contiguous. This facility can be used to overcome the addressing
limitations imposed in Windows NT environments, where currently allowable volume size is
15GB. With Symmetrix system’s 255 logical volumes, meta volumes of up to 3.8TB are
possible.
Global Cache Director At the heart of EMC Symmetrix is the Global Cache Director with CacheStorm™ technology,
a multi-functional, high-performance, parallel-designed, solid-state subsystem that delivers
unmatched high-end performance and data integrity. CacheStorm technology enhances system
performance, improves responsiveness, and manages peak I/O requests through a series of
techniques that reduces contention for shared cache and optimizes utilization of system
resources. The underlying principles are fairly simple:
11
13. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
• Cache memory is partitioned into 16 separately addressable regions
• Requests for cache are expedited to reduce locking
• Requests are intelligently arbitrated to optimize available resource usage
CacheStorm consists of two major functional components, described as follows.
Parallel Cache The Symmetrix Global Cache Director with CacheStorm technology accommodates four
Memory Regions separately addressable, simultaneously accessible regions. So, in a Symmetrix system with four
cache directors, there are 16 separately addressable and accessible cache regions. Compared to
single region cashing, this greatly reduces the probability of contention for cache access that
results in cache queuing and lower performance.
CacheStorm ASICs The Global Cache Director expedites transactions between process requests and cache.
CacheStorm technology Application Specific Integrated Circuits (ASICs) on the Global Cache
Director act as intelligent offload engines to perform repetitive system critical functions.
One function ASICs performs is buffering service requests for cache. These buffers have a
region to store reads, a place to store writes, and an area to store address and
command/instructions. As soon as a process gets access to the cache region it needs to access,
the intelligent ASIC buffers the incoming request and frees up the cache region. Then, within
the ASIC, it performs the instructed operation e.g., read/write to cache. Buffering incoming
requests locally on ASICs and freeing up blocked cache regions as soon as possible results in a
truly non-blocking architecture that is capable of massive performance scaling.
CacheStorm ASICs also arbitrate incoming requests for cache resources in a way that optimally
allocates cache regions to incoming requests by appropriately timing and intelligently pre-
12
14. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
fetching required information from cache into the ASIC buffer. This results in optimal
utilization of available resources.
Instruction Set Logic, routine cache-related activities being requested from processors mounted
on channel (front-end) and disk (back-end) directors are built into the logic in the ASIC. This
expedites the process of cache-related transactions and reduces the time for which cache is to
be blocked for servicing a process request. The result is that cache requests are processed
through ASIC hardware instantaneously without waiting for PowerPC processors on the
channel and disk directors.
Proactive Cache Maintenance EMC makes every effort to provide the most highly reliable hardware in the industry. However,
all hardware is subject to the effects of aging and occasional failures. The unique methods used
by Symmetrix for detecting and preventing these hard failures in a proactive way set it apart
from all others in providing continuous data integrity and high availability.
Symmetrix 8000 actively monitors I/O operations for temporary errors. By tracking these soft,
or temporary, errors during normal operation, Symmetrix can recognize patterns of error
activity and predict a potential hard failure before it occurs. This proactive error tracking can
usually prevent an error in global memory by fencing off, or removing from service, a failing
memory segment before data errors occur.
Constant cache scrubbing to detect and correct single- and double-bit errors dramatically
reduces the potential for multi-bit or hard errors. In addition to monitoring recoverable
conditions during normal access, all locations in global memory are periodically read and
rewritten to detect, and correct, single- and double-bit errors. A Symmetrix system’s global
memory scrubbing technique maintains a record of errors for each memory segment.
If the predetermined error threshold is reached, the segment’s contents are moved to another
area in global memory, and the segment is ‘fenced’ and removed from service. A service
processor call-home function alerts EMC to the unacceptable level of errors, and a non-
disruptive memory replacement is ordered. A Customer Service engineer is dispatched with the
appropriate parts for a speedy repair.
Should a multi-bit error be detected during the scrubbing process, it is considered a permanent
error, and the segment is immediately fenced. Data affected by the error is recovered from disk
or flagged as invalid in the case of write-pending data. A service processor call home is placed
as previously noted.
Cache Chip-Level Redundancy Traditional cache memory systems usually provide for 8 bits of parity information to support
bit error correction and detection in a 64-bit long word. EMC’s Global Cache Director
incorporates Single Nibble Correction Double Nibble Detection. (A nibble is four consecutive
bits of information.) This is achieved by internally generating 16 bits of ECC parity
information and replacing existing 8 bits of incoming ECC information. This enables the
system to correct up to four bit errors associated with a 64-bit long word.
Symmetrix Global Cache Directors can also detect up to eight bit errors. Another benefit is that it
interleaves 64 bits of information plus 16 CacheStorm parity information (total 80 bits) across 20
13
15. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
memory chips on the cache board. This results in each memory chip storing only a nibble of
information corresponding to a word. So, a chip-level error will disable access only to the nibble
stored on that faulty chip. However, CacheStorm enables regeneration of data from the faulty chip.
This leads to chip-level redundancy making every chip on the cache memory board redundant.
Longitude Redundancy Symmetrix Global Cache Directors also incorporate Sector Level longitudinal redundancy
Code (LRC) Checks checks, which further assure data integrity. The check bytes are the XOR (exclusive OR) value
of the accumulated bytes in a 4KB sector. LRC checking can detect both data errors and
incorrect block access problems.
Cache Access Path Protection Before Symmetrix cache can accept data from a host connection, it must ensure that the area to
which the data is to be written is without error. Symmetrix assures the highest level of data
integrity by checking data validity through the various levels of the data transfer in and out of
cache.
Byte-Level Parity Checking All data and control paths have parity generating and checking circuitry that verify data
integrity at the byte or word level. All data and command words passed on the system bus, and
within each director and global memory board, include parity bits used to check integrity at
each stage of the data transfer.
System-Wide Error Checking Both channel and disk directors correct single-bit errors and detect and report double-bit
and Correction (ECC) errors. Error detection and correction circuits on each director continuously check all transfers
within Symmetrix.
A service processor call-home function alerts EMC Global Service Call Centers whenever an
unacceptable level of errors has been detected and a non-disruptive replacement is ordered.
Customer Service is immediately notified of all call-home alerts, and a customer engineer can
be dispatched with the appropriate parts for speedy repair. Even in cases where errors are
occurring and are easily corrected, if they exceed a preset level, the call home is executed. This
represents the EMC philosophy of not accepting any errors.
Efficient Use of Available In early design testing, EMC discovered that cache mirroring is an inefficient way of creating
Cache Memory redundancy for failsafe operations. Cache mirroring results in two cache operations in the case
of system read events and five cache operations in the case of system writes. In addition to this,
mirroring wastes 50 percent of useful memory on the mirror. EMC analysis revealed that
memory boards themselves do not fail, however, memory chips on memory boards start
misbehaving over time. This leads to a design to ensure that each and every chip on the memory
board is redundant - eliminating any single point of failure on cache boards. This also results in
higher utilization of available memory resources resulting in higher system throughput.
To achieve the goal of making each and every memory chip redundant on the memory board, 8
bits of extra parity information are stored in addition to usual 8-bit parity information that
goes with a 64-bit long word. The result is 10 percent of extra memory capacity to create chip-
level redundancy as compared to 50 percent waste in the case of mirrored cache boards.
14
16. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Online Maintenance Every Symmetrix is configured with a minimum of two global memory directors to allow for
and Replacement online hot replacement of a failing board. If a hard error is detected, or the temporary errors
reach a predetermined threshold, the Symmetrix service processor calls home to request an
immediate maintenance action. When board replacement is required, global memory usage is
redirected to the remaining good boards in the system, and the suspect board is removed and
replaced non-disruptively while the system remains online.
Cached Data Protection Symmetrix Enterprise Storage systems provide 100 percent system non-volatility. If there is
any power interruption, EMC’s fully redundant battery backup system fully powers the entire
system, flushes the cache, completes all pending writes, parks the drives, and gracefully powers
the system down into a known good state. Symmetrix batteries are “N+1” and are not only
voltage tested but also continuously “load tested” as part of the normal internal preventive
monitoring performed by the Symmetrix to ensure the highest level of data protection.
Enginuity: EMC’s Storage The Symmetrix Enginuity storage operating environment consists of over 1.6 million lines of
Operating Environment system software executing on over 61,760 MIPS of processing power (EMC Symmetrix 8830).
Enginuity orchestrates all hardware, onboard functionality (such as SRDF, TimeFinder, Data
Mobility, etc.) and application workloads concurrently, while maintaining the highest levels of
end user responsiveness and system availability.
The combination of Symmetrix hardware architecture and Enginuity operating system software
has been continuously updated over time to deliver advancements across all aspects of storage
operations, including performance, functionality, connectivity, capacity, and availability.
Customers’ real-world workloads are very different than most benchmarks used to measure
the performance envelopes of many competing storage subsystems. Real-world workloads are
composed of many different types of I/O activity. They can be read or write requests, they have
different data block sizes, they can be skewed (some disks or host channels doing more work
than others), they can be highly random, sequential or mixed, and they are often “bursty”
(peak reads or writes can come at unexpected times). The workloads used for envelope
measurements are normally static, simple, and designed to always yield certain levels of hit
ratio (access of r/w data directly out of cache), regardless of the cache size and algorithms. In
real life, the actual application behavior is greatly influenced by the performance optimization
algorithms.
Enginuity contains extensive algorithmic intelligence that is designed to achieve the following goals:
• Maximize the read hit (read access from cache memory) ratio...leading to fast application
response time
• Minimize data de-stages to the disks...improving write hit (write access to cache memory) ratios,
optimizing use of internal resources and improving response time
• Avoid extreme situations...to not over consume and to optimize use of internal resources
• Allow end user definition (and future assignment) of priorities for Symmetrix operations...to set
service levels for specific workloads
• Be Efficient...to reuse valuable information for multiple purposes, balance the load evenly
among Symmetrix components, and save valuable resources
• Be proactive...to identify patterns or sequences as soon as possible to optimize operations
• Optimize data layout based on detection of long-term workload patterns
15
17. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Mainframe Host
Directory
Open Systems Host
Cache
Memory
Channel Director Disk Director Disk
Symmetrix 8000 Systems
Optimized Data Flow Symmetrix 8000 models optimize the movement of data for the highest performance possible.
There are four internal buses-top high, top low, bottom high, and bottom low. Symmetrix 8000
systems greatly exceed the throughput and response time performance of conventional disk
storage systems, because the majority of data is transferred to and from global memory at
electronic memory speeds, not at the dramatically slower speeds of physical disk devices.
Director boards, both those connecting to a host and those connecting to the disks, are the
means by which data interfaces with global memory. Director boards are designed to work in
pairs, where each director is connected to two buses. This ensures access to data in the event of
an unlikely failure of any bus.
Symmetrix 8000 systems optimize the flow of data between hosts and disks by:
• Minimizing the number of accesses to the disks
• Executing I/Os in an order that minimizes the time the disks spend for seek and latency, whenever
disk access is unavoidable
Optimizing Response Times The data inside Symmetrix is logically organized in tracks. These tracks are organized into
logical volumes, which are presented to hosts. All data travels through the global memory
directors. The global memory is logically divided into slots. A slot in global memory is
associated with a track of data. A slot may contain an entire track of data, or just part of it.
The slots in the Symmetrix global memory are divided into three logical groups. This division
of data is very flexible. A cache slot can move from one group to another by merely changing a
few pointers without having to move any data.
1. Least Recently Used (LRU) Chain
An LRU chain is a bi-directional linked list dynamically sorted by age of the linked slot. The LRU
chain is the main contributor to read hits. The Symmetrix supports multiple (up to sixteen) simul-
taneous LRU chains. The LRU in these chains are de-staged to the disk in order to create more
room in global memory.
16
18. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
2. Permacache
Permacache is a collection of cache slots that is “permanently” associated with tracks. These
tracks contain critical information that needs rapid response whenever it is needed. Users can spec-
ify which tracks need an association with Permacache. In addition, whenever Enginuity storage
operating environment running on Symmetrix systems can predict that certain data is likely to be
accessed extensively in the near future, it creates a Permacache association for that piece of data.
3. Write Pending Slots and Write Pending Indicators (WPI)
Write pending slots contain data that was written to global memory but has not been destaged to
disks. These slots are removed from the LRU chain. The WPI indicates which slots are waiting for
a disk destage.
Depending on the I/O pattern at any moment, the portion of cache dedicated to the LRU or to
Write Pending varies significantly. The tracks designated by the user to be Permacache remain
in Permacache until the user changes their designations. The other Permacache tracks, those
that were automatically selected by Symmetrix, will change their status automatically when
the likelihood of reusing them does not justify their Permacache status.
Symmetrix Read Four basic types of operations occur in a Symmetrix system: Read Hit, Read Miss, Fast Write, and
and Write Operations Delayed Fast Write. The following diagrams illustrate these operations.
Read Hit A Read Hit occurs on a read operation when all data necessary to satisfy the host I/O request
is in global memory. The Channel Director immediately transfers the requested data from
global memory to the host and updates the cache directory. Since the data is in global memory,
there are no mechanical delays, and data is transferred at electronic speeds. With the large
amounts of global memory offered on Symmetrix 8000 systems, it is common for applications
to attain a read hit ratio (requested data is in global memory) of 90 to 95 percent.
Read Hit
1] Directory Search- Hit
2] Transfer to
Host
3] Update
Directory
Host Channel
2 1
Directory
3
Channel Cache
Director
Disk Director Disk
17
19. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Read Miss In a Read Miss, data necessary to satisfy the host I/O request is not in global memory, so it must
be retrieved from disk. The Disk Director reads the block(s) containing the data from disk,
transfers them to global memory, and updates the cache directory. Simultaneously, the Channel
Director reconnects to the host and transfers the requested data to the host.
Read Miss
1] Directory
Search- Miss
2] Position
Read/Write
Head, Stage
Data to Cache
3] Transfer to
Host
Host Channel 4] Update
Directory
3 1
Directory
4
Channel Cache
Director
2
Disk Director Disk
Fast Write A Fast Write occurs whenever there is global memory available to accept the data being written.
On a host write command, the Channel Director places the incoming block(s) directly in global
memory and immediately sends a ‘write complete’ message to the host. Since Symmetrix Fast
Writes are complete when the data is written to global memory, there are no mechanical delays.
The Disk Director will asynchronously write the data to disk.
Fast Write
1 Search-hit cache directory
2 Transfer to Cache
3 Update directory
Host Channel 4 Destage asynchronously
Directory
Channel
Director Cache
Disk Director Disk
18
20. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Delayed Fast Write A Delayed Fast Write occurs only when the Fast Write threshold has been exceeded (that is, the
percentage of global memory containing modified data, unwritten to disk, is too high to
accommodate the Fast Write data). The Disk Directors immediately destage data to disk as a high-
priority task. When sufficient global memory space is available, the Channel Director processes the
host I/O request as a Fast Write. With sufficient global memory installed, this type of global
memory operation will rarely occur.
Delayed Fast Write
1 Search cache directory
(cache is full)
2 Destage page
3 Update cache directory
Host Channel 4 Transfer to cache
5 Update directory
6 Destage asynchronously
Directory
Channel
Director Cache
Disk Director Disk
Destaging Operation A background operation also occurs in Symmetrix systems. This background operation destages
blocks of data to disk. Frequently used data is maintained in two locations: global memory for high
performance in the occurrence of reuse of that data and on disk to maintain the highest levels of
data integrity. All pending writes are assured of arrival to the intended disk even in the event of
power failure. (See the Non-Volatile Power System section.) The following diagram illustrates this
destaging operation.
Destaging Operation
1] Destage
Block(s)
2] Update Directory
Host Channel
Directory
2
Channel Cache
Director
1
Disk Director Disk
19
21. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Enginuity Performance Simply having these robust cache configurations is not enough. One of the fundamental differences
Optimization Algorithms between Symmetrix products and all other data storage systems is the advanced caching
algorithms that allow intelligent use of the installed global memory for high performance. A
potential problem with increasingly large global memory configurations is that search time
increases proportionally, since this search time is added to every I/O request, read hit, read miss, or
write. This is a considerable penalty for every I/O request, especially in performance-critical
applications. In some data storage systems, the controller may actually disconnect from the
channel during this process and must then reconnect if there is a cache hit.
Symmetrix systems perform the global memory search via advanced patented algorithms,
determining-in microseconds-if a record is in global memory. As well as searching quickly and
efficiently to determine whether the requested data is in global memory, they also understand how
the application is accessing the data and tune themselves accordingly in real time. These advanced
algorithms allow the search time to remain constant regardless of application workload.
With global memory searches performed at electronic speed, there is no reason to disconnect
from the channel during the search. In fact, it takes longer to disconnect and reconnect than it
does to perform the global memory search. In normal operation, the only time that a
Symmetrix system will disconnect from the channel is in the event of a read miss. This is a
complex series of tasks and requires the advanced global memory management algorithms of
Symmetrix to be accomplished effectively.
Symmetrix global memory management is based on the principle that the working-set of data
at any given time is relatively small when compared to the total system storage capacity. When
this working-set of data is in global memory, there is a significant improvement in I/O
performance. The performance improvement achieved is dependent on both:
• Locality of Reference-If a given piece of data is used, there is a high probability that a nearby
piece of data will be used shortly thereafter.
• Data Reuse-If a given piece of data is used, there is a high probability that it will be reused
shortly thereafter.
This cache principle has been in use for years on host processor systems. The following figure
illustrates this type of host cache use. The cache used in this manner is often a high-speed, high-
cost storage unit used as an intermediary between the CPU and main storage.
CPU Cache Memory
Intelligent Prefetch Algorithm This algorithm prefetches data from disks to the cache before the host issues a read command
to this data, in anticipation that the host will shortly want to read this data. It works by
identifying sequential reads. EMC’s prefetch algorithm will reduce response time and improve
the utilization of the disks. The prefetch algorithm maintains, per each logical volume, an array
of statistics and parameters based on the latest sequential patterns observed on the logical
volume. Prefetch dynamically adjusts based on workload demand across all resources in the
backend of the Symmetrix. This algorithm also ensures that cache resources are never overly
consumed in order to maintain optimal performance.
20
22. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Enginuity algorithms continually monitor I/O activity and proactively look for access patterns.
When a second sequential I/O to a track read occurs, the sequential prefetch process is invoked, and
the next track of data is automatically read into global memory. The intent of this process is to avoid
a Read Miss by anticipating the data that will be requested. Once the first track is completely read
by the host processor, the third track is read and reuses the same global memory location as the first.
This process of using the cache track slots in a round-robin fashion prevents cache pollution
caused by conventional sequential caching algorithms. Should a Read Miss occur, the
Symmetrix global memory management will increase the number of track slots read from two
to five. If a Read Miss still occurs, the Symmetrix prefetch routines will continue to increase the
number of track slots read. The maximum number of track slots that will be allocated for a
sequential operation is 12. Should I/O activity reduce, the number of track slots will be reduced
accordingly. When the host processor returns to a random I/O pattern, the Symmetrix system
will discontinue the sequential prefetch process.
Whenever the workload presented to the storage system contains sequential read patterns, it is
very beneficial to prefetch data from the disks to the cache before this data is actually requested
by the host. This helps in two major ways:
• If the data resides in cache when the host is actually reading it, then the response time for this
operation is reduced by about 10 times. Reading from cache takes a few hundred microseconds,
while accessing the physical disk takes several milliseconds.
• The utilization of the physical disk drive is improved, since large portions of data are read from
the disk each time, seek and latency times are reduced to almost zero.
It’s no wonder that all storage vendors employ a prefetch algorithm to achieve these
improvements. However, a bad prefetch algorithm can have a devastating effect on the overall
performance of the system. For sequential I/O performance measurements, most benchmarks
use workloads with very long sequences. Even a simple prefetch algorithm can be made to look
good in these situations. But, in real-life cases where sequences are of various lengths,
customers want a sophisticated and self-adjusting algorithm that on one hand, does not
prefetch too much, and on the other hand, prefetches all the data that is needed and does it on
time without affecting the response times of the other I/Os.
Various storage vendors use different approaches to prefetching. Most vendors use a very
simple algorithm: they prefetch a very large (e.g., 1MB) amount of data from disks to cache
upon detecting a certain number of sequential read operations. Some of the simple algorithms
are very aggressive about prefetch. They prefetch after detecting a sequence of two I/Os. Others
are more conservative. They start to prefetch only after detecting a sequence of eight I/Os.
The Symmetrix adaptive intelligent algorithm automatically adjusts to the workload and
constantly monitors the success rate of its decisions. In real-life workloads, the Symmetrix
approach is significantly superior to the others. The conservative approach fails to detect 90
percent of the sequences, and thus fails to use the disks more efficiently and improve host
response times. The aggressive approach may prefetch significant amount of data that will
never be used by the host computers.
Least Recently Cache Least Recently Used (LRU) is a list of slots (a pre-defined piece of cache that relates to
Used Algorithm data areas on disk) with application data that was recently used. Numerous studies have
proven that data that was more recently accessed has a higher chance of being accessed again
shortly. The LRU algorithms in Symmetrix are designed to maximize hit ratio in the most
efficient manner. There are sixteen independent LRUs in a Symmetrix system.
21
23. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
Write Pending Indicator Cache Write Pending Indicator controls all the slots that have written data that has not been
destaged to the disks. Like the Read Hit case, numerous studies have proven that data that was
written recently has a higher chance of being written again shortly. Therefore, it is beneficial to
keep this data in cache before it is de-staged to the disk. The write destage algorithm constantly
adjusts itself to the existing workload. It is designed to improve the overall performance by
taking into account the effect of keeping written data in cache on the Read and Write Hit ratios
and by optimizing the order in which the tracks are being destaged.
In Symmetrix, the preferred mode of data protection is RAID 1. In RAID 1, each logical volume is duplicated
into at least two mirrors. Each mirror resides on a different hard drive or drives. In most cases, the different
mirrors reside on different disk directors that are serviced by different memory buses. This duplication of
pathing allows Symmetrix to decide from which mirror the data should be read. Symmetrix allows users to
manually set the mirror service policy for each logical volume. However, because workloads change over
time, and because the number of logical volumes in a system is permanently growing, setting one policy as
optimal, or close to being optimal, is practically impossible. When the user sets the Mirror Service Policy
(MSP), he or she determines which of the mirrors of a given logical volume should service a Read Miss
operation.
The two possible policies are:
• M1/M2: One of the mirrors should service all the reads from this logical volume.
• Interleave: The different mirrors alternate on each cylinder. Mirror 1 (M1) serves the odd num-
bered cylinders, while Mirror 2 (M2) serves the even numbered cylinders.
Generally speaking, the Interleave policy benefits sequential patterns, because under this
policy, all the physical drives transfer data. The M1/M2 policy benefits random patterns,
because it limits the distance the disk actuator needs to travel.
DMSP is a dynamic approach to setting the optimal mirror service policy. The DMSP
algorithm monitors the access patterns to the different logical volumes in the back-end, and
based on these access patterns, determines a policy for the next short time interval. As of
Enginuity 5x68, DMSP takes into account all the local mirrors of the logical volume, including
its Business Continuance Volumes (BCVs). The DMSP algorithm tries to achieve two goals:
• Balance the load among all the disks and other Symmetrix back-end components.
• Minimize the time the physical drives spend on seek and latency.
22
24. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
The challenge is to achieve the two goals simultaneously or to achieve the goal that is more
relevant to the current situation. Assume, for example, that a mirrored physical drive has two
logical volumes, one doing 10 I/Os per second, while the other is doing 40 I/Os per second.
Intuition will mislead us to use a policy that will balance the load between the drives. That
way, each physical drive will execute 25 reads per second, 20 from one logical volume, and 5
from the other. But deeper analysis, or simple disk simulation, proves that in this case, we will
be much better off if each physical drive serves one logical volume. This is so because whenever
I/Os are limited to a smaller portion of the disk, the disk performance is much improved, and
because executing 40 I/Os per second on a physical drive does not create any significant
queues. If the expected load on the logicals was doubled (80 and 20), then the considerations
may be different, based on the physical disk characteristics.
The DMSP algorithm has three distinct stages:
• The first stage is geared towards load balancing the different Symmetrix components. These
components include the Disk (DA) directors, the interfaces to the disk drives, and the disk
drives themselves.
• The second stage starts with the policy determined by the first stage, and derives from it several
other potential policies in which seek and latency times are improved.
• The third stage uses a simple simulation to evaluate all the policies produced at the previous
stages, taking into account the actual characteristics of the workload, like random versus sequen-
tial, write percentage, etc. The policy that scores the best is chosen for the next time interval.
Back-End Layout Optimization Similar to DMSP, SymmOptimizer is designed to improve disk utilization by balancing the
or SymmOptimizer load and minimizing the disk seek time. While DMSP is focusing on the short term (every few
minutes), Optimizer examines the workload patterns over extended periods of time and
optimizes disk performance for the long term. It achieves this by moving logical volumes to
different disks or to different locations on the same disk. Decision making data is collected at a
granularity of 5-15 minute intervals. The optimization algorithm module uses this data to
identify overloaded physical volumes, or hot spots. It then determines a series of logical
volume moves that would relieve these hot spots. The data-moving module is responsible to
control the actual moving of logical volumes on the physical drives.
Like the DMSP algorithm, SymmOptimizer is designed to improve disk utilization by
balancing the load among the hard drives, while minimizing the disks’ seek and latency times.
DMSP focuses on the near real time. It examines the workload patterns of the last few minutes
and sets the mirror service policy for the next few minutes. SymmOptimizer, on the other hand,
examines the workload patterns over extended periods of time, usually days or weeks, and
optimizes disk performance for the long term. It does this by moving logical volumes to
different disks or to different locations on the same disk.
SymmOptimizer has three modules:
• Data collection
• Optimization algorithms
• Data moving
23
25. EMC SYMMETRIX 8000 ENTERPRISE PLUS STORAGE SYSTEMS PRODUCT DESCRIPTION GUIDE
The data collection module collects back-end activity statistics of each logical volume and of
each physical drive. The optimization algorithm module uses this data to identify overloaded
physical volumes or hot spots. It then determines a series of logical volume moves that would
relieve these hot spots. The data moving module is responsible for controlling the actual
moving of logical volumes on the physical drives.
The SymmOptimizer algorithm is based on a very interesting observation. The workloads that
run on a given Symmetrix vary over time. In general, the workload characteristics observed in
the last few minutes are a good predictor of the workload characteristics of the next few
minutes. This is the basis for DMSP. But beyond a few minutes, most of the workload
characteristics may change considerably. The one characteristic that is most stable in the
workloads running on the same Symmetrix is the activity correlation between the logical
volumes. If logical volumes X and Y are active at the same times today, they are very likely to
be active at the same times tomorrow. Similarly, if logical volume X is not active when Y is,
then there is a very good chance that this correlation will remain.
Given this observation, the SymmOptimizer goal puts highly correlated volumes on different
hard drives as much as possible. A second goal puts the busiest logical volumes in the most
optimal location on the drive, which is close to the outermost tracks. A third goal is that, if
positively correlated volumes need to reside on the same hard drive, then they should reside
close to one another. All these goals are translated to a cost function that the SymmOptimizer
algorithm tries to minimize.
The SymmOptimizer algorithm performs two functions. Based on the cost function described
above, the SymmOptimizer algorithm first calculates an optimized layout of data on the
physical drives. Next, the SymmOptimizer algorithm calculates an optimal series of data
moving steps to achieve the desired layout. The focus of the second function is to execute the
moves in an order that yields better performance as soon as possible.
Quality of Service Quality of Service, or QoS, lets Symmetrix users control, to a great degree, the performance
level that selected applications receive from Symmetrix. The settings of Quality of Service can
be adjusted at any time to adapt to a system’s I/O requirements. For instance, by reducing the
“quality of service” for BCV or SRDF copy operations on selected devices, customers free
Symmetrix resources and increase the overall performance of the other Symmetrix devices.
One of these Quality of Service features, nLRU-QoS, enables users of Symmetrix systems to
allocate a portion of cache for a subset of the logical volumes. Being able to control how cache
is allocated guarantees that these logical volumes, and the applications they are used for,
achieve a high hit ratio, regardless of the other applications running at the same time. This
feature also lets customers specify when an application can lend portions of its cache to other
applications.
With the nLRU-QoS, customers can guarantee a certain level of performance for applications
or users that demand certain levels of performance, regardless of other applications running on
the system at the same time. The nLRU-QoS feature is implemented through the nLRU
mechanism. The cache slots can be divided among up to 16 independent LRU rings. Customers
can assign a different size for each LRU and map sets of logical volumes to sets of LRU rings.
Another QoS feature permits Symmetrix users to specify the time when a background activity,
such as a Copy, Backup, or BCV Establish, needs to complete. Customers set the time period
24
