Note to Presenter: This provides the benefits of using EMC FAST Suite for Oracle OLTP databases in an optimized scalable virtual environment. The Oracle Real Application Cluster (RAC) 11g database is configured to access EMC® VNX7500™ storage using the Oracle direct NFS (dNFS) Client with enhanced availability. Oracle VM provided the virtualization platform. EMC’s SnapSure™ technology and the Oracle dNFS clonedb feature enable rapid provisioning of Oracle databases.
Please review the companion whitepaper “EMC VNX7500 SCALING PERFORMANCE FOR ORACLE 11gR2 RAC ON VMWARE VSPHERE 5.1”
on the “Everything Oracle” Community - https://community.emc.com/docs/DOC-20771
We will discuss the following items:
Business case
Solution overview
Architecture design
Test scenarios and results
Conclusion
Oracle mission-critical applications for your business have service levels that require high performance, a fast end-user experience (low latency) and resilience. As a result, Oracle environments must address an increasingly broad range of business demands, including the ability to:
Scale Oracle online transaction processing (OLTP) workloads for performance.
Maximize performance while reducing the cost of ownership of the system.
Traditional methods of using Flash drive technology are manual, complex, repetitive, and disruptive
EMC FAST Suite automatically and nondisruptively tunes an application, based on the access patterns.
FAST Cache services active data with fewer Flash drives, while FAST VP optimizes disk utilization and efficiency with Serial Attached SCSI (SAS) and Near-Line SAS (NL-SAS) drives.
Meet rapid on-demand Oracle provisioning requirements to create, deploy, and manage numerous production, development, and testing environments.
This solution addresses all these challenges for a scalable virtualized Oracle RAC 11g database deployment.
This solution uses the following technologies to support the demands of the growing enterprise infrastructure:
EMC® VNX®7500 series
EMC Unisphere®
EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP)
EMC FAST Cache
EMC SnapSure™ checkpoint
VMware vSphere
Oracle Direct NFS Client (dNFS)
Oracle dNFS clonedb
This slides shows the environment profile of the solution.
With VMware vSphere version 5.1 installed, the ESXi server farm consists of two ESXi servers; four virtual machines (two on each ESXi server) were deployed as a four-node RAC database. As Oracle Support's suggestion, we deployed Oracle RAC 11.2.0.3 for this virtualized solution. The storage and cluster interconnect networks used 10 Gigabit Ethernet (GbE).
This slide details the hardware resources for the solution.
Vault disk:
There drivers are also referred to as the system drivers. The first 4 drivers 0 through 3 in DAE0 in a VNX that contain the saved write cache in the event of a failure, the storage system's operating system files, the Persistent Storage Manager(PSM), and the FLARE configuration database, the NAS OS Disks.
This slide details the software resources for the solution.
The disk configuration uses four back-end 6 Gb SAS ports within the VNX7500 storage array.
The diagram shows a logical representation of the layout of the file system used for the Oracle data files. We used four Data Movers in a 2+2 active/standby configuration. Two active Data Movers were used to access the file systems, which were distributed evenly across the four SAS ports. The back-end configuration was based on the I/O requirements.
The table details the Oracle file system storage allocation on VNX7500. All the RAID groups were created on 300 GB 10k SAS drives.
VNX7500 is a member of the VNX series next-generation storage platform, which is powered by Intel quad-core Xeon 5600 series processors and delivers five 9s availability. The VNX series is designed to deliver maximum performance and scalability for enterprises, enabling them to dramatically grow, share, and cost-effectively manage multi-protocol file and block systems.
The VNX Operating Environment (VNX OE) allows Microsoft Windows and Linux/UNIX clients to share files in multi-protocol NFS and Common Internet File System (CIFS) environments. VNX OE also supports iSCSI, FC, and FCoE access for high-bandwidth and latency-sensitive block applications.
FAST Cache is part of the FAST Suite for VNX arrays, which also includes Fully Automated Storage Tiering for Virtual Pools (FAST VP).
FAST VP automatically moves data to the most appropriate storage tier based on sustained data access and demands over time. FAST Cache automatically absorbs unexpected spikes in application workloads, providing immediate performance benefits for burst-prone data.
FAST Cache and FAST VP can be used alone or together.
This solution demonstrates the benefits of FAST Cache only.
FAST Cache
FAST Cache uses Flash drives to add an extra layer of cache between DRAM cache and rotating disk drives, thereby creating a faster medium for storing frequently accessed data. FAST Cache is an extendable, read/write cache. It boosts application performance by ensuring that the most active data is served from high-performing Flash drives and can reside on this faster medium for as long as is needed.
FAST Cache is most effective when application workloads exhibit high data activity skew. This is where a small subset of data is responsible for most of the data set's activity. Fast Cache is more effective when the primarily block reads and writes are small, fits within the 64K FAST Cache track. The storage system is able to take advantage of such data skew by dynamically placing data according to its activity. For those applications whose data sets exhibit a high degree of skewing FAST Cache can be assigned to concentrate a high percentage of application IOPs on Flash capacity.
FAST Cache tracks data activity at a granularity of 64 KB and promotes hot data into FAST Cache by copying it from the HDDs to the Flash drives assigned to FAST Cache. Subsequent I/O access to that data is handled by the Flash drives and is serviced at Flash drive response times—this ensures very low latency for the data.
As data ages and becomes less active, it is flushed from FAST Cache to be replaced by more active data.
A small number of Flash drives implemented as FAST Cache provides a greater performance increase than a large number of short-stroked HDDs.
FAST Cache is particularly suited to applications that randomly access storage with high frequency, such as Oracle OLTP databases. In addition, OLTP databases have inherent locality of reference with varied I/O patterns. Applications with these characteristics benefit most from deploying FAST Cache.
High locality of data is important to realize the benefits of FAST VP. When FAST VP relocates data, it moves the entire slice to the new storage tier. To successfully identify and move the correct slices, FAST VP automatically collects and analyzes statistics before relocating data. Customers can initiate the relocation of slices manually or automatically by using a configurable, automated scheduler that can be accessed from the Unisphere management tool. The multi-tiered storage pool allows FAST VP to fully use all three storage tiers: Flash, SAS, and NL-SAS.
Like FAST Cache, FAST VP works best on data sets that exhibit a high degree of skew. FAST VP is very flexible and supports several tiered configurations, such as single tiered, multi-tiered, with or without a Flash tier, and FAST Cache support. Adding a Flash tier can locate “hot data” on Flash storage in 1 GB slices.
FAST Cache provides read/write caching using a private RAID 1 LUN consisting of Flash disks. A LUN on a RAID group without Flash disks can use the FAST Cache if FAST Cache is enabled for the LUN.
Prerequisites for using FAST Cache
The storage system must have the FAST Cache enabler installed.
The storage system must have Flash disks that are not already in a storage pool.
FAST Cache must be configured on the storage system.
FAST Cache must be enabled for the RAID group LUNs and/or the storage pools that are to use the FAST Cache
These are the steps to configure FAST Cache on the VNX:
Step 1- Analyze The Application Workload The decision to implement FAST Cache should only be made after the application workload characteristics are measured and analyzed. Array-level tools are available to EMC field and support personnel for determining both the suitability of FAST Cache for a particular environment and the right size FAST Cache to configure. Contact your EMC sales teams for guidance.
Whether a particular application will benefit from using FAST Cache, and what the optimal cache size should be, is determined by the size of the application’s active working set, the IOPS requirement, the RAID type, and the read/write ratio. The workload characteristics of OLTP databases make them especially suitable for using FAST Cache. For further information, refer to the white papers: EMC CLARiiON, Celerra Unified, and VNX FAST Cache and Deploying Oracle Database on EMC VNX Unified Storage.
Step 2 - Create FAST Cache You can create FAST Cache using the Unisphere GUI:
Access the FAST Cache tab under System Management > Manage Cache and click Create.
In the Create FAST Cache dialog box, specify the FAST Cache drive criteria (RAID type and number of disks) and select the disks to be used for FAST Cache. The Disks section lists the disks available for FAST Cache and their properties—the system can select the FAST Cache disks for you, based on the specified drive criteria, or you can select them manually.
When the create operation is complete, the storage-system write cache and the FAST Cache are enabled.
Step 3 - Enable FAST Cache at the LUN Level To enable or disable the FAST Cache for a LUN, display the LUN properties in Unisphere and select or deselect the FAST Cache option, as shown in the second figure.
Create FAST VP.
To create FAST VP using the Unisphere GUI, access the Storage Pool link under Storage > Storage configuration, and click Create to Create a new Fast VP pool or choose the existing pool click Expand for more different tiers.
Configure FAST VP.
Set the Tiering Policy for each LUN if necessary to the following:
Auto-tier (recommended)
Highest available tier
Lowest available tier
No data movement
Set the Auto-Tiering policy to scheduled or manual on demand
The goal of the node scalability test is to demonstrate the performance scalability with both nodes and users scaled out on Oracle RACs with dNFS and 10 GbE in a virtualized environment. An OLTP-like workload was run against a single node, users and nodes were then added.
We used Swingbench to generate the OLTP-like workload. The testing included the following steps:
Ran the workload on the first node by gradually increasing the number of concurrent users from 50 to 250 in increments of 50.
Added the second node into the workload, and ran the same workload as the previous step on each node separately, which means the total users scaled from 100 (50 on each node) to 500 (250 on each node).
Repeated the previous two steps after adding the third and fourth nodes separately.
For each user iteration, we recorded the front-end IOPS and TPM and captured the performance statistics from Oracle AWR reports and Unisphere for VNX.
The Swingbench workload was run against the database from four RAC nodes at the same time, and scaled it from 250 concurrent users to 750 on each node.
In this test, the write hit ratio increased to 92 percent while the read hit ratio increased gradually to 89 percent after a warm-up period of approximately four and a half hours.
When enabling FAST VP, five Flash drives were added to the data pool as RAID 5, the TPM increased by about 20 percent and stabilized at around 290,000, and the read response time was reduced by 42 percent.
When enabling FAST Cache, we used four Flash drives. The TPM surged to around 510,000 and stabilized at that level. The response time was dramatically decreased to less than 10 ms. See the Wait statistics from Oracle AWR reports section for detailed analysis from the database side.
The other two test results in Figure 17 show the performance of combining FAST Cache and FAST VP, which are two complementary technologies. When configuring four Flash drives for FAST Cache and five Flash drives for FAST VP, the TPM is slightly higher than that of using four Flash drives for FAST Cache only, and the read response time was reduced by 14 percent accordingly. When configuring two Flash drives for FAST Cache and five Flash drives for FAST VP, the TPM is slightly lower than when using four Flash drives for FAST Cache only, and the read response time was tripled.
The TPM and Read response time comparison demonstrated the big performance improvement using EMC FAST cache or FAST Suite.
Oracle foreground wait statistics highlight potential bottlenecks in Oracle RAC environments.
The figures show that the I/O performance was greatly improved when using FAST Cache or FAST Suite combination - the average wait time for db file sequential read decreased dramatically. Because of the increase of supported concurrent user transactions, the commit operations grew rapidly.
Oracle foreground wait statistics highlight potential bottlenecks in Oracle RAC environments.
The figures show that the I/O performance was greatly improved when using FAST Cache or FAST Suite combination - the average wait time for db file sequential read decreased dramatically. Because of the increase of supported concurrent user transactions, the commit operations grew rapidly.
Customers often need to clone a production database to develop and test new application patches. A new feature called dNFS clonedb enables users to clone test databases instantaneously.
Note To avoid failure of the dbms_dnfs.clonedb_renamefile, you must set cloned=true in the initialization parameter file for the cloned database.
Test Objectives
The goal is to outline the availability and resilience of the dNFS architecture by demonstrating the database availability during the physical NIC failure and a Data Mover panic.
Up to four network paths defined in the oranfstab file for an NFS server can be used with Oracle dNFS features. The dNFS Client performs load balancing across all specified paths. If one of the paths fails, dNFS reissues I/O commands over any other remaining paths.
Physical NIC failure
We manually shut down the NIC to simulate physical NIC failure.
When simulating a physical NIC failure, we observed no database outages because Oracle dNFS provided proactive failover operations when using multiple paths. In this solution, we configured two paths to each Data Mover. When one path was down, the other path was still available.
When shut down one of the physical NIC, Oracle dNFS completed the failover operation in two minutes automatically. When start up the physical NIC, the second path reconnected automatically and workload rebalanced among available paths within one minute.
Data Mover panic
We manually failed over one Data Mover to the standby one to simulate a Data Mover panic.
The process of Data Mover failover and failback was within one minute and no database outage was observed. We checked the database status as well as the Swingbench status, no error was found in the database log and the Swingbench log.
This paper demonstrates a highly scalable system in a virtualized environment using Oracle 11g RAC deployed on Oracle VM virtualization platform with Fast Suite technology, SnapSure checkpoint, and dNFS on EMC VNX 7500. It also shows the high availability of dNFS multipath and demonstrates the ease and quick provisioning of a production database by using Oracle dNFS clonedb.
The key findings of the testing performed for the solution demonstrate:
Efficiency
Automate Oracle performance tuning—Compared with the baseline, the performance enhancements offered by FAST Suite include:
FAST Cache—Creating a FAST Cache with four Flash drives, the performance of the TPM improves by 133 percent and the average read response time reduces from 96.49 ms to 5.57 ms. Using FAST Cache as a secondary cache delivers a 247 percent improvement in IOPS.
FAST VP—Enabling FAST VP by using only five Flash drives can improve the performance by 38 percent while reducing the average read response time from 96.49 ms to 55.51 ms. It delivers a 62 percent improvement in IOPS.
FAST VP and FAST Cache (configuration 1)—Combining FAST Cache and FAST VP using seven Flash drives increases the performance by 124 percent and decreases the average response time from 96.49 ms to 18.4 ms. It delivers a 213 percent improvement in IOPS.
FAST VP and FAST Cache (configuration 2)—Combining FAST Cache and FAST VP when using nine Flash drives increases the performance by 135 percent and decreases the average response time from 96.49 ms to 4.78 ms. It delivers a 260 percent improvement in IOPS.
Performance
Scale OLTP workloads—The TPM increased almost linearly when adding additional RAC nodes. Customers can take this solution as a baseline or foundation and scale it in a flexible, predicable, and near-linear way, by adding additional storage network, front-end ports, and RAC nodes, to provide higher throughput, based on the configuration in this solution.
Performance improvement with FAST Suite:
2 times improvement in transactions per minute (TPM)
3.5 times improvement in IOPS
20 times decrease in average read response times
92 percent hit ratio after a warm-up period of FAST Cache
Agility
Rapid provisioning of Oracle databases—In comparison with the traditional way of database cloning, using EMC SnapSure checkpoint, the Oracle dNFS clonedb feature can quickly and simply provision database clones for test/development purposes, minimizing the impact on the performance of the production database. In addition, it saves DBA time and reduces the storage requirement.
Resilience
Automatic failover—dNFS Client optimizes multiple network paths to not only load balance I/O across all available storage paths but also provide high availability. EMC VNX 7500 integrates with the Oracle dNFS feature seamlessly to provide the high database availability. The database is still alive during the physical NIC failure and the data mover panic.
The key findings of the testing performed for the solution demonstrate:
Efficiency
Automate Oracle performance tuning—Compared with the baseline, the performance enhancements offered by FAST Suite include:
FAST Cache—Creating a FAST Cache with four Flash drives, the performance of the TPM improves by 133 percent and the average read response time reduces from 96.49 ms to 5.57 ms. Using FAST Cache as a secondary cache delivers a 247 percent improvement in IOPS.
FAST VP—Enabling FAST VP by using only five Flash drives can improve the performance by 38 percent while reducing the average read response time from 96.49 ms to 55.51 ms. It delivers a 62 percent improvement in IOPS.
FAST VP and FAST Cache (configuration 1)—Combining FAST Cache and FAST VP using seven Flash drives increases the performance by 124 percent and decreases the average response time from 96.49 ms to 18.4 ms. It delivers a 213 percent improvement in IOPS.
FAST VP and FAST Cache (configuration 2)—Combining FAST Cache and FAST VP when using nine Flash drives increases the performance by 135 percent and decreases the average response time from 96.49 ms to 4.78 ms. It delivers a 260 percent improvement in IOPS.
Performance
Scale OLTP workloads—The TPM increased almost linearly when adding additional RAC nodes. Customers can take this solution as a baseline or foundation and scale it in a flexible, predicable, and near-linear way, by adding additional storage network, front-end ports, and RAC nodes, to provide higher throughput, based on the configuration in this solution.
Performance improvement with FAST Suite:
2 times improvement in transactions per minute (TPM)
3.5 times improvement in IOPS
20 times decrease in average read response times
92 percent hit ratio after a warm-up period of FAST Cache
Agility
Rapid provisioning of Oracle databases—In comparison with the traditional way of database cloning, using EMC SnapSure checkpoint, the Oracle dNFS clonedb feature can quickly and simply provision database clones for test/development purposes, minimizing the impact on the performance of the production database. In addition, it saves DBA time and reduces the storage requirement.
Resilience
Automatic failover—dNFS Client optimizes multiple network paths to not only load balance I/O across all available storage paths but also provide high availability. EMC VNX 7500 integrates with the Oracle dNFS feature seamlessly to provide the high database availability. The database is still alive during the physical NIC failure and the data mover panic.
Links to White Paper : EMC Protection Platform for Oracle Exadata
:Everything Oracle Community: https://community.emc.com/docs/DOC-19000
