Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Database 12c with
Real Application Clusters (RAC)
High Availability (HA) Best Practices
Markus Michalewicz
Director of Product Management
Oracle Real Application Clusters
July 15, 2014
@OracleRACpm
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz

Copyright © 2014 Oracle and/or its affiliates. All rights reserved.
Program Agenda
1
2
3
4
Introduction & Overview
Bronze-Level HA
Silver-Level HA
Gold & Platinum-Level HA
Oracle RAC & Virtualization
Additional Information
5
6

Eliminate risk of downtime
and data loss
High Availability (HA) Business Challenges
Reduce cost and increase
return on investment

A simplified Two-Step Implementation Approach
Standardize
Set of reference
HA architectures
Reduce costs
Consolidate

Oracle Maximum Availability Architecture (MAA)
Edition-based Redefinition,
Online Redefinition, DG / GG
– Minimal downtime upgrades & migrations
Active Data Guard
– Data Protection, DR
– Query Offload
GoldenGate
– Active-active replication
– Heterogeneous
Active Replica
RMAN, Oracle Secure Backup
– Backup to disk, tape or cloud
Application Continuity
– Application HA
Global Data Services
– Service Failover / Load Balancing
Enterprise Manager Cloud Control
– Coordinated Site Failover
RAC
– Scalability
– Server HA
Flashback
– Human error
correction
Production Site
ASM
– Storage management
Database
– Core Technology

Oracle Database 12c MAA
Four Standard Reference Architectures
GOLD
Comprehensive HA and Disaster Protection
Zero or near-zero data loss
SILVER
High Availability (HA) for Recoverable Local Outages
Data protected as of last backup
BRONZE
Single Instance Database, Basic Service Restart
PLATINUM
Zero Outage for Platinum Ready Applications
Zero data loss

BRONZE

SILVER

GOLD

PLATINUM

Standardize on Clusters
Clusters increase scalability and HA regardless of DB types used later
High Availability
Scalability
License Costs

Standardize on Oracle RAC
High Availability
Scalability
For all the flexibility you need

Standardize on Oracle RAC and Oracle Multitenant
High Availability
Scalability
The new standard for consolidation

15
Commonwealth Bank
 The Commonwealth Bank (CBA) is one of Australia’s leading providers of
integrated financial services including retail, business and institutional banking,
funds management, superannuation, insurance, investment and broking services.
The Bank is one of the largest listed companies on the Australian Stock Exchange.

16
Introduction
 In 2007 CBA set out to create an Oracle database shared service offering for the bank
• The offering has been highly successful by several measures
• Oracle-as-a-Service has continued to be developed through several iterations
 Oracle as a Service (OaaS) v1 – went live May 2008

17
 Host many Oracle database applications
on a cluster of hardware
 Processor consolidation
• Run each server hotter
• Take advantage of complimentary workload peaks.
 Higher Availability
• Load balancing
• HA failover for component failure
• Standby DR
• Most apps do not implement these features – too expensive
 Cost Reduction
 Better Service
• Full time experts
• Always on-call
 Reduced Risk
• Whole environment is managed
• Operated as a “business”
Oracle as a Service In A Nutshell

18
Reduce Risk, Improve Time to Market
 For new Projects:
• Remove a phase from the project – infrastructure already in place
• Remove reliance on expensive/scarce SME resources for design and build
• No longer need to manage risk associated with procurement and build
• Time to instantiate a new Production quality environment: 3 months -> 2 minutes.
 Example: New ISV Application introduced into our Online Share Trading platform
• Required to test performance under the workload
& data volume conditions projected in 2 years time.
Dedicated Infrastructure OaaS
Implementation Time 3-4 months few hours
$ Cost to Project Several hundred thousand < $10K
On Project Completion Under-utilized asset remains Environment turned-off

BRONZE
• Minimize the cost of HA
– Use HA features included with Oracle Database
– Utilize backups to protect against media failures
– Secure offsite tape storage (in the cloud) for DR
• Optionally
– Consolidate with Oracle Multitenant
– Use VMs for dedicated resources
– Self-Service provisioning with
Enterprise Manager Database as a Service
Bronze Level High Availability

BRONZE
High Availability at the Bronze Tier
Features Integrated with the Oracle Database
Physical Data Protection
Data Corruption Protection
& Oracle ASM (Mirroring)
Human Error Correction
Flashback Technologies: Drop,
Query, Transaction, Table, and
Database
Planned Maintenance
Online Redefinition
Online Reorganization
Edition Based Redefinition
Online File Move
Online Patching
Fast Backup & Recovery
Recovery Manager (RMAN)
Fast Recovery Area
Oracle Secure Backup

Oracle Database Inherent Protection
Data Corruption Protection
• Oracle Data blocks have a well-defined structure
• Block header is kept consistent with payload
• Enables validation of both physical
and logical intra-block consistency
• Oracle ensures block validity is maintained as it traverses I/O path
• Extensive corruption checks:
• DB_BLOCK_CHECKSUM:
http://docs.oracle.com/cd/E11882_01/server.112/e40402/initparams049.htm
• DB_BLOCK_CHECKING:
http://docs.oracle.com/cd/E18283_01/server.112/e17110/initparams046.htm
• For more information see My Oracle Support (MOS) note 1302539.1

Oracle Automatic Storage Management (ASM)
3rd Party FSApplication
Automatic Storage Management
ASM Cluster & Single Node File System (ACFS)
Database
ACFS Snapshot
ASM Disk
Group
DB Datafiles, OCR and Voting Files Oracle Binaries 3rd Party File Systems
Dynamic Volume Manager
ASM Instance Managing Oracle
DB Files
• ASM supports ALL data – database files, file systems, Clusterware files (OCR, Voting Disk)
• Built-in mirroring protects from disk failures
• Auto-repair of corrupt blocks using a valid mirror copy

Small ASM Features – Big Difference
Failure Group repair time
 Similar to existing disk repair time
 Default is 24 hours
 Disk group attribute: failgroup_repair_time
Disk Resync
 Accepts power limit
 Range 1 (least resources)
to 1024 (most resources)
 Checkpoints: failed operations
restart at the last checkpoint
Fast Disk Replacement
 Efficient, atomic operation
 No disk group reorganization required
Data Scrubbing
 Check for logical consistency
(and repair from mirror)
 during rebalance
 areas on demand
 Protects seldom-accessed data

Fine-grained Table Recovery
From Backup using a simple RMAN command
1 0 1
0 1 0 1 0
1 1 1 0 1 0 0 1
1 0 1 1 0 1 1 1 1 0
0 0 1 0 1 0 0 0 0
1 0 0 1 1 0 1
1 0 1 0
0 1 RMAN
Backups
• One simple RECOVER TABLE command to
• recover one or more tables
• from an RMAN backup (most recent or older)
• Eliminates time and complexity associated
with manual restore, recover & export
• Enables fine-grained point-in-time recovery of
• individual tables
• instead of the contents of the entire tablespace.

Cross-Platform Backup & Restore
Simplified Platform Migration
11101
10110111101
0010100001
1001
11101
10110111101
0010100001
1001
Source Database (AIX)
Backup
to Disk/Tape
(data files, optional
endian conversion,
metadata export)
Restore Backup
(optional endian
conversion,
metadata import)
Destination Database (Solaris)
• Simplified procedure for platform migration

Program Agenda
A
B
C
Silver-Level HA
Free-of-Charge Cluster Features
Stabilize your Cluster
Oracle RAC One Node as a Standard

SILVER
Silver Level High Availability
• Active-passive with Oracle RAC One Node
• Automatic failover
• Fast restart on a second node
• Real-time failover & scalability with Oracle RAC
• All nodes active at all times
• Zero downtime rolling maintenance
• Hardware and OS maintenance
• For qualified Oracle Database patches

Oracle ASM – Overview
Oracle Database 11.2 or earlier configuration
ASM Cluster Pool of Storage
Node2
Disk Group BDisk Group A
Node3 Node5Node4
Shared Disk
Groups
Wide File Striping
One to One
Mapping of ASM
Instances to
Servers
Node1
ASM Instance
Database Instance
ASM Disk
RAC Cluster
DBA DBA DBB DBB DBCDBB
ASM ASM ASM ASM ASM

Oracle Automatic Storage Management (ASM)
Oracle ASM 12c Standard Deployment
Disk Group BDisk Group AShared Disk
Groups
Wide File Striping
One to One
Mapping of ASM
Instances to
Servers
ASM Instance
Database Instance
ASM Disk
RAC Cluster
Node4Node3Node2Node1 Node5ASM ASM ASM ASM ASM
ASM Instance
Database Instance
DBA DBA DBCDBB DBBDBB

Introducing Oracle Flex ASM
Removal of One to One Mapping and HA
Databases share
ASM instances
ASM Instance
Database Instance
ASM Disk
RAC Cluster
Node5Node4Node3Node2Node1
Node1
runs as
ASM
Client to
Node2
Node1
runs as
ASM
Client to
Node4
Node2
runs as
ASM
Client to
Node3
ASM ASM ASM
ASM Instance
DBA DBA DBCDBB DBBDBB

ACFS – The ASM based Cluster File System
ACFS is part of Oracle GI, hosts all data and is fully integrated
Databases share
ASM instances ASM Instance
Database Instance
ASM Disk
Cluster
Node5Node4Node3Node2Node1 ASM ASM ASM
SI
DB
RAC
DBA
RAC
DBA
RAC
DBA
Applications use
ACFS for all data
ACFS ACFSACFS ACFS ACFS

Stabilize your Cluster - Reminder: Network Configuration
vixencomet
racdb1_3 racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
• Size Interconnect for aggregated throughput
– Foresee approx. 1Gb/sec per 32 cores
• 10GbE or IB are recommended for database
consolidation and large SMP nodes (use HA)
• Bonded 1GbE with load balancing
are most common and supported
– Use same type NICs for LB and failover
– Use different subnets (especially using HAIPs)
• Check for IP & NIC send and receive buffer size
• Use Jumbo frames wherever possible
• Interconnect should be stress
tested usign Iperf or netperf

Stabilize Your Cluster – Part 1
Avoid number of sessions related resource contention
vixencomet
racdb1_3 racdb1_4
Connection Pool
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
• Control the number of concurrent sessions
• Foreground processes are in time-share class
• Scheduling delays on high context switch rates
on busy systems may increase the variation in
the cluster traffic times
• More processes imply higher memory
utilization and higher risk of paging
•How to control concurrent sessions:
• Use connection pooling
• Avoid connection storms
(pool and process limits )
• Ensure that load is well-balanced over nodes

Optimize memory locally
• Avoid memory pressure!
– Paging and Swapping activity on one node
affects performance on all nodes
– Severe Paging and Swapping activity on
one node can cause instance evictions
• #1 cause for service disruptions in clusters
• Use Memory Guard
– Oracle Quality of Service Management (QoS)
feature – available in monitoring only mode
– Prevents new connections from coming in to a
server that is already under memory pressure
vixencomet
racdb1_3 racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

Use Solid State Disks to host swap space
• Use Solid State Disks (SSDs) to host swap
space in order to increase node availability
– Memory pressure can cause node evictions.
– Preventing memory pressure is the solution.
– If prevention is not successful and swapping
is performed by the Operating System (OS),
• hosting the swap space can mitigate the impact that
extensive swapping can have on cluster operations
on the on the affected server(s).
• More information:
– My Oracle Support Note Doc ID: 1671605.1 –
“Use Solid State Disks to host swap space in
order to increase node availability”
vixencomet
racdb1_3 racdb1_4
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Swapping

Configure Huge Pages for Oracle RAC
• Use Huge pages for SGA (Linux)
– Dramatic reduction in memory for page tables
– SGA pages pinned in memory
• More information:
– My Oracle Support note 361323.1 –
HugePages on Linux: What It Is... and What It Is Not...
– My Oracle Support note 401749.1 – Shell Script to
Calculate Values Recommended Linux HugePages /
HugeTLB Configuration
• Engineered systems provide templates for
pre-configuration of huge pages for the SGA

Oracle RAC One Node Oracle RAC One Node 12c provides:
1. Best in-class single instance
Oracle Database availability
2. Better database consolidation
3. Better database virtualization
Using
• A standardized and improved
deployment and management
• A familiar and matured HA stack
• Oracle Grid Infrastructure

Oracle RAC One Node Basic Features
• Oracle RAC One Node is
– an Oracle RAC enabled database
– with only one instance running
under normal operation
• In case of a server or instance failure,
the one running instance is started on
another server in the cluster (failover)
• Online Database Relocation
(Oracle RAC One Node specific feature)
minimizes downtime during scheduled
maintenance operations.

Oracle RAC One Node vs. Oracle RAC
“Always On” vs. “Always Running”
 Oracle RAC One Node: one instance
per database running at a time
 Oracle RAC: multiple instances
per database running concurrently

Online Database Relocation for better HA
Oracle RAC One Node-specific functionality that makes the difference
Node3Node2
Oracle (Flex) ASM based pool of shared storage
Node1
Public Network
DBB DBC
Oracle Grid Infrastructure
DBEDBD
DBA
DBAOnline DB Relocation
•Online Database Relocation minimizes downtime
during (scheduled) maintenance operations such as:
•Patching
•Resource re-allocation due to demand
http://www.oracle.com/goto/racone

Standardize on Oracle RAC (One Node)
To simplify Oracle RAC testing
vixencomet
RON RON_2
Online
upgrade
RON_1
AWR
---
ASH
AWR
---
ASH
Decide
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle RAC testing made simple:
1. Standardize on Oracle RAC (One Node)
2. Run tests on Oracle RAC One Node instance
• RAC One Node behaves basically like a single instance
3. Online Upgrade to Oracle RAC for testing only
4. Run tests on Oracle RAC
5. Decide whether the application provides
service goals when load balanced over multiple
active instances
• Identify bottlenecks and tuning actions as required

Standardize on Oracle RAC (One Node)
The simplest way of converting to RAC: unplug/plug
Node2
CDB Instance 2
Node1
CDB Instance 1
CDB

Oracle RAC
Oracle RAC 12c provides:
1. Better Business Continuity
and High Availability (HA)
2. Agility and Scalability
3. Cost-effective Workload
Management
Using
• A standardized and improved
deployment and management
• A familiar and matured HA stack

Oracle Database 12c MAA
Four Standard Reference Architectures
GOLD
Comprehensive HA and Disaster Protection
Zero or near-zero data loss
SILVER
High Availability (HA) for Recoverable Local Outages
BRONZE
Single Instance Database, Basic Service Restart
PLATINUM
Zero Outage for Platinum Ready Applications
Zero data loss
✔
✔

This is Oracle RAC
Storage Management
1. Oracle ASM
2. Oracle ACFS
Quality of Service
1. Quality of Service
Management (QoS)
2. Application Continuity
High Availability and Scalability
1. Oracle Clusterware
2. Oracle RAC (One Node)
Consolidation
1. Oracle RAC
with Oracle Multitenant
Core

Optimizing the use of DBs in virtual environments
Bare-Metal Server
DOM-0 Hypervisor
Guest
Dom-0
Guest
Bare-Metal Server
DOM-0Hypervisor
Dom-0
GuestGuest
DBC1DBA1
+1
DBC2 DBA2
+1
Oracle RAC provides additional benefit in any virtual environment
Core

Optimizing the use of DBs in virtual environments
Guest Guest GuestGuest
DBC1DBA1
+1
DBC2 DBA2
+1
Core

Providing an alternative to virtual environments
Node1 Node2 NodeNNode..
DBC1DBA1 DBC2 DBA2
+1+1
Core

Tools for a Lasting HA Experience
dasher dancer
vixencomet
LOGGERD
sysmond
sysmond
Oracle GI Oracle GI
Oracle GI Oracle GI
sysmond
sysmond
sysmond
LOGGERD
Cluster Health Monitor (CHM) – CHM/OS
1. First failure diagnostics: Collects as much
system metrics and data as feasible
2. Collects system metrics on all
nodes in the cluster synchronized
3. Persistent storage of all
collected data in a database
4. Integrated with Oracle Grid Infrastructure
starting with 11.2.0.2 (12c: uses GIMR)
5. Robust collection in situations
of resource starvation
6. Data dump or visualization

Tools for a Lasting HA Experience
Guest GuestGuest
DBC1DBA1 DB DBA2
• Trace File Analyzer (TFA)
– Improved comprehensive
first failure diagnostics
collection
– Efficient collection,
packaging and transfer
of data for customers
– Collect for all relevant
components (OS, Grid
Infra., ASM, RDBMS)
– One command to
collect all information
– More information:
MOS note ID 1513912.1
• OraChk
(formerly RACcheck)
– RAC Configuration
Audit Tool
– For details see MOS
note ID 1268927.1
– Checks the Oracle
Database:
– Standalone
Database
– Grid Infrastructure
& RAC
– Maximum
Availability
Architecture
(MAA) Validation

More Information: www.oracle.com/goto/rac
• Presentations:
•Oracle RAC 12c Collaborate Best Practices
•http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-collaborate-best-practices-ioug-2014-version
•Oracle RAC 12c Practical Performance Management and Tuning
•http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-practical-performance-management-and-tuningoow13con8825
•The Oracle RAC Family of Solutions
•http://www.slideshare.net/MarkusMichalewicz/the-oracle-real-application-clusters-rac-family-of-solutions
• White Papers:
•Oracle RAC 12c Best Practices
•http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-best-practices

• Database and Middleware streams available
• Videos
– Feature different groups and presenters
– Cover a broad range of topics and products
• Oracle RAC PM / Dev contributions so far:
– Oracle Flex Cluster: Optimized Resource Management for the Cloud - Ian Cookson
– Oracle Grid Infrastructure 12c Bundled Agents – Shankar Iyer
– ACFS Product Overview and Use Cases - Ara Shakian
– The Oracle Real Application Clusters (RAC) Family of Solutions - A User Guide – Markus M.
– Next Generation Oracle Automatic Storage Management - Jim Williams
– Implementing DBaaS with Oracle RAC 12c and Quality of Service Management - Mark Scardina
– Oracle RAC Practical Performance Management and Tuning – Markus Michalewicz
Oracle Learning Streams: http://education.oracle.com/streams/

• Presentation: Oracle RAC on Extended Distance Clusters
– http://www.slideshare.net/MarkusMichalewicz/oracle-rac-on-extended-distance-clusters
• WP: Oracle Real Application Clusters on Extended Distance Clusters
– http://www.oracle.com/technetwork/database/options/clustering/overview/extendedracversion11-435972.pdf
• WP: Using standard NFS to support a third voting file for extended cluster configurations
– http://www.oracle.com/technetwork/database/database-technologies/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
• Presentation: Oracle Clusterware Node Management and Voting Disks
– http://www.slideshare.net/MarkusMichalewicz/oracle-clusterware-node-management-and-voting-disks
More Information: www.oracle.com/goto/rac

Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices

Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (19)

Más de Markus Michalewicz

Más de Markus Michalewicz (20)

Último

Último (20)

Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices