This version of "Oracle Real Application Clusters (RAC) 19c & Later – Best Practices" was first presented in Oracle Open World (OOW) London 2020 and includes content from the OOW 2019 version of the deck. The deck has been updated with the latest information regarding ORAchk as well as upgrade tips & tricks.

1. Oracle Real Application Clusters
(RAC) 19c & Later – Best Practices
Markus Michalewicz
Senior Director of Database HA & Scalability Product Management
@KnownAsMarkus
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz
Copyright © 2019 Oracle and/or its affiliates.

2. The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
Copyright © 2019 Oracle and/or its affiliates.

3. Agenda
• What’s New in Oracle RAC?
• How to best upgrade to Oracle RAC 19c?
• Prepare for Changes

4. Agenda
• What’s New in Oracle RAC?
• How to best upgrade to Oracle RAC 19c?
• Prepare for Changes

5. Oracle RAC is Ready for a New Generation
Ease of use
established with
Oracle RAC 12.2
Best ever
scalability &
performance in
Oracle RAC 18c +
Best ever High
Availability in
Oracle RAC 19c
24 7
https://www.oracle.com/technetwork/database/options/clustering/overview/new-generation-oracle-rac-5975370.pdf

6. Best In-Class
Copyright © 2019 Oracle and/or its affiliates.
1
2
3
4
5
6
UCP, JDBC, ODP.Net,
OCI Session Pool,
Tuxedo, WebLogic
✓
High Availability
0
5000
10000
15000
20000
25000
30000
35000
40000
4 8 32 48 64 80
2035
4010
15520
22416
30016
37040
# of Cores across Oracle RAC Nodes
Users
2 Nodes
3 Nodes
4 Nodes
5 NodesSAP certified SD Benchmark results
Scalability

7. Best in Class High Availability
• Oracle RAC reconfiguration times
reduced by
• 4x between Oracle RAC 11.2.0.4 & 12 Rel. 2
• 1.5x between Oracle RAC 12c Rel. 2 & 18c
• for a total of 6x improvements
between Oracle RAC 11.2.0.4 and 18c
• Selection of contributing features:
• Remastering Slaves (*12.1)
• Support for 100 LMS’s (*12.2)
• Cache Fusion Hardening
• Near Zero Brownout for
Planned Maintenance
Copyright © 2019 Oracle and/or its affiliates.
Maximum Availability Architecture
4x
faster
1.5x
faster
More details: https://www.slideshare.net/MarkusMichalewicz/oracle-rac-roadmap-for-new-features

8. What Needs to be Configured for
Continuous Availability?
• RAC or RAC One, Active Data Guard, GoldenGate …Server Stack
• All databases on Flex ASMFlex ASM
• Services for Location TransparencyServices
• Connections appear continuousContinuous Connections
• FAN or 18c Database for drainingDraining
• Application ContinuityInflight work continues
• Drain in a timely mannerSLA’s
https://www.oracle.com/technetwork/database/options/clustering/applicationcontinuity/continuous-service-for-apps-on-atpd-5486113.pdf

9. The TNS Connector String for High Availability
ALIAS =(DESCRIPTION =
(CONNECT_TIMEOUT=90) (RETRY_COUNT=20)(RETRY_DELAY=3)
(TRANSPORT_CONNECT_TIMEOUT=3)
(ADDRESS_LIST =
(LOAD_BALANCE=on)
( ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521)))
(ADDRESS_LIST =
(LOAD_BALANCE=on)
( ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME = gold-cloud)))
Standard for All Drivers starting with Oracle Database 12.2
https://www.slideshare.net/MarkusMichalewicz/oracle-maa-best-practices-applications-considerations

10. Better Scalability & Performance
• Oracle RAC performance
has improved up to
• 5x between Oracle RAC 11.2.0.4 & 18.1
• for high contention workloads
• Exadata-based optimizations:
• “Smart Fusion Block Transfer” (*12.2)
• Undo RDMA-Read (*18c)
• Buffer Cache RDMA-Read
Copyright © 2019 Oracle and/or its affiliates.
5x
faster
BLOCKBLOCK
RDMA RDMA
Inst
1
Inst
3
Inst
2
More details: https://www.slideshare.net/MarkusMichalewicz/oracle-rac-roadmap-for-new-features

11. Oracle RAC Scales the World’s
Most Complex Enterprise Workloads
Due to its market leading Cache Fusion algorithm, Oracle RAC scales
1. any feature – e.g. Pluggable Databases, Oracle In-Memory and Oracle Data Guard
2. most enterprise applications – e.g. Ebusiness Suite, SAP, Oracle Hospitality
3. nearly all custom applications as used by many of Oracle’s >18000 RAC customers
• Without the need for significant application changes
• Especially on Oracle Exadata Database Machines
Copyright © 2019 Oracle and/or its affiliates.
https://www.oracle.com/a/otn/docs/oracle-rac-cache-fusion-performance-optimization-on-exadata-wp.pdf

12. Copyright © 2019 Oracle and/or its affiliates.
Expectation: ~90% scalability rate across the system as defined
Oracle RAC Scales Nearly Linearly
For test and application changes
• Focus on applications scaling with <80% scalability rate across the system in first go.
• Best results come from application optimization. Schema optimization can complement.

13. The Must Dos
Avoid Pipes
DBMS_PIPE can be
used with RAC, but
application changes
required to make it
work outweigh
benefits.
Use Advanced
Queuing (AQ) instead.
Use Connection Pools
Control the number of
concurrent processes
using a well-balanced
workload across nodes.
Hence, use (FAN
enabled) Connection
Pools.
Use Scalable Sequences
Use non-ordered &
cached sequences if
sequence is used to
generate primary key.
Use Scalable
Sequences (new in 18c)
[see hidden slide]
Avoid Write Contention
Frequent transactional
changes to the same
data blocks in all
instances may result in
“write hot spots”.
Some derivation of this
occurrence is likely the
reason for a less than
expected scaling result.
NUM
INC1()
Sequence
Connection
Pool
https://www.oracle.com/technetwork/database/options/clustering/overview/racappdevbp-3124851.pdf

14. New option in Create/Alter Sequence
SCALE {Extend |NoExtend | NoScale }
When Scale is specified an internal
algorithm uses the instance id and other
session specific information to generate an
offset
DBAs need to alter existing sequences
DBA/User/ALL_sequences will reflect the
additional attributes
Scalable Sequences Operational Details
Copyright © 2019 Oracle and/or its affiliates.

15. Contention – The Simplest Solution: RAC 19c
• Oracle RAC performance
has improved up to
• 5x between Oracle RAC 11.2.0.4 & 18.1
• for high contention workloads
• Exadata-based optimizations:
• “Smart Fusion Block Transfer” (*12.2)
• Undo RDMA-Read (*18c)
Copyright © 2019 Oracle and/or its affiliates.
5x
faster
https://www.slideshare.net/MarkusMichalewicz/make-your-application-oracle-rac-ready-test-for-it

16. Agenda
• What’s New in Oracle RAC?
• How to best upgrade to Oracle RAC 19c?
• Prepare for Changes

17. Check First!
Copyright © 2019 Oracle and/or its affiliates.
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2550798.1

18. Why ORAchk?
• ORAchk=ORAchk+cluvfy+Autoupgrade.jar*
• No need to individually download autoupgrade.jar or cluvfy (CVU)
• One report with results from autoupgrade.jar, orachk & CVU
• orachk -preupgrade -targetversion 19.3.0.0.0 –showpass
• cluvfy –stage pre/post
• orachk includes preupgrade.jar and cluvfy pre-upgrade checks
• ORAchk includes checks for additional component
• Application Continuity and Security related checks
Copyright © 2019 Oracle and/or its affiliates.

19. Linux 7 Required for RAC 19c
• ./gridSetup.sh run
on pre-Linux 7
releases likely to fail
• Failure is reported as a
perl error message due
to glibc dependcy
• Best solution: rolling
(OS) upgrade using
deleteNode / addNode
Copyright © 2019 Oracle and/or its affiliates.
https://www.linkedin.com/pulse/high-level-steps-upgrade-oracle-19c-rac-anil-nair/

20. Bonding Alert!?
• Oracle recommends
bonding modes 0
(“balance-rr”) or 1
(“active-backup”) for
simplicity.
• Other modes are
allowed, but may
require experience in
configuring them.
Copyright © 2019 Oracle and/or its affiliates.

21. Installing GIMR is Optional with 19c
• Choose to Install Grid
Infrastructure Mgmt
Repository (GIMR)
• Upgrades depend on
Initial GIMR config
• Eventual goal is to
move GIMR into its
own, separate home
Copyright © 2019 Oracle and/or its affiliates.

22. Copyright © 2019 Oracle and/or its affiliates.
-dryRunForUpgrade
Still Not Sure Whether to Upgrade?
$./gridSetup.sh –dryRunForUpgrade

23. Agenda
• What’s New in Oracle RAC?
• How to best upgrade to Oracle RAC 19c?
• Prepare for Changes

24. De-support of Non-Container Database Architecture
• The Oracle Database non-CDB architecture will be
de-supported from Oracle Database 20c onwards
• To ease the migration to this architecture,
from Oracle Database 19c onwards, the multitenant architecture
supports up to 3 user-created Pluggable Databases of any type
• The Multitenant Option is required for 4 or more user-created PDBs

25. Automatic PDB Resource Placement
• Two new features:
1. A ”first class” PDB resource
2. Automatic placement of PDBs
• Oracle Clusterware uses a PDB resource
• Instead of using services to manage PDBs
Copyright © 2019 Oracle and/or its affiliates.
NodeA
Oracle Grid Infrastructure
Oracle RAC
cons_1

26. Read Only Oracle HOME (ROOH)
• Read Only Oracle Database Homes separate configuration files from
binaries. Not applicable to Grid Infrastructure homes.
• ROOH is a logical concept – the homes remains writeable on OS-level
• Separation of configuration files
• Allows for faster cloning and improves security as running processes cannot
create new files under an Oracle Home
• ROOH
• Can be used for Single Instance and RAC Database Homes
• $roohctl can be used in versions 18c and 19c for manual conversion
• Oracle Database 20c enables ROOH by default
• Plan to remove configuration files used by applications such as
tnsnames.ora or individual files before converting to ROOH
Copyright © 2019 Oracle and/or its affiliates.

27. Zero Downtime Grid Infrastructure Patching
• Zero-Downtime Oracle Grid Infrastructure (GI) Patching
• Patch Oracle GI without interrupting database operations
• Patches are applied in a rolling fashion (one node at a time) while
the database instance(s) on that node remain up and running
• Limited availability with Oracle Database 18c (18.4 & RAC) on Linux
• General availability with Fleet Patching and Provisioning in Oracle Database 19c
• General availability with Oracle Database 20c (& 19.6+) on Linux + RU & RUR support
• Oracle Grid Infrastructure SwitchHome
• Oracle Installer provides a "switchHome" operation that
allows you to easily switch from one home to another.
• Most beneficial when used to switch from an
original version home to its patched equivalent.
Copyright © 2019 Oracle and/or its affiliates.

28. Oracle Clusterware Startup Diagnostics
• Environment changes and incorrect
permissions of binaries can prevent
stack startup
• Oracle Clusterware 19c attempts to
auto-diagnose unsuccessful startup
issues
• Provides detailed logging as needed:
CRS-41053: checking Oracle Grid Infrastructure for file
permission issues
PRVG-2031 : Owner of file ”…gipcd.bin" did not match the
[Expected="grid(54320)" Found="oracle(54325)"]
*NOT all daemons are shown in illustration above
init
cssdAgent
cssdmonitor
ohasd
oraAgent
cssd
crsd
ctssd
HAIP
ACFS
mdnsd
gipcd
evmd
ASM
oraRootAgent

29. Oracle Clusterware Runtime Diagnostics
• Oracle Clusterware 19c processes maintain histograms and statistics
such as trace file rotation frequency, time taken for rotation, etc.
• Preserves critical information on very busy systems
• Severity tagging provides human readable criticality of messages
2019-08-20 08:36:13.142 : CSSD:1871161088: [ ERROR] clssgmclienteventhndlr:
(SENDCOMPLETE) No proc found for ClientID
2019-08-20 08:36:13.188 : CSSD:1871161088: [ INFO] clssgmDeadProc: Removing clientID
2:43454:0 (0x7fda802df820), with GIPC
• New diagnostics monitor thread ensures in-memory logs (UTS) are
periodically written to disk, ensuring diagnostics are available in case
of process crashes

30. Oracle RAC Performance Automation
• My Oracle Support Note 1619155.1 describes “Best Practices and
Recommendations for RAC databases with SGA size over 100GB”
• With RAC 19c, most of the recommendations are auto-configured:
• Dynamic CR slaves to deal with changing workload
• LMS CR slaves (_max_cr_rollbacks deprecated)
• See Note 1630755.1 to reduce “LMS process busy” event in AWR report
• LMS CR Slaves
• Dynamic DLM ticket adjustment to prevent hangs
• Remove _lm_tickets
For RAC Databases with >100 GB SGA

31. Copyright © 2019 Oracle and/or its affiliates.
Autonomous Database = RAC on Exadata (& More)
Autonomous
Database
Automated
Data Center Operations
Oracle Cloud
• Oracle RAC has been enhanced to meet Autonomous Transaction Processing Dedicated (ATP-D) requirements.
• RAC Resource Runtime Management – allows for (memory) structures to be dynamically modified.
• Colocation of workload – to define that workload should run in 1 instance independent of the service
• Connect_Data SESSION_SETTINGS – define actions and set parameters as part of the connect string
https://www.slideshare.net/MarkusMichalewicz/oracle-rac19c-the-basis-for-the-autonomous-database

32. Thank you!
Markus Michalewicz (Markus.Michalewicz@oracle.com)
Senior Director of Database Product Management
@KnownAsMarkus
www.linkedin.com/in/markusmichalewicz
www.slideshare.net/MarkusMichalewicz