1. Exadata Consolidation Success Story
Getting the kids to play nice with each other…
Presented by:
Karl Arao
1

2. whoami
Karl Arao
• Senior Technical Consultant @ Enkitec
• Performance and Capacity Planning Enthusiast
6 years 11 months 12 days DBA experience
Oracle ACE, OCP-DBA, RHCE, OakTable
Blog: karlarao.wordpress.com
Wiki: karlarao.tiddlyspot.com
Twitter: @karlarao
www.enkitec.com 2

3. Agenda
• Architecture
• Tools and Methodology
• War Stories
www.enkitec.com 3

4. General Architecture
Primary Site Standby Site
Production Future Growth
Test & Dev Disaster Recovery
www.enkitec.com 4

5. General Architecture
www.enkitec.com 5

6. The Stats
Three Half Rack Exadata clusters with High Cap. drives
Cluster #1
36 Dev/Test Databases
Cluster #2
11 Production Databases
Cluster #3
13 Dev/Test Databases
6 Standby Databases
Still more databases to come…
www.enkitec.com 6

7. Why Consolidate?
Primary drivers for consolidation center around cost savings
• Reduces Oracle software licensing
• 3rd party products such as backup agents, ETL tools, etc…
• More efficient use of system resources
• Soft Costs
– Floor space
– Power & Cooling
– Administration, Staffing Costs
(training, etc.)
www.enkitec.com 7

8. A Simple Consolidation Example
7 Databases
www.enkitec.com 8

9. A Simple Consolidation Example
Let’s say we have the following databases to migrate on Exadata:
Cluster Level
Utilization
For example, the first row should read…
Database ‘A’ requires 4 CPU’s and will run on nodes 1 and 2 (2 CPU’s each)
www.enkitec.com 9

10. A Simple Consolidation Example
Let’s say we have the following databases to migrate on Exadata:
Per compute node
Utilization
For example, the first row should read…
Database ‘A’ requires 4 CPU’s and will run on nodes 1 and 2 (2 CPU’s each)
www.enkitec.com 10

11. A Simple Consolidation Example
25% 42% 33% 17%
Cluster Level Per compute node Utilization
Utilization = 29.2%
www.enkitec.com 11

12. A Simple Consolidation Example
8% 83% 17% 8%
Cluster Level Per compute node Utilization
Utilization = 29.2%
www.enkitec.com 12

13. Tools And Methodology
• Gather Utilization Metrics (usage history)
• Create Provisioning Plan
• Implement Plan
• Audit Your Implementation
www.enkitec.com 13

14. www.enkitec.com 14

15. Provisioning Worksheet
• Capacity Planning
U =R tilization equirements /C apacity
• Communication Tool
**Supplement to existing Exadata installation tools:
• Site planning checklist
• Configuration Worksheet
• Exadata Configurator sheet
• CheckIP
• OneCommand
• Hand off
www.enkitec.com 15

16. Capacity
2 = quarter rack CPU_COUNT,
Space will also depend on:
• ASM redundancy
4 = half rack threads, & cores
• DATA/RECO allocation
http://goo.gl/CunHN
8 = full rack http://goo.gl/I3fjn
96 to 144GB
Query Low (4x)
(frequency of the
SPECint_rate2006 Query High (6x)
memory DIMMs
http://goo.gl/doBI5 Archive Low (7x)
drops to 800 MHz
Archive High (12x)
from 1333 MHz)
www.enkitec.com 16

17. CPU Core Comparison
Sun Fire X4170 M2 X5670@2.93GHz
Source Destination
chip efficiency factor = source SPEC rating / Exadata SPEC rating
= 16/26 how much of the
multiplier for amount of CPU
equivalent resources that will
= .6154 source CPU cores
are being used
database be offloaded to the
machine cores storage cells
EXA cores requirement = source host cores * utilization * chip efficiency factor * offload factor
= 32 * .7 * .6154 * .5
= --------- 6.89
13.78
www.enkitec.com 17

18. The Perfect Storm
(Peoplesoft HR)
Month-end Processing
+ Weekly Time Entry
+ SQL Plan Change
------------------------------------
Uh-oh!
www.enkitec.com 18

19. CPU Allocation
node node node node
DB Uniq Name DB Name 1 2 3 4
4 instance 5 instance 4 instance 3 instance
47% cpu used 75% cpu used 47% cpu used 18% cpu used
49% mem used 66% mem used 71% mem used 54% mem used
BIPRDDAL biprd P P
DBFSPRD DBFSPRD P P P P
HCMPRDDAL hcmprd P P
MTAPRD11DAL mtaprd11 P P
PAPRDDAL paprd P P
RMPRDDAL rmprd P P
dbm dbm F F F F
Fsprddal fsprd P P
= Preferred
= Failover
www.enkitec.com 19

20. Load Map
(our first stop…)
Users Complaint: HR time entry and OBIEE reports painfully slow…
www.enkitec.com 20

21. Top Activity - HCMPRD
www.enkitec.com 21

22. Instance Activity – HCMPRD2
Node 2
Problem: A single SQL stmt. overwhelming HCMPRD Caged SQL Profile Installed
CPU resources. at 12 CPU’s to lock in good plan.
www.enkitec.com 22

23. Memory Exhaustion
(OBIEE)
“1 Report = 1 SQL query, right?”
WRONG!
www.enkitec.com 23

24. Node 1
Overlapping workloads of three databases
across 3 nodes.
Node 2
BIPRD, HCMPRD, and MTAPRD
Node 3
Node 4
www.enkitec.com 24

25. Node Layout Revisited…
www.enkitec.com 25

26. Notice what happens to CPU waits
and the system load average when
this report is run.
www.enkitec.com 26

27. PGA Memory Spikes
www.enkitec.com 27

28. www.enkitec.com 28

29. www.enkitec.com 29

30. Storage Cell Saturation
(OBIEE)
www.enkitec.com 30

31. www.enkitec.com 31

32. I/O Intensive Workload
www.enkitec.com 32

33. Smart Scans as seen in Grid Control
www.enkitec.com 33

34. 25 Sessions Doing Smart Scans
…as seen in gv$sql
www.enkitec.com 34

35. www.enkitec.com 35

36. Smart Scan in Action. The cells are scanning 1T but only returning 144G…
***That’s on each of the highlighted row source below…
www.enkitec.com 36

37. The databases on other nodes see the contention as “System I/O”
Without I/O resource management even critical processes are affected (CKPT, LGWR, …)
www.enkitec.com 37

38. Inter-database IORM Plan
(only kicks in when needed)
I/O requests from critical processes like CKPT, LGWR, LMON get priority automatically.
Without IORM I/O requests from these important processes receive the same priority
as any other process.
*Side Benefit (automatic when IORM is enabled)
www.enkitec.com 38

39. IORM Plan Definition
(on each storage cell)
www.enkitec.com 39

40. Wrap up!
Provisioning Methodology & Tools
– Utilization metrics and requirements
– Provisioning Spreadsheet
Success Stories
– CPU resource management
– Tuning and provisioning adjustments
– I/O resource management
www.enkitec.com 40

41. Q
&
A
www.enkitec.com 41

42. Contact Info…
karl.arao@enkitec.com
Fastest Growing Companies
in Dallas
42