1. Measurement Errors

2. I/O Bottleneck Problem
Response Time Component Time % Elap AvgEla
---------------------------------------- ----------- ------- ---------
CPU service 3934.97s 48.39% 0.000716
un-accounted for time 1363.01s 16.76%
db file sequential read 1122.00s 13.80% 0.032253
gc buffer busy 451.73s 5.56% 0.011746
log buffer space 451.64s 5.55% 0.123974
buffer busy waits 176.79s 2.17% 0.029579
gc cr block 2-way 156.49s 1.92% 0.003287
gc cr grant 2-way 100.20s 1.23% 0.006541
latch: cache buffers chains 98.92s 1.22% 0.005708
gc current grant 2-way 69.68s 0.86% 0.006728
latch: library cache 30.10s 0.37% 0.010030
row cache lock 28.95s 0.36% 0.018727
gc current block 2-way 26.72s 0.33% 0.003828
gc cr block busy 19.35s 0.24% 0.006802
gc current grant busy 15.30s 0.19% 0.004999
latch: row cache objects 14.28s 0.18% 0.006165
gc cr block 3-way 11.73s 0.14% 0.002952
gc current block 3-way 11.34s 0.14% 0.003440
log file sync 10.71s 0.13% 0.315066
enq: SQ - contention 9.14s 0.11% 0.060911

3. Timings for Single Block Reads – 3 node RAC
Single Block Read Times From AWR
0
5
10
15
20
25
30
35
40
45
50
0:00
1:40
3:20
5:00
6:40
8:20
10:00
11:40
13:20
15:00
16:40
18:20
20:00
21:40
23:20
Snapshot ID - 20 minutes interval
SingleBlockReadTimein
Milliseconds
Inst 1

4. Timings for Single Block Reads – 3 node RAC
Single Block Read Times From AWR
0
5
10
15
20
25
30
35
40
45
50
0:00
1:40
3:20
5:00
6:40
8:20
10:00
11:40
13:20
15:00
16:40
18:20
20:00
21:40
23:20
Snapshot ID - 20 minutes interval
SingleBlockReadTimein
Milliseconds
Inst 1
Inst 2
Inst3

5. The Facts About The I/O Bottleneck Problem
• Facts:
– 3 node RAC
– Same storage
– Single block read time for Instance 1 was substantially different from read
times for other instances during off hours
– 5 batch jobs during off hours
• The timings for Instance 1 are obviously not correct
• The sandwich syndrome (output from strace)
gettimeofday({1159440978, 931945}, NULL) = 0
pread(14, "6242003752302+254.000160054001050"...,
8192, 455 057408) = 8192
gettimeofday({1159440978, 944159}, NULL) = 0
• Waiting in runque for CPU exaggerates all wait times of the
process.

6. Database Connection Problems

7. Too many sessions
• 35000 sessions per instance
• database level 70000 (2-node RAC)

10. http://www.youtube.com/watch?v=xNDnVOCdvQ0