2. Brief History
1989 6 Weakness of Ratios
Ratio based guess work – no waits
Utlbstat/Utlestat
Lists of rules – no global picture/perspective
Counters
1993 7 Power of Waits
Time gains and losses clear
Undocumented
“Expert Only”
1999 8.1.6 Statspack
Top 5 Wait Events Timings
Filtered out Idle Events
Compulsive Tuning Disorder
2001 9iR2 CPU
Top 5 Timed Events (includes CPU)
Response Time = Service Time (CPU) + Wait Time
2003 10g Simplicity of AAS
Average Active Session (AAS) = DB Time / Elapsed Time Sampling
2007 11g SQL Row Source
Copyright @
2006 Kyle Hailey
3. V6 Problem
How do you shorten your daily commute?
Analyze Statistics ?
RPM
Number of rotations of wheels?
Average amount of gas in tank?
No !
Solution:
Analyze Time
Where do you get blocked?
Are there alternatives ?
Oracle 6 didn’t have wait times, but you do !
4. Ratios, Example v6
Wait timings unavailable for IO
Db file sequential read
s!
Db file scattered read
Direct path read
Ra t io
Ratio : Cache Buffer Hit
l ac ed
ep
proxy for IO wait time
t sr
en
Unreliable
Ev
ait
High ratio
LowW
Bad – untuned SQL doing mega LIOs
ratio
Good - I/Os are coming from UNIX file cache or disk cache
Reliability: IO Wait Times
5. Oracle 7
Waits introduced in version 7
Quantum Step
1992 slipped in for a benchmark
by kernel architect (now Senior VP) Juan Loaiza in 1992
Benchmark couldn’t find where time was being spent
Wait instrumentation solved the mystery
Originally Undocumented
Still badly documented course should help change that
Experts only
Thus, ratio based tuning dominated
06/05/08
7. Statspack
Introduced in 8.1.6
by Connie Dialeris
Snapshots
Selects from 55 tables and views
x$ structures
v$ views
DBA views
Number of tables has expanded with each version
30 plus tables in Oracle8i
40 plus tables in Oracle9i
55 plus tables in Oracle10g
Calculates Deltas (and ratios)
Reports 1000-2000 lines output, about 30 pages
Copyright @
9. Statspack Sections (10g)
Instance description Session details x 3 Latch parent and child
Host hardware Tablespace I/O Mutex
Snapshot headline File I/O Segment stats x 8
Cache information File I/O histogram Dictionary cache
Load profile Buffer pool Library cache
Instance efficiency Instance Recovery RAC (GES)
Shared pool stats Buffer pool advisory RAC (CR and CUR
Top timed events Buffer busy waits served)
Host CPU load Various PGA RAC (cache xfer x 2)
Host / Instance CPU warning summaries RAC (Remastering)
VM activity PGA histogram Streams x 7
Memory usage PGA advisory Shared pool advisor
Time model stats PGA allocation Java pool advisor
RAC statistics summary SGA resizing
All wait events PGA allocation top SGA target advisor
Background wait events N SGA summary
Event histogram Enqueue (lock) SGA detail
SQL x 9 activity SQL memory summary
Instance activity Undo stats x 2 Resource limits
Log switches Latch activity Parameters
OS stats Latch miss details
10. Statspack Method
Who/When
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ Load profile good for
% Total
Event Waits having a feel for theTime
Time (s) Call
---------------------- ------------ application and---------
-----------
buffer busy waits 2,748 comparing two periods
250 78.72
Summary CPU time for changes32 10.16
free buffer waits 1,588 15 4.63
write complete waits 10 Efficiency ratios
8 2.51
log buffer space 306 misleading 5 carry over
1.51
from version 6 days
----------------------------------------
Big Picture
Waits
06/05/08
11. Oracle 8
Top 5 Wait Events
~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
-------------------------------------------- ------------ ------------ -------
direct path read 4,232 10,827 52.01
db file scattered read 6,105 6,264 30.09
direct path write 1,992 3,268 15.70
control file parallel write 893 198 .95
db file parallel write 40 131 .63
IO Looks horrible, but wait …
Statistic Total per Second per Trans
--------------------------------- ---------------- ------------ ------------
CPU used by this session 358,806 130.5 12,372.6
Total time in secs =
Missing the CPU 108+62+32+1+1+3588=3792
CPU time = 95% 3588/3792=95%
Eliminating all IO gains 3 % ! 108/3792=3%
06/05/08
12. Oracle 9
Statspack adds CPU
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
CPU time 8,026 93.78
direct path write 1,930 421 4.92
db file scattered read 4,746 69 .80
direct path read 77 18 .21
db file parallel write 23 11 .13
------------------------------------------------------------
Method Max gain is 6.22%
by tuning all waits
% wait time = potential performance gain
In other words go
tune the top SQL
% gain = Savings/Service Time
Service Time = CPU time + Wait Time
Service Time =DB Time in 10g
06/05/08
13. Problems in 9
1. Lack of Data to solve issues
Examples
1. Buffer busy wait
2. latch free
3. db file scattered read (IO)
4. Enqueue (lock)
14. Buffer Busy Wait
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Call Time
-------------------------------------------- ------------ ----------- ---------
buffer busy waits 2,748 250 78.72
CPU time 32 10.16
free buffer waits 1,588 15 4.63
write complete waits 10 8 2.51
log buffer space 306 5 1.51
------------------------------------------------------------
Buffer Busy Wait – trying to modify a block a
buffer. Who blocks, what kind of buffer, what
is the SQL?
Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3
Class Waits Wait Time (s) Avg Time (ms)
------------------ ----- ------------- -------------
file header block 264 203 769
data block 6,070 162 27
undo header 355 0 1
Statspack fails for analysis
06/05/08
15. Latch Free
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
latch free 9,652 760 66.10
CPU time 248 21.62
PL/SQL lock timer 41 123 10.72
SQL*Net message from dblink 681 14 1.22
log file parallel write 128 1 .13
-------------------------------------------------------------
What Latch? There are 100s
Latch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4
-> ordered by misses desc
Latch Name Requests Misses Sleeps Sleeps 1->4
-------------------------- -------------- ----------- ----------- ------------
cache buffers chains 8,448,787 649,484 6,930 0/0/0/0/0
library cache pin 8,405,896 82,915 1,427 81537/1330/4
7/1/0
library cache 8,435,488 55,645 1,294 54375/1247/2
2/1/0
shared pool 58,626 7 1 6/1/0/0/0
06/05/08
16. IO
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time 5/7549=0.6ms
------------------------ ------------ ----------- --------
db file scattered read 7,549 5 52.26 Very fast
CPU time 2 23.29 Problem is SQL
db file sequential read 884 2 19.11
control file sequential read 154 0 2.68
control file parallel write 255 0 1.87
Are the disks slow? What files?
Is the buffer cache too small?
IO – what SQL?
06/05/08
17. IO
Are the disks slow? What files?
File IO Stats for DB: ORA9 Instance: ora9 Snaps: 21 -22
Tablespace Filename
------------------------ ----------------------------------------------------
Av Av Av Av Buffer Av Buf
Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
-------------- ------- ------ ------- ------------ -------- ---------- ------
EXAMPLE D:ORACLEORADATAORA9EXAMPLE01.DBF
49 0 3.1 3.0 0 0 0
SYSTEM
8,331 11 0.7 7.3
< 10ms
D:ORACLEORADATAORA9SYSTEM01.DBF
3 0 0
UNDOTBS1 D:ORACLEORADATAORA9UNDOTBS01.DBF
5 0 8.0 1.0 0 0 0
Is the buffer cache too small?
IO – what SQL?
06/05/08
18. IO
Is the buffer cache too small?
Buffer Pool Advisory for DB: ORA9 Instance: ora9 End Snap: 22
Size for Size Buffers for Est Physical Estimated
P Estimate (M) Factr Estimate Read Factor Physical Reads
--- ------------ ----- ---------------- ------------- ------------------
D 4 .1 979 1.05 67,973
D 16 .5 3,916 1.00 64,668
D 32 1.0 7,832 1.00 64,612
D 48 1.5 11,748 1.00 64,612
D 64 2.0 15,664 1.00 64,612
D 68 2.1 16,643 1.00 64,612
D 72 2.3 17,622 0.30 19,132
D 76 2.4 18,601 0.30 19,132
D 80 2.5 19,580 0.30 19,132
IO – what SQL?
06/05/08
19. IO
What SQL?
SQL ordered by Reads for DB: ORA9 Instance: ora9 Snaps: 21 -22
CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
60,197 4 15,049.3 98.4 0.48 5.04 1432830824
Module: SQL*Plus
select count(*) from sys.source$
666 1 666.0 1.1 0.70 1.46 2893531678
Module: SQL*Plus
select count(*) from all_objects
232 24 9.7 0.4 0.53 0.97 238087931
select t.schema, t.name, t.flags, q.name from system.aq$_queue_t
ables t, sys.aq$_queue_table_affinities aft, system.aq$_que
Sorted by Reads, not Time
(reads proxy for IO wait time)
06/05/08
20. Row Locks 10g+
op 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
----------------------------------------- ------------ ----------- ------ ------
enq: TX - row lock contention 59 160 2714 41.8
PL/SQL lock timer 4 117 29291 30.6
CPU time 28 7.2
buffer busy waits 1,217 18 15 4.7
log file parallel write 422 11 27 3.0
Who is waiting
Who is blocking
What is the SQL
What is the row?
06/05/08
21. Current Method
1000 lines of data / 30 pages 188 minutes CPU
What do you look at ? Elapsed 15 minutes
Top 5 Timed Events?
48 Processors or
74% idle
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
------------------------ ------------ ----------- --------
CPU time 11,330 81.95
log file sync 163,725 1,012 7.32
enqueue 4,486 615 4.45
latch free 28,786 236 1.71
db file sequential read 1,420,355 149 1.08
-----------------------------------------
Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html
02/20/13 21
22. Statspack
Top 5 Timed Events missing
CPU Count
Elapsed Time
As well as drill down diagnostic data
Take a step back, take deep breath, and start over
Start with Typical Problems
Identify Solutions
Design a Visual Representation
⇒Put intelligence into the interface
06/05/08