1. Statspack
Kyle Hailey
http://oraclemonitor.com
06/05/08 1

2. Brief History
1989 6 Weakness of Ratios
 Ratio based guess work – no waits
 Utlbstat/Utlestat
 Lists of rules – no global picture/perspective
Counters
1993 7 Power of Waits
 Time gains and losses clear
 Undocumented
 “Expert Only”
1999 8.1.6 Statspack
 Top 5 Wait Events Timings
 Filtered out Idle Events
 Compulsive Tuning Disorder
2001 9iR2 CPU
 Top 5 Timed Events (includes CPU)
 Response Time = Service Time (CPU) + Wait Time
2003 10g Simplicity of AAS
 Average Active Session (AAS) = DB Time / Elapsed Time Sampling
2007 11g SQL Row Source
Copyright @
2006 Kyle Hailey

3. V6 Problem
How do you shorten your daily commute?
Analyze Statistics ?
RPM
Number of rotations of wheels?
Average amount of gas in tank?
No ! 
Solution:
Analyze Time
Where do you get blocked?
Are there alternatives ?
Oracle 6 didn’t have wait times, but you do !

4. Ratios, Example v6
Wait timings unavailable for IO
 Db file sequential read
s!
 Db file scattered read
 Direct path read
Ra t io
Ratio : Cache Buffer Hit
l ac ed
ep
proxy for IO wait time
t sr
en
Unreliable
Ev
ait
High ratio
LowW
Bad – untuned SQL doing mega LIOs
ratio
Good - I/Os are coming from UNIX file cache or disk cache
Reliability: IO Wait Times

5. Oracle 7
Waits introduced in version 7
 Quantum Step
 1992 slipped in for a benchmark
by kernel architect (now Senior VP) Juan Loaiza in 1992
Benchmark couldn’t find where time was being spent
 Wait instrumentation solved the mystery
Originally Undocumented
Still badly documented course should help change that
Experts only
Thus, ratio based tuning dominated
06/05/08

6. Oracle Instrumentation
Database: Guilty until proven innocent
CPU
Redo Lib Buffer
Database
Cache Cache
Locks
Network
*$%@!!
IO
Copyright 2006

7. Statspack
Introduced in 8.1.6
by Connie Dialeris
Snapshots
Selects from 55 tables and views
x$ structures
v$ views
DBA views
Number of tables has expanded with each version
 30 plus tables in Oracle8i
 40 plus tables in Oracle9i
 55 plus tables in Oracle10g
Calculates Deltas (and ratios)
 Reports 1000-2000 lines output, about 30 pages
Copyright @

8. Statspack: Cheat Sheet
Install
Connect as SYSDBA
@?/rdbms/admin/spcreate.sql
Run
Exec statspack.snap;
Generate Reports
@?/rdbms/admin/spreport.sql
 (AWR : @?/rdbms/admin/awrrpt.sql )
Copyright 2006 Kyle Hailey

9. Statspack Sections (10g)
Instance description Session details x 3 Latch parent and child
Host hardware Tablespace I/O Mutex
Snapshot headline File I/O Segment stats x 8
Cache information File I/O histogram Dictionary cache
Load profile Buffer pool Library cache
Instance efficiency Instance Recovery RAC (GES)
Shared pool stats Buffer pool advisory RAC (CR and CUR
Top timed events Buffer busy waits served)
Host CPU load Various PGA RAC (cache xfer x 2)
Host / Instance CPU warning summaries RAC (Remastering)
VM activity PGA histogram Streams x 7
Memory usage PGA advisory Shared pool advisor
Time model stats PGA allocation Java pool advisor
RAC statistics summary SGA resizing
All wait events PGA allocation top SGA target advisor
Background wait events N SGA summary
Event histogram Enqueue (lock) SGA detail
SQL x 9 activity SQL memory summary
Instance activity Undo stats x 2 Resource limits
Log switches Latch activity Parameters
OS stats Latch miss details

10. Statspack Method
Who/When
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ Load profile good for
% Total
Event Waits having a feel for theTime
Time (s) Call
---------------------- ------------ application and---------
-----------
buffer busy waits 2,748 comparing two periods
250 78.72
Summary CPU time for changes32 10.16
free buffer waits 1,588 15 4.63
write complete waits 10 Efficiency ratios
8 2.51
log buffer space 306 misleading 5 carry over
1.51
from version 6 days
----------------------------------------
Big Picture
Waits
06/05/08

11. Oracle 8
Top 5 Wait Events
~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
-------------------------------------------- ------------ ------------ -------
direct path read 4,232 10,827 52.01
db file scattered read 6,105 6,264 30.09
direct path write 1,992 3,268 15.70
control file parallel write 893 198 .95
db file parallel write 40 131 .63
IO Looks horrible, but wait …
Statistic Total per Second per Trans
--------------------------------- ---------------- ------------ ------------
CPU used by this session 358,806 130.5 12,372.6
Total time in secs =
Missing the CPU 108+62+32+1+1+3588=3792
CPU time = 95% 3588/3792=95%
Eliminating all IO gains 3 % ! 108/3792=3%
06/05/08

12. Oracle 9
Statspack adds CPU
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
CPU time 8,026 93.78
direct path write 1,930 421 4.92
db file scattered read 4,746 69 .80
direct path read 77 18 .21
db file parallel write 23 11 .13
------------------------------------------------------------
Method Max gain is 6.22%
by tuning all waits
% wait time = potential performance gain
In other words go
tune the top SQL
% gain = Savings/Service Time
Service Time = CPU time + Wait Time
Service Time =DB Time in 10g
06/05/08

13. Problems in 9
1. Lack of Data to solve issues
Examples
1. Buffer busy wait
2. latch free
3. db file scattered read (IO)
4. Enqueue (lock)

14. Buffer Busy Wait
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Call Time
-------------------------------------------- ------------ ----------- ---------
buffer busy waits 2,748 250 78.72
CPU time 32 10.16
free buffer waits 1,588 15 4.63
write complete waits 10 8 2.51
log buffer space 306 5 1.51
------------------------------------------------------------
Buffer Busy Wait – trying to modify a block a
buffer. Who blocks, what kind of buffer, what
is the SQL?
Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3
Class Waits Wait Time (s) Avg Time (ms)
------------------ ----- ------------- -------------
file header block 264 203 769
data block 6,070 162 27
undo header 355 0 1
Statspack fails for analysis
06/05/08

15. Latch Free
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
latch free 9,652 760 66.10
CPU time 248 21.62
PL/SQL lock timer 41 123 10.72
SQL*Net message from dblink 681 14 1.22
log file parallel write 128 1 .13
-------------------------------------------------------------
What Latch? There are 100s
Latch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4
-> ordered by misses desc
Latch Name Requests Misses Sleeps Sleeps 1->4
-------------------------- -------------- ----------- ----------- ------------
cache buffers chains 8,448,787 649,484 6,930 0/0/0/0/0
library cache pin 8,405,896 82,915 1,427 81537/1330/4
7/1/0
library cache 8,435,488 55,645 1,294 54375/1247/2
2/1/0
shared pool 58,626 7 1 6/1/0/0/0
06/05/08

16. IO
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time 5/7549=0.6ms
------------------------ ------------ ----------- --------
db file scattered read 7,549 5 52.26 Very fast
CPU time 2 23.29 Problem is SQL
db file sequential read 884 2 19.11
control file sequential read 154 0 2.68
control file parallel write 255 0 1.87
Are the disks slow? What files?
Is the buffer cache too small?
IO – what SQL?
06/05/08

17. IO
Are the disks slow? What files?
File IO Stats for DB: ORA9 Instance: ora9 Snaps: 21 -22
Tablespace Filename
------------------------ ----------------------------------------------------
Av Av Av Av Buffer Av Buf
Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
-------------- ------- ------ ------- ------------ -------- ---------- ------
EXAMPLE D:ORACLEORADATAORA9EXAMPLE01.DBF
49 0 3.1 3.0 0 0 0
SYSTEM
8,331 11 0.7 7.3
< 10ms
D:ORACLEORADATAORA9SYSTEM01.DBF
3 0 0
UNDOTBS1 D:ORACLEORADATAORA9UNDOTBS01.DBF
5 0 8.0 1.0 0 0 0
Is the buffer cache too small?
IO – what SQL?
06/05/08

18. IO
Is the buffer cache too small?
Buffer Pool Advisory for DB: ORA9 Instance: ora9 End Snap: 22
Size for Size Buffers for Est Physical Estimated
P Estimate (M) Factr Estimate Read Factor Physical Reads
--- ------------ ----- ---------------- ------------- ------------------
D 4 .1 979 1.05 67,973
D 16 .5 3,916 1.00 64,668
D 32 1.0 7,832 1.00 64,612
D 48 1.5 11,748 1.00 64,612
D 64 2.0 15,664 1.00 64,612
D 68 2.1 16,643 1.00 64,612
D 72 2.3 17,622 0.30 19,132
D 76 2.4 18,601 0.30 19,132
D 80 2.5 19,580 0.30 19,132
IO – what SQL?
06/05/08

19. IO
What SQL?
SQL ordered by Reads for DB: ORA9 Instance: ora9 Snaps: 21 -22
CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
60,197 4 15,049.3 98.4 0.48 5.04 1432830824
Module: SQL*Plus
select count(*) from sys.source$
666 1 666.0 1.1 0.70 1.46 2893531678
Module: SQL*Plus
select count(*) from all_objects
232 24 9.7 0.4 0.53 0.97 238087931
select t.schema, t.name, t.flags, q.name from system.aq$_queue_t
ables t, sys.aq$_queue_table_affinities aft, system.aq$_que
Sorted by Reads, not Time
(reads proxy for IO wait time)
06/05/08

20. Row Locks 10g+
op 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
----------------------------------------- ------------ ----------- ------ ------
enq: TX - row lock contention 59 160 2714 41.8
PL/SQL lock timer 4 117 29291 30.6
CPU time 28 7.2
buffer busy waits 1,217 18 15 4.7
log file parallel write 422 11 27 3.0
Who is waiting
Who is blocking
What is the SQL
What is the row?
06/05/08

21. Current Method
1000 lines of data / 30 pages 188 minutes CPU
What do you look at ? Elapsed 15 minutes
Top 5 Timed Events?
48 Processors or
74% idle
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
------------------------ ------------ ----------- --------
CPU time 11,330 81.95
log file sync 163,725 1,012 7.32
enqueue 4,486 615 4.45
latch free 28,786 236 1.71
db file sequential read 1,420,355 149 1.08
-----------------------------------------
Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html
02/20/13 21

22. Statspack
Top 5 Timed Events missing
CPU Count
Elapsed Time
As well as drill down diagnostic data
Take a step back, take deep breath, and start over
 Start with Typical Problems
 Identify Solutions
 Design a Visual Representation
⇒Put intelligence into the interface
06/05/08

23. OEM 10g
Database
Load
Load Source:
SQL
Session
Wait
Simple, Powerful