Sql Performance Tuning with ASH & AWR: Real World Use Cases

The following is intended to outline our general product direction.
It is intended for information purposes only, and may not be
incorporated into any contract. It is not a commitment to deliver
any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and
timing of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.

All the views expressed in this presentation are the opinion of the
authors and do not necessarily reflect the views of Oracle.

The queries are provided "as is", without warranty of any kind,
expressed or implied.


SQL Performance Tuning with Oracle AWR & ASH
Real World Use Cases


About myself

• Vlado Barun, M.Sc.
– Sr. Principal Performance Engineer with Oracle Cloud Services
– OCP, OCE SQL, MCP
– 15 years in the Database Arena (Oracle DB - 10 years)


Objective

• Enable you to start exploring the rich repository of
performance metrics in ASH & AWR independent of any
specific GUI and related limitations
• Provide list of resources that enable you to “hit the
ground running”


Agenda

• Introduction to ASH & AWR (very brief)
• Review 2 real world production use cases
– step-by-step from data gathering to resolution
– write ASH/AWR queries from scratch
– use pre-existing free scripts
– Note: check license requirements
• Resources


Automatic Workload Repository (AWR)

• Collects database performance statistics
– Objects (access and usage statistics)
– SQL Statement statistics
– Wait events statistics
– System statistics
– ASH Statistics (DBA_HIST_ACTIVE_SESS_HISTORY)
– Time Model Statistics based on time usage for activities
– Every 1 hr by default
• Stored in DBA_HIST% views
• How to access?
– $ORACLE_HOME/rdbms/admin/awrrpt.sql, SQLT, custom
scripts,…
– OEM, SQL Developer
– Other tools


Active Session History (ASH) Introduction

v$active_session_history dba_hist_active_sess_history
(1 second intervals) (10 second intervals)

3 ASH Samples in
AWR

Browse Read Reviews
Add to Cart Checkout
Books For One Book

21 ASH Samples

TIME

SQL Trace (Microsecond level)
Application Logging (custom level)
= time spent in database


ASH - Dimensions
When Row Source Object
SAMPLE_ID SQL_PLAN_LINE_ID CURRENT_OBJ#
SAMPLE_TIME SQL_PLAN_OPERATION CURRENT_FILE#
CURRENT_BLOCK#
CURRENT_ROW#
Session Waits
SESSION_ID EVENT
SESSION_SERIAL# EVENT_ID Application
SESSION_TYPE EVENT# SERVICE_HASH
SESSION_STATE SEQ# PROGRAM
USER_ID P1TEXT MODULE
QC_INSTANCE_ID P1 ACTION
QC_SESSION_ID P2TEXT MACHINE
QC_SESSION_SERIAL# P2
CLIENT_ID P3TEXT
IN and IS columns P3
WAIT_CLASS Statistics
WAIT_CLASS_ID TM_DELTA_TIME
SQL WAIT_TIME TM_DELTA_CPU_TIME
SQL_ID TIME_WAITED TM_DELTA_DB_TIME
IS_SQLID_CURRENT CURRENT_OBJ# DELTA_TIME
SQL_CHILD_NUMBER CURRENT_FILE# DELTA_READ_IO_REQUESTS
SQL_OPCODE CURRENT_BLOCK# DELTA_WRITE_IO_REQUESTS
SQL_OPNAME CURRENT_ROW# DELTA_READ_IO_BYTES
TOP_LEVEL_SQL_ID DELTA_WRITE_IO_BYTES
TOP_LEVEL_SQL_OPCODE DELTA_INTERCONNECT_IO_BYTES
SQL_EXEC_ID Blocking Session PGA_ALLOCATED
SQL_EXEC_START BLOCKING_SESSION_STATUS TEMP_SPACE_ALLOCATED
SQL_PLAN_HASH_VALUE BLOCKING_SESSION
SQL_PLAN_OPTIONS BLOCKING_SESSION_SERIAL#
BLOCKING_INST_ID
BLOCKING_HANGCHAIN_INFO


ASH - Summary

• Samples the current state of all active session
– 1 second samples
– Stores various attributes (dimensions) of a session
• Stored in
– V$ACTIVE_SESSION_HISTORY
• 1-second sampling rate
– DBA_HIST_ACTIVE_SESS_HISTORY
- Sub-sampling to disk into AWR Snapshot
- 1-in-10 samples

• How to access?
- $ORACLE_HOME/rdbms/admin/ashrpt.sql, ashrpti.sql, SQLT, custom
scripts,…
- OEM, SQL Developer, Real Time SQL Monitoring, …
- Other tools


GUI or command line
• It is about knowing the right tool for the right situation
• Understand the strengths and weaknesses of each
• GUI tools strengths => cmd line weaknesses
– Easy to use
– Good visualization of data
– Provides concise summary of system activity
Performance
– etc… data

• GUI tools weaknesses=> cmd line strength
Performance data
– Slow - click…wait…click…wait… exposed by GUI

– Availability – GUI infrastructure is down
– Limited exposure – subset of available data
– etc…


ASH dimensions growth

See
https://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history


Understand the issue
What Identify
(app is slow)
Initial
Data
How Much Quantify collection
(takes 1m)
AWR/ASH, Trace,
Apps Logs/Tables,
When Timeline OS/IO stats etc
(during peak hours)

Where Component
(Web Tier, App Tier,
DB , Network, Storage, Analyze
etc.)

How Flow chart

Additional
Data
SQL/Object Stats, Collection
and
Root Cause Execution Plans,
Why (Stats not updated) etc
Analysis


Use Case 1

What Slow ETL

How Much 20+ hours

When Once a month

Informatica + DB
Where
Extract, Transform, Load
How


Find slow SQL (ASH)

SQL> SELECT sql_id
2 , COUNT(*) as sample_count
3 FROM v$active_session_history
4 WHERE sample_time BETWEEN TO_DATE('2011-06-29 07:57', 'yyyy-mm-dd hh24:mi')
5 AND TO_DATE('2011-06-30 04:30', 'yyyy-mm-dd hh24:mi')
6 AND user_id = 61
7 AND program like 'pmdtm@%'
8 GROUP BY sql_id
9 ORDER BY 2 DESC
10 ;
no rows selected

SQL> SELECT min(sample_time)
2 FROM v$active_session_history
3 ;

MIN(SAMPLE_TIME)
----------------------------------------
2011-07-01 19:14:48

1 row selected.


Find slow SQL (ASH in AWR)
SELECT sql_id
, COUNT(*) as sample_count
FROM dba_hist_active_sess_history
v$active_session_history

JOIN dba_hist_snapshot USING (dbid, instance_number, snap_id)
WHERE sample_time BETWEEN TO_DATE('2011-06-29 07:57', 'yyyy-mm-dd hh24:mi')
AND TO_DATE('2011-06-30 04:30', 'yyyy-mm-dd hh24:mi')
AND user_id = 61
AND program like 'pmdtm@%‘
GROUP AND
BY begin_interval_time BETWEEN TO_DATE('2011-06-29 07:00','yyyy-mm-dd hh24:mi')
sql_id
ORDER BY 2 DESC AND TO_DATE('2011-06-30 05:00', 'yyyy-mm-dd hh24:mi')
;

SQL_ID SAMPLE_COUNT
------------- ------------
dkwucgaxxhm2z 7362
grksgbxxuacaf 6041
…
24 rows selected.


Find slow SQL (ASH in AWR) - readable
SQL>SELECT sql_id, sql_type
2 , round(sample_cnt_sql/sample_cnt_total*100) as sample_pct
3 , round(sample_cnt_sql /6/60,1) hours
4 FROM(SELECT DISTINCT sql_id
5 , aa.name AS sql_type
6 , COUNT(*) over (PARTITION BY sql_id) AS sample_cnt_sql
7 , COUNT(*) over () AS sample_cnt_total
8 FROM dba_hist_active_sess_history ash
9 JOIN dba_hist_snapshot s USING (dbid, INSTANCE_NUMBER, snap_id)
10 JOIN audit_actions aa on (ash.sql_opcode = aa.action)
11 WHERE begin_interval_time BETWEEN
… snip …
18 ORDER BY 3 desc)
19 WHERE rownum <= 3;

SQL_ID SQL_TYPE SAMPLE_PCT HOURS
------------- -------- ---------- -----
dkwucgaxxhm2z SELECT 25 20.4
grksgbxxuacaf SELECT 21 16.8
gh6w2fm3avn3j SELECT 12 10.1

3 rows selected.


Active Session History (ASH) - Dimensions
When Row Source Object
SAMPLE_ID SQL_PLAN_LINE_ID CURRENT_OBJ#
SAMPLE_TIME SQL_PLAN_OPERATION CURRENT_FILE#
CURRENT_BLOCK#
CURRENT_ROW#
Session Waits
SESSION_ID EVENT
SESSION_SERIAL# EVENT_ID Application
SESSION_TYPE EVENT# SERVICE_HASH
SESSION_STATE SEQ# PROGRAM
USER_ID P1TEXT MODULE
QC_INSTANCE_ID P1 ACTION
QC_SESSION_ID P2TEXT MACHINE
QC_SESSION_SERIAL# P2
CLIENT_ID P3TEXT
IN and IS columns P3
WAIT_CLASS Statistics
WAIT_CLASS_ID TM_DELTA_TIME
SQL WAIT_TIME TM_DELTA_CPU_TIME
SQL_ID TIME_WAITED TM_DELTA_DB_TIME
IS_SQLID_CURRENT CURRENT_OBJ# DELTA_TIME
SQL_CHILD_NUMBER CURRENT_FILE# DELTA_READ_IO_REQUESTS
SQL_OPCODE CURRENT_BLOCK# DELTA_WRITE_IO_REQUESTS
SQL_OPNAME CURRENT_ROW# DELTA_READ_IO_BYTES
TOP_LEVEL_SQL_ID DELTA_WRITE_IO_BYTES
TOP_LEVEL_SQL_OPCODE DELTA_INTERCONNECT_IO_BYTES
SQL_EXEC_ID Blocking Session PGA_ALLOCATED
SQL_EXEC_START BLOCKING_SESSION_STATUS TEMP_SPACE_ALLOCATED
SQL_PLAN_HASH_VALUE BLOCKING_SESSION
SQL_PLAN_OPTIONS BLOCKING_SESSION_SERIAL#
BLOCKING_INST_ID
BLOCKING_HANGCHAIN_INFO


AWR SQL stats
SQL> SELECT PLAN_HASH_VALUE,
2 , sum(EXECUTIONS_DELTA) as executions
3 , sum(ROWS_PROCESSED_DELTA) as rows_processed
4 , sum(ELAPSED_TIME_DELTA) as elapsed_time
5 , sum(CPU_TIME_DELTA) as cpu_time, sum(IOWAIT_DELTA) as iowait_time
6 , sum(DIRECT_WRITES_DELTA) as direct_writes
7 , sum(BUFFER_GETS_DELTA) as LIO, sum(DISK_READS_DELTA) as PIO
8 FROM dba_hist_sqlstat
9 WHERE snap_id between 132042 and 132124
10 AND sql_id = 'dkwucgaxxhm2z'
11 GROUP BY sql_id, PLAN_HASH_VALUE, OPTIMIZER_ENV_HASH_VALUE;

Plan IO
Hash Rows Elapsed CPU WAIT Direct
Value EXECUTIONS Processed Time TIME TIME Writes LIO PIO
---------- ---------- ---------- ---------- ---------- ----------- -------- -------- -------
1700726234 1 5997889 7.3806E+10 903837577 73047955458 219418 36152215 6427679

1 row selected.


AWR SQL stats - readable
SQL> SELECT PLAN_HASH_VALUE, executions
2 , round(elapsed_time /1000000/60/60,1) as elapsed_hours
3 , rows_processed
4 , round(cpu_time /elapsed_time * 100) as cpu_time_pct
5 , round(iowait_time /elapsed_time * 100) as iowait_pct
6 , LIO, PIO, direct_writes
7 , round(iowait_time /PIO/1000,1) as mili_sec_PIO
8 FROM(SELECT PLAN_HASH_VALUE
18 GROUP BY sql_id, PLAN_HASH_VALUE);
Plan CPU IO Milisec
Hash Elapsed Rows TIME WAIT Direct /
Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes Pio
---------- ---------- -------- ---------- ---- ----- ---------- ---------- -------- -------
1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4


AWR SQL plan
SQL>select * from table(dbms_xplan.display_awr('dkwucgaxxhm2z', 1700726234, null,
'ALL LAST'));
----------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | | 9553K(100)| |
| 1 | NESTED LOOPS OUTER | | 8011K| 46G| | 9553K (1)| 31:50:39 |
| 6 | HASH JOIN RIGHT OUTER | | 8011K| 44G| | 2338K (1)| 07:47:43 |
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | | |
| 10 | INDEX FAST FULL SCAN | S_SRC_M10 | 6633 | 285K| | 10 (0)| 00:00:01 |
| 12 | INDEX FAST FULL SCAN | S_SRC_P1 | 6633 | 285K| | 8 (0)| 00:00:01 |
| 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| 228M| 2338K (1)| 07:47:42 |
| 14 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 7488K| 142M| | 38163 (2)| 00:07:38 |
| 15 | HASH JOIN RIGHT OUTER | | 8011K| 43G| | 51545 (1)| 00:10:19 |
| 16 | TABLE ACCESS FULL | S_INDUST_LANG | 4023 | 113K| | 199 (0)| 00:00:03 |
| 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8 (0)| 00:00:01 |
| 19 | NESTED LOOPS OUTER | | 8011K| 42G| | 51224 (1)| 00:10:15 |
| 20 | VIEW | | 8011K| 42G| | 51169 (1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 |
| 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | 1 (0)| 00:00:01 |
| 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35 (3)| 00:00:01 |
| 25 | TABLE ACCESS BY INDEX ROWID| S_ORG_EXT | 421K| 419M| | 2693 (1)| 00:00:33 |
| 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | 1 (0)| 00:00:01 |
| 27 | INDEX UNIQUE SCAN | S_OU_PRTNR_TYPE_P1 | 1 | | | 1 (0)| 00:00:01 |
| 28 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 |
| 29 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 |
| 34 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 |
| 35 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------------------
…


Find slow SQL (ASH in AWR)
SQL>SELECT sql_id, sql_type
2 , round(sample_cnt_sql/sample_cnt_total*100) as sample_pct
3 , round(sample_cnt_sql /6/60,1) hours
4 FROM(SELECT DISTINCT sql_id
5 , aa.name AS sql_type
6 , COUNT(*) over (PARTITION BY sql_id) AS sample_cnt_sql
10 JOIN audit_actions aa on (ash.sql_opcode = aa.action)
11 WHERE begin_interval_time BETWEEN
… snip …
18 ORDER BY 3 desc)
19 WHERE rownum <= 3;

------------- -------- ---------- -----
dkwucgaxxhm2z SELECT 25 20.4
grksgbxxuacaf SELECT 21 16.8
gh6w2fm3avn3j SELECT 12 10.1

3 rows selected.


ASH rowsource data (plan line id)
SQL>SELECT sql_id, sql_plan_line_id
2 , round(sample_cnt_plan_line/sample_cnt_total*100) AS sql_plan_line_pct
3 , round(sample_cnt_plan_line/6) minutes
4 FROM(
5 SELECT DISTINCT sql_id, sql_plan_line_id
6 , COUNT(*) over (PARTITION BY sql_plan_line_id) AS sample_cnt_plan_line
10 WHERE sql_id = 'dkwucgaxxhm2z‘ and begin_interval_time BETWEEN
… snip …

SQL_ID SQL_PLAN_LINE_ID SQL_PLAN_LINE_PCT MINUTES
------------- ---------------- ----------------- ----------
dkwucgaxxhm2z 34 48 593
…


ASH rowsource (plan line id + event)
SQL>SELECT sql_id, sql_plan_line_id, event
2 , round(sample_cnt_plan_line_event/sample_cnt_total*100) as plan_line_event_pct
3 , round(sample_cnt_plan_line_event/6) minutes
4 FROM (SELECT DISTINCT sql_id
5 , sql_plan_line_id, nvl(event, 'CPU') as event
6, COUNT(*) over (PARTITION BY sql_plan_line_id,event)AS sample_cnt_plan_line_event
7, COUNT(*) over () AS sample_cnt_total
10 WHERE sql_id = 'dkwucgaxxhm2z' and begin_interval_time BETWEEN …
… snip …

SQL_ID SQL_PLAN_LINE_ID EVENT PLAN_LINE_EVENT_PCT MINUTES
------------- ---------------- ------------------------- ------------------- ----------
dkwucgaxxhm2z 34 db file sequential read 48 591
… snip …

22 rows selected.


ASH rowsource (10g) (object + event)
SQL>SELECT sql_id, object_name, event
2 , round(sample_cnt_object_event/sample_cnt_total*100) as object_event_pct
3 , round(sample_cnt_object_event/6) minutes
4 FROM(SELECT DISTINCT sql_id, object_name, event
5 , count(*) over (partition by object_name,event) as sample_cnt_object_event
6 , count(*) over () as sample_cnt_total
7 FROM ( SELECT sql_id, nvl(event,'CPU') as event
8 , case
9 when wait_class_id in (3871361733,1740759767,427450380,3875070507)
10 then object_name else 'N/A‘ end as object_name
… snip …

SQL_ID OBJECT_NAME EVENT OBJECT_EVENT_PCT MINUTES
------------- -------------------- ------------------------- ---------------- ----------
dkwucgaxxhm2z S_ADDR_PER db file sequential read 52 634
dkwucgaxxhm2z S_ADDR_PER_P1 db file sequential read 28 347
dkwucgaxxhm2z S_ORG_EXT db file sequential read 16 195
… snip …


AWR SQL plan
----------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------------
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | • IO is 99% of elapsed time
| |
| 11 | INDEX FAST FULL SCAN | S_SRC_M12 | 6633 | 285K| | • 75% of time in nested loop
10 (0)| 00:00:01 |
| 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| (2,34,35)
228M| 2338K (1)| 07:47:42 |
| 15 |
| 16 |
HASH JOIN RIGHT OUTER
TABLE ACCESS FULL
|
| S_INDUST_LANG
| 8011K|
| 4023 |
43G|
113K| |
• Query is processing a large
| 51545
199
(1)| 00:10:19 |
(0)| 00:00:03 |
| 17 | HASH JOIN RIGHT OUTER | | 8011K| 43G| data set
| 51289 (1)| 00:10:16 |
| 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8 (0)| 00:00:01 |
| 20 | VIEW | | 8011K| 42G| • Nested loop is not an efficient
| 51169 (1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
method for joining large data
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 |
| 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | sets
1 (0)| 00:00:01 |
| 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35 (3)| 00:00:01 |
| 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | • Why does the CBO choose this
1 (0)| 00:00:01 |
inefficient plan ?!
| 29 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 – (0)| 00:00:01 |
Bad stats?
| 31 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 – (0)| 00:00:01 |
Incorrect optimizer settings?
– Or something else?
----------------------------------------------------------------------------------------------------------------------
…


AWR SQL stats
SQL> SELECT PLAN_HASH_VALUE, executions
2 , round(elapsed_time /1000000/60/60,1) as elapsed_hours
3 , rows_processed
4 , round(cpu_time /elapsed_time * 100) as cpu_time_pct
5 , round(iowait_time /elapsed_time * 100) as iowait_pct
6 , LIO, PIO, direct_writes
7 , round(iowait_time /PIO/1000,1) as mili_sec_PIO
8 FROM(SELECT PLAN_HASH_VALUE
18 GROUP BY sql_id, PLAN_HASH_VALUE);
Plan CPU IO Milisec
Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes Pio
---------- ---------- -------- ---------- ----- ---- ---------- ---------- -------- -------
1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4


AWR SQL Optimizer Env values
SQL>select * from table(dbms_xplan.display_awr('dkwucgaxxhm2z', 1700726234, null,
'ALL LAST +OUTLINE'));
---------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------

..snip…

Outline Data
-------------
/*+
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE('11.1.0.7')
DB_VERSION('11.1.0.7')
OPT_PARAM('_b_tree_bitmap_plans' 'false')
OPT_PARAM('_optim_peek_user_binds' 'false')
OPT_PARAM('optimizer_index_cost_adj' 10)
ALL_ROWS
OUTLINE_LEAF(@"SEL$335DD26A")
MERGE(@"SEL$3")
..snip…


AWR SQL plan
----------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | | |
| 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| 228M| 2338K (1)| 07:47:42 |
| 16 | TABLE ACCESS FULL | S_INDUST_LANG | 4023 | 113K| | 199 (0)| 00:00:03 |
| 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8Direct
(0)| 00:00:01 |
| 20 | VIEW | | 8011K| 42G| | 51169Writes
(1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169--------
(1)| 00:10:15 |
| 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | 1 (0)| 00:00:01 |
| 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35219,418
(3)| 00:00:01 |
| 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | 1 (0)| 00:00:01 |
| 28 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1workarea_size_policy=manual
(0)| 00:00:01 |
| 30 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1hash_area_size=2147483647
(0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------------------
…


New SQL Plan
select * from table(dbms_xplan.display_cursor(null, null, 'ALLSTATS LAST'));
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E -Rows |E-Bytes| Cost (%CPU)| A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | | 674K(100)| 0 |00:11:40.62 | 16M| 3031K| | | |
|* 1 | HASH JOIN RIGHT OUTER | | 1 | 17M| 98G| 674K (2)| 5352K|01:20:42.29 | 16M| 3031K| 1143K| 1143K| 74M (0)|
| 2 | VIEW | index$_join$_010 | 1 | 6633 | 285K| 383 (1)| 6707 |00:00:01.93 | 749 | 725 | | | |
|* 3 | HASH JOIN | | 1 | | | | 6707 |00:00:01.93 | 749 | 725 | 1143K| 1143K| 73M (0)|
|* 4 | HASH JOIN | | 1 | | | | 6707 |00:00:01.07 | 680 | 659 | 1117K| 1117K| 73M (0)|
| 5 | INDEX FAST FULL SCAN | S_SRC_M10 | 1 | 6633 | 285K| 103 (0)| 6707 |00:00:00.08 | 338 | 328 | | | |
| 6 | INDEX FAST FULL SCAN | S_SRC_M12 | 1 | 6633 | 285K| 100 (0)| 6707 |00:00:00.17 | 342 | 331 | | | |
| 7 | INDEX FAST FULL SCAN | S_SRC_P1 | 1 | 6633 | 285K| 76 (0)| 6707 |00:00:00.04 | 69 | 66 | | | |
|* 8 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 41G| 674K (2)| 5352K|01:20:08.10 | 16M| 3030K| 1208M| 33M| 1455M (0)|
| 9 | TABLE ACCESS FULL | S_ADDR_PER | 1 | 13M| 744M| 107K (2)| 14M|00:01:53.38 | 392K| 392K| | | |
| 11 | TABLE ACCESS FULL | S_ADDR_PER | 1 | 13M| 744M| 107K (2)| 14M|00:02:21.69 | 392K| 392K| | | |
| 13 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 1 | 7488K| 142M| 38163 (2)| 6777K|00:01:55.31 | 173K| 138K| | | |
| 15 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:04:56.19 | 154K| 102K| | | |
| 17 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:00:48.23 | 102K| 19057 | | | |
| 19 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:00:00.01 | 102K| 0 | | | |
|* 20 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 40G| 340K (2)| 5352K|01:05:35.55 | 15M| 1986K| 915K| 915K| 74M (0)|
| 21 | TABLE ACCESS FULL | S_INDUST_LANG | 1 | 4023 | 113K| 199 (0)| 4023 |00:00:00.04 | 713 | 709 | | | |
| 23 | TABLE ACCESS FULL | S_ORG_PRTNR | 1 | 324 | 43740 | 8 (0)| 319 |00:00:00.02 | 23 | 21 | | | |
| 25 | VIEW | index$_join$_014 | 1 | 1 | 94 | 1 (100)| 0 |00:00:00.06 | 3 | 2 | | | |
|* 26 | HASH JOIN | | 1 | | | | 0 |00:00:00.06 | 3 | 2 | 1269K| 1269K| 64M (0)|
| 27 | INDEX FAST FULL SCAN| S_OU_PRTNR_TYPE_P1 | 1 | 1 | 94 | 0 (0)| 0 |00:00:00.02 | 3 | 2 | | | |
| 28 | INDEX FAST FULL SCAN| S_OU_PRTNR_TYPE_U1 | 0 | 1 | 94 | 0 (0)| 0 |00:00:00.01 | 0 | 0 | | | |
| 29 | VIEW | | 1 | 7460K| 38G| 340K (2)| 5352K|01:05:12.63 | 15M| 1985K| | | |
|* 30 | HASH JOIN | | 1 | 7460K| 7527M| 340K (2)| 5352K|01:05:12.59 | 15M| 1985K| 1078K| 1078K| 72M (0)|
| 31 | INDEX FULL SCAN | SO_ANA_BU_U2 | 1 | 19 | 285 | 1 (0)| 19 |00:00:00.02 | 1 | 1 | | | |
| 32 | TABLE ACCESS FULL | S_ORG_EXT | 1 | 7068K| 7030M| 339K (2)| 6887K|01:05:18.93 | 15M| 1985K| | | |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Evaluating the changes

Before:
Plan CPU IO Milisec
Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- ---------- ---- ---- ----------- ---------- -------- ----------
1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4

After:
Plan CPU IO Milisec
Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- ---------- ---- ---- ----------- ---------- -------- ----------
2491926841 1 1.3 5,352,381 5 96 13,134,981 2,651,076 0 2

 Reduced elapsed time by 94%, Logical IO by 63% and Physical IO by 59%
 But increased IOPS by 6x => ensure that disk subsystem can handle it !!!


Use Case 2 – Find sql & stats
Top 3 SQL for the time period
------------- ---------- ---------- ----------
dvaxm0c9tdj80 INSERT 11 3.9
77uh2gt7ytnm9 SELECT 9 3.2
12dy8wkk6x5b5 INSERT 7 2.6

Stats for SQL dvaxm0c9tdj80
Plan CPU IO milisec
Hash Elapsed Rows time WAIT Direct /
DAY Value executions Minutes processed PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- -------- ---------- ---- ---- ----------- ---------- -------- --------
2011-06-29 4061473922 1 118 9,289 1 99 449,940 406,585 0 17
2011-06-30 4061473922 1 236 6,754 0 100 408,245 406,481 0 35
2011-07-01 4061473922 1 89 352 1 99 406,564 406,349 0 13
2011-07-02 4061473922 1 145 7,383 1 99 410,586 406,358 0 21
2011-07-03 4061473922 1 220 13,944 1 100 413,684 406,027 0 32


Use Case 2 – SQL details
SQL PLan
SQL>select * from table(dbms_xplan.display_awr('dvaxm0c9tdj80', 4061473922, null,
'ALL LAST’));

-------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 24155 (100)| |
| 1 | INDEX FULL SCAN | S_PARTY_W1 | 911K| 17M| 24155 (1)| 00:04:50 |
-------------------------------------------------------------------------------

SQL Text
INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
(ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
SELECT ROW_ID, 1, 'I', LAST_UPD
FROM S_PARTY
WHERE S_PARTY.LAST_UPD > :1;


Use Case 2 – SQL rowsource

SQL_ID SQL_PLAN_LINE_ID EVENT PLAN_LINE_EVENT_PCT MINUTES
------------- ---------------- ------------------------- ------------------- ----------
dvaxm0c9tdj80 2 db file sequential read 100 235
dvaxm0c9tdj80 2 CPU 0 1

Stats for SQL dvaxm0c9tdj80

Plan CPU IO milisec
Hash Elapsed Rows time WAIT Direct /
DAY Value executions Minutes processed PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- -------- ---------- ---- ---- ----------- ---------- -------- --------
2011-06-29 4061473922 1 118 9,289 1 99 449,940 406,585 0 17
2011-06-30 4061473922 1 236 6,754 0 100 408,245 406,481 0 35
2011-07-01 4061473922 1 89 352 1 99 406,564 406,349 0 13
2011-07-02 4061473922 1 145 7,383 1 99 410,586 406,358 0 21
2011-07-03 4061473922 1 220 13,944 1 100 413,684 406,027 0 32


Use Case 2 – Index Structure
*SQL> @i S_PARTY

Type Unq? Part? Index name STATUS Column name DATA_TYPE
------ ---- ------ ----------- -------- -------------- ---------
NORMAL NO NO S_PARTY_F1 VALID PAR_PARTY_ID VARCHAR2
S_PARTY_M1 VALID ROOT_PARTY_FLG CHAR
… snip …
NO NO S_PARTY_W1 VALID LAST_UPD DATE
VALID ROW_ID VARCHAR2
*Output based on modified version of Tim Gorman’s script at
www.evdbt.com/tool.htm
 IO is 99% of elapsed time
SQL Text  Using perfect index, but wrong
access method
INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70  Why does the CBO choose this
(ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
inefficient plan ?!
SELECT ROW_ID, 1, 'I', LAST_UPD
FROM S_PARTY  Bad stats?
WHERE S_PARTY.LAST_UPD > :1;  Incorrect optimizer settings?
 Or something else?
INDEX FULL SCAN | S_PARTY_W1


Use Case 2 – Bad stats?
SQL>select * from table(dbms_xplan.display_awr('dvaxm0c9tdj80', 4061473922, null,
'ALL LAST’));

-------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 24155 (100)| |
| 1 | INDEX FULL SCAN | S_PARTY_W1 | 911K| 17M| 24155 (1)| 00:04:50 |
-------------------------------------------------------------------------------


Use Case 2 – Replicate take 1
SQL>explain plan for
2 INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
3 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
4 SELECT ROW_ID, 1, 'I', LAST_UPD
5 FROM S_PARTY
6 WHERE S_PARTY.LAST_UPD > :x;
Explained.
SQL>select * from table(dbms_xplan.display(null, null, 'ALLSTATS LAST'));
… snip …
------------------------------------------------------------
| Id | Operation | Name | E-Rows |
------------------------------------------------------------
| 0 | INSERT STATEMENT | | 913K|
| 1 | LOAD TABLE CONVENTIONAL | S_ETL_I_IMG_70 | |
|* 2 | INDEX RANGE SCAN | S_PARTY_W1 | 913K|
------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("S_PARTY"."LAST_UPD">:X)

Object Statistics History
 WRI$_OPTSTAT_TAB_HISTORY
 WRI$_OPTSTAT_IND_HISTORY
 Etc.


Use Case 2 – bind data type
SQL>select distinct name, was_captured, last_captured
2 , sys.anydata.gettypename(value_anydata) as data_type
3 , sys.anydata.accesstimestamp(value_anydata) as data_value
4 from DBA_HIST_SQLBIND where sql_id = 'dvaxm0c9tdj80’ order by last_captured;

was
NAME captured LAST_CAPTURED DATA_TYPE DATA_VALUE
----- -------- ------------------------- -------------------- --------------------
:1 YES 2011-06-29 17:51:44 SYS.TIMESTAMP 2011-06-28 08:12:58

Type Unq? Part? Index name STATUS Column name DATA_TYPE
------ ---- ------ ----------- -------- -------------- ---------
NORMAL NO NO S_PARTY_F1 VALID PAR_PARTY_ID VARCHAR2
S_PARTY_M1 VALID ROOT_PARTY_FLG CHAR
… snip …
NO NO S_PARTY_W1 VALID LAST_UPD DATE
VALID ROW_ID VARCHAR2


Use Case 2 – Replicate (2nd try)
SQL>declare
2 x timestamp;
3 begin
4 INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
5 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
6 SELECT ROW_ID, 1, 'I', LAST_UPD
7 FROM S_PARTY
8 WHERE S_PARTY.LAST_UPD > :x;
9 end;
10 /

----------------------------------------------------------
| Id | Operation | Name | E-Rows |
----------------------------------------------------------
| 0 | INSERT STATEMENT | | |
| 1 | LOAD TABLE CONVENTIONAL | | |
|* 2 | INDEX FULL SCAN | S_PARTY_W1 | 913K|
----------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter(INTERNAL_FUNCTION("S_PARTY"."LAST_UPD")>:B1)


Use Case 2: Visualizing the
improvement

IMG Process Duration
10

9

8

7
Elapsed Time (Hours)

6

5

4

3

2

1

0
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
ETL Day


Free script collections

• Tanel Poder
• tech.e2sn.com/oracle-scripts-and-tools
• Kerry Osborne -
• kerryosborne.oracle-guy.com/papers/my_favorite_scripts_2010.zip
• Tim Gorman
• evdbt.com/tools.htm
• Adrian Billington
• oracle-developer.net/utilities.php
• Jonathan Lewis
• jonathanlewis.wordpress.com/2009/12/18/simple-scripts/


Resources
SQL Tuning Forms Tuning
– Trace files – Forms Tracing (MOS Doc : 373548.1)
– SQLT output (MOS Doc: 215187.1) – Generic MOS Doc : 438652.1
– Trace Analyzer (MOS Doc :
224270.1)
Reports Tracing
– AWR Report (MOS Doc : 748642.1)
– MOS Doc: 111311.1
– AWR SQL Report (awrsqrpt.sql)
– 11g Real-Time SQL Monitoring
Database Tuning
– SQL Tuning Advisor
– AWR Report (MOS Doc : 748642.1)
– ADDM report (MOS Doc : 250655.1)
PL/SQL Tuning – Active Session History (ASH) Report
– Product logs – LTOM output (MOS Doc : 352363.1)
– PL/SQL Profiler (MOS Doc :
808005.1)
OS
– OSWatcher (MOS Doc : 301137.1)
Middletier Tuning
– JVM Logs
– Third Party Tools


To learn even more

• Performance Fundamentals for Oracle Database 10g and 11g
by Graham Wood and John Beresniewicz
(http://www.oracle.com/technetwork/database/focus-areas/manageability/db-perf-tuning-ow08-131582.pdf)

• ASH website by Kyle Hailey
(https://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history)

• Expert Oracle Database Architecture: Oracle Database
9i, 10g, and 11g Programming Techniques and Solutions by
Thomas Kyte

• Troubleshooting Oracle Performance by Christian Antognini

• Blogs
– Tanel Poder, Kerry Osborne, Jonathan Lewis, Cary Millsap, Greg
Rahn, Randolf Geist, etc.


Summary

• Ask questions
• Understand the database internals
• Use multiple tools depending on the situation
• Combine the best of both (i.e. PerfSheet from Tanel
Poder)


Q&A


Sql Performance Tuning with ASH & AWR: Real World Use Cases

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (7)

Similar a Sql Performance Tuning with ASH & AWR: Real World Use Cases

Similar a Sql Performance Tuning with ASH & AWR: Real World Use Cases (20)