Presentation on 11g Database Upgrade Best Practices by Ashish Agrawal and Mirella Gazzoni given in Cleveland, Ohio on August 20th, 2009.

1. <Insert Picture Here>
Upgrading to 11g – Best Practices
Ashish Agrawal Mirella Gazzoni
Senior Principal Technical Support Engineer Customer Support Manager
Center of Excellence Customer Support Education

2. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
Real Application Testing
Q&A

3. Lifetime Support Policy
today
August 2012 August 2015
July 2010 July 2013
R2
January 2009 January 2012
Sustaining Support
Premier Support Extended Support
R2 July 2007 July 2008 July 2010
t
2011
2013
2014
2002
2003
2004
2005
2006
2007
2008
2009
2010
2012
2016
2015
http://www.oracle.com/support/library/brochure/lifetime-support-technology.pdf

4. Lifetime Support Policy Deliverables
Feature Premier Extended Sustaining
Support Support Support
Major Product and Technology Releases
Technical Support
Access to Knowledge Base
(MetaLink/Customer Connection/SupportWeb)
Updates, Fixes, Security Alerts and Critical Patch Pre-existing Only
Pre-
Updates
Tax, Legal and Regulatory Updates No
Upgrade Scripts No
Certification with existing Third Party No
Products/Versions
Certification with New Third Party No No
Products/Versions
Certification with new Oracle Products No

5. Documentation
• Note: 466181.1 Upgrade Companion 10gR2
• Note: 601807.1 Upgrade Companion 11g

6. Documentation
• Note: 601807.1 Upgrade Companion 11g

7. Documentation
• Note: 601807.1 Upgrade Companion 11g

8. My Oracle Support Configuration Manager
What is it?
A Support capability that automates configuration information
exchange between Oracle and our customers enabling faster
resolution and proactive detection of customer issues. My Oracle
Support utilizes core configuration management capabilities available
from Oracle Enterprise Manager and provides:
• The ability to define configurations and organize projects
• A view of System details and changes
• Create, track, and status Service Requests
• Advanced Knowledge Management capabilities
• Proactive problem avoidance with HealthChecks
• Proactive Product and Security Alerts

9. OCM: The Big Picture
My Oracle Support or MetaLink
Support
Config View
SCM
Knowledge
Management
Oracle CCR
HTTPS
Proxy
Server
Oracle Live Link ™
Oracle Live Link
Oracle Support Live Link™
Oracle Live Link helps Oracle provide you timely and
focused Support.
A small amount of information about your environment is
sent to Oracle support to better handle your requests.
Simple install/
To see an example of the information, select this link.
Enable Oracle Live Link
Disable Oracle Live Link
command UI < Back Next > Cancel
Customer Site

10. What is a system?
Systems are named collections of hardware, software, and
operating system running on a machine

11. System Health – Health Checks and Patch Advice

12. System Health Dashboard

13. Inventory and Usage - Details Page
To change from graphical view

14. Sample System Data being pulled …
Application Server
Application Server JServ level Database level eBusiness Suite
eBusiness Suite
level JServ level
• Configuration files
Database level
• Config version
• # of active users
• # of active users
•
level
Configuration files
• Configuration files
• Zones •
• Config version
Control files
• Applied patches
• Applied patches
• Configuration files • Zones • Control files • Custom forms
• Console configuration • Data files • Custom forms
• Console configuration • Data files • Applications system summary
• DCM configuration • Instance information • Applications system summary
• DCM configuration • Instance information • Applications topology
• Deployed applications • Feature usage • Applications topology
• Deployed applications • Feature usage • Comprising patches
• General • High water mark • Comprising patches
• General • High water mark • Component versions
• OPMN configuration • Initialization parameters • Component versions
• OPMN configuration • Initialization parameters • Config version
• Session high water mark • Config version
• Session high water mark • Custom applications
• Options • Custom applications
• Options • Profile options registered under custom app.
HTTP Server level
HTTP Server level
• Redo logs
• Redo logs •
• Profile options registered under custom app.
Custom database objects
• Cipher suites • Rollback segments • Custom database objects
• Rollback segments • Cust. Hooks
• Cipher suites • SGA • Cust. Hooks
• Configuration files • SGA • Custom message count
• Configuration files • Tablespaces • Custom message count
• General • Tablespaces • Custom objects owned by Apps
• General • Database users • Custom objects owned by Apps
• Listen addresses • Database users • Custom request sets
• Listen addresses • Statspack configuration • Custom request sets
• Performance related settings • Statspack configuration • Responsibility count
• Performance related settings • Responsibility count
• Virtual hosts • Custom triggers
• Virtual hosts • Custom triggers
• Custom value sets – table validated
• Custom value sets – table validated
• Custom value sets – other
• Custom value sets – other
• Custom workflows
Infrastructure level
Infrastructure level •
• Custom workflows
Data groups
Host level
Host level • Apps web configuration
• Apps web configuration •
• Data groups
Database version
• Operating system initialized services • Apps web configuration files • Database version
• Operating system initialized services • Apps web configuration files • End-Dated users
• Operating system ULIMITs • Context files • End-Dated users
• Operating system ULIMITs • Context files • Generic service components parameters
• Hardware • Database configuration • Generic service components parameters
• Hardware • Database configuration • High priority patches not applied
• CPUs • Database configuration files • High priority patches not applied
• CPUs • Database configuration files • JTF hooks
• IO devices • Custom context variables • JTF hooks
• IO devices • Custom context variables • Localization modules
• Network interfaces • Environment context variables • Localization modules
• Network interfaces • Environment context variables • NSL languages
• Operating system • Global context variables • NSL languages
• Operating system • Global context variables • Other customizations
• Component • Host context variables • Other customizations
• Component • Host context variables • Application patch levels
• File system • Install context variables • Application patch levels
• File system • Install context variables • Printers
• Modules • Process context variables • Printers
• Modules • Process context variables • Printer drivers
• OS parameters • System context variables • Printer drivers
• OS parameters • System context variables • Profile options
• Registered software • Technology stack inventory (file system) • Profile options
• Registered software • Technology stack inventory (file system) • Registered Oracle users
• Workflow mailer configuration • Registered Oracle users
• Workflow mailer configuration • Technology stack inventory (DB)
• Workflow mailer configuration files • Technology stack inventory (DB)
• Workflow mailer configuration files

15. Configuration Manager
Oracle Configuration Manager and Oracle Enterprise Manager
• You do not need to install Enterprise Manager (EM) in
order to use the Oracle Configuration Manager.
• If EM Agent is already installed, you may also install the
Oracle Configuration Manager
• EM and Oracle Configuration Manager send data to
different repositories: Enterprise Manager sends to EM
Repository, Oracle Configuration Manager sends to
Customer Configuration Repository at Oracle Support.
• To manage your Oracle system with EM and also send
Configuration data back to configuration manager in My
Oracle Support or MetaLink, each must be installed on the
system.

16. Service Request Severity Definitions
Severity Level 4 No Business Impact
No loss of service or resources
Severity Level 3 Minor Business Impact
Minor loss of service or resources
Severity Level 2
Serious Business Impact
Severe loss of service or resources w/o acceptable workaround
Severity Level 1 Critical Business Impact
Complete loss of service or resources and work cannot
reasonably continue - the work is considered “mission critical”

17. Escalation Process
Call US: 800 223 1711
Other Global Support Hot Lines:
V.P. or
http://www.oracle.com/support/contact.html
Executive
Senior Manager
or Director
Manager /
Escalation Manager
Support Representative
Customer

18. Global Customer Care
Call US: 800 223 1711
Other Global Support Hot Lines:
http://www.oracle.com/support/contact.html
Global Customer Care is a highly available expert
resource that resolves customers' business-related
inquiries while capturing and sharing feedback to
enhance the customer relationship.
Responsibilities include but are not limited to:
• Access and navigation of Oracle Support websites (including Metalink)
• Placing software orders
• Assisting with CSI questions
• Assisting with product or platform issues related to SR logging
• Locate and provide published information
• Create software upgrade orders

19. Oracle Support Services
http://www.oracle.com/support

20. Oracle Advisor Webcasts Now Available
Through My Oracle Support and MetaLink!
• A new way to receive information about your Oracle E-
Business Suite solutions
• Go to MetaLink Note 398877.1 - Welcome to the
Oracle Advisor Webcast Program!
• Includes links to the Advisor Webcast page
• Instructions for registering and viewing both live and archived
webcasts
• The current menu of scheduled webcasts to choose from

21. Process and Tools Advisor Webcasts
http://www.oracle.com/support/seminars.html

22. Customer Support Education
Delivery Options
• One-on-One Customer meetings and Webcasts
• Support Webcasts – targeting a core Customer audience
• Regularly scheduled free Internet Seminars
• Regional Support Education Workshops
• User Groups, SIGs, RUGs, etc.
Value to you
Following attendance at one of our presentations, Customers have
experienced, on average:
• Up to a 40% reduction in volume of new Service Requests
• As much as a 70% reduction in issues requiring further
escalation
• A 10% increase in overall Customer Satisfaction

23. Customer Support Education
Contact us via the email addresses below, or contact a
member of the CSE Team directly
• External email:
• North America:
• support-training_us@oracle.com

24. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
Real Application Testing
Q&A

25. Upgrade Paths
UPGRADE
UPGRADE
Export/Import
Export/Import
Y
Y OS Change?
OS Change? N
N
CTAS, COPY
CTAS, COPY
N
N Downtime >45min?
Downtime >45min?
SQL Apply
SQL Apply
Y
Y
Oracle Streams
Oracle Streams
DBUA
DBUA
Transportable Tablespaces
Transportable Tablespaces
OR
ORA
CLI
CLI
ACL
CLEE rrec
ecom SQL> @catupgrd
Transportable Database
Transportable Database omm
men
ende
dedd

26. Challenges
• >90% of highly visible problems attributed to an upgrade do
not occur while upgrading but appear as unanticipated
performance degradations days or weeks after the upgrade
operation completed
• Usually too late to schedule another extended outage to downgrade
• Other changes have been introduced elsewhere to make the
downgrade even more complex
• Typical root causes:
• Optimizer regression - execution plan chosen is worse
• Queries are slow
• Reports or batches take longer to complete
• Increased overall resource requirements
• Changed behavior / new bottlenecks under high load

27. Best Practices – Preserve
• Preserve as much information as possible BEFORE you
upgrade the production environment to the new release -
that's of vital importance!!!
• Capture accurate performance statistics to act as baseline
• Allows comparison: before ↔ after
• Measure query and batch timings
• System level performance stats (OS Watcher / statspack /
AWR)
• Save execution plans
• Save old configuration information (RDA / OCM)

28. Best Practices – Preserve
• Keep a backup copy of your current optimizer
statistics. See Note 465787.1
• Incorrect / inaccurate optimizer statistics is one cause
of suboptimal execution plans.
• Save execution plans

29. Best Practices – Pre Upgrade
• Upgrade and stress test a test database first
• Do functional testing
• Always use real production data
• Create a copy of the production database with RMAN
• Or leverage a standby DB if multiple exist
• Run performance tests with real life loads!!!
• Rehearse both upgrade and back out procedures
• Consider other dependent components that have to be shut
down too
• Should be able to do steps in your sleep
• Most production systems upgrade during off-hours
• Use a buddy system to make sure checklists are followed

30. Pre Upgrade
• Create dictionary statistics prior to the upgrade - otherwise
it will take significantly longer
SQL> EXECUTE dbms_stats.gather_schema_stats
SQL> EXECUTE dbms_stats.gather_schema_stats
('SYS',
('SYS',
options
options =>
=> 'GATHER',
'GATHER',
estimate_percent
estimate_percent =>
=> DBMS_STATS.AUTO_SAMPLE_SIZE,
DBMS_STATS.AUTO_SAMPLE_SIZE,
method_opt
method_opt =>
=> 'FOR ALL COLUMNS SIZE AUTO',
'FOR ALL COLUMNS SIZE AUTO',
cascade
cascade =>
=> TRUE);
TRUE);
• Detailed scripts can be found here:
Oracle® Database Upgrade Guide Appendix B
• Or in / :
SQL> EXECUTE dbms_stats.gather_dictionary_stats;
SQL> EXECUTE dbms_stats.gather_dictionary_stats;

31. Post Upgrade
• Create system statistics during a regular workload period -
otherwise non-appropriate values for the CBO will be used:
SQL>
SQL> EXECUTE dbms_stats.gather_system_stats('start');
EXECUTE dbms_stats.gather_system_stats('start');
...
...
SQL>
SQL> EXECUTE dbms_stats.gather_system_stats('stop');
EXECUTE dbms_stats.gather_system_stats('stop');
SQL> select pname NAME, pval1 WERT, pval2 INFO
SQL> select pname NAME, pval1 WERT, pval2 INFO
from aux_stats$;
from aux_stats$;
NAME
NAME WERT INFO
WERT INFO
-------------------- ---------- ------------------------------
-------------------- ---------- ------------------------------
STATUS
STATUS COMPLETED
COMPLETED
DSTART
DSTART 04-07-2008 12:30
04-07-2008 12:30
DSTOP
DSTOP 05-07-2008 12:30
05-07-2008 12:30
FLAGS
FLAGS 1
1
CPUSPEEDNW
CPUSPEEDNW 1392,39
1392,39
IOSEEKTIM
IOSEEKTIM 11,405
11,405
IOTFRSPEED
IOTFRSPEED 25595,605
25595,605
...
...

32. Post Upgrade
• Create fixed table statistics soon after the upgrade:
SQL> execute
SQL> execute
dbms_stats.gather_fixed_objects_stats;
dbms_stats.gather_fixed_objects_stats;

33. Validate statistics gathering strategy
Default for DBMS_STATS is
DBMS_STATS.GATHER_SCHEMA_STATS(OWNNAME=>‘NAME',-
DBMS_STATS.GATHER_SCHEMA_STATS(OWNNAME=>‘NAME',-
ESTIMATE_PERCENT=>DBMS_STATS.AUTO_SAMPLE_SIZE,-
ESTIMATE_PERCENT=>DBMS_STATS.AUTO_SAMPLE_SIZE,-
AUTO',-
METHOD_OPT=>'FOR ALL COLUMNS SIZE AUTO',-
=>'AUTO',-
GRANULARITY =>'AUTO',-
CASCADE=>DBMS_STATS.AUTO_CASCADE,-
CASCADE=>DBMS_STATS.AUTO_CASCADE,-
NO_INVALIDATE=>DBMS_STATS.AUTO_INVALIDATE);
ESTIMATE_PERCENT is set to AUTO_SAMPLE_SIZE.
In 11g You are encouraged to use AUTO_SAMPLE_SIZE for ESTIMATE_PERCENT .
ESTIMATE_PERCENT
versions.
In 11g AUTO_SAMPLE_SIZE is very fast compared to earlier versions.
and gives accuracy of close to 100 % sample size.
AUTO_SAMPLE_SIZE uses a new Hash-based Sampling for Column Statistics.
Hash-

34. Validate statistics gathering strategy
• New Features in dbms_stats in 11g
• Multicolumn Statistics
• Expression Statistics
• Statistic Preferences

35. Best Practices
• Possibilities for performance analysis and
prediction:
• STATSPACK
• AWR
• SQL Plan Management
• Real Application Testing
• SQL Performance Analyzer
• Database Replay

36. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
Real Application Testing
Q&A

37. Automatic Workload Repository
• Since Oracle 10g: AWR (Automatic Workload Repository)
• Statistics repository in the SYSAUX tablespace
• Contains AWR snapshots (current and past periods)
• Requires a license for Diagnostic Pack
• Command line report:
• @?/rdbms/admin/awrrpt.sql
• Export the AWR:
begin
begin
DBMS_SWRF_INTERNAL.AWR_EXTRACT(
DBMS_SWRF_INTERNAL.AWR_EXTRACT(
dmpfile => 'awr_data.dmp',
dmpfile => 'awr_data.dmp',
dmpdir => 'TMP_DIR',
dmpdir => 'TMP_DIR',
bid
bid => 302,
=> 302,
eid
eid => 305);
=> 305);
end;
end;
/
/

38. STATSPACK
• Database performance tool - pre-10g
• Note:394937.1 STATSPACK guide
• Export the PERFSTAT user

39. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
Real Application Testing
Q&A

40. Challenge
How do I ensure
plan stability???

41. Plan Stability Techniques
• Optimizer - prevent execution plan changes:
• Classical approach:
• Rule Based Optimizer (Please note: RBO desupport since Oracle 10g)
• Hints
• Stored Outlines
• Rewriting SQL statements
• optimizer_features_enabled=n.n.n
• Change specific optimizer parameters
• Import and fix object and system statistics
• Modern, efficient and better resource consumption:
• SQL Plan Management
• SQL Profiling

42. Without SQL Plan Management
• Challenging to "freeze" execution plans and statistics
• Difficulty:
• Statement has been parsed and a plan got created.
Verification happens during execution:
GB
Parse Execute Plan acceptable
HJ
HJ
• Now some conditions get changed (e.g. upgrade)
• A new plan will be created - is it better or worse???
GB
Parse Execute Plan possibly
NL
not acceptable
NL

43. SQL Plan Management
• First preventive and fully transparent database mechanism
to ensure plan stability
• SQL execution plans will be recorded
• A SQL Baseline will be created
• Consists of accepted execution plans
• Contains only plans for statements being parsed/executed more
than once
• Only known and verified and accepted plans will be used
• Package: DBMS_SPM

44. SQL Plan Management
• 3 phases for plan stability:
• Capture
• Create a SQL Baseline representing trusted execution plans
• Baseline is stored in SQL Management Base in tablespace
SYSAUX
• Selection
• Only accepted plans will be used
• New execution plans will be recorded in the plan history
• Evolution
• Evaluate all unverified plans for a given statement in the plan
history to become either accepted or rejected

45. SQL Plan Management
• White Paper:
• http://www.oracle.com/technology/products/
manageability/database/pdf/ow07/
spm_white_paper_ow07.pdf

46. SQL Plan Management
• Phase 1 - Capture
• OPTIMIZER_CAPTURE_SQL_PLAN_BASELINE=TRUE (Default: FALSE)
GB
Parse Execute Plan acceptable
HJ
HJ
Initial plan SQL MANAGEMENT BASE
will be accepted Plan History Residing in SYSAUX TS.
Will occupy max. 10% of SYSAUX.
if repeated and Weekly job will delete plans
will be added to the not used since 53 weeks [default].
SQL Plan Baseline Plan Baseline
GB
HJ
HJ

47. SQL Plan Management
• Phase 2 - Selection
• OPTIMIZER_USE_SQL_PLAN_BASELINE=TRUE (Default: TRUE)
• Same statement parsed again but a different plan will be created
GB
Parse
NL
NL
New plan will be
added to the Plan
History but it won't Plan History
be used unless it
has been verified GB
NL
Plan Baseline
NL
GB
HJ
HJ

48. SQL Plan Management
• Phase 2 - Selection
• OPTIMIZER_USE_SQL_PLAN_BASELINE=TRUE
GB
Parse Execute Plan known and
HJ
acceptable
HJ
The optimizer will use Plan History
only one of the
VERIFIED plans stored GB
in the SQL Baseline NL
Plan Baseline
because only these will NL
guarantee PLAN GB
STABILITY HJ
HJ

49. SQL Plan Management
• Phase 3 - Evolution
Plan History
Worse plan will Plan History
be kept in the
Plan History GB
GB
GB Plan Baseline and deleted NL Plan Baseline
NL
NL
automatically NL
GB GB
NL GB
NL NL HJ
HJ
NL
HJ
HJ
Equal or better plans
can be added to the SQL
Plan Baseline
DBA schedules
verification
DBA
Optimizer

50. SQL Plan Management
• DatabaseControl:

51. SQL Plan Management
• DatabaseControl - Configuration:
Start with 5-14 weeks:
exec DBMS_SPM.CONFIGURE('plan_retention_weeks',5);

52. SQL Plan Management
• DatabaseControl - details:

53. SQL Plan Management
• DatabaseControl - evolve:

54. SQL Plan Management - Scenarios
• Upgrade scenario 1:
• Parameterize the optimizer back to the "old" behaviour
• Works for all database releases since 8.0.3
• Upgrade scenario 2:
• Transport all well known plans to the new release
• Works since 10gR2
• Ship a new application (module) along with appropriate
SQL plan baselines:
• Ship the best execution plans for the new statements within a
staging table and ensure that they'll be used
• Works since 10gR2

55. SQL Plan Management - Upgrade 1
• Upgrade scenario 1
STS
Repeatable plans will be added
to the Plan Baseline upon 2nd Now: Different
execution plans created
with OFE=11
Plan History will be added to
GB the Plan History
NL for later verification
Plan Baseline
NL
GB GB GB
NL HJ NL
NL NL
HJ
OPTIMIZER_FEATURES_ENABLE=9.2.0 OPTIMIZER_FEATURES_ENABLE=11.1.0
OPTIMIZER_CAPTURE_SQL_PLAN_BASELINE=TRUE OPTIMIZER_CAPTURE_SQL_PLAN_BASELINE=FALSE

56. SQL Plan Management - Upgrade 2
• Upgrade scenario 2
Staging
STS exp imp
STS
Table expdp impdp
DB-Link ...
10.2 plans will be added
to the SQL Plan Baseline
Plan History
GB
NL
Plan Baseline
NL
GB GB GB
NL HJ NL
NL NL
HJ
Every new better plan found in 11g
will be added to the Plan History

57. SQL Plan Management - New Application
• New application (module) gets delivered
@Vendor @Customer
DBMS_SPM.CREATE_STGTAB_BASELINE
Staging Staging
exp imp
expdp impdp Table
Table
DBMS_SPM.UNPACK_STGTAB_BASELINE
DBMS_SPM.PACK_STGTAB_BASELINE
Plan Baseline Plan Baseline
GB GB GB GB
GB GB
NL HJ NL NL NL
HJ
NL NL NL NL
HJ HJ

58. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
Real Application Testing
Q&A

59. Real Application Testing
• Goal:
• Enable successful adoption of new technology and features -
specifically new releases of the database - at low-cost and low-risk
by:
• Record and replay a real workload to see how the new system performs
• Finding regressions and changing plans before the upgrade
• Licensable database pack "Real Application Testing"
Available with Oracle Database 11.1.0.6
Available with patch set 10.2.0.4
Available as single patch for 9.2.0.8 and 10.2.0.3
For patch numbers please see Note:560977.1
Can be used for upgrades from
9.2.0.8 to 10.2.0.3/4
9.2.0.8 to 11.1.0.x
10.2.0.3/4 to 11.1.0.x

60. Real Application Testing
• Real Application Testing consists of:
• Database Replay
• Package DBMS_WORKLOAD_CAPTURE
• Package DBMS_WORKLOAD_REPLAY
• SQL Performance Analyzer (SPA)
• Package DBMS_SQLPA
• SQL Tuning Sets (STS)
• Package DBMS_SQLTUNE

61. Real Application Testing
• White Paper:
• SQL Performance Analyzer:
• Techinical White Paper: SQL Performance Analyzer
http://www.oracle.com/technology/products/manageability/database/pdf
/ow07/spa_white_paper_ow07.pdf
• Database Replay:
• Techinical White Paper: Database Replay
http://www.oracle.com/technology/products/manageability/database/pdf
/ow07/db_replay_white_paper_ow07_1.pdf

62. Real Application Testing: Tools of the Trade
SQL Performance Analyzer Database Replay
What is it? • Predicts SQL performance deviations • Replays real database workload
before end-users can be impacted, on test system, helps assess
helps assess impact of change on impact of change on workload
SQL response time throughput
How it works? • Executes each SQL, stored in SQL • Captures workloads and replays it
Tuning Set, in isolation using with production characteristics
production context and then including concurrency,
compares before and after execution synchronization &
plans and run-time statistics dependencies
When to use? • Unit testing of SQL with the goal to • Comprehensive testing of all
identify the set of SQL statements sub-systems of the database
with improved/regressed performance server using real production
workload
SQL Concurrency
SQL
Dependency Speed up/down

63. Real Application Testing
• Database Control:

64. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
SQL Performance Analyzer
Real Application Testing Database Replay
Q&A

65. Challenge
How do I find out
which plans will
be changed?
How do I verify
parameter
changes?

66. Common SPA Usage Scenarios
• Common SPA usage scenario examples
• Database upgrades, patch set changes
• Parameter changes
• Schema changes
• Optimizer statistics refresh, etc.
• SPA interface
• Rich GUI through Enterprise Manager
• Command line interface through DBMS_SQLPA PL/SQL
package

67. SQL Performance Analyzer: Overview
Production Test
Client
…
Middle Tier Re-execute SQL
… …
Capture Transport
SQL SQL
Oracle DB
Storage
Make Changes /
Tuning Regressions
• If adequate spare cycles available, * No middle & application tier setup required
optionally execute SQL here

68. SQL Performance Analyzer: Workflow
Production Test
Make Change
Steps (1) (2) (3) (4) (5)
Capture Transport Execute SQL Execute SQL Compare
SQL (STS) STS Pre-change Post-change Perf.
(6) Reiterate
No
Done?
(7)
Yes
Tuned System
Production Change /
(7)
Tuning Deployment

69. SPA Workflows
• Possible workflows:
• Optimizer upgrade simulation (Start: 8.0.0)
• Init parameter change
• Guided workflow for manual changes
• All are based on SQL Tuning Sets

70. Database Upgrade: 10g to 11g
Scenario 1:
I want to upgrade from 10.x to 11.x database release to
benefit from 11g functionality. How can I best
accomplish the upgrade?
Goal:
Assess impact of upgrade on SQL workload
performance so that there are no surprises after
upgrade. Once upgraded to 11g new features can be
enabled one at a time.

71. Database Upgrade: 10g to 11g
• Step 1:
• Capture workload into STS:
Preferred method (> 10.2.0.1)
EXEC DBMS_SQLTUNE.CAPTURE_CURSOR_CACHE_SQLSET( .. )
EXEC DBMS_SQLTUNE.CAPTURE_CURSOR_CACHE_SQLSET( .. )
• Create STS from Top SQL in AWR / AWR Baseline
• For 9i database enable SQL tracing:
alter system set events
alter system set events
'10046 trace name context forever, level 4';
'10046 trace name context forever, level 4';

72. Database Upgrade: 10g to 11g
• Step 1 in EM:

73. Database Upgrade: 10g to 11g
• Step 2:
Upgrade a copy of 10g production to 11g test database
• Step 3:
Run SPA in 11g test database
• Select “Optimizer Upgrade” workflow link

74. Database Upgrade: 10g to 11g
• Step 3 in EM: Run SPA task

75. Database Upgrade: 10g to 11g
• SPA results
3
2
4
1

76. Database Upgrade: 10g to 11g
• After using SQL Plan Management with a baseline
2
1

77. Parameter Changes
• SPA Recommendation: SQL Tuning Advisor
2
1
3

78. Parameter Changes
• SQL Tuning Advisor offers SQL Profiles

79. Parameter Changes
• SPA run after SQL Profiles had been applied
1

80. SQL Performance Analyzer
• SPA: SQL statement details example

81. SPA for a 9i workload against 11g
• Upgrade scenario from 9i directly to 11g
• See Note: 562899.1 for the White Paper and Note:455889.1 for a SPA example
Event 10046
ORCL_ora_133674.trc
ORCL_ora_133674.trc copy
ORCL_ora_133674.trc
ORCL_ora_133674.trc
STS
...
SQL Performance Analyzer
Mapping Table
Id Owner Name
123 SH1 SAL1
124 SH2 CUST
exp / imp

82. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
SQL Performance Analyzer
Real Application Testing Database Replay
Q&A

83. Challenge
How will my system
perform after an
upgrade?
Will all
application calls
work in 11g?

84. Database Replay
• Capture and replay an actual production database workload
• Identify, analyze and fix potential instabilities before making changes to
production
• Capture Workload in Production
• Capture full production workload with real load, timing & concurrency
characteristics
• Move the captured workload to test system
• Replay Workload in Test
• Make the desired changes in test system
• Replay workload with full production characteristics
• Honor commit ordering
• Analyze & Report
• Errors
• Data divergence
• Performance divergence

85. Step 1: Workload Capture
• All external client requests Production System
captured in binary files
• System background and internal Client Client Client
activity excluded …
• Minimal overhead
File System
• Avoids function call when possible
• Buffered I/O Middle Tier
• Independent of client protocol
• Can capture on 9.2.0.8 and replay File 1
on 11g
• Capture load for interesting time File 2
period, e.g., peak workload,
month-end processing, etc.
…
File n
Storage

86. Step 2: Process Workload Files
• Setup test system
Test System
• Application data should be
same as production system as
of capture start time
• Use RMAN, Snapshot Standby, File 1
imp/exp, Data Pump, etc. to File 1
create test system File 2
• Make change: upgrade db File 2
…
and/or OS, change storage,
migrate platforms, etc. … File n
• Pre-processing transforms File n
captured data into replayable
format
Metadata
• Once pre-processed, workload Replay Files
Capture Files
can be replayed many times

87. Step 3: Replay Workload
• Replays workload preserving Test System
timing, concurrency and
dependencies of the captured Replay Clients
system
• Replay Client is a special
program that consumes
processed workload and
sends requests to the replay
File 1
system
File 2
• For high concurrency
workloads, it may be …
necessary to start
File n
multiple clients
Metadata
Replay Files

88. Database Replay
• Step-By-Step

89. Database Replay
1. Create a copy of the database to replay the workload
• RMAN Duplicate
• Snapshot Standby

90. Database Replay
2. Workflow in EM: Workload Capture

91. Database Replay
2. Preparation steps in EM: Workload Capture

92. Database Replay
3. Workload Filter:

93. Database Replay
4. Prepare an empty directory for capture files

94. Database Replay
• Job scheduling - specify time frame for capture:

95. Database Replay
• Database Control - capture review and submit:

96. Database Replay
• Database Control - capture summary:

97. Database Replay
• Database Control - capture statistics:

98. Database Replay
• Preprocessing captured workload:

99. Database Replay
• Replaying the captured workload:

100. Database Replay
• Choose directory containing preprocessed capture files

101. Database Replay
• Workload Replay - choose replay name:

102. Database Replay
• Workload Replay: Wait for replay clients to connect

103. Database Replay
• Workload Replay Clients: Calibrate
• $> wrc mode=calibrate replaydir=/tmp/testing
• Workload Replay Clients: Start
• $> wrc system/oracle mode=replay

104. Database Replay
• Workload Replay: WRC client(s) connected

105. Database Replay
• Monitor Workload Replay during replay

106. Database Replay
• Monitor divergence stats:

107. Database Replay
• Workload Replay result:

108. Database Replay
• Workload replay reporting:

109. Additional Pieces of Info

110. Important Upgrade Notes at a glance
Upgrade to Oracle 10gR2
Note: 466181.1 Upgrade Companion 10gR2
Note: 316889.1 Complete Checklist for Manual Upgrade to 10gR2
Note: 555579.1 Known Issues 10.2.0.4
Upgrade to Oracle 11g
Note: 601807.1 Upgrade Companion 11g
Note: 429825.1 Complete Checklist for Manual Upgrades to 11g
Note: 454506.1 Known Issues and Alerts 11.1.0.6
Performance Testing
Note: 560977.1 Real Application Testing available for earlier releases
Note: 562899.1 Using SQL Performance Analyzer for upgd. 9.2 to 10.2
Note: 394937.1 Statspack Guide
Note: 295819.1 Potential Query Tuning Related Issues
Note: 345048.1 GROUP BY does not sort if you don't use ORDER BY
Upgrading to Oracle Database 10g: What to expect from the Optimizer

111. Upgrade Webpage on OTN
• http://www.oracle.com/technology/products/database/oracle11g/upgrade/index.html

112. OTN Upgrade Forum
• http://forums.oracle.com/forums/forum.jspa?forumID=583&start=0

113. Latest Patchset 11.1.0.7.0
Patch Number 6890831
Available for all major platforms

114. Upgrade Information / Alerts
• Note 738538.1 Known Issues specific to the 11.1.0.7 Patch Set

115. Real World Experience
• White Paper available on OTN
• http://www.oracle.com/technology/products/database/oracle11g/upgrade/
presentations/9i_to_11g_real_world_customer_experience.pdf

116. Guidelines
• Never ever change too many system components at once!
• Document all changes detailed and clearly into a change log!
• Always (!!!) use real world data for testing!
• Don't underestimate the test efforts. Reserve enough time and
resources for testing.
• ALWAYS collect as much performance data BEFORE the
upgrade will be started!!
• Create a fallback strategy!
• PLEASE test your fallback strategy - does it really work??
• Please remember:
Upgrade has never been easier - but you still have to test!!!

117. Agenda
Recap
Challenges & Best Practices
AWR & STATSPACK
SQL Plan Management
Real Application Testing
Q&A