SlideShare una empresa de Scribd logo
1 de 68
Ben Prusinski
ben@ben-oracle.com
About the Speaker
 Ben Prusinski , Oracle ACE Director
 Oracle R12 EBS OCP
 More than 14 years experience with Oracle
 Author of three books on Oracle Database Technology
Agenda
 Oracle R12 E-Business Suite Performance Tuning
 Focus on Methodology
 Best Practices
 Holistic Tuning Approach
 Proactive Versus Reactive Tuning Methods
 Basic techniques to Advanced Tuning
Methodology
•Load testing before, during, and after Go-Live
•Functional Requirements and Testing
•Map the Business and Functional to the Technical
Environment
•Business people think different than technical people
•Goal: Unify as one to resolve tuning issues (Method R)
•Stress Testing
•Integration Testing and Unit Test
Methodology: Load Testing
 Why do we Load test?
 Load testing allows you to generate metrics before,
during and post-Go Live to optimize performance and
availability
 Fact: you really do not know the true performance
until you test!
 Tools for Load testing: Oracle 11g Real Application Test
(RAT) Suite excellent for database tier
 Oracle 10g/11g Application Management Pack (AMP)
useful for measuring application tier performance
metrics.
Methodology: Functional to
Technical Specifications and Testing
 Eliminate the disconnect between the functional,
development and technical teams as soon as possible
in the deployment process to reduce bottlenecks in
performance.
 Establish stake holders to review and test
requirements from functional to tech teams
 Dependent on business requirements such as modules
deployed (eg: AP, AR, GL for Oracle Financials) to
really understand what to test and measure for
performance tuning!
How to Tune?
Knowing is half the battle!
Start with business problem eg) users slow response time for invoices processed
(AP/AR modules) with Oracle Financials
Drill down from macro to micro areas
Never assume!
Myth : The database is slow or “The database is guilty until proven innocent!”
Fact: it could be the application server or concurrent manager issue
Infrastructure First
 Hardware Tuning
 Storage and Disk Tuning
 Network Tuning
 Operating System Performance
Oracle R12 EBS Technology Stack
 Concurrent Manager Tuning
 Oracle R12 EBS Module tuning: AR, AP, GL, HR, etc.
 Oracle 10g Application Server Tuning- Apache, Java tuning
 Servlet vs. Socket mode (11i versus R12 EBS)
Database Tuning
 Instance tuning: SGA, PGA sizing is critical for database tier performance
 Statistics- FND_STATS
Application Tuning
 SQL, PL/SQL, Oracle R12 Customizations and Code Performance
Focus on Methodology
 Holistic Tuning Approach
 Proactive Versus Reactive Tuning Methods
 Basic techniques to Advanced Tuning
Key Lesson: Do not mistake the forest for the trees and avoid tunnel vision!
Areas to Tune: Start Here!
Holistic Picture
 Understand the “real” problem first!
 Cary Millsap, Method R
 Oak Table “Battle against any Guess (BAAG)”- Alex Gorbachev
 Avoid the Three Blind Men and Elephant Fallacy
 Drill down to the root cause and analyze
 Duplicate the problem with test cases if possible Oracle R12 E-Business
Suite Performance Tuning
 Focus on Methodology not “Silver Bullets”
 Best Practices
 Holistic Tuning Approach
 Proactive Versus Reactive Tuning Methods
 Basic techniques to Advanced Tuning
 MOS Note: 69565.1 : A Holistic Approach to Performance Tuning
Oracle Applications Systems
Hardware Tuning
 Common area overlooked when sizing Oracle R12 EBS
 Starting point when sizing hardware:
 Get sizing recommendations for vendors and compare
benchmarks for performance
 Big centralized servers versus many servers, which is
best?
 For application tier, recommend multiple smaller
servers with hardware load balancers
 For database tier, advise large single server.
Virtualization and Performance
 Virtual servers with VMWare and Oracle VM require
additional hardware resources to optimize
performance
 Common question:
Why is my Oracle R12 EBS Slow on Virtual Servers?
 Answer: Virtual Servers consume additional memory
resources beyond just the application
 Watch out for issues with virtualization and
performance
 VMWare not officially supported by Oracle
Hardware and Performance: Disk I/O tuning
 Solid State Disk offers best overall disk performance
but is more expensive.
 Optimize performance with Disk I/O with RAID 0+1 or
RAID 1+0 and Avoid RAID 5 for best overall disk
performance.
 Implement Automatic Storage Management (ASM) for
Oracle 10g and Oracle 11g on database tier for
performance and availability
 Usually vendor centric depends on storage vendor eg)
EMC, SUN, HP, Hitachi, etc.
Hardware and Performance: Memory and CPU
 More memory not always better but you do want to
ensure that you have enough
 Sizing recommendations from vendors helps
 Initial load testing is best bet to assess if you have
sufficient memory and CPU resources.
 Tools such as top, sar, vmstat, iostat and Oracle
Enterprise Manager (OEM) are useful yardsticks to
measure resources and consumption
 Multiple cores (ie: dual core) excellent for CPU
performance but may have additional Oracle licensing
requirements.
Operating System Performance
 Tuning the operating system a common area that is
neglected with performance analysis!
 Varies based on platform: Linux Windows!
 MOS Note: 224176.1
How to use OS commands to diagnose Database
Performance issues?
Tuning Linux For Oracle R12 EBS
 Start simple with top and iostat
 Check memory and configuration for kernel settings
with dmesg and sysctl commands.
 Drill down if necessary with strace utility.
Linux Performance Tips
Network Performance:
 MOS Note: 274953.1 Tuning TCP/IP parameter in Linux Box for
SQLNET
Use netstat and ping to identify latency issues with networks
 MOS Note: 560590.1 How and When to use the net.ipv4.tcp_rmem and
net.ipv4.tcp_wmem Linux kernel Parameters
Kernel Tuning:
 MOS Note: 434351.1 Linux Kernel: The SLAB Allocator
Disk Tuning:
 MOS Note: 175980.1 Tuning Disk Throughput Using hdparm in Linux
 sar and iostat are your friends for identifying disk I/O contention
issues.
 Also check for disk I/O issues with SAN tools (varies with vendor such
as EMC, Hitachi, etc).
UNIX Performance Tips
 My Oracle Support (MOS) available from
http://support.oracle.com
 MOS Note: 144638.1 Relationship Between Common
Init.ora Parameters and Unix, Linux Kernel Parameters
 Verify that kernel parameters are set correctly and verify
with the vendor platform for UNIX.
 Assess performance with top, iostat, vmstat, sar with UNIX
systems.
 OS Watcher tool:
MOS Note: 301137.1 OS Watcher User Guide
Unix Performance Tuning
Find the performance bottleneck with top and ps to drill
down to the root cause
 Solaris: use truss and DTRACE for tuning system calls and
processes
http://www.sun.com/bigadmin/features/articles/dtrace_truss.jsp
 HP-UX: use the sam utility for tuning
http://docs.hp.com/en/B2355-90692/sam.1M.html
IBM AIX: SMIT (System Management Information Tool)
is useful tool for performance tuning
http://www.ibm.com/developerworks/aix/library/au-smit/index.html
Note: you may need root OS privileges to run these tools or
elevated privileges.
Linux Performance Tuning: top
top - 15:05:08 up 33 days, 21:49, 10 users, load average: 0.09, 0.05, 0.01
Tasks: 492 total, 2 running, 490 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.9%us, 1.1%sy, 0.0%ni, 81.2%id, 0.7%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 4044596k total, 3683136k used, 361460k free, 272104k buffers
Swap: 2064376k total, 76k used, 2064300k free, 1488836k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9455 applmgr 15 0 219m 112m 13m R 27.2 2.9 6233:08 npviewer.bin
10914 applmgr 19 0 546m 50m 7704 S 1.9 1.3 0:28.01 java
13701 applmgr 15 0 13020 1316 720 R 1.9 0.0 0:00.01 top
1 root 15 0 10348 704 588 S 0.0 0.0 0:01.51 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.23 ksoftirqd/0
4 root 10 -5 0 0 0 S 0.0 0.0 0:01.17 events/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
46 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
50 root 10 -5 0 0 0 S 0.0 0.0 0:05.01 kblockd/0
Linux Performance Tuning: iostat
$ iostat
Linux 2.6.18-194.8.1.0.1.el5 (app01.ben.com) 09/19/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.13 0.37 0.00 99.22
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 0.08 0.29 1.79 21019459 131483716
sda1 0.00 0.00 0.00 2496 28
sda2 0.08 0.29 1.79 21016659 131483688
sdb 0.00 0.00 0.00 23141 5072
sdb1 0.00 0.00 0.00 22253 5072
dm-0 0.23 0.29 1.79 21000250 131361616
dm-1 0.00 0.00 0.00 16120 122072
$ iostat -x
Linux 2.6.18-194.8.1.0.1.el5 (db01.ben.com) 09/19/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.13 0.37 0.00 99.22
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm
%util
sda 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05
0.02
sda1 0.00 0.00 0.00 0.00 0.00 0.00 23.37 0.00 3.22 2.70
0.00
sda2 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05
0.02
Windows Performance Tips
 Recommend using system monitoring tools available
with Windows 2003 and Windows 2008 Server
 Sysinternal tools excellent for diagnosis as well
http://technet.microsoft.com/en-us/sysinternals/default.aspx
MOS Note: 231159.1: How to configure Windows to
improve performance database oracle
MOS Note: 1054009.1 Instructions for setting up the
Performance Monitor Counters (PERFMON) for
Windows
Perfmon (Performance Monitor) excellent tool for
tuning Windows environments.
Using Perfmon for Windows Performance Tuning
Tuning Concurrent Managers
 Concurrent Managers source of many performance
issues.
 Useful My Oracle Support Notes: 104452.1:
Troubleshooting Concurrent Manager
 My Oracle Support Note: 1057802.1
Best Practices for Performance for Concurrent
Managers in E-Business Suite
 Myth: More concurrent managers are better
 Keep it simple and use only few concurrent managers
Tuning Concurrent Managers
 OAM excellent starting point to find performance
bottlenecks for concurrent managers and other tuning
issues with Oracle R12 EBS.
Using OAM for tuning Concurrent Processing
 Check for long running concurrent requests for potential
bottlenecks by using Oracle Application Manager
Tuning Concurrent Managers
Oracle Applications Manager (OAM) to measure performance with Concurrent
Managers for throughput to get the big picture
Tuning Internal Concurrent Manager (ICM)
Tuning the Internal Concurrent Manager (ICM)
Performance is based on the following parameters for Oracle R12
 PMON sleep cycle — The number of sleep cycles that ICM
waits between polling status for failures in concurrent
processing. You should change the PMON cycle to a lower value
if your concurrent managers experience chronic terminations.
 Queue Size — is the number of PMON sleep cycles that the
ICM waits before polling for new concurrent managers.
 Sleep Time — parameter that indicates the time in seconds that
the ICM waits before polling for requests that are waiting to run.
Concurrent Manager Tuning
 Tune queue size for concurrent managers
 Tune sleep cycle (PMON)
 Tune cache size
 Tune number of processes
 Configure Specialization Rules
 Define Workshifts to balance performance
Queue Size Tuning: Concurrent Managers
 Queue size
 Default queue size is too small after R12 installation
 How to tune?
 View statistics from scripts
 Focus on tuning for ICM and Standard Managers first
How to Tune Sleep Cycle for PMON?
During Peak Usage for business hours:
1) Set the sleep cycle for PMON to low value when
number of concurrent requests is high.
Why? With low sleep cycle for PMON, less need for
concurrent managers to check for new requests thus
increasing throughput.
During Off hours with low business activity:
2) Recommend setting PMON sleep cycle to higher
value during non-business hours such as night time
with value such as 5 minutes.
Tuning PMON Sleep Cycle
How to set value for PMON sleep cycle?
 Startup parameter in adcmctl.sh script for Oracle R12
EBS
Located under $ADMIN_SCRIPTS_HOME or
$INST_TOP/admin/scripts/ directory in
applications tier.
 Parameters:
adcmctl.sh {start|stop|abort|status} [<APPS username/APPS
password>] [sleep=<seconds>] [restart=<N|minutes>]
[pmon=<iterations>] [quesiz=<pmon_iterations>] [diag=Y|N]
[wait=Y|N]
 Tune by modify the pmon and sleep parameters
 You can also use OAM to set these values as well.
Finding value for PMON sleep cycle
 Login to OAM as SYSADMIN then System Administrator
responsibility then navigate System Administrator->
Concurrent: Manager->Define->WorkShifts
Tuning cache size for Concurrent Processing
How to Tune cache size for concurrent managers?
 Increase cache size (number of requests cached) to
twice the number of target processes for each manager.
Example
If we have standard manager with 1 target process and a
cache value of 5, it will read five requests, and then
attempt to run those five requests before reading any
new requests.
Recommendation
 Use a value of 1 when defining a manager that has
long running jobs, and a higher value of 4-5 for
managers that run shorter transaction jobs.
Cache Size and Performance for Concurrent Processing
 Tune by setting in OAM for Oracle R12 environment
Tuning Concurrent Managers Number of Processes
 Examine the number of processes for Concurrent Managers
 Check the columns for actual, target and running in OAM
Login as SYSADMIN then navigate path Concurrent:Manager-> Administer
Specialization Rules for Concurrent Manager Performance
 Use INCLUDE and EXCLUDE rules to optimize performance since Concurrent Managers are
batch type processes
 Login as SYSADMIN and navigate System Administrator Concurrent:Manager->Define in OAM
Located under Specialization Rules
Setting values for number of processes with Concurrent Managers
 Located under in OAM under navigation path
System Administrator-> Concurrent: Manager->Define->WorkShifts
Specialization Rules Continued
 Use combined specialization rules for complex tasks
Workshifts for Performance with Concurrent Processing
 Setup different workshifts for balancing performance
load with different critical tasks
 Example: schedule AP during day and nightly
processing on different workshifts
 Login to OAM as SYSADMIN then navigate path
System Administrator Concurrent:Manager->
WorkShifts
Example: Workshifts with Concurrent Processing
Concurrent Manager Tuning
 Check for high number of requests
 Script to find high number of concurrent requests
SELECT v.USER_CONCURRENT_QUEUE_NAME,COUNT(PHASE_CODE) v1
FROM APPS.FND_CONCURRENT_QUEUES_V1 v,
APPS.FND_CONCURRENT_WORKER_REQUESTS r
WHERE r.queue_application_id = 0
AND r.PHASE_CODE = 'P' -- Pending Concurrent Requests
AND r.HOLD_FLAG != 'Y' -- Concurrent Requests not on hold
AND r.REQUESTED_START_DATE <= SYSDATE -- No Future Concurrent jobs
AND r.CONCURRENT_QUEUE_ID=v.CONCURRENT_QUEUE_ID
GROUP BY v.USER_CONCURRENT_QUEUE_NAME
HAVING COUNT (PHASE_CODE) >=20
/
 Use Oracle Application Manager (OAM) charts to measure
performance
Useful Scripts for Tuning Concurrent Managers
 Some Key Scripts for monitoring the concurrent managers:
afcmstat.sql Shows all defined managers, capacity, process id, etc.
afimchk.sql Status for Internal Concurrent Manager (ICM) & PMON
afcmcreq.sql Shows concurrent manager basics
afrqwait.sql Shows pending, held and scheduled requests
afrqstat.sql Concurrent request execution times and status
afqpmrid.sql Process id (PID) for FNDLIBR processes
afimlock.sql Find concurrency and locking issues
Execute these while connected to SQL*PLUS as APPS schema.
 The above scripts are located under the $FND_TOP/sql directory
Additional Scripts for Tuning Concurrent Managers and Requests
select
f.application_short_name app,
substr(p.user_concurrent_program_name,1,55) description,
substr(p.concurrent_program_name,1,20) program,
r.priority,
count(*) cnt,
sum(actual_completion_date - actual_start_date) * 24 elapsed,
avg(actual_completion_date - actual_start_date) * 24 average,
max(actual_completion_date - actual_start_date) * 24 max,
min(actual_completion_date - actual_start_date) * 24 min,
stddev(actual_completion_date - actual_start_date) * 24 stddev,
stddev(actual_start_date - requested_start_date) * 24 wstddev,
sum(actual_start_date - requested_start_date) * 24 waited,
avg(actual_start_date - requested_start_date) * 24 avewait,
c.request_class_name type
from fnd_concurrent_queues fcq,
fnd_concurrent_queue_content fcqc,
fnd_concurrent_request_class c,
fnd_application f,
fnd_concurrent_programs_vl p,
fnd_concurrent_requests r
where r.program_application_id = p.application_id
and r.concurrent_program_id = p.concurrent_program_id
and r.status_code in ('C','G','E')
and p.application_id = f.application_id
and r.program_application_id = f.application_id
and r.request_class_application_id = c.application_id(+)
and r.concurrent_request_class_id = c.request_class_id(+)
and r.request_class_application_id = fcqc.type_application_id(+)
and r.concurrent_request_class_id = fcqc.type_id(+)
and fcqc.queue_application_id = fcq.application_id(+)
and fcqc.concurrent_queue_id = fcq.concurrent_queue_id(+)
group by
c.request_class_name,
f.application_short_name,
p.concurrent_program_name,
p.user_concurrent_program_name,
r.priority
/
TOTAL AVG MAX MIN RUN WAIT #WAITED AVG REQ APP DESCRIPTION PROGRAM PRI R
UN HOURS HOURS HOURS HOURS STDDEV STDDEV HOURS WAIT TYPE
------ ---------------------------------------- -------------------- ---- ------ -- ------- ------ ------ ------ ------- --
----- --------- -------- ------------- --
SQLGL Periods - Open Period: Child Process GLOOAPC 50 1 2 .00 .00 .00 .00 .00
.00 .10 .01
BOM Cost Manager CMCTCM 50 1 ####### ###### # ##### ###### .00
.00 -16403.89 ########
ALR Check Event Alert ALECTC 1 8 .02 .00 .00 .00 .00
.01 .16 .02
OFA Asset Inventory Report FAS410 1 2 .01 .01 .01 .00 .00
.00 .01 .00
SQLGL Open Encumbrance Year GLEOYR 1 2 .09 .05 .05 .04 .01
.00 .03 .01
Manually Gather Statistics for Oracle R12 EBS
 FND_STATS for schema statistics collection
Example: Collect statistics for AP schema:
exec fnd_stats.gather_schema_statistics('AP');
 Collect stats for all schemas for Oracle R12 EBS:
exec fnd_stats.gather_schema_statistics('ALL');
 How to gather stats for temp tables with Oracle R12 EBS:
exec fnd_stats.gather_table_stats('<schema>','<temp_table_name>');
 Example to collect stats for temp tables in AR schema:
exec fnd_stats.gather_table_stats('AR','temp_ap');
Statistics Collection for Oracle R12 EBS
Why?
 Because cost based optimizer requires best execution
path with latest current statistics.
 Reference Point on how to collect statistics
My Oracle Support http://support.oracle.com
Note: 419728.1 How To Gather Statistics On Oracle
Applications 11.5.10(and above) - Concurrent
Process,Temp Tables, Manually
Do not use DBMS_STATS with Oracle R12 EBS to
collect statistics!
Concurrent Manager Statistics
Be sure to Gather Table Statistics on these CM tables:
 FND_CONCURRENT_PROCESSES
 FND_CONCURRENT_PROGRAMS
 FND_CONCURRENT_REQUESTS
 FND_CONCURRENT_QUEUES.
Gather Stats for Oracle R12 EBS
 Either run FND_STATS or Concurrent Request via OAM
System Administrator|Concurrent Requests|
Options for STATS Collection
• Options to set parameters for estimate percent,
parallelization, schema for Concurrent request to gather
statistics for Oracle R12 E-Business Suite
• Recommend that you do not set estimate percent too high
otherwise the load to collect stats will impact performance
and collection time.
Tuning Application Tech Stack
 Oracle 10g Application Server tuning issues
 Oracle HTTP Server (OHS): “Apache” web server
 Timeout settings for Apache
 Under httpd.conf file
 Configuration settings for JVM with OC4J under both
the Oracle R12 EBS Context file for application tech
stack and opmn.xml values.
Tuning Apache OHS
Oracle R12 EBS Application Tech Stack
 Values to check for the Apache httpd.conf file located under
$IAS_ORACLE_HOME/Apache/Apache/conf directory
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
KeepAliveTimeout 15
Application Tech Stack Tuning
 Tune values for timeouts to prevent and resolve errors
in performance for network delays for forms, oacore,
OC4J
 Avoid and resolve infamous “Uninterrupted Exception
150 error” when starting and stopping app tier services
 Values to check in Oracle R12 EBS Context file
 Consider shared APPL_TOP and staged APPL_TOP for
performance
 Implement Parallel Concurrent Processing (PCP) so
distribute performance load for Concurrent processing
Application Tech Stack tuning
 Performance metrics for OC4J groups
 Use server for concurrent processing server,
application server in split tier configuration to provide
more performance and availability
 Consider load balancer: hardware or software to
improve performance and availability
 Consider deploying Oracle RAC for scale out and scale
up in performance if you need it for large deployments
 Oracle Application Management Pack (AMP) for
tuning and monitoring Oracle R12 EBS environment
Load Balancing for Application tier performance
 Hardware Load Balancing options: Cisco and Big IP
some vendors that offer solutions
 Software Load Balancing
 Load balancing is complex and requires coordination
with network, security and systems administration
teams and the database team to implement
successfully.
 Reference Point:
My Oracle Support Note 727171.1:
Implementing Load Balancing On Oracle E-Business Suite -
Documentation For Specific Load Balancer Hardware
Software Load Balancing for Oracle R12 EBS
Application Tier performance
 Not as robust as hardware load balancers but more
cost effective.
 Uses web cache option for Oracle 10gAS application
server
 My Oracle Support Note: 380486.1
Installing and Configuring Web Cache 10g and Oracle
E-Business Suite 12
Software Load Balancing Oracle R12 EBS
 Requires configuration at both 10g Application Server level and within
context file for Oracle R12 EBS
The following autoconfig variables must be set within the Oracle R12 EBS
application tier context file:
 s_webentryhost- webentry points to Oracle 10gAS Web Cache 10g
Server
 s_webentrydomain- webentry domain to domain name for Oracle
10gAS Web Cache 10g Server
 s_active_webport-active webport to port for Oracle 10gAS Server
Web Cache where it listens for client requests
 s_webentryurlprotocol - webentry protocol to protocol values
that clients will use to access Oracle 10gAS Server Web Cache 10g
 s_login_page - login page to
hostname:port/OA_HTML/AppsLogin
Java Tuning for Oracle R12 EBS
 Tuning Java components is essential for performance
with Oracle R12 EBS
 Oracle R12 EBS is dependent on Java technology
 Default values are too low for Java with OC4J settings
 Watch out for issues with garbage collection
 Avoid full garbage collection (GC) as this impacts
performance!
 My Oracle Support Note: 567551.1:
Troubleshooting : Configuring various JVM tuning
parameters for Oracle E-Business suite 11i and R12
Tuning Java for R12 EBS: JVM Sizing
 How many JVMs (Java Virtual Machines) ?
OACoreGroup
 For 2-tier environment with application middle tier on separate host server and database server
 Best not to have more than 100 active users per JVM/OC4J instance for middle tier server.
No more than one active JVM/OC4J instance per CPU
 Best practice: there should be ratio of 1 JVM to 2 CPUs for middle tier server
 Use top and other system tools to check that sufficient memory is available. You want avoid memory
swapping issues.
 Common mistake for application middle tier server to not be sized correctly for memory and CPU!
FormsGroup
 For the forms servlet, the actual JVM/servlet usage is minimal, and a single JVM can normally handle
up to 100-250 forms users.
 JVM settings are configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs,
s_forms_servlet_nprocs and s_xmlsvcs_nprocs
Java Tuning: Oracle R12 EBS
Heap configuration for JVMs with middle tier
 OACORE starting point for tuning J VM heap sizes:
-Xmx512M -Xms256M -XX:MaxPermSize=128M
-XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB
 Goal is to minimize garbage collection frequency and to reduce user response
time by minimizing full garbage collection.
 For JDK 1.5, if the JVM detects that you have a server with multiple CPUs it
will enable Parallel Throughput Garbage Collector by default.
 Rule of thumb: number of GC (garbage collection) threads defaults to number
of CPUs per machine.
 If you run multiple JVMs on the same server, or if you have more than 2 CPUs,
to avoid excessive GC, reduce the number of GC threads:
-XX:+UseParallelGC -XX:ParallelGCThreads=2
 Two GC threads should be enough for most environments
 For heap sizes over 1GB, you can test with 4 GC threads and measure
performance.
 FormsGroup values for Garbage Collection (GC) as rule of thumb:
-Xmx256M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2
Key Oracle R12 EBS Application Middle Tier Server Autoconfig Variables for Java Heap tuning with JVMs:
 s_oacore_jvm_start_option
 s_forms_jvm_start_options
Tools for Measuring JVM Performance with
Oracle R12 EBS
 Excellent free tool to graph performance for JVM with
Oracle R12 EBS and garbage collection.
 http://www.tagtraum.com/gcviewer.html
Tuning Java for Oracle R12 EBS
 My Oracle Support Note: 362851.1: -
Guidelines to setup the JVM in Apps E-Business Suite 11i and R12
JVM Behavioral Options:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#BehavioralOptions
JVM Options List:
http://blogs.sun.com/watt/resource/jvm-options-list.html
Monitoring Java Performance: Oracle R12 EBS
 Oracle 10g Grid Control (OEM) with Application
Management Pack (AMP).
Note: These items require additional cost licenses.
 MOS Note 557194.1 provides a script called
monitor_jdbc_conn.sql to monitor JDBC
connections.
Java Tuning for Oracle R12 EBS
 Collect metrics to analyze current performance
 Good information from ADDM and AWR in 11g
Statistic Total per Second per Trans
-------------------------------- ------------------ -------------- -------------
java call heap collected bytes 683,300,160 1,300.0 877.8
java call heap collected count 3,693,358 7.0 4.7
java call heap gc count 356 0.0 0.0
java call heap live object count 24,810 0.1 0.0
java call heap live object count 142,439 0.3 0.2
Instance Activity Stats DB/Inst: SBOX/SBOX Snaps: 274-420
java call heap live size 2,358,144 4.5 3.0
java call heap live size max 33,260,184 63.3 42.7
java call heap object count 56,992 0.1 0.1
java call heap object count max 345,992 0.7 0.4
java call heap total size 42,729,472 81.3 54.9
java call heap total size max 115,343,360 219.5 148.2
java call heap used size 6,144,496 11.7 7.9
java call heap used size max 72,864,528 138.6 93.6
Database Tuning for Oracle R12 EBS
 Similar to tuning regular Oracle 11g database but
unique in how statistics collected
 Remember to use FND_STATS and NOT
DBMS_STATS to collect and update statistics!
 Watch out for concurrency and locking issues for
runaway user processes!
 Schedule backup jobs off hours to avoid impact
business users
 Don’t forget to use tools like AWR and STATSPACK if
possible.
Database Tuning
 Suggested values for database initialization parameters for
Oracle R12 EBS
 My Oracle Support Note: 396009.1
Database Initialization Parameters for Oracle
Applications Release 12
 Oracle R12 by default installs either a 10gR2 or an 11gR1
database for fresh (new) installation.
 For example: Oracle 12.0 to 12.04 uses 10.2.x database
 While Oracle 12.1.1 and later release install Oracle 11gR1
database (11.1.x).
 Watch out for upgrade issues from 11i to R12 EBS for
database!
Database Parameters for Oracle R12 EBS
Parameter Name
Development or
Test Instance
11-100 Users 101-500 Users 501-1000 Users 1001-2000 Users
processes 200 200 800 1200 2500
sessions 400 400 1600 2400 5000
sga_target 1G 1G 2G 3G 14G
shared_pool_size
(csp)
N/A N/A N/A 1800M 3000M
shared_pool_reser
ved_size (csp)
N/A N/A N/A 180M 300M
shared_pool_size
(no csp)
400M 600M 800M 1000M 2000M
shared_pool_reser
ved_size (no csp)
40M 60M 80M 100M 100M
pga_aggregate_tar
get
1G 2G 4G 10G 20G
Total Memory
Required
~ 2 GB ~ 3 GB ~ 6 GB ~ 13 GB ~ 25 GB
Upgrade Performance considerations for Oracle R12 EBS
 When you upgrade from older releases such as 10.x and 11i
to Oracle R12 must account for performance items
 Rule based Optimizer (RBO) in 10.x and early 11i releases to
the Cost Based Optimizer (CBO) in R12
 Different way of tuning for stats and database
 Consider changes for SGA and PGA sizing
 Migrate from old tablespace model to Oracle Application
Managed Tablespaces (OATM)
 My Oracle Support Note: 761570.1
Database Preparation Guidelines for an E-Business
Suite Release 12.1.1 Upgrade
Q&A Discussion
Thank You!
 Please provide feedback
 Email: ben@ben-oracle.com
 My blog has further tips on Oracle R12 EBS
 http://oracle-magician.blogspot.com
 I will be available in the OTN ACE Director Office
Hours this week for questions

Más contenido relacionado

La actualidad más candente

How to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata EnvironmentsHow to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata EnvironmentsSandesh Rao
 
Oracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret InternalsOracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret InternalsAnil Nair
 
Oracle Database performance tuning using oratop
Oracle Database performance tuning using oratopOracle Database performance tuning using oratop
Oracle Database performance tuning using oratopSandesh Rao
 
What's new in Oracle 19c & 18c Recovery Manager (RMAN)
What's new in Oracle 19c & 18c Recovery Manager (RMAN)What's new in Oracle 19c & 18c Recovery Manager (RMAN)
What's new in Oracle 19c & 18c Recovery Manager (RMAN)Satishbabu Gunukula
 
ORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIES
ORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIESORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIES
ORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIESLudovico Caldara
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsJohn Kanagaraj
 
Oracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAsOracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAsGokhan Atil
 
Oracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLONOracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLONMarkus Michalewicz
 
Understand oracle real application cluster
Understand oracle real application clusterUnderstand oracle real application cluster
Understand oracle real application clusterSatishbabu Gunukula
 
Performance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and UnderscoresPerformance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and UnderscoresJitendra Singh
 
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Kyle Hailey
 
Scaling paypal workloads with oracle rac ss
Scaling paypal workloads with oracle rac ssScaling paypal workloads with oracle rac ss
Scaling paypal workloads with oracle rac ssAnil Nair
 
Oracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsOracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsEnkitec
 
Oracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAsOracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAsGokhan Atil
 
Oracle EBS R12.2 - Deployment and System Administration
Oracle EBS R12.2 - Deployment and System AdministrationOracle EBS R12.2 - Deployment and System Administration
Oracle EBS R12.2 - Deployment and System AdministrationMozammel Hoque
 
Standard Edition High Availability (SEHA) - The Why, What & How
Standard Edition High Availability (SEHA) - The Why, What & HowStandard Edition High Availability (SEHA) - The Why, What & How
Standard Edition High Availability (SEHA) - The Why, What & HowMarkus Michalewicz
 
Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the TradeCarlos Sierra
 

La actualidad más candente (20)

Analyzing awr report
Analyzing awr reportAnalyzing awr report
Analyzing awr report
 
How to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata EnvironmentsHow to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata Environments
 
Oracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret InternalsOracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret Internals
 
Oracle Database performance tuning using oratop
Oracle Database performance tuning using oratopOracle Database performance tuning using oratop
Oracle Database performance tuning using oratop
 
What's new in Oracle 19c & 18c Recovery Manager (RMAN)
What's new in Oracle 19c & 18c Recovery Manager (RMAN)What's new in Oracle 19c & 18c Recovery Manager (RMAN)
What's new in Oracle 19c & 18c Recovery Manager (RMAN)
 
ORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIES
ORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIESORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIES
ORACLE 12C DATA GUARD: FAR SYNC, REAL-TIME CASCADE STANDBY AND OTHER GOODIES
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
 
Oracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAsOracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAs
 
Oracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLONOracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLON
 
AWR & ASH Analysis
AWR & ASH AnalysisAWR & ASH Analysis
AWR & ASH Analysis
 
Understand oracle real application cluster
Understand oracle real application clusterUnderstand oracle real application cluster
Understand oracle real application cluster
 
Performance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and UnderscoresPerformance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and Underscores
 
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle
 
Scaling paypal workloads with oracle rac ss
Scaling paypal workloads with oracle rac ssScaling paypal workloads with oracle rac ss
Scaling paypal workloads with oracle rac ss
 
Oracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsOracle Performance Tuning Fundamentals
Oracle Performance Tuning Fundamentals
 
ASH and AWR on DB12c
ASH and AWR on DB12cASH and AWR on DB12c
ASH and AWR on DB12c
 
Oracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAsOracle Enterprise Manager Cloud Control 13c for DBAs
Oracle Enterprise Manager Cloud Control 13c for DBAs
 
Oracle EBS R12.2 - Deployment and System Administration
Oracle EBS R12.2 - Deployment and System AdministrationOracle EBS R12.2 - Deployment and System Administration
Oracle EBS R12.2 - Deployment and System Administration
 
Standard Edition High Availability (SEHA) - The Why, What & How
Standard Edition High Availability (SEHA) - The Why, What & HowStandard Edition High Availability (SEHA) - The Why, What & How
Standard Edition High Availability (SEHA) - The Why, What & How
 
Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the Trade
 

Destacado

Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the TradeEnkitec
 
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2SolarWinds
 
Monitoring and Tuning Oracle FMW 11g
Monitoring and Tuning Oracle FMW 11gMonitoring and Tuning Oracle FMW 11g
Monitoring and Tuning Oracle FMW 11gMatthias Furrer
 
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1SolarWinds
 
Double the Performance of Oracle SOA Suite 11g? Absolutely!
Double the Performance of Oracle SOA Suite 11g? Absolutely!Double the Performance of Oracle SOA Suite 11g? Absolutely!
Double the Performance of Oracle SOA Suite 11g? Absolutely!Revelation Technologies
 
Top 10 tips for Oracle performance (Updated April 2015)
Top 10 tips for Oracle performance (Updated April 2015)Top 10 tips for Oracle performance (Updated April 2015)
Top 10 tips for Oracle performance (Updated April 2015)Guy Harrison
 
OOW15 - Getting Optimal Performance from Oracle E-Business Suite
OOW15 - Getting Optimal Performance from Oracle E-Business SuiteOOW15 - Getting Optimal Performance from Oracle E-Business Suite
OOW15 - Getting Optimal Performance from Oracle E-Business Suitevasuballa
 
Ebs performance tuning session feb 13 2013---Presented by Oracle
Ebs performance tuning session  feb 13 2013---Presented by OracleEbs performance tuning session  feb 13 2013---Presented by Oracle
Ebs performance tuning session feb 13 2013---Presented by OracleAkash Pramanik
 
How to find what is making your Oracle database slow
How to find what is making your Oracle database slowHow to find what is making your Oracle database slow
How to find what is making your Oracle database slowSolarWinds
 
Oracle performance tuning
Oracle performance tuningOracle performance tuning
Oracle performance tuningvksgarg
 
Getting optimal performance from oracle e-business suite presentation
Getting optimal performance from oracle e-business suite presentationGetting optimal performance from oracle e-business suite presentation
Getting optimal performance from oracle e-business suite presentationBerry Clemens
 

Destacado (12)

Performance in the Oracle Cloud
Performance in the Oracle CloudPerformance in the Oracle Cloud
Performance in the Oracle Cloud
 
Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the Trade
 
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 2
 
Monitoring and Tuning Oracle FMW 11g
Monitoring and Tuning Oracle FMW 11gMonitoring and Tuning Oracle FMW 11g
Monitoring and Tuning Oracle FMW 11g
 
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1
Stop the Chaos! Get Real Oracle Performance by Query Tuning Part 1
 
Double the Performance of Oracle SOA Suite 11g? Absolutely!
Double the Performance of Oracle SOA Suite 11g? Absolutely!Double the Performance of Oracle SOA Suite 11g? Absolutely!
Double the Performance of Oracle SOA Suite 11g? Absolutely!
 
Top 10 tips for Oracle performance (Updated April 2015)
Top 10 tips for Oracle performance (Updated April 2015)Top 10 tips for Oracle performance (Updated April 2015)
Top 10 tips for Oracle performance (Updated April 2015)
 
OOW15 - Getting Optimal Performance from Oracle E-Business Suite
OOW15 - Getting Optimal Performance from Oracle E-Business SuiteOOW15 - Getting Optimal Performance from Oracle E-Business Suite
OOW15 - Getting Optimal Performance from Oracle E-Business Suite
 
Ebs performance tuning session feb 13 2013---Presented by Oracle
Ebs performance tuning session  feb 13 2013---Presented by OracleEbs performance tuning session  feb 13 2013---Presented by Oracle
Ebs performance tuning session feb 13 2013---Presented by Oracle
 
How to find what is making your Oracle database slow
How to find what is making your Oracle database slowHow to find what is making your Oracle database slow
How to find what is making your Oracle database slow
 
Oracle performance tuning
Oracle performance tuningOracle performance tuning
Oracle performance tuning
 
Getting optimal performance from oracle e-business suite presentation
Getting optimal performance from oracle e-business suite presentationGetting optimal performance from oracle e-business suite presentation
Getting optimal performance from oracle e-business suite presentation
 

Similar a Oracle R12 EBS Performance Tuning

Using preferred read groups in oracle asm michael ault
Using preferred read groups in oracle asm michael aultUsing preferred read groups in oracle asm michael ault
Using preferred read groups in oracle asm michael aultLouis liu
 
Troubleshooting SQL Server
Troubleshooting SQL ServerTroubleshooting SQL Server
Troubleshooting SQL ServerStephen Rose
 
[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』
[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』
[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』Insight Technology, Inc.
 
Sql server troubleshooting
Sql server troubleshootingSql server troubleshooting
Sql server troubleshootingNathan Winters
 
Hardware planning & sizing for sql server
Hardware planning & sizing for sql serverHardware planning & sizing for sql server
Hardware planning & sizing for sql serverDavide Mauri
 
Extreme Availability using Oracle 12c Features: Your very last system shutdown?
Extreme Availability using Oracle 12c Features: Your very last system shutdown?Extreme Availability using Oracle 12c Features: Your very last system shutdown?
Extreme Availability using Oracle 12c Features: Your very last system shutdown?Toronto-Oracle-Users-Group
 
שבוע אורקל 2016
שבוע אורקל 2016שבוע אורקל 2016
שבוע אורקל 2016Aaron Shilo
 
Oracle Enterprise Manager 12c - OEM12c Presentation
Oracle Enterprise Manager 12c - OEM12c PresentationOracle Enterprise Manager 12c - OEM12c Presentation
Oracle Enterprise Manager 12c - OEM12c PresentationFrancisco Alvarez
 
Sql server 2016 it just runs faster sql bits 2017 edition
Sql server 2016 it just runs faster   sql bits 2017 editionSql server 2016 it just runs faster   sql bits 2017 edition
Sql server 2016 it just runs faster sql bits 2017 editionBob Ward
 
Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006Sal Marcus
 
SQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teamsSQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teamsSumeet Bansal
 
SOUG_SDM_OracleDB_V3
SOUG_SDM_OracleDB_V3SOUG_SDM_OracleDB_V3
SOUG_SDM_OracleDB_V3UniFabric
 
Ceph Day Taipei - Accelerate Ceph via SPDK
Ceph Day Taipei - Accelerate Ceph via SPDK Ceph Day Taipei - Accelerate Ceph via SPDK
Ceph Day Taipei - Accelerate Ceph via SPDK Ceph Community
 
Strata + Hadoop 2015 Slides
Strata + Hadoop 2015 SlidesStrata + Hadoop 2015 Slides
Strata + Hadoop 2015 SlidesJun Liu
 
Collaborate07kmohiuddin
Collaborate07kmohiuddinCollaborate07kmohiuddin
Collaborate07kmohiuddinSal Marcus
 
Performance tuning intro
Performance tuning introPerformance tuning intro
Performance tuning introaioughydchapter
 
Webinar: Does Your Data Center Need NVMe?
Webinar: Does Your Data Center Need NVMe?Webinar: Does Your Data Center Need NVMe?
Webinar: Does Your Data Center Need NVMe?Storage Switzerland
 

Similar a Oracle R12 EBS Performance Tuning (20)

Using preferred read groups in oracle asm michael ault
Using preferred read groups in oracle asm michael aultUsing preferred read groups in oracle asm michael ault
Using preferred read groups in oracle asm michael ault
 
Troubleshooting SQL Server
Troubleshooting SQL ServerTroubleshooting SQL Server
Troubleshooting SQL Server
 
[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』
[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』
[db tech showcase Tokyo 2018] #dbts2018 #B17 『オラクル パフォーマンス チューニング - 神話、伝説と解決策』
 
Sql server troubleshooting
Sql server troubleshootingSql server troubleshooting
Sql server troubleshooting
 
Hardware planning & sizing for sql server
Hardware planning & sizing for sql serverHardware planning & sizing for sql server
Hardware planning & sizing for sql server
 
Extreme Availability using Oracle 12c Features: Your very last system shutdown?
Extreme Availability using Oracle 12c Features: Your very last system shutdown?Extreme Availability using Oracle 12c Features: Your very last system shutdown?
Extreme Availability using Oracle 12c Features: Your very last system shutdown?
 
שבוע אורקל 2016
שבוע אורקל 2016שבוע אורקל 2016
שבוע אורקל 2016
 
Oracle Enterprise Manager 12c - OEM12c Presentation
Oracle Enterprise Manager 12c - OEM12c PresentationOracle Enterprise Manager 12c - OEM12c Presentation
Oracle Enterprise Manager 12c - OEM12c Presentation
 
Sql server 2016 it just runs faster sql bits 2017 edition
Sql server 2016 it just runs faster   sql bits 2017 editionSql server 2016 it just runs faster   sql bits 2017 edition
Sql server 2016 it just runs faster sql bits 2017 edition
 
Performance Tuning intro
Performance Tuning introPerformance Tuning intro
Performance Tuning intro
 
Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006
 
11g R2
11g R211g R2
11g R2
 
SQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teamsSQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teams
 
SOUG_SDM_OracleDB_V3
SOUG_SDM_OracleDB_V3SOUG_SDM_OracleDB_V3
SOUG_SDM_OracleDB_V3
 
Ceph Day Taipei - Accelerate Ceph via SPDK
Ceph Day Taipei - Accelerate Ceph via SPDK Ceph Day Taipei - Accelerate Ceph via SPDK
Ceph Day Taipei - Accelerate Ceph via SPDK
 
Strata + Hadoop 2015 Slides
Strata + Hadoop 2015 SlidesStrata + Hadoop 2015 Slides
Strata + Hadoop 2015 Slides
 
Collaborate07kmohiuddin
Collaborate07kmohiuddinCollaborate07kmohiuddin
Collaborate07kmohiuddin
 
Performance tuning intro
Performance tuning introPerformance tuning intro
Performance tuning intro
 
Webinar: Does Your Data Center Need NVMe?
Webinar: Does Your Data Center Need NVMe?Webinar: Does Your Data Center Need NVMe?
Webinar: Does Your Data Center Need NVMe?
 
Hptf 2240 Final
Hptf 2240 FinalHptf 2240 Final
Hptf 2240 Final
 

Último

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsNanddeep Nachan
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Zilliz
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024The Digital Insurer
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusZilliz
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024The Digital Insurer
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Angeliki Cooney
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 

Último (20)

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 

Oracle R12 EBS Performance Tuning

  • 2. About the Speaker  Ben Prusinski , Oracle ACE Director  Oracle R12 EBS OCP  More than 14 years experience with Oracle  Author of three books on Oracle Database Technology
  • 3. Agenda  Oracle R12 E-Business Suite Performance Tuning  Focus on Methodology  Best Practices  Holistic Tuning Approach  Proactive Versus Reactive Tuning Methods  Basic techniques to Advanced Tuning
  • 4. Methodology •Load testing before, during, and after Go-Live •Functional Requirements and Testing •Map the Business and Functional to the Technical Environment •Business people think different than technical people •Goal: Unify as one to resolve tuning issues (Method R) •Stress Testing •Integration Testing and Unit Test
  • 5. Methodology: Load Testing  Why do we Load test?  Load testing allows you to generate metrics before, during and post-Go Live to optimize performance and availability  Fact: you really do not know the true performance until you test!  Tools for Load testing: Oracle 11g Real Application Test (RAT) Suite excellent for database tier  Oracle 10g/11g Application Management Pack (AMP) useful for measuring application tier performance metrics.
  • 6. Methodology: Functional to Technical Specifications and Testing  Eliminate the disconnect between the functional, development and technical teams as soon as possible in the deployment process to reduce bottlenecks in performance.  Establish stake holders to review and test requirements from functional to tech teams  Dependent on business requirements such as modules deployed (eg: AP, AR, GL for Oracle Financials) to really understand what to test and measure for performance tuning!
  • 7. How to Tune? Knowing is half the battle! Start with business problem eg) users slow response time for invoices processed (AP/AR modules) with Oracle Financials Drill down from macro to micro areas Never assume! Myth : The database is slow or “The database is guilty until proven innocent!” Fact: it could be the application server or concurrent manager issue
  • 8. Infrastructure First  Hardware Tuning  Storage and Disk Tuning  Network Tuning  Operating System Performance Oracle R12 EBS Technology Stack  Concurrent Manager Tuning  Oracle R12 EBS Module tuning: AR, AP, GL, HR, etc.  Oracle 10g Application Server Tuning- Apache, Java tuning  Servlet vs. Socket mode (11i versus R12 EBS) Database Tuning  Instance tuning: SGA, PGA sizing is critical for database tier performance  Statistics- FND_STATS Application Tuning  SQL, PL/SQL, Oracle R12 Customizations and Code Performance Focus on Methodology  Holistic Tuning Approach  Proactive Versus Reactive Tuning Methods  Basic techniques to Advanced Tuning Key Lesson: Do not mistake the forest for the trees and avoid tunnel vision! Areas to Tune: Start Here!
  • 9. Holistic Picture  Understand the “real” problem first!  Cary Millsap, Method R  Oak Table “Battle against any Guess (BAAG)”- Alex Gorbachev  Avoid the Three Blind Men and Elephant Fallacy  Drill down to the root cause and analyze  Duplicate the problem with test cases if possible Oracle R12 E-Business Suite Performance Tuning  Focus on Methodology not “Silver Bullets”  Best Practices  Holistic Tuning Approach  Proactive Versus Reactive Tuning Methods  Basic techniques to Advanced Tuning  MOS Note: 69565.1 : A Holistic Approach to Performance Tuning Oracle Applications Systems
  • 10. Hardware Tuning  Common area overlooked when sizing Oracle R12 EBS  Starting point when sizing hardware:  Get sizing recommendations for vendors and compare benchmarks for performance  Big centralized servers versus many servers, which is best?  For application tier, recommend multiple smaller servers with hardware load balancers  For database tier, advise large single server.
  • 11. Virtualization and Performance  Virtual servers with VMWare and Oracle VM require additional hardware resources to optimize performance  Common question: Why is my Oracle R12 EBS Slow on Virtual Servers?  Answer: Virtual Servers consume additional memory resources beyond just the application  Watch out for issues with virtualization and performance  VMWare not officially supported by Oracle
  • 12. Hardware and Performance: Disk I/O tuning  Solid State Disk offers best overall disk performance but is more expensive.  Optimize performance with Disk I/O with RAID 0+1 or RAID 1+0 and Avoid RAID 5 for best overall disk performance.  Implement Automatic Storage Management (ASM) for Oracle 10g and Oracle 11g on database tier for performance and availability  Usually vendor centric depends on storage vendor eg) EMC, SUN, HP, Hitachi, etc.
  • 13. Hardware and Performance: Memory and CPU  More memory not always better but you do want to ensure that you have enough  Sizing recommendations from vendors helps  Initial load testing is best bet to assess if you have sufficient memory and CPU resources.  Tools such as top, sar, vmstat, iostat and Oracle Enterprise Manager (OEM) are useful yardsticks to measure resources and consumption  Multiple cores (ie: dual core) excellent for CPU performance but may have additional Oracle licensing requirements.
  • 14. Operating System Performance  Tuning the operating system a common area that is neglected with performance analysis!  Varies based on platform: Linux Windows!  MOS Note: 224176.1 How to use OS commands to diagnose Database Performance issues?
  • 15. Tuning Linux For Oracle R12 EBS  Start simple with top and iostat  Check memory and configuration for kernel settings with dmesg and sysctl commands.  Drill down if necessary with strace utility.
  • 16. Linux Performance Tips Network Performance:  MOS Note: 274953.1 Tuning TCP/IP parameter in Linux Box for SQLNET Use netstat and ping to identify latency issues with networks  MOS Note: 560590.1 How and When to use the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem Linux kernel Parameters Kernel Tuning:  MOS Note: 434351.1 Linux Kernel: The SLAB Allocator Disk Tuning:  MOS Note: 175980.1 Tuning Disk Throughput Using hdparm in Linux  sar and iostat are your friends for identifying disk I/O contention issues.  Also check for disk I/O issues with SAN tools (varies with vendor such as EMC, Hitachi, etc).
  • 17. UNIX Performance Tips  My Oracle Support (MOS) available from http://support.oracle.com  MOS Note: 144638.1 Relationship Between Common Init.ora Parameters and Unix, Linux Kernel Parameters  Verify that kernel parameters are set correctly and verify with the vendor platform for UNIX.  Assess performance with top, iostat, vmstat, sar with UNIX systems.  OS Watcher tool: MOS Note: 301137.1 OS Watcher User Guide
  • 18. Unix Performance Tuning Find the performance bottleneck with top and ps to drill down to the root cause  Solaris: use truss and DTRACE for tuning system calls and processes http://www.sun.com/bigadmin/features/articles/dtrace_truss.jsp  HP-UX: use the sam utility for tuning http://docs.hp.com/en/B2355-90692/sam.1M.html IBM AIX: SMIT (System Management Information Tool) is useful tool for performance tuning http://www.ibm.com/developerworks/aix/library/au-smit/index.html Note: you may need root OS privileges to run these tools or elevated privileges.
  • 19. Linux Performance Tuning: top top - 15:05:08 up 33 days, 21:49, 10 users, load average: 0.09, 0.05, 0.01 Tasks: 492 total, 2 running, 490 sleeping, 0 stopped, 0 zombie Cpu(s): 16.9%us, 1.1%sy, 0.0%ni, 81.2%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4044596k total, 3683136k used, 361460k free, 272104k buffers Swap: 2064376k total, 76k used, 2064300k free, 1488836k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9455 applmgr 15 0 219m 112m 13m R 27.2 2.9 6233:08 npviewer.bin 10914 applmgr 19 0 546m 50m 7704 S 1.9 1.3 0:28.01 java 13701 applmgr 15 0 13020 1316 720 R 1.9 0.0 0:00.01 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.51 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.23 ksoftirqd/0 4 root 10 -5 0 0 0 S 0.0 0.0 0:01.17 events/0 5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 46 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread 50 root 10 -5 0 0 0 S 0.0 0.0 0:05.01 kblockd/0
  • 20. Linux Performance Tuning: iostat $ iostat Linux 2.6.18-194.8.1.0.1.el5 (app01.ben.com) 09/19/2010 avg-cpu: %user %nice %system %iowait %steal %idle 0.29 0.00 0.13 0.37 0.00 99.22 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 0.08 0.29 1.79 21019459 131483716 sda1 0.00 0.00 0.00 2496 28 sda2 0.08 0.29 1.79 21016659 131483688 sdb 0.00 0.00 0.00 23141 5072 sdb1 0.00 0.00 0.00 22253 5072 dm-0 0.23 0.29 1.79 21000250 131361616 dm-1 0.00 0.00 0.00 16120 122072 $ iostat -x Linux 2.6.18-194.8.1.0.1.el5 (db01.ben.com) 09/19/2010 avg-cpu: %user %nice %system %iowait %steal %idle 0.29 0.00 0.13 0.37 0.00 99.22 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05 0.02 sda1 0.00 0.00 0.00 0.00 0.00 0.00 23.37 0.00 3.22 2.70 0.00 sda2 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05 0.02
  • 21. Windows Performance Tips  Recommend using system monitoring tools available with Windows 2003 and Windows 2008 Server  Sysinternal tools excellent for diagnosis as well http://technet.microsoft.com/en-us/sysinternals/default.aspx MOS Note: 231159.1: How to configure Windows to improve performance database oracle MOS Note: 1054009.1 Instructions for setting up the Performance Monitor Counters (PERFMON) for Windows Perfmon (Performance Monitor) excellent tool for tuning Windows environments.
  • 22. Using Perfmon for Windows Performance Tuning
  • 23. Tuning Concurrent Managers  Concurrent Managers source of many performance issues.  Useful My Oracle Support Notes: 104452.1: Troubleshooting Concurrent Manager  My Oracle Support Note: 1057802.1 Best Practices for Performance for Concurrent Managers in E-Business Suite  Myth: More concurrent managers are better  Keep it simple and use only few concurrent managers
  • 24. Tuning Concurrent Managers  OAM excellent starting point to find performance bottlenecks for concurrent managers and other tuning issues with Oracle R12 EBS.
  • 25. Using OAM for tuning Concurrent Processing  Check for long running concurrent requests for potential bottlenecks by using Oracle Application Manager
  • 26. Tuning Concurrent Managers Oracle Applications Manager (OAM) to measure performance with Concurrent Managers for throughput to get the big picture
  • 27. Tuning Internal Concurrent Manager (ICM) Tuning the Internal Concurrent Manager (ICM) Performance is based on the following parameters for Oracle R12  PMON sleep cycle — The number of sleep cycles that ICM waits between polling status for failures in concurrent processing. You should change the PMON cycle to a lower value if your concurrent managers experience chronic terminations.  Queue Size — is the number of PMON sleep cycles that the ICM waits before polling for new concurrent managers.  Sleep Time — parameter that indicates the time in seconds that the ICM waits before polling for requests that are waiting to run.
  • 28. Concurrent Manager Tuning  Tune queue size for concurrent managers  Tune sleep cycle (PMON)  Tune cache size  Tune number of processes  Configure Specialization Rules  Define Workshifts to balance performance
  • 29. Queue Size Tuning: Concurrent Managers  Queue size  Default queue size is too small after R12 installation  How to tune?  View statistics from scripts  Focus on tuning for ICM and Standard Managers first
  • 30. How to Tune Sleep Cycle for PMON? During Peak Usage for business hours: 1) Set the sleep cycle for PMON to low value when number of concurrent requests is high. Why? With low sleep cycle for PMON, less need for concurrent managers to check for new requests thus increasing throughput. During Off hours with low business activity: 2) Recommend setting PMON sleep cycle to higher value during non-business hours such as night time with value such as 5 minutes.
  • 31. Tuning PMON Sleep Cycle How to set value for PMON sleep cycle?  Startup parameter in adcmctl.sh script for Oracle R12 EBS Located under $ADMIN_SCRIPTS_HOME or $INST_TOP/admin/scripts/ directory in applications tier.  Parameters: adcmctl.sh {start|stop|abort|status} [<APPS username/APPS password>] [sleep=<seconds>] [restart=<N|minutes>] [pmon=<iterations>] [quesiz=<pmon_iterations>] [diag=Y|N] [wait=Y|N]  Tune by modify the pmon and sleep parameters  You can also use OAM to set these values as well.
  • 32. Finding value for PMON sleep cycle  Login to OAM as SYSADMIN then System Administrator responsibility then navigate System Administrator-> Concurrent: Manager->Define->WorkShifts
  • 33. Tuning cache size for Concurrent Processing How to Tune cache size for concurrent managers?  Increase cache size (number of requests cached) to twice the number of target processes for each manager. Example If we have standard manager with 1 target process and a cache value of 5, it will read five requests, and then attempt to run those five requests before reading any new requests. Recommendation  Use a value of 1 when defining a manager that has long running jobs, and a higher value of 4-5 for managers that run shorter transaction jobs.
  • 34. Cache Size and Performance for Concurrent Processing  Tune by setting in OAM for Oracle R12 environment
  • 35. Tuning Concurrent Managers Number of Processes  Examine the number of processes for Concurrent Managers  Check the columns for actual, target and running in OAM Login as SYSADMIN then navigate path Concurrent:Manager-> Administer
  • 36. Specialization Rules for Concurrent Manager Performance  Use INCLUDE and EXCLUDE rules to optimize performance since Concurrent Managers are batch type processes  Login as SYSADMIN and navigate System Administrator Concurrent:Manager->Define in OAM Located under Specialization Rules
  • 37. Setting values for number of processes with Concurrent Managers  Located under in OAM under navigation path System Administrator-> Concurrent: Manager->Define->WorkShifts
  • 38. Specialization Rules Continued  Use combined specialization rules for complex tasks
  • 39. Workshifts for Performance with Concurrent Processing  Setup different workshifts for balancing performance load with different critical tasks  Example: schedule AP during day and nightly processing on different workshifts  Login to OAM as SYSADMIN then navigate path System Administrator Concurrent:Manager-> WorkShifts
  • 40. Example: Workshifts with Concurrent Processing
  • 41. Concurrent Manager Tuning  Check for high number of requests  Script to find high number of concurrent requests SELECT v.USER_CONCURRENT_QUEUE_NAME,COUNT(PHASE_CODE) v1 FROM APPS.FND_CONCURRENT_QUEUES_V1 v, APPS.FND_CONCURRENT_WORKER_REQUESTS r WHERE r.queue_application_id = 0 AND r.PHASE_CODE = 'P' -- Pending Concurrent Requests AND r.HOLD_FLAG != 'Y' -- Concurrent Requests not on hold AND r.REQUESTED_START_DATE <= SYSDATE -- No Future Concurrent jobs AND r.CONCURRENT_QUEUE_ID=v.CONCURRENT_QUEUE_ID GROUP BY v.USER_CONCURRENT_QUEUE_NAME HAVING COUNT (PHASE_CODE) >=20 /  Use Oracle Application Manager (OAM) charts to measure performance
  • 42. Useful Scripts for Tuning Concurrent Managers  Some Key Scripts for monitoring the concurrent managers: afcmstat.sql Shows all defined managers, capacity, process id, etc. afimchk.sql Status for Internal Concurrent Manager (ICM) & PMON afcmcreq.sql Shows concurrent manager basics afrqwait.sql Shows pending, held and scheduled requests afrqstat.sql Concurrent request execution times and status afqpmrid.sql Process id (PID) for FNDLIBR processes afimlock.sql Find concurrency and locking issues Execute these while connected to SQL*PLUS as APPS schema.  The above scripts are located under the $FND_TOP/sql directory
  • 43. Additional Scripts for Tuning Concurrent Managers and Requests select f.application_short_name app, substr(p.user_concurrent_program_name,1,55) description, substr(p.concurrent_program_name,1,20) program, r.priority, count(*) cnt, sum(actual_completion_date - actual_start_date) * 24 elapsed, avg(actual_completion_date - actual_start_date) * 24 average, max(actual_completion_date - actual_start_date) * 24 max, min(actual_completion_date - actual_start_date) * 24 min, stddev(actual_completion_date - actual_start_date) * 24 stddev, stddev(actual_start_date - requested_start_date) * 24 wstddev, sum(actual_start_date - requested_start_date) * 24 waited, avg(actual_start_date - requested_start_date) * 24 avewait, c.request_class_name type from fnd_concurrent_queues fcq, fnd_concurrent_queue_content fcqc, fnd_concurrent_request_class c, fnd_application f, fnd_concurrent_programs_vl p, fnd_concurrent_requests r where r.program_application_id = p.application_id and r.concurrent_program_id = p.concurrent_program_id and r.status_code in ('C','G','E') and p.application_id = f.application_id and r.program_application_id = f.application_id and r.request_class_application_id = c.application_id(+) and r.concurrent_request_class_id = c.request_class_id(+) and r.request_class_application_id = fcqc.type_application_id(+) and r.concurrent_request_class_id = fcqc.type_id(+) and fcqc.queue_application_id = fcq.application_id(+) and fcqc.concurrent_queue_id = fcq.concurrent_queue_id(+) group by c.request_class_name, f.application_short_name, p.concurrent_program_name, p.user_concurrent_program_name, r.priority / TOTAL AVG MAX MIN RUN WAIT #WAITED AVG REQ APP DESCRIPTION PROGRAM PRI R UN HOURS HOURS HOURS HOURS STDDEV STDDEV HOURS WAIT TYPE ------ ---------------------------------------- -------------------- ---- ------ -- ------- ------ ------ ------ ------- -- ----- --------- -------- ------------- -- SQLGL Periods - Open Period: Child Process GLOOAPC 50 1 2 .00 .00 .00 .00 .00 .00 .10 .01 BOM Cost Manager CMCTCM 50 1 ####### ###### # ##### ###### .00 .00 -16403.89 ######## ALR Check Event Alert ALECTC 1 8 .02 .00 .00 .00 .00 .01 .16 .02 OFA Asset Inventory Report FAS410 1 2 .01 .01 .01 .00 .00 .00 .01 .00 SQLGL Open Encumbrance Year GLEOYR 1 2 .09 .05 .05 .04 .01 .00 .03 .01
  • 44. Manually Gather Statistics for Oracle R12 EBS  FND_STATS for schema statistics collection Example: Collect statistics for AP schema: exec fnd_stats.gather_schema_statistics('AP');  Collect stats for all schemas for Oracle R12 EBS: exec fnd_stats.gather_schema_statistics('ALL');  How to gather stats for temp tables with Oracle R12 EBS: exec fnd_stats.gather_table_stats('<schema>','<temp_table_name>');  Example to collect stats for temp tables in AR schema: exec fnd_stats.gather_table_stats('AR','temp_ap');
  • 45. Statistics Collection for Oracle R12 EBS Why?  Because cost based optimizer requires best execution path with latest current statistics.  Reference Point on how to collect statistics My Oracle Support http://support.oracle.com Note: 419728.1 How To Gather Statistics On Oracle Applications 11.5.10(and above) - Concurrent Process,Temp Tables, Manually Do not use DBMS_STATS with Oracle R12 EBS to collect statistics!
  • 46. Concurrent Manager Statistics Be sure to Gather Table Statistics on these CM tables:  FND_CONCURRENT_PROCESSES  FND_CONCURRENT_PROGRAMS  FND_CONCURRENT_REQUESTS  FND_CONCURRENT_QUEUES.
  • 47. Gather Stats for Oracle R12 EBS  Either run FND_STATS or Concurrent Request via OAM System Administrator|Concurrent Requests|
  • 48. Options for STATS Collection • Options to set parameters for estimate percent, parallelization, schema for Concurrent request to gather statistics for Oracle R12 E-Business Suite • Recommend that you do not set estimate percent too high otherwise the load to collect stats will impact performance and collection time.
  • 49. Tuning Application Tech Stack  Oracle 10g Application Server tuning issues  Oracle HTTP Server (OHS): “Apache” web server  Timeout settings for Apache  Under httpd.conf file  Configuration settings for JVM with OC4J under both the Oracle R12 EBS Context file for application tech stack and opmn.xml values.
  • 50. Tuning Apache OHS Oracle R12 EBS Application Tech Stack  Values to check for the Apache httpd.conf file located under $IAS_ORACLE_HOME/Apache/Apache/conf directory # Timeout: The number of seconds before receives and sends time out. # Timeout 300 # # KeepAlive: Whether or not to allow persistent connections (more than # one request per connection). Set to "Off" to deactivate. # KeepAlive On # # MaxKeepAliveRequests: The maximum number of requests to allow # during a persistent connection. Set to 0 to allow an unlimited amount. # We recommend you leave this number high, for maximum performance. # MaxKeepAliveRequests 100 # # KeepAliveTimeout: Number of seconds to wait for the next request from the # same client on the same connection. KeepAliveTimeout 15
  • 51. Application Tech Stack Tuning  Tune values for timeouts to prevent and resolve errors in performance for network delays for forms, oacore, OC4J  Avoid and resolve infamous “Uninterrupted Exception 150 error” when starting and stopping app tier services  Values to check in Oracle R12 EBS Context file  Consider shared APPL_TOP and staged APPL_TOP for performance  Implement Parallel Concurrent Processing (PCP) so distribute performance load for Concurrent processing
  • 52. Application Tech Stack tuning  Performance metrics for OC4J groups  Use server for concurrent processing server, application server in split tier configuration to provide more performance and availability  Consider load balancer: hardware or software to improve performance and availability  Consider deploying Oracle RAC for scale out and scale up in performance if you need it for large deployments  Oracle Application Management Pack (AMP) for tuning and monitoring Oracle R12 EBS environment
  • 53. Load Balancing for Application tier performance  Hardware Load Balancing options: Cisco and Big IP some vendors that offer solutions  Software Load Balancing  Load balancing is complex and requires coordination with network, security and systems administration teams and the database team to implement successfully.  Reference Point: My Oracle Support Note 727171.1: Implementing Load Balancing On Oracle E-Business Suite - Documentation For Specific Load Balancer Hardware
  • 54. Software Load Balancing for Oracle R12 EBS Application Tier performance  Not as robust as hardware load balancers but more cost effective.  Uses web cache option for Oracle 10gAS application server  My Oracle Support Note: 380486.1 Installing and Configuring Web Cache 10g and Oracle E-Business Suite 12
  • 55. Software Load Balancing Oracle R12 EBS  Requires configuration at both 10g Application Server level and within context file for Oracle R12 EBS The following autoconfig variables must be set within the Oracle R12 EBS application tier context file:  s_webentryhost- webentry points to Oracle 10gAS Web Cache 10g Server  s_webentrydomain- webentry domain to domain name for Oracle 10gAS Web Cache 10g Server  s_active_webport-active webport to port for Oracle 10gAS Server Web Cache where it listens for client requests  s_webentryurlprotocol - webentry protocol to protocol values that clients will use to access Oracle 10gAS Server Web Cache 10g  s_login_page - login page to hostname:port/OA_HTML/AppsLogin
  • 56. Java Tuning for Oracle R12 EBS  Tuning Java components is essential for performance with Oracle R12 EBS  Oracle R12 EBS is dependent on Java technology  Default values are too low for Java with OC4J settings  Watch out for issues with garbage collection  Avoid full garbage collection (GC) as this impacts performance!  My Oracle Support Note: 567551.1: Troubleshooting : Configuring various JVM tuning parameters for Oracle E-Business suite 11i and R12
  • 57. Tuning Java for R12 EBS: JVM Sizing  How many JVMs (Java Virtual Machines) ? OACoreGroup  For 2-tier environment with application middle tier on separate host server and database server  Best not to have more than 100 active users per JVM/OC4J instance for middle tier server. No more than one active JVM/OC4J instance per CPU  Best practice: there should be ratio of 1 JVM to 2 CPUs for middle tier server  Use top and other system tools to check that sufficient memory is available. You want avoid memory swapping issues.  Common mistake for application middle tier server to not be sized correctly for memory and CPU! FormsGroup  For the forms servlet, the actual JVM/servlet usage is minimal, and a single JVM can normally handle up to 100-250 forms users.  JVM settings are configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs, s_forms_servlet_nprocs and s_xmlsvcs_nprocs
  • 58. Java Tuning: Oracle R12 EBS Heap configuration for JVMs with middle tier  OACORE starting point for tuning J VM heap sizes: -Xmx512M -Xms256M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB  Goal is to minimize garbage collection frequency and to reduce user response time by minimizing full garbage collection.  For JDK 1.5, if the JVM detects that you have a server with multiple CPUs it will enable Parallel Throughput Garbage Collector by default.  Rule of thumb: number of GC (garbage collection) threads defaults to number of CPUs per machine.  If you run multiple JVMs on the same server, or if you have more than 2 CPUs, to avoid excessive GC, reduce the number of GC threads: -XX:+UseParallelGC -XX:ParallelGCThreads=2  Two GC threads should be enough for most environments  For heap sizes over 1GB, you can test with 4 GC threads and measure performance.  FormsGroup values for Garbage Collection (GC) as rule of thumb: -Xmx256M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 Key Oracle R12 EBS Application Middle Tier Server Autoconfig Variables for Java Heap tuning with JVMs:  s_oacore_jvm_start_option  s_forms_jvm_start_options
  • 59. Tools for Measuring JVM Performance with Oracle R12 EBS  Excellent free tool to graph performance for JVM with Oracle R12 EBS and garbage collection.  http://www.tagtraum.com/gcviewer.html
  • 60. Tuning Java for Oracle R12 EBS  My Oracle Support Note: 362851.1: - Guidelines to setup the JVM in Apps E-Business Suite 11i and R12 JVM Behavioral Options: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#BehavioralOptions JVM Options List: http://blogs.sun.com/watt/resource/jvm-options-list.html
  • 61. Monitoring Java Performance: Oracle R12 EBS  Oracle 10g Grid Control (OEM) with Application Management Pack (AMP). Note: These items require additional cost licenses.  MOS Note 557194.1 provides a script called monitor_jdbc_conn.sql to monitor JDBC connections.
  • 62. Java Tuning for Oracle R12 EBS  Collect metrics to analyze current performance  Good information from ADDM and AWR in 11g Statistic Total per Second per Trans -------------------------------- ------------------ -------------- ------------- java call heap collected bytes 683,300,160 1,300.0 877.8 java call heap collected count 3,693,358 7.0 4.7 java call heap gc count 356 0.0 0.0 java call heap live object count 24,810 0.1 0.0 java call heap live object count 142,439 0.3 0.2 Instance Activity Stats DB/Inst: SBOX/SBOX Snaps: 274-420 java call heap live size 2,358,144 4.5 3.0 java call heap live size max 33,260,184 63.3 42.7 java call heap object count 56,992 0.1 0.1 java call heap object count max 345,992 0.7 0.4 java call heap total size 42,729,472 81.3 54.9 java call heap total size max 115,343,360 219.5 148.2 java call heap used size 6,144,496 11.7 7.9 java call heap used size max 72,864,528 138.6 93.6
  • 63. Database Tuning for Oracle R12 EBS  Similar to tuning regular Oracle 11g database but unique in how statistics collected  Remember to use FND_STATS and NOT DBMS_STATS to collect and update statistics!  Watch out for concurrency and locking issues for runaway user processes!  Schedule backup jobs off hours to avoid impact business users  Don’t forget to use tools like AWR and STATSPACK if possible.
  • 64. Database Tuning  Suggested values for database initialization parameters for Oracle R12 EBS  My Oracle Support Note: 396009.1 Database Initialization Parameters for Oracle Applications Release 12  Oracle R12 by default installs either a 10gR2 or an 11gR1 database for fresh (new) installation.  For example: Oracle 12.0 to 12.04 uses 10.2.x database  While Oracle 12.1.1 and later release install Oracle 11gR1 database (11.1.x).  Watch out for upgrade issues from 11i to R12 EBS for database!
  • 65. Database Parameters for Oracle R12 EBS Parameter Name Development or Test Instance 11-100 Users 101-500 Users 501-1000 Users 1001-2000 Users processes 200 200 800 1200 2500 sessions 400 400 1600 2400 5000 sga_target 1G 1G 2G 3G 14G shared_pool_size (csp) N/A N/A N/A 1800M 3000M shared_pool_reser ved_size (csp) N/A N/A N/A 180M 300M shared_pool_size (no csp) 400M 600M 800M 1000M 2000M shared_pool_reser ved_size (no csp) 40M 60M 80M 100M 100M pga_aggregate_tar get 1G 2G 4G 10G 20G Total Memory Required ~ 2 GB ~ 3 GB ~ 6 GB ~ 13 GB ~ 25 GB
  • 66. Upgrade Performance considerations for Oracle R12 EBS  When you upgrade from older releases such as 10.x and 11i to Oracle R12 must account for performance items  Rule based Optimizer (RBO) in 10.x and early 11i releases to the Cost Based Optimizer (CBO) in R12  Different way of tuning for stats and database  Consider changes for SGA and PGA sizing  Migrate from old tablespace model to Oracle Application Managed Tablespaces (OATM)  My Oracle Support Note: 761570.1 Database Preparation Guidelines for an E-Business Suite Release 12.1.1 Upgrade
  • 68. Thank You!  Please provide feedback  Email: ben@ben-oracle.com  My blog has further tips on Oracle R12 EBS  http://oracle-magician.blogspot.com  I will be available in the OTN ACE Director Office Hours this week for questions