This document provides an agenda and overview for a presentation on best practices for large Oracle Applications R12 implementations. The agenda covers application tier topics like Forms, response time, concurrent processing, and workflow. It also covers database tier topics such as AWR and ADDR reports, ORACHK health checks, and cluster callout scripts. The goal is to review strategies for ensuring performance, scalability, and proactive issue identification.

1. Imagination at work.
Ajith Narayanan
Technical Lead-Oracle ERP Configuration Management
GE Healthcare, Bangalore, India
8th Dec-2014, Liverpool, U.K.
Best Practices For Large Oracle
Apps R12 Implementations

2. Disclaimer

3. The views/contents in this slides are those of the author and do not
necessarily reflect that of General Electric Company or Oracle
Corporation and/or their respective affiliates/subsidiaries.
The material in this document is for informational purposes only, and is
provided on an AS-IS basis.
11/14/2014
3

4. Who Am I?
Ajith Narayanan
Technical Lead – Oracle ERP Configuration Management
GE Healthcare, Bangalore, India
• 10 years of Oracle [APPS] DBA/Technical Architect experience.
• Oracle ACE Associate
• Blogger :- http://oracledbascriptsfromajith.blogspot.com
• Member:-OAUG, AIOUG
• Website Chair (2011-2013) – Oracle RACSIG http://www.oracleracsig.org
• AIOUG Real Application Clusters SIG Leader
11/14/2014
4

5. Agenda
Why do we need to implement best practices?
• Application Tier
• R12 Tech Stack
• Forms
• Response Time
• Web Applications
• Concurrent Manager
• Workflow Engine
• Taking Diagnostics Help
• CBO Statistics
• Application Patching
• Network & Latency Check
11/14/2014
5

6. Agenda
• Database Tier
• CPU Utilization Observations and analysis
• High DB_IO observation and correlation with CPU usage.
• Identify Top SQLs
• Effective usage of AWR, ADDM Reports
• ORACHK Check (From Oracle RAC assurance team)
• ADR Command Interpreter (ADRCI)
• Cluster Callout scripts example.
• Looking At Issues Proactively & From Greater Heights
• References
• Q & A
11/14/2014
6

7. Why do we need to implement best
practices?
11/14/2014
7

8. Oracle Applications R12
• Huge infrastructure
– Oracle Applications R12
– 8 Node middle tiers
– 16 Node RAC database tier.
– Database Size 40+ TB
– Annual DB growth of 2 TB
– Daily Orders: 45 K –3 M order
lines
– 1500 online users
– 1M+ XML messages/day
Best Practices
• Benefits
– Enables us being proactive
– simple in terms of manageability
– Helps in support cost reduction.
– Time & effort saving
Why do we need to implement best
practices?
11/14/2014
8

9. Upgrade Tech Stack
• R12 Tech Stack
• OC4J: Oracle AS 10g 10.1.3./ Weblogic Managed Servers (FMW)
• Forms: Oracle AS 10.1.2
• JAVA, JRE Plugin
Useful MOS Articles
Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1)
Oracle E-Business Suite Technology Stack Release Notes for Release 12.2*
Oracle E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1)
Troubleshooting Assistant: EBS Technology Stack (Doc ID 1607365.2)
Application Tier
11/14/2014
9

10. Application Tier
11/14/2014
10

11. Forms
• Socket Mode for Internal users (MOS 384241.1)
• Servlet mode is default in R12 and recommended for external access.
– cat $FORMS_WEB_CONFIG_FILE|grep serverURL=
If the serverURL parameter has no value then Forms is implemented in socket mode else it is servlet
• Forms dead client detection (DCD) and Abnormal termination handler.
– Value specified in minutes : FORMS_TIMEOUT=10
env | grep -i FORMS_TIMEOUT
– Terminates fwebmx processes for dead clients.
– Enable Forms Abnormal Termination Handle
– Do not set FORMS_CATCHTERM
– SQLNET.EXPIRE_TIME – Probes client connection at regular interval.
Useful MOS Articles
Application Tier
11/14/2014
11
Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID 745711.1)

12. Response Time/CPU Issues
• When users complain about response time, configure Apache to log the time it
takes to service a request.
• Access logs monitoring should be good way to keep a tab on the response times of
user requests.
– Configure Apache to log the time it takes to service a request
– Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*
– Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
LogFormat "%h %T
(Or)
– Enable Execution Content ID's (ECID) for the access_log in Release 12. ECID's allow correlation of
requests across log files so that one can map the flow of a given request across the various
components of iAS.
LogFormat "%h %l %u %t [ecid: %{Oracle-ECID}i] "%r" %>s %b [%T (secs)]" combined
Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for getting http return code definitions
Note - R12.2 replaces Oracle 10gAS OC4J containers with Weblogic managed servers.
Application Tier
11/14/2014
12

13. Web Applications
• Train users to logout when transactions are complete.
– Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.)
Releases process resources sooner
• JVM/OC4J Sizing + Heap Size
– OACORE, OAFORMS, OAFM etc..
– Start with no more than 100 active users per JVM/OC4J instance.
– Start with 1 active JVM/OC4J instance per CPU core
Note:- Each 'core' in a multi-core system is considered to be a separate CPU. For example, a dual quad-
core chip would be the same as stating 2 x 4 x CPU or 8 CPUs.
Useful MOS Articles
JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1)
MOS Note 357597.1 on how to enable SQL trace for OA Framework applications
Application Tier
11/14/2014
13

14. EBS Concurrent Processing Analyzer
• The EBS Concurrent Processing Analyzer is a self-service health-check script
• Reviews the overall Concurrent Processing
• Analyzes the current configurations & settings for the environment.
• Providing feedback and recommendations on best practices.
• APPS Workload (Concurrent, FORM, ICX) observations.
• Mandatory Concurrent Programs fro purging.
Sample Output
Useful MOS Articles
Application Tier
2DEMO_cp_analyzer.html
11/14/2014
14
Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1]
Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)

15. EBS Workflow Analyzer
• The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint.
• Analyzes the workflow configurations & Provides feedback and makes
recommendations on best practices and areas of concern.
• Provides immediate analysis and output of the EBS Workflow environment.
• Workflow Process Stuck & Work Flow Checks
Sample Output
Useful MOS Articles
Application Tier
2DEMO_wf_analyzer.html
11/14/2014
15
How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1)
11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc
ID 1369938.1)
How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)

16. OAM Diagnostics Help
• Oracle Application Object Library Best Practices: Database Security Tests Health
Check Test
• Oracle Application Object Library Best Practices: E-Business Suite Security Tests
Health Check Test
• Oracle Application Object Library Best Practices: E-Business Suite Diagnostic
Tests Health Check Test
• And many more……
Application Tier
11/14/2014
16

17. CBO Statistics
• FND_STATS uses the standard the DBMS_STATS package, but enhances the
functionality as it provides several additional PL/SQL procedures for
gathering statistics at the table, schema, or database level.
• The FND_STATS package also supports histograms, table exclusions,
extended stats, incremental stats gathering for partitioned tables, and
concurrent statistics gathering.
Useful MOS Articles
Application Tier
11/14/2014
17
Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1)
bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)

18. Application Patching
• Thorough patch analysis with timing reports on sand box & non-prods
environments before proceeding with the Prod patching.
• In Release 12, you register your customized files in the Register Flagged Files tool
in Oracle Applications Manager (OAM), which replaces the
$APPL_TOP/admin/applcust.txt file.
Useful MOS Articles
Application Tier
11/14/2014
18
See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“
Patching Best Practices And Reducing Downtime (Doc ID 225165.1)

19. • There are two diagnostic tools available in the Oracle E-Business Suite:
The Network test Form – This form can be added to user’s menu , So that they can
run the tests when they feel a performance degradation
Note:- Network latency should be always low (<3ms) and bandwidth should be high.
Application Tier
Network & Latency
11/14/2014
19

20. Useful MOS Articles
The Client Analyzer
Application Tier
Network & Latency
11/14/2014
20
Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)

21. CPU Utilization Observations and
analysis
• CPU spikes for more than 30 minutes
continuously is a candidate for RCA.
• High DB_IO observation and
correlation with CPU usage.
• I/O Thresholds: Logical RDS >
300K/sec, Physical RDS> 3000/Sec are
candidate for RCA.
– How many of you think RAM
access is 10,000 times faster than
Physical disk access?
– In real world, LIO is only 25-100
times cheaper than PIO
– Internal locks & latch serialization
mechanisms involved.
– Targeting only PIO counts(or high
cache hit ratios) important pitfall
to avoid. Even with no PIOs, a
query can still be outrageously
inefficient
– LIO are a critical component of
query cost
Database Tier
11/14/2014
21

22. AWR, ASH, ADDM Reports
• Identify Top SQL, Latch Spin Counts
etc., Memory Advisors
• Identify top wait events (System, User
IO, Cluster, Concurrency,
Configuration, Applications, Network)
• Fragmented Interface tables
• Object to be pinned – (Frequently used
database objects)
• RAC Check (From Oracle RAC
assurance team) RACcheck - RAC
Configuration Audit Tool (Doc ID
1268927.1)
• ORAchk - Health Checks for the Oracle
Stack (Doc ID 1268927.2)
• Sample ORACHK report
• ORAchk-Collection Manager
– ORAchk daemon auto-start mode
after node reboot (init integration)
– Merge multiple ORAchk collection
reports
– Exclude checks based on profile
– Upload of installed patches to
database
– Collection Manager for ORAchk,
RACcheck and Exachk (Doc ID
1602329.1)
– ORAchk signature file in /tmp on
all nodes to verify last ORAchk run
– New checks and bug fixes,
including o 30 Oracle Ebusiness
AP module data integrity checks
– 12 new Database checks
– 8 new Solaris system checks
Database Tier
ORAchk-Report-Example.html
11/14/2014
22

23. • ADRCI has multiple features such as
– Analyze diagnostic data within the Automatic Diagnostic Repository (ADR).
– Create a package with incident / problem information for Oracle Support.
– Diagnostic data includes incident and problem descriptions, trace files,
dumps, health monitor reports, alert log entries, and more.
– Run the Health Checks
– Helps in purging old diagnostics trace & dump files
– Simple to use a rich command set
Set correct ORACLE_HOME and just enter adrci
The utility starts and displays the following prompt:
adrci>
Database Tier
11/14/2014
23
ADR Command Interpreter (ADRCI)

24. • At the ADRCI prompt, enter the following command:
adrci> HELP
To get help for a specific ADRCI command:
adrci> HELP command
• For example, to get help on the SHOW TRACEFILE command, enter the following:
adrci> HELP SHOW TRACEFILE
adrci> set homepath diag/rdbms/orclbi/orclbi2
adrci> show homes
ADR Homes:
diag/rdbms/orclbi/orclbi2
• To view the alert log:
1. Start ADRCI in interactive mode, and ensure that the homepath points to the
correct directory within the ADR base directory hierarchy.
Database Tier
11/14/2014
24

25. Useful MOS Articles
2. At the ADRCI prompt, enter the following command:
SHOW ALERT
The following are variations on the SHOW ALERT command:
SHOW ALERT -TAIL
SHOW ALERT -TAIL 50
SPOOL /tmp/MYALERT.LOG
SHOW ALERT
SPOOL OFF
This outputs the alert log, without XML tags, to the file
/tmp/MYALERT.LOG.
SHOW ALERT -P "MESSAGE_TEXT LIKE '%ORA-600%'"
Database Tier
11/14/2014
25
11G ADR DOCUMENTATION (Doc ID 445022.1)
How to Invoke ADR Command Interpreter (adrci) in FMW home? (Doc ID 1669923.1)

26. Database Tier
#!/bin/ksh
# # Author: Ajith Narayanan
## http://oracledbascriptsfromajith.blogspot.com
## Version 1.0
## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs.
##
umask 022
FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.log
EVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.log
EVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.log
MAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.log
WEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_ds
echo $* "reported="`date` >> $FAN_LOGFILE &
tail -1 $FAN_LOGFILE > $EVENTLINE
awk '{
for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END {
for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } }
}' $EVENTLINE > $EVENTLINE_MID
SER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'`
DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'`
INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'`
HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'`
STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'`
if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; then
if [ $STAT = nodedown ]; then
cat $EVENTLINE_MID > $MAIL_CONT
11/14/2014
26
Cluster callout scripts. – Quickly get alerted on any cluster events.

27. echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONT
elif [ $STAT = up ]; then
cat $EVENTLINE_MID > $MAIL_CONT
echo "**============================SERVICES RESTORED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT
else
cat $EVENTLINE_MID > $MAIL_CONT
echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT
#fi
cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.com
fi
#cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.com
fi
rm $EVENTLINE $EVENTLINE_MID $MAIL_CONT
Database Tier
11/14/2014
27

28. Looking At Issues Proactively & From
Greater Heights
11/14/2014
28

29. References
• My Oracle Support
• Steven Chan’s blog (https://blogs.oracle.com/stevenChan)
11/14/2014
29

30. Q&A
11/14/2014
30

31. 11/14/2014
31
Thank You!