This is my old presentation about internals of Concurrent managers in e-Business Suite. Presented at OUG Harmony 2011 in Helsinki, Finland.

1. Inside
Concurrent
Managers
OUG Harmony 2011
© 2011 Tieto Corporation
Māris Elsiņš
Lead Oracle [Applications] DBA
Tieto Latvia,
Maris.Elsins@tieto.com

2. Who I am?
• 9 years in IT
• 3 years – PL/SQL developer
• 6 years – Oracle [Apps] DBA (started with 11.5.7 and 8.1.7)
• Certificates
• 10g OCM
• 9i / 10g / 11g OCP
• 11i Applications Database Administrator OCP
• 11i System Administrator OCE
• Conferences
• UKOUG 2007/2008/2010
• LVOUG 2009/2010
• EMEA Harmony 2010/2011 
• Current employer – Tieto Latvia
• All kinds of oracle DBA tasks - patching, upgrade, performance tuning,
troubleshooting, planning and implementation of backup and recovery procedures,
cross platform migration, etc
• Planning of sydtem architecture, design and implementation of HA solutions (RAC,
Data Guard, Custom cold failover)
• Implementation of system specific monitoring, automation of routine tasks
• Technical project planning and coordination, management of team of DBAs
2 © 2011 Tieto Corporation 2010-11-29

3. Purpose of this session
• Provide background knowledge for successful
troubleshooting of concurrent manager problems
• What this presentation is about?
• Life cycle of a concurrent request
• Implementation details of the most interesting phases of a life cycle
of a concurrent request
• Internals of Concurrent Managers and Conflict Resolution Managers
• DB objects that can be used for troubleshooting
• Why it is important?
• DBA has to be quick when solving real issues
• Knowing the process before the problems happen decreases solving
time
• Not everything is visible using the GUI
• Querying DB objects is faster then navigating through Forms
• Understanding the process is important when tuning the setup of
Concurrent Managers
3 © 2011 Tieto Corporation 2010-11-29

4. Most interesting DB objects
• Objects
• FND_CONCURRENT_REQUESTS (FCR)
• FND_CONCURRENT_PROGRAMS (FCP)
• FND_CONCURRENT_QUEUES (FCQ)
• FND_CONCURRENT_PROCESSES (FCPROC)
• FND_CONCURRENT_PROGRAM_SERIAL (FCPS)
• FND_CONC_RELEASE_CLASSES (FCRC)
• FND_LOOKUPS (FL)
• FND_CONCURRENT_WORKER_REQESTS (FCWR)
• FND_CRM_HISTORY (FCH)
• FND_CONC_WAITING_REQUESTS
• Some examples how the tables are used will be uncovered
in this presentation
4 © 2011 Tieto Corporation 2010-11-29

5. Types of concurrent managers
• Types of concurrent managers
• Internal Concurrent Manager
• Service Manager
• Concurrent Manager
• Conflict Resolution Manager
• Internal Monitor
• Transaction Manager
• Scheduler/Prereleaser Manager
• …
• Today the accent is on
• Concurrent Manager
• Conflict Resolution Manager
5 © 2011 Tieto Corporation 2010-11-29

6. Life cycle of a concurrent request
INACTIVE (I) PENDING (P) RUNNING (R) COMPLETED (C)
• Disabled (U) • Normal (I) • Normal (R) • Normal (C)
• On Hold (H) • Standby (Q) • Paused (W) • Error (E)
• No Manager (M) • Scheduled (P) • Resuming (B) • Warning (G)
• Waiting (Z) • Terminating (T) • Cancelled (D)
• Terminated (X)
FND_LOOKUPS contains the values of Phase/Status Codes
«Oracle® E-Business Suite System Administrator's Guide – Maintenance»
describes the meaning of each phase and status
6 © 2011 Tieto Corporation 2010-11-29

7. Typical life cycle of most requests
Pending / Scheduled
• Request is scheduled for the execution in future
Pending / Standby
• Time to execute the request has arrived
• Request waits for to be evaluated by the Conflict Resolution Manager
Pending / Normal
• Request is allowed to be executed
• Request is waiting to be picked up by Concurrent Manager
Running / Running
• Request is beeing executed by a concurrent manager
Completed / Normal
7 © 2011 Tieto Corporation 2010-11-29

8. Phase/Status representation in tables
• FND_CONCURRENT_REQUESTS
• PHASE_CODE
• STATUS_CODE
• Is that so simple? NO!
• Phase = Inactive practically not used
• Inactive/Disabled: FCR.PHASE_CODE=„P‟, FCP.ENABLED_FLAG=„N‟
• Inactive/On Hold: FCR.PHASE_CODE=„P‟, FCR.HOLD_FLAG=„Y‟
• Inactive/No manager: FCR.PHASE_CODE=„P‟ and nonexistance of
specific rows in FND_CONCURRENT_WORKER_REQESTS view
• Pending/Scheduled is actually
• STATUS_CODE=‘Q’+(FCR.REQUESTED_START_DATE>SYSDATE)
• Why so complicated?
• No need to update statuses too often
8 © 2011 Tieto Corporation 2010-11-29

9. Pending / Scheduled
9 © 2011 Tieto Corporation 2010-11-29

10. Schedule types
• As Soon as Possible/Once
FCR.REQUESTED_START_DATE
• Periodically/On Specific Days
FND_CONC_RELEASE_CLASSES (FCRC)
• Join with FCR on column RELEASE_CLASS_ID
• FCRC.DATE1 - “Start at” field in the form
• FCRC.DATE2 - “End at” field in the form
• FCRC.CLASS_TYPE – «P» for «Periodically», «S» for «On Specific
days»
• FCRC.CLASS_INFO – actual schedule data
• FCR.REQUESTED_START_DATE always contains the time
of next execution, only this field used to determine when the
request should be run by the concurrent manager
10 © 2011 Tieto Corporation 2010-11-29

11. Periodic schedules
FCRC.CLASS_INFO contents
• Values like «X:Y:Z»
• X – number of months/weeks/days/hours/minutes the request has to
be rescheduled from prior run.
• Y – time units: “M” – months, “D” – days, “H” – hours, “N” – minutes
• Z – rescheduling type: «S» – from the start of the prior run, «C» –
from the completion of the prior run.
• Samples
• 30:N:S – Repeat every 30 minutes from the start of the prior run
• 5:N:C – Repeat every 5 minutes from the completion of the prior run
• 12:H:S – Repeat every 12 hours from the start of the prior run
11 © 2011 Tieto Corporation 2010-11-29

12. «On Specific Days» schedules
FCRC.CLASS_INFO contents
• Values like
«XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYZZZZZZZ»
• X, Y, Z contains values 0 or 1, representing if the option is selected
or not
• X – digit in each position represent dates 1 – 31
• Y – Last Day of month
• Z – day of week, Su - Sa
• Samples
• 000000000000000000000000000000000000001 – Days of week: Sa
• 111111111000000000000000000000000111110 – Dates: 1 2 3 4 5
6 7 8 9. Days of week: Mo Tu We Th Fr
• 000000000000000000000000000000010000000 – Last day of
month
12 © 2011 Tieto Corporation 2010-11-29

13. Reporting the schedules
• Easy to schedule requests, but hard to keep track
• Check if there are no obsolete schedules that can be removed or tuned
• Might be few repeting schedules of the same concurrent program can be
consolidated into one
13 © 2011 Tieto Corporation 2010-11-29

14. Pending / Standby
14 © 2011 Tieto Corporation 2010-11-29

15. Conflict Resolution Manager (CRM)
• CRM resolves conflicts in execution of requests enforced by
incompatibility rules
• Simplified CRM workflow
• Startup – load all incompatibility rules
1. Loaded from FND_CONCURRENT_PROGRAM_SERIAL
2. Stored in memory by FNDCRM (CRM binary executable)
• Each itereation
1. If FCQ.CONTROL_CODE=„V‟, reload incompatibility rules (?)
2. Lock STATUS_CODE for all Pending/Normal and Pending/Scheduled
requests
3. Check each Pending request against the incompatibility rules and verify
there are running concurrent managers that can process them
(FND_CONCURRENT_WORKER_REQUESTS)
1. «Release» Pending/Standby requests, which don‟t break any rules
2. «Return» Pending/Normal requests, which break some rules (?)
4. Record statistics in FND_CRM_HISTORY
5. Sleep for «Sleep Seconods»
15 © 2011 Tieto Corporation 2010-11-29

16. Interesting DB objects
• FND_CONCURRENT_WORKER_REQUESTS
• View returning mappings of requests and concurrent managers able
to execute them
• Definition of the view contains hardcoded IDs according to the
Specialization Rules of Concurrent Managers
• Rebuilt by «Build Concurrent Request Queue View» request on
change of Specilaization rules
• FND_CRM_HISTORY
• Good information for tuning and troubleshooting
• Records statistics for each CRM run, stotistics include:
• REQUESTS_EXAMINED
• REQUESTS_STANDBY
• REQUESTS_RELEASED
• REQUESTS_RETURNED
• Purged by «Purge Concurrent Request and/or Manager Data»,
leaving 1 day of history (R12.1.3)
• Check FCR.CRM_RELEASE_DATE to find when the
request has been released for execution by CRM
16 © 2011 Tieto Corporation 2010-11-29

17. Graphing FND_CRM_HISTORY
17 © 2011 Tieto Corporation 2010-11-29

18. Why is my request still pending?
• Check the status!
• Pending/Scheduled – time for execution has not come yet
• Pending/Normal –waiting to be picked up by concurrent manager
• The request has just been released
• All concurrent manager processes are busy executing requests
• Long request execution queue
• Pending/Standby – Waiting for CRM to be released
SELECT WREQID, PHASE, STATUS, WHY
FROM FND_CONC_WAITING_REQUESTS
WHERE REQID = &REQUEST_ID --5820658
18 © 2011 Tieto Corporation 2010-11-29

19. Pending / Normal
19 © 2011 Tieto Corporation 2010-11-29

20. Workflow of «Concurrent Managers»
20 © 2011 Tieto Corporation 2010-11-29

21. Building SQL for querying the requests queue
Select R.Rowid …
From Fnd_Concurrent_Requests R And (P.Execution_Method_Code != 'S' OR
Where R.Hold_Flag = 'N' (R.PROGRAM_APPLICATION_ID,
And R.Status_Code = 'I' R.CONCURRENT_PROGRAM_ID) IN
And R.Requested_Start_Date <= Sysdate ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757)))
And (R.Node_Name1 is null or AND ((R.PROGRAM_APPLICATION_ID,
R.CONCURRENT_PROGRAM_ID) NOT IN
(R.Node_Name1 is not null and
((510, 40032),
FND_DCP.target_node_mgr_chk(R.request_id) = 1))
(510, 40033),
AND EXISTS
(510, 42156),
(Select Null
(510, 42157),
From Fnd_Concurrent_Programs P
(530, 43793),
Where P.Enabled_Flag = 'Y'
(530, 43794),
And R.Program_Application_Id = P.Application_Id
(535, 42626),
And R.Concurrent_Program_Id = P.Concurrent_Program_Id
(535, 42627),
AND EXISTS
(535, 42628)) AND
(Select Null
((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND
From Fnd_Oracle_Userid O
R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR
Where R.Oracle_Id = O.Oracle_Id
(R.REQUEST_CLASS_APPLICATION_ID,
AND EXISTS (Select Null
R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2)))))
From Fnd_Conflicts_Domain C
ORDER BY NVL(R.priority, 999999999), R.Priority_Request_ID,
Where P.Run_Alone_Flag = C.RunAlone_Flag R.Request_ID
And R.CD_Id = C.CD_Id))
…
Implementation ofConcurrentrulesrulewithRequestprograms
Distributed only «Pending/Normal»implementation
Execution order Processing requests
«Run Alone» flag implementationexecutables (Subroutines)
Exclusion specializationfor concurrentConflict type
Exclusion «Immediate» type concurrent domains
Query Specialization for for
21 © 2011 Tieto Corporation 2010-11-29

22. Building SQL for querying the requests queue
• The query is built at the startup of the concurrent manager
• It hardcodes all specialization rules for the manager
• Any changes to specialization rules force restart of the
concurrent manager processes (and runs «Build Concurrent
Request Queue View» concurrent program too)
• Be careful! It does restart automatically! What happens if there are
long running requests?
• Use «request types», define one for each custom concurrent
manager, and one for Standard manager exclusions (DEMO?)
22 © 2011 Tieto Corporation 2010-03-26

23. Locking the STATUS_CODE
SELECT ... …
FROM fnd_concurrent_requests R, And R.Requested_Start_Date <= Sysdate
fnd_concurrent_programs P, And (R.Enforce_Seriality_Flag = 'N' OR
fnd_application A, (C.RunAlone_Flag = P.Run_Alone_Flag And
fnd_user U, (P.Run_Alone_Flag = 'N' OR Not Exists
fnd_oracle_userid O, (Select Null
fnd_conflicts_domain C, From Fnd_Concurrent_Requests Sr
fnd_concurrent_queues Q, Where Sr.Status_Code In ('R', 'T')
fnd_application A2, And Sr.Enforce_Seriality_Flag = 'Y'
fnd_executables E, And Sr.CD_id = C.CD_Id))))
fnd_conc_request_arguments X And Q.Running_Processes <= Q.Max_Processes
WHERE R.Status_code = 'I' And R.Rowid = :reqname
And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or And ((P.Execution_Method_Code != 'S' OR
(R.OPS_INSTANCE = (R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN
decode(:dcp_on, 1, FND_CONC_GLOBAL.OPS_INST_NUM, ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) AND
R.OPS_INSTANCE))) ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN
And R.Request_ID = X.Request_ID(+) ((510, 40032),
And R.Program_Application_Id = P.Application_Id(+) (510, 40033),
And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+) (510, 42156),
And R.Program_Application_Id = A.Application_Id(+) (510, 42157),
And P.Executable_Application_Id = E.Application_Id(+) (530, 43793),
And P.Executable_Id = E.Executable_Id(+) (530, 43794),
And P.Executable_Application_Id = A2.Application_Id(+) (535, 42626),
And R.Requested_By = U.User_Id(+) (535, 42627),
And R.Cd_Id = C.Cd_Id(+) (535, 42628)) AND
And R.Oracle_Id = O.Oracle_Id(+) ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND
And Q.Application_Id = :q_applid R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR
And Q.Concurrent_Queue_Id = :queue_id (R.REQUEST_CLASS_APPLICATION_ID,
And (P.Enabled_Flag is NULL OR P.Enabled_Flag = 'Y') R.CONCURRENT_REQUEST_CLASS_ID) NOT IN
And R.Hold_Flag = 'N„ ((0, 2)))))
… FOR UPDATE OF R.status_code NoWait
23 © 2011 Tieto Corporation 2010-11-29

24. Locking the STATUS_CODE
• Query for locking the STATUS_CODE reimplements the same validation
criteria to make sure situation has not changed
• All processes of a concurrent manager use the same query to fetch the
«cache size» number of requests
• As more processes of the same manager are run, as higher the competition
for requests («ORA-00054: resource busy and acquire with NOWAIT
specified», or 0 rows updated by the query if the status has been changed
already)
• As higher the competition, as faster runs out the list of cached request ids for
each manager
• As sooner the list of cached queries runs out, as more often FCR is queried
• We want to query FCR as saldom as possible
• Not hard to get to point where FCR queries are TOP SQLs in DB
• Even more important if you have RAC
• The key is to minimize the number of concurrent manager processes
• Cache size and sleep seconds have some effect
24 © 2011 Tieto Corporation 2010-11-29

25. Cache size and Sleep seconds
• No silver bullets for tuning 
• «Sleep seconds»
• sleep time of Conflict Resolution Manager affects how soon the
request will be passed to execution
• are spent only when no requests are in pending/normal status in
FCR
• should be chosen based on the average time the request is allowed
to spend in the queue
• Average time 20s and 5 managers? == 100s sleep seconds?
• «Cache size»
• Large cache sizes make changes of request priorities less effective
(do you use different priorities)
• Small cache size might be better for Long-running requests
• Larger cache sizes might be better for Short-running requests that
have few concurrent manager instances.
• Large cache size increases the number of failed attempts to lock the
status code.
25 © 2011 Tieto Corporation 2010-11-29

26. Running / Normal
26 © 2011 Tieto Corporation 2010-11-29

27. How to find processes and sessions?
select r.request_id req_id, from applsys.fnd_concurrent_requests r,
r.phase_code p, gv$session ss,
r.status_code s, gv$process pp,
(select node_name || ':' gv$instance gi,
from applsys.fnd_concurrent_processes cp applsys.fnd_concurrent_processes cp
where concurrent_process_id = r.controlling_manager) || where request_id = &request_id
r.os_process_id cp_process, and ss.audsid(+) = r.oracle_session_id
gi.INSTANCE_NAME || ':' || ss.sid || ',' || ss.serial# and pp.inst_id(+) = ss.inst_id
inst_sid_serial#, and pp.addr(+) = ss.paddr
gi.HOST_NAME || ':' || pp.spid db_process, and gi.inst_id(+) = ss.inst_id
ss.program, and cp.concurrent_process_id(+) = r.controlling_manager
ss.status,
ss.sql_id || ':' || ss.sql_child_number sql_id_chld,
ss.event,
ss.WAIT_TIME,
ss.STATE
• For completed requests CP_PROCESS field is still visible
27 © 2011 Tieto Corporation 2010-11-29

28. Do we still have time for some
bonus material?
28 © 2011 Tieto Corporation 2010-11-29

29. Troubleshooting performance of a
concurrent program
• Trace it! – but how?
• How do we switch on tracing for a concurrent program and
what are we looking for to get a complete picture?
• waits information
• bind variables information
• row source operation statistics
• easy to switch on
• transparent to the end-user
• traces only what we want (only the concurrent program)
• trace the program execution from the beginning
• Lot of options...
29 © 2011 Tieto Corporation 2010-11-29

30. Tracing the concurrent program 1
• Concurrent -> Program -> Define -> «Enable Trace»
• Binds
30 © 2011 Tieto Corporation 2010-11-29

31. Tracing the concurrent program 2
• Profile Option “Concurrent: Allow Debugging”
• transparent to the end-user
31 © 2011 Tieto Corporation 2010-11-29

32. Tracing the concurrent program 3
• Profile Option “SQL Statement - Custom“
• BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION
SET TRACEFILE_IDENTIFIER='||''''||'USERID' ||''''||' EVENTS ='||''''||'
10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;
• easy to switch on
• traces only what we want (only the concurrent program)
32 © 2011 Tieto Corporation 2011-05-19

33. Tracing the concurrent program 4
• «Catch» the session (v$session) and enable tracing
manually
• dbms_monitor.session_trace_enable
• dbms_system.set_sql_trace_in_session
• ...
• trace the program execution from the beginning
33 © 2011 Tieto Corporation 2011-05-19

34. Best way I have found so far...
• Concurrent manager sets MODULE information (concurrent program
short name) for the DB session when the execution of the request starts
and unsets MODULE information when it ends.
• Need to find the service_name and module from v$sessions before
select r.request_id, s.service_name, s.module
from fnd_concurrent_requests r, gv$session s
where r.status_code = 'R' and s.AUDSID = r.oracle_session_id;
begin
dbms_monitor.serv_mod_act_trace_enable(service_name => 'TEST',
module_name => 'e:SYSADMIN:cp:fnd/FNDCPRT_PLSQL',
action_name => dbms_monitor.all_actions,
waits => true,
binds => true,
plan_stat => 'ALL_EXECUTIONS'); --<==starting from 11g
end;
• What‟s enabled? DBA_ENABLED_TRACES
• Find the trace file!
• find -mmin -20 -exec grep -l 391541 {} ;
• select oracle_process_id from fnd_concurrent_requests where request_id=391541;
34 © 2011 Tieto Corporation 2011-05-19

35. Where to get more information?
• OTN - Oracle E-Business Suite System Administrator's
Guide Documentation Set
• http://etrm.oracle.com – ER Diagrams and information about
the DB Objects
• http://appsdbalife.wordpress.com – Comment and ask the
questions, I will answer!
• Twitter @MarisElsins
Thank you!
?
35 © 2011 Tieto Corporation 2010-11-29

36. Māris Elsiņš
© 2011 Tieto Corporation
Lead Oracle [Applications] DBA
Tieto Latvia,
Maris.Elsins@tieto.com