Test

Oracle9i Application Server:
mod_oc4j Technical Overview

An Oracle White Paper
May 2002

Oracle9i Application Server: mod_oc4j
Technical Overview

Introduction to Oracle 9iAS and Clustering .......................................... 3
Audience............................................................................................ 3
Oracle 9iAS Clustering Framework ................................................... 3
The Terminology ........................................................................... 3
The Clustering Framework ............................................................ 3
OPMN – Oracle Process Management and Notification Service ...... 4
Process Management ..................................................................... 4
Process Notification ...................................................................... 4
OPMN Configuration Files............................................................ 5
DCMCTL Commands.................................................................... 5
mod_oc4j: An Overview ................................................................... 7
Identifying the requests to Route................................................... 7
Identifying which OC4J to Route the request to............................ 7
Communicating with OC4J............................................................ 7
Setting the Cookie ......................................................................... 7
Transparent Updates to mod_oc4j Routing Table ......................... 8
Clusters, Instances, and Islands ..................................................... 8
Mod_oc4j Configuration................................................................ 8
Scenario A: Routing Between Instances in Same Farm...................... 9
Routing Configuration ................................................................... 9
Extending to multiple instances or multiple clusters ................... 11
Associated Best practices ............................................................ 11
Caveats ........................................................................................ 11
Scenario A: Summary................................................................... 12
Scenario B: Configuring Instances in different farms into a cluster.. 12
Caveats ........................................................................................ 13
Scenario B: Summary ................................................................... 13
Scenario C: Routing (or clustering) Instances Across Firewalls ....... 14
Firewall Configuration for mod_oc4j........................................... 14
Firewall Configuration for OPMN............................................... 15
Firewall Configuration for DCM.................................................. 15
Finding the Ports Used ................................................................ 15
Scenario C: Summary ................................................................... 16
Summary .............................................................................................. 16

Oracle9i Application Server Release 2:
mod_oc4j: Technical Overview Page 2

INTRODUCTION TO ORACLE 9IAS AND CLUSTERING
In 9iAS Release 2, several new components have been introduced with respect to
clustering. This paper focuses on one of the components – mod_oc4j.

Audience
This paper is geared towards the engineer or support analyst actually modifying
configuration files for different implementation scenarios. For a higher-level view
appropriate for managers and architects please see the paper: “Oracle 9iAS
Release 2: Availability, Scalability, and Manageability of 9iAS Clusters.”

Oracle 9iAS Clustering Framework

FW/LB/Web Cache The Terminology

Oracle9iAS Instance is a collection of component processes, spawned from the
same ‘Oracle Home’.
A Cluster is a collection of Oracle9iAS instances, all configured identically.
OHS
OHS OHS
A Node is a server machine. It could theoretically run multiple Oracle9iAS
OC4J
instances that may or may not be part of the same cluster.
OC4J
OC4J OC4J

A Component Instance is a set of identically configured component processes within an
OC4J OC4J
OC4J
OC4J OC4J
Oracle9iAS instance.
An island is a group of processes in the cluster, which replicate session state
amongst each other. This, in conjunction with mod_oc4j, enables transparent
failover for the client.

The Clustering Framework

The different components that constitute the clustering framework are:
FW/LB/WC
Oracle Process Manager and Notification Server (OPMN) manages
(starts/stops/monitors) the processes within a 9iAS instance and channels
notifications between the different processes.
http

Distributed Configuration Management (DCM) distributes the configuration
9iAS Instance
information across the components in the cluster, and also saves it to the
events
OHS OPMN1
repository.
3
events
mod_oc4j
Infrastructure Repository contains all the relevant configuration information. All
DCM2
ajp
instances that have their configuration in the same repository are said to be
events
members of a farm. The repository can either be a database or be file based.
Mod_oc4j plugs into Oracle HTTP Server (OHS) and routes to all OC4J
OC4J
OC4J
instances using AJP. Its routing table is continuously kept updated by OPMN.
OC4J

Thus OPMN provides fault tolerance, DCM and infrastructure repository provide
distributed deployment, and mod_oc4j provides load balancing. Now, we will cover


each of these components in more detail.

OPMN – Oracle Process Management and Notification Service

Process Management

OPMN can manage (start/stop/monitor) processes it knows about – including
external ones, that may not related to 9iAS in any way.
There is one OPMN process per 9iAS instance.
http

OPMN uses a combination of ping, reverse ping, and
9iAS Cluster
FW/LB/WC
system specific process id information to monitor a process.
9iAS Instance 9iAS Instance
If it determines it is dead, it restarts the process. External
OHS OHS
OPMN OPMN
OPMN 3

processes that may not have been instrumented for a
reverse ping (heart-beat), are also adequately monitored
DCM DCM
DCM 2

using the ping and pid mechanisms.
1

In addition, when an instrumented process (specifically,
Join
OC4J OC4J
OC4J
OC4J
OC4J
OC4J
OC4J OC4J
OHS and OC4J) is down, internal bookkeeping and
notification ensures a cluster wide update of the state.
Thus, mod_oc4j always contains updated routing information.
There is also an OPMN buddy process that monitors and restarts OPMN if
required. The buddy process itself is not monitored.

Process Notification

OPMN also provides a simple, lightweight publish/subscribe notification system. It
is leveraged by other components (such as DCM) to communicate with each
other across instances in the cluster.
All notification is point to point, using HTTP. By default it is sent to all instances
in the farm. On receiving a notification, OPMN filters out the ones not targeted
for that instance. In a section below, we will cover how to tweak the default event
notification destinations.

Internal Mechanisms for Notification

OPMN uses two distinct ports for notification purposes:
Intra instance notifications – All processes within a 9iAS instance communicate with
OPMN on the local node at this port. Typically, this node will not need to be
manually configured.
Inter instance notifications – The OPMN on different instances work together to
keep their status in synch. By default, these notifications are sent to other
instances in the same farm. OPMN, using DCM, is able to determine the different
instances and their corresponding host/port configurations. On receiving a
notification, OPMN determines if there is a local subscriber for the event – and if
not, it drops the notification.


OPMN Configuration Files

OPMN uses two configuration files, located in the
ORACLE_HOME/opmn/conf/ directory. DCM manages these files and may
overwrite any manual updates of these files during the next DCM event. Hence, it
is recommended that the files should not be updated manually. A utility - dcmctl
(or OEM) - should be used to modify these files.
This configuration information will get used to create different configuration
scenarios in the next few sections of this paper.
In situations where the files are manually modified (ex. through an editor), a dcmctl
updateConfig command should be used so that the file gets saved in the
infrastructure repository.
Ons.conf – This file contains information on which instances should OPMN
communicate with. By default, its values are filled by DCM with the host and port
information of other instances in the farm. To get this information about an
instance, you can issue the command:

cd <ORACLE_HOME_2>/dcm/bin
dcmctl getOPMNPort

The output of the above will be of the format – Ipaddress:PortNumber – example:

127.2.141.148:6200

With this information, you can (if required) modify an ons.conf of another instance,
such that now the two OPMN, even if they are not in the same farm, will
communicate with each other:

cd <ORACLE_HOME_1>/dcm/bin
dcmctl addOPMNLink IP-address:Port

Example:

dcmctl addOPMNLink 127.2.141.148:6200

The changes have to be done in the ons.conf files on both the machines, so that
both machines know of each other.
Opmn.xml – OPMN uses this file to know which and how many processes (ex.
Oc4j, OHS, etc.) to start. OPMN also manages these processes. DCM manages
this configuration file too.

DCMCTL Commands

The dcmctl commands are documented in detail in the 9iAS Administrator’s Guide
(Part No. A92171-01, Appendix F). This document assumes you have access to
that Appendix. Only the relevant dcmctl commands are mentioned in this
document.


mod_oc4j: An Overview
mod_oc4j sits within Oracle HTTP Server and (i) identifies the requests it needs to
act on, (ii) determines which OC4J to route those requests to, and (iii) communicates
with that process.

Identifying the requests to Route

Every J2EE based (web) application, when deployed, needs to be associated with a
root context. This root context (i.e. URL prefix) acts as the identifier of requests that
need to be handled by mod_oc4j.

Identifying which OC4J to Route the request to

An island is a set of OC4J processes within the cluster that replicate session
information among each other. The J2EE section of the paper goes in more
depth on the island concept.
For the purposes of this routing discussion, it is important to understand that once
a (stateful) request is routed to any member of an island, it cannot be later failed-
over to an OC4J process outside the island.
1. The mod_oc4j routing algorithm for stateless requests is simple round
robin. If this is a new request (with no valid routing cookie), it picks the
next process from the list as the destination.
2. If an attempt to send the request to that process fails, mod_oc4j picks
the next process from the list, until all processes in that instance are
exhausted. And if so, it returns failure to the client.
FW/LB/WC
3. If it is a request with a valid routing cookie (see discussion on “setting the
cookie” below), the cookie already identifies the process to route to
uniquely. This target process is changed only if the target OC4J process is
http
dead. In that case, mod_oc4j picks the next process from the same island
9iAS Instance
and routes to it.
OHS OPMN
OPMN
Communicating with OC4J
DCM Mod_oc4j now extracts some relevant parameters (ex. SSL info, certain
environment variables etc.) and forwards them to OC4J, using AJP13 protocol.
Mod_oc4j analyzes the response from OC4J and takes appropriate actions - ex. if
OC4J
a single sign on redirect is required.
OC4J
OC4J

Setting the Cookie

If the request to OC4J sets a cookie (i.e. a stateful request), mod_oc4j adds
jvmroute information to JSESSIONID cookie. jvmroute cookie contains the
following information in an obfuscated format:


oc4j_host_name#oc4j_port_number#ias_cluster_name#ias_ins
tance_name#opmn_generated_id#opmn_group_id#oc4j_instance
_name#oc4j_island_id

This format is documented here to (i) better understand the information available
from the cookie for routing purposes, and (ii) assist in determining the length of
the cookie (for firewall, web-cache or other configurations). This format is subject
to change.

Transparent Updates to mod_oc4j Routing Table

By default, OPMN processes on all 9iAS instances in the farm notify each other
of the up/down status of OC4J within their instance. In turn, every OPMN also
notifies its local mod_oc4j of changes in the OC4J status on all machines within
the cluster. This allows mod_oc4j to keep its routing table updated, without any
intervention from an administrator.

Clusters, Instances, and Islands

This figure contains one cluster – C1 – which in turn contains
two 9iAS instances. These could be installed on the same
machine (different Oracle Homes), or, on different machines.
The two 9iAS instances – I1 and I2 – in turn contain an OHS
instance (unnamed), and two Oc4J instances, O1 and O2.
Again, an OC4J instance is a set of processes with identical
configuration.
In this example, within an Oc4J instances there is just one
island – Is1 in O1, and Is2 in O2. Island Is1 contains one
process (#1) on each 9iAS instances, while Is2 contains a total
of four processes (#2 and #3) – two on each 9iAS instance.
Typically, there will be multiple islands within an OC4J
instance. The session state is replicated across all processes
within an island.
With these details, we now get into a little more of the mod_oc4j configuration.

Mod_oc4j Configuration

All mod_oc4j related configuration information is kept in
<ORACLE_HOME>/Apache/Apache/conf/mod_oc4j.conf
OHS uses an Oc4jMount command to map the root context to the OC4J instance
into which the application was deployed:

Oc4jMount /j2ee/* home
Oc4jMount /ojspdemos OC4J_Demos
Mod_oc4j Directive
Oc4jMount /ojspdemos/* OC4J_Demos
Root Context


Oc4jMount /cabo home
OC4J Instance
Oc4jMount /cabo/* home

This causes OHS to route a request that contains the regular expression (such as
/j2ee/*) to the home OC4J instance. By default, the instance or cluster
specification is omitted, and it is assumed to point to the local OC4J 9iAS instance
(or cluster).
To route to an OC4J instance (say O3) in a different 9iAS instance (say I3) or
cluster (C3), two keywords – instance and cluster – are used as follows:

Oc4jMount /app/* instance://I3:O3
Keywords
Oc4jMount /app1/* cluster://C3:O3

This document is focused on the specific scenarios that can be created to leverage
the power in configuration file.

Scenario A: Routing Between Instances in Same Farm
In many cases it may be desirable to have a single OHS in an instance
talk to OC4J processes in other instances, without clustering the two
9iAS instances together.
The instances might have different applications deployed and totally
different root context mappings.
This scenario may be valuable when deploying multiple non-clustered
9iAS instances on the same machine without having multiple OHS
listeners.

Routing Configuration

For this configuration it is necessary to edit the Oc4jMount directives
in the 9iAS instance I1 to point to the OC4J instances in 9iAS instance
I2 (see the figure). Let us further assume that the OC4J on I2
instance are running applications under the /i2app/ root context on
Oc4J instance O4. OH_1 is used to refer to the value of environment
variable ORACLE_HOME. The steps to achieve this routing, at a
coarse level are:
1. Determine the instance and component names in I2, and
2. Inform mod_oc4j in I1 of those instance and component names

Determining instance and component names in I2

These can be determined by the dcmctl commands below, or, through OEM.

On which Sample Result
What steps to perform
Instance


I2 Change to the right directory.
cd <OH_1>/dcm/bin

I2 dcmctl Ias-machine.oracle.com.
This is referred to as I2 in this
whichInstance
example.

I2 This will list all the components on
Dcmctl
the 9iAS instance. For example:
listComponents
Home
OC4J_Demos
…

We will refer to these component
instances as O1, O2, O3, O4 etc.

Informing mod_oc4j in I1

Instance

cd <OH_1>/
Apache/Apache/conf

I1 Edit mod_oc4j.conf to include the line
Oc4jMount /i2app/* instance://I2:O4
This will cause mod_oc4j in I1 to route to
I1
the O4 Oc4J instance in 9iAS instance I2.
Save the file.
I1
dcmctl
I1 This will cause the configuration to
updateConfig
be saved in the repository
dcmctl restart
I1 To restart OHS to pick the
configuration. It is possible to start
only OHS. To keep the commands
simple, we are restarting the
instance.

Extending the setup to a cluster

If I2 was part of a cluster, say C2, and you wanted to route to all processes that
mapped to the O4 instance in that cluster, you could instead edit mod_oc4j.conf
on I1 with the following:
Oc4jMount /i2app/* cluster://C2:O4


Extending to multiple instances or multiple clusters

If there were additional clusters (say C3) or 9iAS instances (say I3) with OC4J
instance (O5), it could be added to the routing as follows:
Oc4jMount /i2app/* instance://I2:O4, I3:O5

Or,

Oc4jMount /i2app/* cluster://C2:O4, C3:O5

This configuration automatically makes these instances fail-over candidates for
each other for stateless servlets (or JSP).
However, if fail-over of stateful servlets was an added requirement, you will need
to use OEM 1 to create islands with the same name in both the instances. This
allows mod_oc4j to fail-over requests to other processes within the same named
island but on different instances.
The different instances to which mod_oc4j is going to route have to have identical
configuration. This has to be ensured by configuring each instance separately –
either through OEM or manually. DCM cannot be leveraged to copy
configuration.

Associated Best practices

• Always use OEM to edit mod_oc4j.conf, since it will not only edit the file but
also run dcmctl updateConfig – so that the change in configuration is committed
to a local repository.
• If editing by the file by hand, you should run the command dcmctl updateConfig
after making the change.

Caveats

• All instances have to be in the same Farm – This scenario assumes that the
clusters and instances being used here are part of the same farm – i.e.
they share the same repository for their configuration. If this is not the
case, and the instances have their configurations in different repositories,
additional configuration steps are required – as described in Scenario B.
• Patch 2367550 – This patch is required for correct failover of stateful
requests across instances that are not in the same cluster. This patch is
available at MetaLink, and is included for releases beyond 9.0.2.0.

It is also possible to set islands up manually, but it is recommended to use OEM.
1


Scenario A: Summary

This configuration allows an OHS on one instance to communicate to OC4J
instances across several 9iAS instances or clusters – even if the OHS and OC4J
are not in the same cluster.
It allows load balancing and fail-over, but does not allow distributed configuration.

Scenario B: Configuring Instances in different farms into a cluster
In this scenario we want to allow routing between instances even when they are
not in the same farm. And, by ensuring all instances have identical
configuration, we effectively convert this into a clustered setup, but
without any distributed configuration management.
The different farms may both use a file-based repository, or, both
may use a database-based repository, or, they may be a mix
between the file based and database-based repositories. This setup
can also be extended to support several OHS and OC4J instances.
At a coarse level, the steps are:
1. Determine the different instance names, port numbers
etc., so you can do the manual setup.
2. Update the ons.conf of all instances so they have the
information about the other instances
3. Update mod_oc4j.conf routing, to send the requests to the
OC4J in other instances.

Determining host and port for an OPMN on an instance

Instance

cd <OH_1>/dcm/bin

I1 127.2.141.148:6200. We will refer to
dcmctl
this as “IP of I1:Port of I1”
getOPMNPort

cd <OH_2>/dcm/bin

I2 127.2.142.331:6200 We will refer to
dcmctl
this as “IP of I2:Port of I2”
getOPMNPort

Setting host and port for all ons.conf

On which Sample Command / Result
Instance

cd <OH_1>/dcm/bin


I1 Example Command:
dcmctl addOPMNLink
dcmctl addOPMNLink
<IP of I2>:<Port of
127.2.142.331:6200
I2>
cd <OH_2>/dcm/bin

I2 dcmctl addOPMNLink
dcmctl addOPMNLink
127.2.141.148:6200
<IP of I1>:<Port of
I1>

At this stage, both the instances are set to receive OPMN notifications from each
other. However, we still need to tell mod_oc4j to route requests from I1 to I2
and I2 to I1, just as any standard 9iAS cluster setup would.

Mod_oc4j Routing

This is identical to the description in Scenario A (see page 8). In Scenario A, we
described routing from I1 to I2, and not vice versa - the only difference for
scenario B is that both instances – I1 and I2 – have to be configured to route to
one another.

Extending to Clusters

This refers to routing from 9iAS instances in a cluster – say C1 - to all instances
in a cluster – say C2, even when C1 and C2 are on different farms.
In this case, the IP and port of each instance in the target cluster (C2) have to be
configured in ons.conf for all 9iAS instances in C1 as described on page 12. To
complete the configuration, perform the configuration as described in Scenario A
(Page 9: Extending to multiple instances or multiple clusters).

Caveats

• Identical Configuration – It has to be ensured that all the instances that are glued
in this fashion have identical configuration. DCM cannot be leveraged to do
this, since the instances are not in the same farm. However, DCM or OEM
may be used to deploy the same set of applications to each 9iAS instance.

Scenario B: Summary

In this scenario, we clustered instances together even if they were on different
farms – including the cases when each instance was just using a file based
repository.


Scenario C: Routing (or clustering) Instances Across Firewalls
In this scenario, we will have two
clusters – one composed mainly of
OHS instance(s), and the other of
OC4J instance(s). These may or
may not be in the same farm – i.e.
either of Scenario A or Scenario B
may apply. These clusters will have
a firewall in between – so the focus
is to understand what ports and
protocols are in use and how these
can be manually configured. This
will help us understand which ports
should be opened on the firewall
for the system to work correctly.

Firewall Configuration for mod_oc4j

Mod_oc4j uses AJP to talk to the OC4J processes. Each OC4J process needs one
port for AJP13 communication with mod_oc4j.
These ports are specified in the opmn.xml file, which is located in
ORACLE_HOME/opmn/conf/ directory.
A section of the file is cut-and-paste here for analysis purposes.

<oc4j instanceName=quot;OC4J_Demosquot; gid=quot;OC4J_Demosquot;>
<config-file
path=quot;/ias_m17g/j2ee/OC4J_Demos/config/server.xmlquot; />
<java-option value=quot;-Xmx512Mquot; />
<oc4j-option value=quot;-userThreads -propertiesquot; />
<port ajp=quot;3001-3100quot; rmi=quot;3101-3200quot; jms=quot;3201-3300quot; />
<environment>
<prop name=quot;%LIB_PATH_ENV%quot;
value=quot;%LIB_PATH_VALUE%quot; />
</environment>
</oc4j>
The port tag lists all the port in use. The ajp attribute lists all the ports to be used
for ajp communication by this particular OC4J instance (OC4J_Demos in this
example).
There are two ways to specify the port:
• A range, as is done in the example file above. In this case, at startup time,
OPMN assigns OC4J one of the available ports from the range. The firewall
then needs to be configured to open all the ports in this range for this
particular OC4J instance. The range needs to include at least as many ports as
the number of processes in this particular OC4J instance.


• Specific Ports (comma separated list) – If it is undesirable to specify a range, an
exact number of ports may be specified, such as: port1, port2, port3. OPMN
picks a port from this list when starting OC4J. This allows a more stringent
firewall configuration, since only these specific ports have to be opened for
communication from the OHS machines to the OC4J machines.

Firewall Configuration for OPMN

OPMN, as discussed earlier, uses http to communicate with other OPMN. In the
setup involving instances across the firewalls, ons.conf will need to be configured (as
discussed in scenario B) to point to these different instances.
To correctly extend the Scenario B configuration of ons.conf to Scenario C - the
firewall will need to be opened up for http communication on the OPMN ports.
Just to repeat, the specific ports used by different OPMN processes can be found
using the dcmctl command: dcmctl getOPMNPort.
Since OPMN communication is bi-directional, the correct firewall configuration
will need to allow http communication from the OHS machines to the OPMN
port on OC4J machines and from OC4J machines to the OPMN port on OHS
machines.

Firewall Configuration for DCM

DCM uses JDBC to talk to the backend (Oracle based) repository. This requires
configuring the firewall to allow the requisite ports (ex. 1521) so DCM can
communicate with the repository.
Also DCM bootstraps with information from Oracle Internet Directory (OID)
and hence needs access to an LDAP port too. This port will thus have to be
opened through firewall for DCM to function.
If it is not desirable to open ports to the
database, one possibility is to run OHS
instances (or cluster) with a file based
repository (see Scenario B on how to do
this). Thus, DCM on only the OC4J
instances will communicate with the
infrastructure repository.

Finding the Ports Used

At installation time, Oracle Installer picks
the available ports and assigns them to the
relevant processes. This eliminates port
conflicts and startup failures. Moreover,
to make it easier to determine these port
assignments, the default home page (and
OEM) displays the ports used by the


different components.
A sample screen of these ports is shown here.

Scenario C: Summary

This scenario covered the case when an OHS cluster is on one side of the
firewall, routing to an OC4J cluster on the other side – a common deployment
configuration.

SUMMARY
This paper introduced mod_oc4j, the new routing component and described
several deployment configurations where the component’s powerful configuration
capabilities could be used.
Several other permutations of the deployment setups are possible – but we
described three key ones, which can be used as building blocks for other kinds of
setup.


Oracle9i Application Server: Clustering Core Components
May 2002
Author: Mukul Paithane
Contributing Authors: Chet Fryjoff, John Lang, Wayne Milsted, Sanket Atal, Mark Nelson, Jerry Bortvedt, Jeremy Lizt, Juliana Button
Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.

Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
www.oracle.com

Oracle Corporation provides the software
that powers the internet.

Oracle is a registered trademark of Oracle Corporation. Various
product and service names referenced herein may be trademarks
of Oracle Corporation. All other product and service names
mentioned may be trademarks of their respective owners.

Copyright © 2002 Oracle Corporation
All rights reserved.


Test

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (10)

Similar a Test

Similar a Test (20)

Último

Último (20)

Test