HANFS Part-II article published in IOUG SELECT journal Q1 2016

1. 12 ◾ Q1-16 www.ioug.orgH
Clusterware Dependencies Between
HANFS Component Resources
Listings 1, 2 and 3 show start and stop dependencies for the
HANFS component resources — more specifically, the vol1.acfs
ACFS file system, the export1 Export FS and the havip1 HAVIP,
respectively.
[root@host01 ~]# crsctl stat res ora.data.vol1.acfs -f | grep
DEPENDENCIES
START_DEPENDENCIES=hard(ora.DATA.VOL1.advm) pullup(ora.DATA.VOL1.advm)
pullup:always(ora.asm)
STOP_DEPENDENCIES=hard(intermediate:ora.DATA.VOL1.advm)
[root@host02 ~]# lsmod |grep oracle
oracleacfs 2837904 1
oracleadvm 342512 1
oracleoks 409560 2 oracleacfs,oracleadvm
oracleasm 84136 1
Listing 1: Dependencies of ora.data.vol1.acfs
[root@host01 ~]# crsctl stat res ora.export1.export -f | grep
DEPENDENCIES
START_DEPENDENCIES=hard(ora.data.vol1.acfs) attraction(ora.havip1.havip)
pullup(ora.data.vol1.acfs)
STOP_DEPENDENCIES=hard(intermediate:ora.data.vol1.acfs)
[root@host02 ~]# lsmod |grep oracle
oracleacfs 2837904 1
oracleadvm 342512 1
oracleoks 409560 2 oracleacfs,oracleadvm
oracleasm 84136 1
Listing 2: Dependencies of ora.export1.export
Oracle ACFS High Availability NFS
Services (HANFS): Part II
By Anju Garg ◾ Jim Czuprynski, Editor
In Part I of this article, we discussed various
prerequisites and features of Oracle’s ASM
Clustered File System (ACFS) with Highly Available
IP (HVIP) providing high availability for network file
systems (HANFS).
If the ACFS file system on volume
VOL1 stops running on a node,
the export1 resource must also
stop on that node.
Having successfully configured all the component resources
for an ACFS file system and successfully mounted our HANFS
exported ACFS file system via NFS client, Part II of this series
explores the start and stop dependencies among various HANFS
component resources.

2. www.ioug.org Q1-16 ◾ 13H
[root@host01 ~]# crsctl stat res ora.havip1.havip
NAME=ora.havip1.havip
TYPE=ora.havip.type
TARGET=ONLINE
STATE=ONLINE on host02
[root@host01 ~]# crsctl stat res ora.export1.export
NAME=ora.export1.export
TYPE=ora.havip1.export.type
TARGET=ONLINE
STATE=ONLINE on host02
[root@host01 ~]# crsctl stat res ora.data.vol1.acfs
NAME=ora.data.vol1.acfs
TYPE=ora.acfs.type
TARGET=ONLINE , ONLINE
STATE=ONLINE on host01, ONLINE on host02
Listing 4: Baseline Status of HANFS Resources
Considering the dependencies shown above, if the ACFS file system
on volume VOL1 is stopped on node host02 where both export1 and
havip1 are currently running, Oracle Clusterware should perform the
following actions:
◾◾ export1 should stop on host02 because of its hard stop
dependency on vol1.acfs.
◾◾ havip1 should also stop on host02 as it has a hard stop
dependency on any associated export type resources.
◾◾ Also, as vol1.acfs is running on host01, export1 should start on
host01 due to its pull up start dependency on vol1.acfs.
◾◾ Because havip1 has a pull-up start dependency on resource(s)
of type export associated with it, havip1 should start on host01
where export1 has already started.
Verifying Hard Stop Dependencies and Pull
Up Start Dependencies of HANFS Resources
To demonstrate these dependencies, let’s stop ACFS on host02, as
shown in Listing 5.
[root@host01 ~]# crsctl stop res ora.data.vol1.acfs -n host02
CRS-2529: Unable to act on 'ora.data.vol1.acfs' because that would
require stopping or relocating 'ora.export1.export', but the force option
was not specified
CRS-4000: Command Stop failed, or completed with errors.
[root@host01 ~]# crsctl stop res ora.data.vol1.acfs -n host02 -f
CRS-2673: Attempting to stop 'ora.havip1.havip' on 'host02'
CRS-2677: Stop of 'ora.havip1.havip' on 'host02' succeeded
CRS-2673: Attempting to stop 'ora.export1.export' on 'host02'
CRS-2677: Stop of 'ora.export1.export' on 'host02' succeeded
CRS-2673: Attempting to stop 'ora.data.vol1.acfs' on 'host02'
CRS-2672: Attempting to start 'ora.export1.export' on 'host01'
CRS-2676: Start of 'ora.export1.export' on 'host01' succeeded
CRS-2672: Attempting to start 'ora.havip1.havip' on 'host01'
CRS-2677: Stop of 'ora.data.vol1.acfs' on 'host02' succeeded
CRS-2676: Start of 'ora.havip1.havip' on 'host01' succeeded
[root@host01 ~]# crsctl stat res ora.havip1.havip -f | grep DEPENDENCIES
START_DEPENDENCIES=hard(ora.net1.network,uniform:type:ora.havip1.export.type)
weak(global:ora.gns)
attraction(ora.data.vol1.acfs) dispersion:active(type:ora.havip.type) pullup(ora.net1.
network)
pullup:always(type:ora.havip1.export.type)
STOP_DEPENDENCIES=hard(intermediate:ora.net1.network,uniform:intermediate:type:ora.havip1.
export.type)
oracleacfs 2837904 1
oracleadvm 342512 1
oracleoks 409560 2 oracleacfs,oracleadvm
oracleasm 84136 1
Listing 3: Dependencies of ora.havip1.havip
As these listings illustrate, these are the hard stop dependencies and
pull up start dependencies that Oracle Clusterware has established
between these HANFS resources:
◾◾ Export FS resource ora.export1.export has a hard stop
dependency on resource ora.data.vol1.acfs, so if the ACFS file
system on volume VOL1 stops running on a node, the export1
resource must also stop on that node.
◾◾ Export FS resource ora.export1.export has a pull-up start
dependency on resource ora.data.vol1.acfs. Therefore, the
export1 resource will be automatically started on a node where
the ACFS file system on volume VOL1 is running.
◾◾ HAVIP resource ora.havip1.havip has a hard stop dependency
on resource type ora.havip1.export.type. In other words, if all the
exports associated with havip1 are stopped on a server, havip1
must also stop on that server. Currently, only one export (export1)
is associated with havip1, so if export1 stops running on a node,
havip1 must also stop on that node.
◾◾ Finally, HAVIP resource ora.havip1.havip has a pull-up start
dependency on resource type ora.havip1.export.type. This
means that Oracle Clusterware will start the havip1 only on a
server where resource(s) of type export associated with it are
running. Currently, we have only one export (export1) associated
with havip1, so havip1 will be started on a node where export1
is running.
Now let’s verify these dependencies experimentally. Listing 4 shows
the current state of our three HANFS resources. It can be seen that
whereas ACFS file system on volume VOL1 is mounted on both the
nodes, havip1 and export1 are currently executing on host02.
Currently, only one export
(export1) is associated with
havip1, so if export1 stops
running on a node, havip1
must also stop on that node.

3. 14 ◾ Q1-16 www.ioug.orgH
TARGET=ONLINE , OFFLINE
STATE=ONLINE on host01, OFFLINE
Listing 8: Impact of Stopping havip1
Verifying Hard Start Dependencies of
HANFS Resources
As can be seen from Listings 2 and 3, respectively:
◾◾ Export FS resource ora.export1.export has a hard start
dependency on resource ora.data.vol1.acfs, so for the export1
resource to start on a node, the ACFS file system on volume
VOL1 must be running on that node.
◾◾ HAVIP resource ora.havip1.havip has a hard start dependency
on resource type type:ora.havip1.export.type. This means that
for resource havip1 to start on a server, at least one resource
of type export associated with it must first be running on that
server. Currently, we have only one export (export1) associated
with havip1, so for it to start on a node, export1 must be running
on that node.
Considering the start dependencies mentioned above, if havip1
is started on node host02 where neither export1 nor vol1.acfs are
currently running, export1 should automatically start on host01
because havip1 has a hard start dependency on export1. Also,
vol1.acfs should start on host02 as export1 has hard start
dependency on that resource too. We can verify this by starting
havip1 on node host02, as Listing 9 shows.
[root@host01 ~]# crsctl start res ora.havip1.havip -n host02
CRS-2805: Unable to start 'ora.havip1.havip' because it has a 'hard'
dependency on resource type 'ora.havip1.export.type' and no resource of
that type can satisfy the dependency
CRS-2525: All instances of the resource 'ora.export1.export' are already
running; relocate is not allowed because the force option was not
specified
CRS-4000: Command Start failed, or completed with errors.
[root@host01 ~]# crsctl start res ora.havip1.havip -n host02 -f
CRS-2672: Attempting to start 'ora.data.vol1.acfs' on 'host02'
CRS-2676: Start of 'ora.data.vol1.acfs' on 'host02' succeeded
CRS-2673: Attempting to stop 'ora.export1.export' on 'host01'
CRS-2677: Stop of 'ora.export1.export' on 'host01' succeeded
CRS-2672: Attempting to start 'ora.export1.export' on 'host02'
CRS-2676: Start of 'ora.export1.export' on 'host02' succeeded
CRS-2672: Attempting to start 'ora.havip1.havip' on 'host02'
CRS-2676: Start of 'ora.havip1.havip' on 'host02' succeeded
[root@host01 ~]# crsctl stat res ora.havip1.havip
NAME=ora.havip1.havip
TYPE=ora.havip.type
TARGET=ONLINE
STATE=ONLINE on host02
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
Listing 9: Starting HAVIP Resource on host02
[root@host01 ~]# crsctl stat res ora.data.vol1.acfs
NAME=ora.data.vol1.acfs
TYPE=ora.acfs.type
TARGET=ONLINE , OFFLINE
STATE=ONLINE on host01, OFFLINE
Listing 5: Stopping ACFS on host02
As Listing 6 shows, the end result is that export1 stops on host02
because of its hard stop dependency on vol1.acfs, but vol1.acfs,
which is executing on host01, causes export1 to start on host01
because export1 has a pull up start dependency on vol1.acfs.
[root@host01 ~]# crsctl stat res ora.export1.export
NAME=ora.export1.export
TYPE=ora.havip1.export.type
TARGET=ONLINE
STATE=ONLINE on host01
Listing 6: Checking Status of Export FS export01
Note that resource havip1 also stops on host02 because of its hard
stop dependency on all associated export type resources. But
because export1 has already started on host01, Oracle Clusterware
also starts havip1 on host01, because havip1 has a pull up start
dependency on resource type ora.havip1.export.type so that if any
resource of type export associated with havip1 is started on a node, it
will cause havip1 to also start on the same node, as Listing 7 shows.
[root@host01 ~]# crsctl stat res ora.havip1.havip
NAME=ora.havip1.havip
TYPE=ora.havip.type
TARGET=ONLINE
STATE=ONLINE on host01
Listing 7: Checking Status of HAVIP havip1
Now, what happens when we stop the havip1 resource in this
situation? Because neither export1 nor ACFS on VOL1 have any
stop dependencies on havip1, they will continue to execute on
their current nodes, as Listing 8 shows.
[root@host01 ~]# crsctl stop res ora.havip1.havip
CRS-2673: Attempting to stop 'ora.havip1.havip' on 'host01'
CRS-2677: Stop of 'ora.havip1.havip' on 'host01' succeeded
[root@host01 ~]# crsctl stat res ora.havip1.havip
NAME=ora.havip1.havip
TYPE=ora.havip.type
TARGET=OFFLINE
STATE=OFFLINE
[root@host01 ~]# crsctl stat res ora.export1.export
NAME=ora.export1.export
TYPE=ora.havip1.export.type
TARGET=ONLINE
STATE=ONLINE on host01
[root@host01 ~]# crsctl stat res ora.data.vol1.acfs
NAME=ora.data.vol1.acfs
TYPE=ora.acfs.type

4. www.ioug.org Q1-16 ◾ 15H
◾◾ Load balancing of HAVIPs
◾◾ Migration of HAVIPs and Exports across the Cluster
◾◾ Controlling the location of Exports in the Cluster.
◾◾ Guidelines for ensuring that HANFS provides the maximum
scalability and availability
◾◾ Potential practical use cases of this feature.
References
Oracle ACFS File System Resource Management
Oracle Database 12c SRVCTL Command Reference, August 2013
Helmut’s RAC / JEE Blog
Oracle Database 12c Clusterware Administration and Deployment
Guide, August 2013
Oracle Database 12c Automatic Storage Management Guide,
August 2013
Benefits of Oracle ACFS White Paper, January 2015
OOW 2011 presentation Managing Storage in Private Clouds with
Oracle Cloud File System
Oracle ACFS White Paper, July 2014
Oracle White Paper on “Highly Available NFS over Oracle ASM
Cluster File System (ACFS) “
Oracle Database 12c Automatic Storage Management Guide,
July 2014
Listing 10 shows the end result: export1 is automatically relocated
from host01 to host02 to honor the hard start dependency of havip1
on resource type ora.havip1.export.type.
[root@host01 ~]# crsctl stat res ora.export1.export
NAME=ora.export1.export
TYPE=ora.havip1.export.type
TARGET=ONLINE
STATE=ONLINE on host02
Listing 10: Automatic Relocation of Export Resource
Finally, Listing 11 shows that Oracle Clusterware starts the
vol1.acfs resource on host02, which was earlier running only
on host01, because export1 has a hard start dependency on
vol1.acfs resource.
[root@host01 ~]# crsctl stat res ora.data.vol1.acfs
NAME=ora.data.vol1.acfs
TYPE=ora.acfs.type
TARGET=ONLINE , ONLINE
STATE=ONLINE on host01, ONLINE on host02
Listing 11: Automatic Startup of ACFS Resource
Note that the ACFS file system will remain unmounted until the file
system resource is actually started, as Listing 9 shows.
Next Steps
In the next and final article in this series, we will explore:
◾◾ Management of HAVIP and associated Export FS resources as a
group by Oracle Clusterware
◾◾ Execution of HAVIP on the node in where the largest number of
file systems are available.
Contact
Anju Garg is an Oracle ACE associate with over 12 years of experience in the IT industry
in various roles. Since 2010, she has trained over 100 DBAs from across the world
in various core DBA technologies such as RAC, Data Guard, Performance Tuning, SQL
statement tuning, database administration and more.
Because neither export1 nor
ACFS on VOL1 have any stop
dependencies on havip1, they
will continue to execute on their
current nodes.
If havip1 is started on node
host02 where neither export1 nor
vol1.acfs are currently running,
export1 should automatically
start on host01.