Architecture considerations and best practices for architecting an Oracle RAC solution with Serviceguard and SGeRAC
9
RAC also has its own group membership. When you configure RAC, you choose all nodes or a subset
of nodes in the cluster to run RAC instances for a specific RAC database. By default, RAC Instance
Membership Recovery (IMR) is set to 17 minutes. This means that when a critical RAC resource fails,
RAC will wait for 17 minutes before performing its own recovery. This time delay is to ensure the
underlying clustering technology (either Serviceguard or Oracle Clusterware) has enough time to
perform cluster recovery before RAC handles it.
Now let’s look at critical resources for each component and the impact to the cluster when they fail.
Serviceguard:
• Cluster heartbeat network
If all heartbeat networks fail for a cluster node, the impacted node will be evicted from the cluster
via a system Transfer of Control (TOC). The time for node eviction depends on the cluster’s
configurable parameter MEMBER_TIMEOUT
5
value. The default value is 14 seconds, resulting in a
failover time of 18 seconds. The minimum MEMBER_TIMEOUT value is 3 seconds, which would
result in a fastest failover time of 4 seconds. When the node is evicted, the remaining nodes reform
the cluster and the new cluster membership is provided to Oracle Clusterware, which is then
propagated to RAC as appropriate. In the event of a failure of a heartbeat network on all nodes,
SGeRAC will continue to run RAC on a single-cluster node. Please be aware that the Cluster
Interconnect Subnet Monitoring
6
• Storage monitoring
feature, available with SGeRAC A.11.18 and later, must be
configured to enable this functionality. If this feature is not enabled, the entire cluster will fail.
Beginning with Serviceguard A.11.20, the Serviceguard Volume Monitor provides package-
configured monitoring for CVM/VxVM Volumes and LVM LVs. The volume monitor replaces the
CVM/VxVM Volume Monitor, which was introduced with Serviceguard A.11.18. This is the
preferred solution, over the deprecated EMS HA Monitors, and is recommended for use monitoring
Oracle Clusterware storage, root volumes, and data volumes, for fastest detection and recovery
from storage array/enclosure or complete failures of redundant storage links.
Oracle Clusterware:
• CSS heartbeat network
If all CSS heartbeat networks for a node fail and if Serviceguard doesn’t monitor the CSS heartbeat
network, the impacted node will be evicted from the cluster by CSS via a TOC.
7
• Voting disk
The time for the
node eviction depends on the CSS MISSCOUNT value; the default value in a SGeRAC environment
is 600 seconds. When the node executes a TOC, the node has crashed from Serviceguard’s
perspective and Serviceguard will reform the cluster. The new cluster membership is then provided
to Oracle Clusterware via Serviceguard.
If a node loses access to the voting disk, CSS evicts the impacted node from the cluster by causing
a TOC on the node. The time for the node’s eviction depends on the disk heartbeat time out value;
the default value is 200 seconds.
8
When the node executes a TOC, the node has crashed from
Serviceguard’s perspective and Serviceguard will reform the cluster. The new cluster membership is
then provided to Oracle Clusterware via Serviceguard.
5
Since Serviceguard A.11.19, MEMBER_TIMEOUT replaced two attributes: NODE_TIMEOUT and HEARTBEAT_INTERVAL. For further information on
optimizing MEMBER_TIMEOUT, please see Optimizing Serviceguard failover time,
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02056218/c02056218.pdf, or for Serviceguard A.11.18, see Optimizing
failover time in a Serviceguard environment,
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02054506/c02054506.pdf
6
For more information on SGeRAC Cluster Interconnect Subnet Monitoring, see Using Serviceguard Extension for RAC
7
Prior to patch 2 of release 2 (10.2.0.2), CSS used reboot to evict the node
8
Prior to patch 2 of release 2 (10.2.0.2), the disk I/O timeout value was the same as CSS MISSCOUNT value