Arbitration For Data Integrity in Serviceguard Clusters Manufacturing Part Number : B3936-90078 July 2007
Legal Notices The information contained in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Arbitration for Data Integrity in Serviceguard Clusters Arbitration for Data Integrity in Serviceguard Clusters Clustering is an architecture of interconnected servers that allows multiple hosts to run the same applications, permitting the individual systems to be up or down. Applications move easily between systems, accessing the same shared data from different nodes at different times. The goal is to provide high availability for the application and the data without endangering the integrity of the data.
Arbitration for Data Integrity in Serviceguard Clusters Cluster Membership Concepts Cluster Membership Concepts What is arbitration? Why is it necessary? When and how is it carried out? To answer these questions, it is necessary to explain a number of clustering concepts that are central to the processes of cluster formation and re-formation. These concepts are membership, quorum, split-brain, and tie-breaking. Membership A cluster is a networked collection of nodes.
Arbitration for Data Integrity in Serviceguard Clusters Cluster Membership Concepts When the cluster is part of a disaster tolerant solution that has nodes located in more than one data center, loss of communication can easily happen unless redundant networking is implemented with different routing for the redundant links. In all the above cases, the loss of heartbeat communication with other nodes in the cluster causes the re-formation protocol to be carried out.
Arbitration for Data Integrity in Serviceguard Clusters Cluster Membership Concepts Quorum Cluster re-formation takes place when there is some change in the cluster membership. In general, the algorithm for cluster re-formation requires the new cluster to achieve a cluster quorum of a strict majority (that is, more than 50%) of the nodes previously running.
Arbitration for Data Integrity in Serviceguard Clusters Cluster Membership Concepts Tie-Breaking Tie-breaking (arbitration) is only required when a failure could result in two equal-sized subsets of cluster nodes each trying to re-form the cluster at the same time. These competitors are essentially tied in the contest for the cluster’s identity. The tie-breaker selects a winner, and the other nodes leave the cluster.
Arbitration for Data Integrity in Serviceguard Clusters To Arbitrate or Not to Arbitrate To Arbitrate or Not to Arbitrate Arbitration is not always used to determine cluster membership. Some cluster software products rely exclusively on the use of multiple cluster membership communication links (heartbeats). These algorithms are described in the following sections.
Arbitration for Data Integrity in Serviceguard Clusters To Arbitrate or Not to Arbitrate In Figure 2, on the other hand, a single node failure would result in the loss of heartbeat communication. In the no-arbitration model, the loss of heartbeat would be interpreted by the cluster manager as a failure of node 1, and therefore the cluster could re-form with packages failing over from node 1 to node 2.
Arbitration for Data Integrity in Serviceguard Clusters To Arbitrate or Not to Arbitrate Figure 3 Multiple Paths with Different Media Note that the configuration could be expanded to include multiple disk links plus multiple LAN links, as in Figure 4. Such a configuration would require the loss of at least 4 links for the heartbeat to be lost.
Arbitration for Data Integrity in Serviceguard Clusters To Arbitrate or Not to Arbitrate No Arbitration—Risks When all is said and done, it may be very unlikely that intra-node communication would be lost in the above configurations, but it is still possible that heartbeat could disappear, with both nodes still running, and this scenario can cause data corruption.
Arbitration for Data Integrity in Serviceguard Clusters How Serviceguard Uses Arbitration How Serviceguard Uses Arbitration Serviceguard employs a lock disk, a quorum server, or arbitrator nodes to provide definitive arbitration to prevent split-brain conditions. This section describes how the software handles cluster formation and re-formation and supplies arbitration when necessary.
Arbitration for Data Integrity in Serviceguard Clusters How Serviceguard Uses Arbitration Dynamic Cluster Re-Formation A dynamic re-formation is a temporary change in cluster membership that takes place as nodes join or leave a running cluster. Re-formation differs from reconfiguration, which is a permanent modification of the configuration files. Re-formation of the cluster occurs under the following conditions (not a complete list): • An SPU or network failure was detected on an active node.
Arbitration for Data Integrity in Serviceguard Clusters How Serviceguard Uses Arbitration The cluster lock is used as a tie-breaker only for situations in which a running cluster fails and, as Serviceguard attempts to form a new cluster, the cluster is split into two sub-clusters of equal size. Each sub-cluster will attempt to acquire the cluster lock. The sub-cluster which gets the cluster lock will form the new cluster, preventing the possibility of two sub-clusters running at the same time.
Arbitration for Data Integrity in Serviceguard Clusters How Serviceguard Uses Arbitration Lock Requirements The cluster lock can be implemented either by means of a lock disk (HP-UX clusters only), a lock LUN (HP-UX and Linux clusters), or by means of a quorum server (HP-UX and Linux clusters). A one-node cluster does not require a cluster lock. A two-node cluster requires a cluster lock. In larger clusters, the cluster lock is strongly recommended.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Lock Disk as the Cluster Lock Use of a Lock Disk as the Cluster Lock The cluster lock disk (used only in HP-UX clusters) is a disk that can be written to by all members of the cluster. When a node obtains the cluster lock, this disk is marked so that other nodes will recognize the lock as “taken.” This mark will survive an off-on power cycle of the disk device unlike SCSI disk reservations.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Lock Disk as the Cluster Lock Serviceguard periodically checks the health of the lock disk and writes messages to the syslog file when a lock disk fails the health check. This file should be monitored for early detection of lock disk problems. You can choose between two lock disk options—a single or dual lock disk—based on the kind of high availability configuration you are building. A single lock disk is recommended where possible.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Lock LUN as the Cluster Lock Use of a Lock LUN as the Cluster Lock The lock LUN is similar to the HP-UX cluster lock disk but different in certain respects. As with the lock disk, a lock LUN is marked when a node obtains the cluster lock, so that other nodes will see the lock as “taken.” This mark will survive an off-on power cycle of the disk device unlike SCSI disk reservations.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Lock LUN as the Cluster Lock The operation of the lock LUN is shown in Figure 6. The node that acquires the lock (in this case node 2) continues running in the cluster. The other node halts.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Quorum Server as the Cluster Lock Use of a Quorum Server as the Cluster Lock A quorum server can be used in clusters of any size. The quorum server is an alternate form of cluster lock that uses a server program running on a separate system for tie-breaking rather than a lock disk.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Quorum Server as the Cluster Lock The quorum server runs on a Linux or HP-UX system.You can configure the quorum server as a Serviceguard package in another cluster; it must be a cluster other than the one for which it is providing quorum services. A quorum server, whether or not it is running as a package, can provide quorum services for multiple clusters.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Quorum Server as the Cluster Lock Specifying a Quorum Server If you will be using a quorum server, be sure to specify the -q qshost option with the cmquerycl command. Example: # cmquerycl -v -n lp1 -n lp2 -q lp-qs -C clus-lp.config Quorum Server Status and State The status of the quorum server can be one of the following: • Up. The quorum server is active. • Down. The quorum server is not active.
Arbitration for Data Integrity in Serviceguard Clusters Use of a Quorum Server as the Cluster Lock Viewing Quorum Server Status and State If the cluster is using a quorum server for tie-breaking services, you can use the cmviewcl command to display the server name, state, and status following the entry for each node, as in the following excerpt from the output of cmviewcl -v: CLUSTER clus-lp NODE lp1 STATUS up STATUS up Quorum Server Status: NAME STATUS lp-qs up STATE running STATE running ...
Arbitration for Data Integrity in Serviceguard Clusters Use of Arbitrator Nodes Use of Arbitrator Nodes One way to ensure that split-brain situations do not arise is to devise an architecture that makes an even partition of the cluster impossible or at least extremely unlikely. A single failure in a four-node cluster could result in two equal-sized partitions, but a single failure in a five-node cluster could not.
Arbitration for Data Integrity in Serviceguard Clusters Arbitration in Disaster-Tolerant Clusters Arbitration in Disaster-Tolerant Clusters Disaster-tolerant clusters are those which are intended to survive the loss of a data center that contains multiple resources.
Arbitration for Data Integrity in Serviceguard Clusters Arbitration in Disaster-Tolerant Clusters with shared data, there is no one disk that is actually connected to both data centers that could act as a lock disk. Arbitration in this case can be obtained by using arbitrator nodes or a quorum server.
Arbitration for Data Integrity in Serviceguard Clusters Arbitration in Disaster-Tolerant Clusters Continental Clusters There are no special arbitration requirements or configurations for the separate clusters within a continental cluster. Each cluster must provide its own arbitration separately, according to the applicable rules for a standard Serviceguard cluster. In other words, the continental cluster can employ any supported method of arbitration for its component clusters.
Arbitration for Data Integrity in Serviceguard Clusters Arbitration in Disaster-Tolerant Clusters Note that if the first lock disk is located in the first data center when the heartbeat is lost, the first data center will normally obtain the lock first because it is closest to the disk. Thus in this scenario, the first data center will re-form the cluster. 3.
Arbitration for Data Integrity in Serviceguard Clusters Arbitration in Disaster-Tolerant Clusters A dual lock disk configuration is shown in Figure 9.
Arbitration for Data Integrity in Serviceguard Clusters Summary Summary This paper has described a number of approaches to arbitration to provide safety for data in high availability clusters. There are advantages and disadvantages to each of the major approaches using a lock disk, a quorum server, or arbitrator nodes. Table 1 Comparison of Different Arbitration Methods Arbitration Mode Lock Disk Lock LUN 30 Advantages Disadvantages • Does not require a separate system.
Arbitration for Data Integrity in Serviceguard Clusters Summary Table 1 Comparison of Different Arbitration Methods (Continued) Arbitration Mode Quorum Server Arbitrator Nodes Advantages Disadvantages • Can be used with clusters of any size. • Requires a separate system not part of the cluster. • Can serve up to 50 clusters/100 nodes. • Only one IP address is used for quorum server. • Can be used HP-UX and Linux clusters at the same time.
Arbitration for Data Integrity in Serviceguard Clusters Summary 32
