Managing HP Serviceguard A.12.00.
© Copyright 2001, 2014 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Printing History ..........................................................................................15 Preface......................................................................................................17 1 Serviceguard for Linux at a Glance.............................................................19 1.1 What is Serviceguard for Linux? .........................................................................................19 1.1.1 Failover.......................................
3.1.2.3 WBEM Query....................................................................................................35 3.1.2.4 WBEM Indications..............................................................................................36 3.2 How the Cluster Manager Works .......................................................................................36 3.2.1 Configuration of the Cluster ........................................................................................36 3.2.
3.5.11 VLAN Configurations................................................................................................68 3.5.11.1 What is VLAN?.................................................................................................68 3.5.11.2 Support for Linux VLAN......................................................................................68 3.5.11.3 Configuration Restrictions....................................................................................68 3.5.11.
4.8.1.4 cmpreparecl......................................................................................................85 4.8.2 Heartbeat Subnet and Cluster Re-formation Time ..........................................................86 4.8.3 About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode.....................86 4.8.3.1 What Is IPv4–only Mode?...................................................................................87 4.8.3.2 What Is IPv6-Only Mode?...........................
4.9.10.8.6 Using Load Sensitive Package Placement....................................................126 4.9.10.8.7 Starting Load Sensitive Package Placement.................................................127 4.9.10.9 Examples......................................................................................................127 4.9.11 About External Scripts.............................................................................................135 4.9.11.
5.1.13.4.1 Propagation of Disk Groups in VxVM..........................................................162 5.1.13.5 Creating Logical Volumes.................................................................................162 5.1.13.6 Creating File Systems.......................................................................................163 5.1.13.7 Monitoring VxVM Disks....................................................................................163 5.1.13.8 Deporting Disk Groups..................
6.1.4.7 auto_run.........................................................................................................186 6.1.4.8 node_fail_fast_enabled.....................................................................................186 6.1.4.9 run_script_timeout.............................................................................................187 6.1.4.10 halt_script_timeout...........................................................................................187 6.1.4.
6.2.2 cmmakepkg Examples.............................................................................................201 6.2.3 Next Step...............................................................................................................202 6.3 Editing the Configuration File............................................................................................202 6.4 Adding or Removing a Module from an Existing Package.....................................................205 6.
7.4.2.1 Halting a Package that Has Dependencies...........................................................230 7.4.2.2 Handling Failures During Package Halt...............................................................230 7.4.3 Moving a Failover Package ......................................................................................231 7.4.4 Changing Package Switching Behavior ......................................................................232 7.5 Maintaining a Package: Maintenance Mode.........
8.1.2.4 Listing the Existing Session.................................................................................259 8.1.3 Managing the Cluster...............................................................................................259 8.1.4 Managing the Nodes in the Simulated Cluster.............................................................259 8.2 Simulation Scenarios for the Package................................................................................260 8.2.
10.8.7.1 Authorization File Problems..............................................................................280 10.8.7.2 Timeout Problems...........................................................................................280 10.8.7.3 Messages.....................................................................................................281 10.8.8 Lock LUN Messages...............................................................................................281 10.
C Blank Planning Worksheets ....................................................................299 C.1 Hardware Worksheet .....................................................................................................299 C.2 Power Supply Worksheet ................................................................................................299 C.3 Quorum Server Worksheet .............................................................................................300 C.
Printing History Printing Date Part Number Edition November 2001 B9903-90005 First November 2002 B9903-90012 First December 2002 B9903-90012 Second November 2003 B9903-90033 Third February 2005 B9903-90043 Fourth June 2005 B9903-90046 Fifth August 2006 B9903-90050 Sixth July 2007 B9903-90054 Seventh March 2008 B9903-90060 Eighth April 2009 B9903-90068 Ninth July 2009 B9903-90073 Tenth June 2012 701460-001 NA December 2012 701460-002 NA May 2013 701460-003 NA Febr
Preface This guide describes how to configure and manage Serviceguard for Linux on HP ProLiant server under the Linux operating system. It is intended for experienced Linux system administrators. (For Linux system administration tasks that are not specific to Serviceguard, use the system administration documentation and manpages for your distribution of Linux.) NOTE: Starting Serviceguard A.12.00.00, legacy packages are obsolete and only modular packages are supported.
• HP Serviceguard Extended Distance Cluster for Linux A.12.00.00 Deployment Guide • Clusters for High Availability: a Primer of HP Solutions. Second Edition. HP Press, 2001 (ISBN 0-13-089355-2) Information about supported configurations is in the HP Serviceguard for Linux Configuration Guide. For updated information on supported hardware and Linux distributions refer to the HP Serviceguard for Linux Certification Matrix. Both documents are available at: http://www.hp.
1 Serviceguard for Linux at a Glance This chapter introduces Serviceguard for Linux and shows where to find different kinds of information in this book. It includes the following topics: • What is Serviceguard for Linux? (page 19) • Using Serviceguard for Configuring in an Extended Distance Cluster Environment (page 21) • Using Serviceguard Manager (page 22) • Configuration Roadmap (page 22) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4 (page 77).
Figure 1 Typical Cluster Configuration In the figure, node 1 (one of two SPU's) is running package A, and node 2 is running package B. Each package has a separate group of disks associated with it, containing data needed by the package's applications, and a copy of the data. Note that both nodes are physically connected to disk arrays. However, only one node at a time may access the data for a given group of disks.
Figure 2 Typical Cluster After Failover After this transfer, the package typically remains on the adoptive node as long the adoptive node continues running. If you wish, however, you can configure the package to return to its primary node as soon as the primary node comes back online. Alternatively, you may manually transfer control of the package back to the primary node at the appropriate time. Figure 2 (page 21) does not show the power connections to the cluster, but these are important as well.
1.3 Using Serviceguard Manager Serviceguard Serviceguard Serviceguard Serviceguard Manager is a web-based GUI management application for Serviceguard clusters. Manager is used to configure, monitor, and administer Serviceguard clusters, and Disaster Recovery clusters like Extended Distance Cluster and Metro cluster. Manager protects critical applications from planned and unplanned downtime. 1.3.1 Launching Serviceguard Manager Serviceguard Manager runs on 5511 and 5522 ports.
Figure 3 Tasks in Configuring a Serviceguard Cluster HP recommends that you gather all the data that is needed for configuration before you start. See Chapter 4 (page 77) for tips on gathering data. 1.
2 Understanding Hardware Configurations for Serviceguard for Linux This chapter gives a broad overview of how the server hardware components operate with Serviceguard for Linux. The following topics are presented: • Redundant Cluster Components • Redundant Network Components (page 25) • Redundant Disk Storage (page 29) • Redundant Power Supplies (page 30) Refer to the next chapter for information about Serviceguard software components. 2.
2.2.1 Rules and Restrictions • A single subnet cannot be configured on different network interfaces (NICs) on the same node. • In the case of subnets that can be used for communication between cluster nodes, the same network interface must not be used to route more than one subnet configured on the same node. • For IPv4 subnets, Serviceguard does not support different subnets on the same LAN interface. ◦ For IPv6, Serviceguard supports up to two subnets per LAN interface (site-local and global).
Figure 4 Redundant LANs In Linux configurations, the use of symmetrical LAN configurations is strongly recommended, with the use of redundant hubs or switches to connect Ethernet segments. The software bonding configuration should be identical on each node, with the active interfaces connected to the same hub or switch. 2.2.3 Cross-Subnet Configurations As of Serviceguard A.11.
failing over across subnets can take longer than failing over on the same subnet. List the nodes in order of preference instead of using the wildcard. • You should configure IP monitoring for each subnet; see “Monitoring LAN Interfaces and Detecting Failure: IP Level” (page 64). 2.2.3.2 Restrictions The following restrictions apply: • All nodes in the cluster must belong to the same network domain (that is, the domain portion of the fully-qualified domain name must be the same.
IMPORTANT: Although cross-subnet topology can be implemented on a single site, it is most commonly used by extended-distance clusters and Metrocluster. For more information about such clusters, see the following documents at http://www.hp.
The following restrictions are applicable when iSCSI LUNs are used as a shared storage: • An iSCSI storage device does not support configuring a lock LUN. • Hardware initiator models does not support iSCSI storage. • An iSCSI storage device that are exposed using SCSI targets is not supported. 2.3.2 Disk Monitoring You can configure monitoring for disks and configure packages to be dependent on the monitor.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
• cmlogd—cluster system log daemon • cmdisklockd—cluster lock LUN daemon • cmresourced—Serviceguard Generic Resource Assistant daemon • cmprd—Persistent Reservation daemon • cmserviced—Service Assistant daemon • qs—Quorum Server daemon • cmlockd—utility daemon • cmsnmpd—cluster SNMP subagent (optionally running) • cmwbemd—WBEM daemon • cmproxyd—proxy daemon Each of these daemons logs to the Linux system logging files.
3.1.1.3 Network Manager Daemon: cmnetd This daemon monitors the health of cluster networks. It also handles the addition and deletion of relocatable package IPs, for both IPv4 and IPv6 addresses. 3.1.1.4 Log Daemon: cmlogd cmlogd is used by cmcld to write messages to the system log file. Any message written to the system log by cmcld it written through cmlogd. This is to prevent any delays in writing to syslog from impacting the timing of cmcld. The path for this daemon is $SGLBIN/cmlogd. 3.1.1.
is killed. It can also be configured as a Serviceguard package in a cluster other than the one(s) it serves; see Figure 9 (page 40). All members of the cluster initiate and maintain a connection to the quorum server; if it dies, the Serviceguard nodes will detect this and then periodically try to reconnect to it. If there is a cluster re-formation while the quorum server is down and tie-breaking is needed, the re-formation will fail and all the nodes will halt (system reset).
3.1.2 Serviceguard WBEM Provider 3.1.2.1 What is WBEM? Web-Based Enterprise Management (WBEM) is a set of management and Internet standard technologies developed to unify the management of distributed computing environments, facilitating the exchange of data across otherwise disparate technologies and platforms.
• HP_SGNodePackage • HP_SGPService • HP_SGPackagePService • HP_SGNodePService • HP_SGLockLunDisk • HP_SGRemoteQuorumService • HP_SGLockObject • HP_SGQuorumServer • HP_SGLockLun • HP_SGLockDisk For more information about WBEM provider classes, see Managed Object Format (MOF) files for properties.
3.2.2 Heartbeat Messages Central to the operation of the cluster manager is the sending and receiving of heartbeat messages among the nodes in the cluster. Each node in the cluster exchanges UDP heartbeat messages with every other node over each IP network configured as a heartbeat device. If a cluster node does not receive heartbeat messages from all other cluster nodes within the prescribed time, a cluster re-formation is initiated; see “What Happens when a Node Times Out” (page 73).
3.2.5 Dynamic Cluster Re-formation A dynamic re-formation is a temporary change in cluster membership that takes place as nodes join or leave a running cluster. Re-formation differs from reconfiguration, which is a permanent modification of the configuration files. Re-formation of the cluster occurs under the following conditions (not a complete list): • An SPU or network failure was detected on an active node. • An inactive node wants to join the cluster.
When a node obtains the cluster lock, this partition is marked so that other nodes will recognize the lock as “taken.” NOTE: • The lock LUN is dedicated for use as the cluster lock, and, in addition, HP recommends that this LUN comprise the entire disk; that is, the partition should take up the entire disk. • An iSCSI storage device does not support configuring a lock LUN. The complete path name of the lock LUN is identified in the cluster configuration file.
Figure 8 Quorum Server Operation A quorum server can provide quorum services for multiple clusters. Figure 9 illustrates quorum server use across four clusters. Figure 9 Quorum Server to Cluster Distribution IMPORTANT: For more information about the quorum server, see the latest version of the HP Serviceguard Quorum Server release notes at http://www.hp.com/go/linux-serviceguard-docs (Select HP Serviceguard Quorum Server Software). 3.2.
In a cluster with four or more nodes, you may not need a cluster lock since the chance of the cluster being split into two halves of equal size is very small. However, be sure to configure your cluster to prevent the failure of exactly half the nodes at one time. For example, make sure there is no potential single point of failure such as a single LAN between equal numbers of nodes, and that you don’t have exactly half of the nodes on a single power circuit. 3.2.
cluster, and the multi-node package, which can be configured to run on all or some of the nodes in the cluster. System multi-node packages are reserved for use by HP-supplied applications. The rest of this section describes failover packages. 3.3.1.2 Failover Packages A failover package starts up on an appropriate node (see node_name (page 186)) when the cluster starts. In the case of a service, network, or other resource or dependency failure, package failover takes place.
3.3.1.2.3 Failover Packages’ Switching Behavior The auto_run parameter (known in earlier versions of Serviceguard as the pkg_switching_enabled parameter) defines the default global switching attribute for a failover package at cluster startup: that is, whether Serviceguard can automatically start the package when the cluster is started, and whether Serviceguard should automatically restart the package on a new node in response to a failure.
Figure 11 Before Package Switching In Figure 12, node1 has failed and pkg1 has been transferred to node2. pkg1's IP address was transferred to node2 along with the package. pkg1 continues to be available and is now running on node2. Also note that node2 now has access both to pkg1's disk and pkg2's disk. NOTE: For design and configuration information about clusters that span subnets, see the documents listed under “Cross-Subnet Configurations” (page 27).
Figure 12 After Package Switching 3.3.1.2.4 Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file. The failover policy governs how the package manager selects which node to run a package on when a specific node has not been identified and the package needs to be started.
Table 1 Package Configuration Data Package Name NODE_NAME List FAILOVER_POLICY pkgA node1, node2, node3, node4 min_package_node pkgB node2, node3, node4, node1 min_package_node pkgC node3, node4, node1, node2 min_package_node When the cluster starts, each package starts as shown in Figure 13.
Figure 14 Rotating Standby Configuration after Failover NOTE: Under the min_package_node policy, when node2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node. If these packages had been set up using the configured_node failover policy, they would start initially as in Figure 13, but the failure of node2 would cause the package to start on node3, as shown in Figure 15. 3.
Figure 15 configured_node Policy Packages after Failover If you use configured_node as the failover policy, the package will start up on the highest-priority eligible node in its node list. When a failover occurs, the package will move to the next eligible node in the list, in the configured order of priority. 3.3.1.2.
Figure 16 Automatic Failback Configuration before Failover Table 2 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 configured_node automatic pkgB node2, node4 configured_node automatic pkgC node3, node4 configured_node automatic node1 panics, and after the cluster reforms, pkgA starts running on node4: 3.
Figure 17 Automatic Failback Configuration After Failover After rebooting, node1 rejoins the cluster. At that point, pkgA will be automatically stopped on node4 and restarted on node1.
NOTE: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period. If you are using automatic failback, you may want to wait to add the package’s primary node back into the cluster until you can allow the package to be taken out of service temporarily while it switches back to the primary node. Serviceguard automatically chooses a primary node for a package when the NODE_NAME is set to '*'.
If there is a common generic resource that needs to be monitored as a part of multiple packages, then the monitoring script for that resource can be launched as part of one package and all other packages can use the same monitoring script. There is no need to launch multiple monitors for a common resource. If the package that has started the monitoring script fails or is halted, then all the other packages that are using this common resource also fail.
3.4.1 What Makes a Package Run? There are 3 types of packages: • The failover package is the most common type of package. It runs on one node at a time. If a failure occurs, it can switch to another node listed in its configuration file. If switching is enabled for several nodes, the package manager will use the failover policy to determine where to start the package. • A system multi-node package runs on all the active cluster nodes at the same time.
Figure 19 Modular Package Time Line Showing Important Events The 1. 2. 3. 4. 5. 6. 7. 8. following are the most important moments in a package’s life: Before the master control script starts. During master control script execution to start the package. While services are running. If there is a generic resource configured and it fails, then the package will be halted. When a service or subnet fails, or a dependency is not met. During master control script execution to halt the package.
3.4.3 During Run Script Execution Once the package manager has determined that the package can start on a particular node, it launches the script that starts the package (that is, a package’s control script or master control script is executed with the start parameter). This script carries out the following steps: 1. 2. 3. 4. 5. 6. 7. Executes any external_pre_scripts. For more information, see “About External Scripts” (page 135)). Activates volume groups or disk groups. Mounts file systems.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled (page 194) is set to yes, it will halt the node on which the package is running.
3.4.7 When a Service or Subnet Fails or Generic Resource or a Dependency is Not Met What happens when something goes wrong? If a service fails and there are no more restarts, or if a configured dependency on another package is not met, then a failover package will halt on its current node and, depending on the setting of the package switching flags, may be restarted on another node.
7. 8. Exits with an exit code of zero (0). Executes any external_pre_script. For more information, see “external_pre_script” (page 200). Figure 21 Modular Package Time Line for Halt Script Execution At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). If the halt script execution is not complete before the time specified in the halt_script_timeout (page 187) , the package manager will kill the script.
3.4.10.1 Package Control Script Error and Exit Conditions Table 3 shows the possible combinations of error condition, failfast setting and package movement for failover packages.
3.5.1 Stationary and Relocatable IP Addresses and Monitored Subnets Each node (host system) should have an IP address for each active network interface. This address, known as a stationary IP address, is configured in the file /etc/sysconfig/network-scripts/ifcfg- on Red Hat or /etc/sysconfig/network/ifcfg- on SUSE. The stationary IP address is not associated with packages, and it is not transferable to another node.
NOTE: It is possible to configure a cluster that spans subnets joined by a router, with some nodes using one subnet and some another. This is called a cross-subnet configuration.
Figure 22 Bonded Network Interfaces The LANs in the non-bonded configuration have four LAN cards, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (eth1, eth2, eth3, or eth4). When these ports are aggregated, all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as bond0, and this is the name by which the bond is known during cluster configuration.
After the failure of a card, messages are still carried on the bonded LAN and are received on the other node, but now eth1 has become active in bond0 on node1. This situation is shown in Figure 24.
Figure 25 Bonded NICs Configured for Load Balancing 3.5.6 Monitoring LAN Interfaces and Detecting Failure: Link Level At regular intervals, determined by the NETWORK_POLLING_INTERVAL (see “Cluster Configuration Parameters” (page 89)), Serviceguard polls all the network interface cards specified in the cluster configuration file (both bonded and non-bonded).
◦ Inbound failures ◦ Errors that prevent packets from being received but do not affect the link-level health of an interface IMPORTANT: You should configure the IP Monitor in a cross-subnet configuration, because IP monitoring will detect some errors that link-level monitoring will not. See also “Cross-Subnet Configurations” (page 27). 3.5.7.
The IP Monitor section of the cluster configuration file will look similar to the following for a subnet on which IP monitoring is configured with target polling. IMPORTANT: By default, the cmquerycl does not verify that the gateways it detects will work correctly for monitoring. But if you use the -w full option, cmquerycl will validate them as polling targets. SUBNET 192.168.1.0 IP_MONITOR ON POLLING_TARGET 192.168.1.254 By default, IP_MONITOR parameter is set to OFF.
node are down), IP monitoring fails and all the subnets are marked down on the operational node. This results in failure of packages running on the operational node. Therefore, peer polling is not suitable when there is only a single peer as exists in a 2-node cluster. In such scenarios, a polling target should always be defined so that a single LAN failure does not affect polling of other LANs. 3.5.
3.5.10 Address Resolution Messages after Switching on the Same Subnet When a relocatable IP address is moved to a new interface, either locally or remotely, an ARP message is broadcast to indicate the new mapping between IP address and link layer address. An ARP message is sent for each IP address that has been moved. All systems receiving the broadcast should update the associated ARP cache entry to reflect the change.
1. 2. 3. VLAN heartbeat networks must be configured on separate physical NICs or Channel Bonds, to avoid single points of failure. Heartbeats are still recommended on all cluster networks, including VLANs. If you are using VLANs, but decide not to use VLANs for heartbeat networks, heartbeats are recommended for all other physical networks or Channel Bonds specified in the cluster configuration file. 3.
Figure 26 Physical Disks Combined into LUNs NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer. Since arrays vary considerably, you should refer to the documentation that accompanies your storage unit. 3.6.2 Monitoring Disks Each package configuration includes information about the disks that are to be activated by the package at startup. If monitoring is used, the health of the disks is checked at package startup.
For more information about how to create a storage infrastructure with VxVM, see Creating a Storage Infrastructure with VxVM. 3.6.4.1 Limitations Following are the limitations of VxVM: • Serviceguard does not support VxVM with VMs (Virtual Machines). • Serviceguard does not support VxVM on iSCSI storage. • Serviceguard supports VxVM only with OS Native naming scheme. To set the OS Native naming scheme: #/usr/sbin/vxddladm set namingscheme=osn 3.
◦ The udev alias names must not be configured in the /dev/mapper/ or/dev/mpath/ directory. ◦ Multipath device alias names must not contain “pN” or “_partN” strings, where N is the number. For example, /dev/mapper/evadskp1 or /dev/mapper/evadsk_part1 • If you accidently run the pr_cleanup command on LUNs belonging to a package that is already running, PR protection is disabled. To enable PR protection, you must restart the package.
shutdown, using the sg_persist command. This command is available, and has a manpage, on both Red Hat 5, Red Hat 6, and SUSE 11 . Serviceguard makes a PR of type Write Exclusive Registrants Only (WERO) on the package's LUN devices. This gives read access to any initiator regardless of whether the initiator is registered or not, but grants write access only to those initiators who are registered. (WERO is defined in the SPC-3 standard.
1. 2. 3. The node contacts the other nodes and tries to re-form the cluster without the failed node. If the remaining nodes are a majority or can obtain the cluster lock, they form a new cluster without the failed node. If the remaining nodes are not a majority or cannot get the cluster lock, they halt (system reset). 3.8.1.1.1 Example Situation. Assume a two-node cluster, with Package1 running on SystemA and Package2 running on SystemB.
Disk monitoring provides additional protection. You can configure packages to be dependent on the health of disks, so that when a disk monitor reports a problem, the package can fail over to another node. See “Creating a Disk Monitor Configuration” (page 209). Serviceguard does not respond directly to power failures, although a loss of power to an individual cluster component may appear to Serviceguard like the failure of that component, and will result in the appropriate switching behavior.
• Any failure of a generic resource of evaluation type "before_package_start" configured in a package will not disable the node switching for the package. • Any failure of a generic resource of evaluation type "during_package_start" configured in a package will disable the node switching for the package. “Choosing Switching and Failover Behavior” (page 106) provides advice on choosing appropriate failover behavior. See “Parameters for Configuring Generic Resources” (page 107). 3.8.4.
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather and record information about all the hardware and software components of the configuration.
your cluster without having to bring it down, you need to plan the initial configuration carefully. Use the following guidelines: • Set the Maximum Configured Packages parameter (described later in this chapter under “Cluster Configuration Planning ” (page 84)) high enough to accommodate the additional packages you plan to add. • Networks should be pre-configured into the cluster configuration if they will be needed for packages you will add later while the cluster is running.
4.2.2 Supported cluster configuration options Following are the supported cluster configuration options when using VMware or KVM guests as cluster nodes: • Cluster with VMware or KVM guests from a single host as cluster nodes (cluster-in-a-box; not recommended) NOTE: This configuration is not recommended because failure of the host brings down all the nodes in the cluster which is a single point of failure.
Subnet Name The IP address for the subnet. Note that heartbeat IP addresses must be on the same subnet on each node. Interface Name The name of the LAN card as used by this node to access the subnet. This name is shown by ifconfig after you install the card. IP Address The IP address to be used on this interface. An IPv4 address is a string of 4 digits separated with decimals, in this form: nnn.nnn.nnn.
Slot Number Indicate the slot number(s) into which the SCSI or FibreChannel interface card(s) are inserted in the backplane of the computer. Address Enter the bus hardware path number, which is the numeric part of the host parameter, which can be seen on the system by using the following command: cat /proc/scsi/scsi Disk Device File Enter the disk device file name for each SCSI disk or LUN. This information is needed when you create the mirrored disk configuration using LVM.
different physical volume groups; this allows you to set up mirroring between physical disks that are not only on different I/O buses, but also connected to different power supplies. To prevent confusion, label each hardware unit and power supply unit clearly with a different unit number. Indicate on the Power Supply Worksheet the specific hardware units you are using and the power supply to which they will be connected.
• Can support a cluster with any supported number of nodes. • Can communicate with the cluster on up to two subnets (a primary and an alternate). IMPORTANT: If you plan to use a Quorum Server, make sure you read the HP Serviceguard Quorum Server Version A.04.00 Release Notes before you proceed. You can find them at http:// www.hp.com/go/linux-serviceguard-docs (Select HP Serviceguard Quorum Server Software). You should also consult the Quorum Server white papers at the same location. 4.5.3.
When designing a storage configuration using VxVM disk groups, consider the following: • High availability applications, services, and data must be placed in separate disk groups from non-high availability applications, services, and data. • You must not group two different high availability applications, services, or data, whose control needs to be transferred independently, onto the same disk group. • Your root disk can belong to an LVM or VxVM volume group that is not shared among cluster nodes. 4.
Examples Create a new LVM volume group “lvmvg” and import it to nodes node1, node2, node3, and node4: cmpreparestg -l lvmvg -n node1 -n node2 -n node3 -n node4 -p /dev/sdz Create a new VxVM disk group “cvmdg” and import it to nodes node1, node2, node3, and node4: cmpreparestg -g cvmdg -n node1 -n node2 -n node3 -n node4 -p /dev/sdz 4.8.1.3 cmdeploycl The cmdeploycl creates the cluster with the previously generated network and storage configuration. For more information, see cmdeploycl (1m) manpage.
1. Verify the prerequisites for cluster configuration: cmpreparecl –n —n -p 2. Run the cmpreparecl command with the nodes on which the cluster needs to be configured: cmpreparecl –n —n 3. The cmpreparecl command performs the following actions: a. Verifies the availability of ports required by Serviceguard. For information about port requirements on Red Hat Enterprise Linux Server and SUSE Linux Enterprise Server, see HP Serviceguard A.12.00.00 for Linux Release Notes. b.
• IPv4-only • IPv6-only • Mixed IPv4-only means that Serviceguard will try to resolve the hostnames to IPv4 addresses only. IMPORTANT: You can configure an IPv6 heartbeat, or stationary or relocatable IP address, in any mode: IPv4-only, IPv6-only, or mixed. You can configure an IPv4 heartbeat, or stationary or relocatable IP address, in IPv4-only or mixed mode. IPv6-only means that Serviceguard will try to resolve the hostnames to IPv6 addresses only.
NOTE: This also applies if HOSTNAME_ADDRESS_FAMILY is set to ANY; Red Hat 5 supports only IPv4-only clusters. • All addresses used by the cluster must be in each node's /etc/hosts file. In addition, the file must contain the following entry: ::1 localhost ipv6-localhost ipv6-loopback For more information and recommendations about hostname resolution, see “Configuring Name Resolution” (page 145).
4.8.3.2.2 Recommendations for IPv6-Only Mode IMPORTANT: Check the latest Serviceguard for Linux release notes for the latest instructions and recommendations. • If you decide to migrate the cluster to IPv6-only mode, you should plan to do so while the cluster is down. 4.8.3.
cluster configuration template file created by means of the cmquerycl command, as described under “Configuring the Cluster” (page 164). NOTE: See “Reconfiguring a Cluster” (page 237) for a summary of changes you can make while the cluster is running. The following parameters must be configured: CLUSTER_NAME The name of the cluster as it will appear in the output of cmviewcl and other commands, and as it appears in the cluster configuration file.
IPv4 or an IPv6 address if HOSTNAME_ADDRESS_FAMILY is set to ANY, but otherwise must match the setting of HOSTNAME_ADDRESS_FAMILY. This parameter is used only when you employ a quorum server for tie-breaking services in the cluster. You can also specify an alternate address (QS_ADDR) by which the cluster nodes can reach the quorum server. For more information, see “Cluster Lock Planning” (page 82) and “Specifying a Quorum Server” (page 166).
Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 41) for important information. QS_TIMEOUT_EXTENSION You can use the QS_TIMEOUT_EXTENSION to increase the time interval after which the current connection (or attempt to connect) to the quorum server is deemed to have failed; but do not do so until you have read the HP Serviceguard Quorum Server Version A.04.
CAPACITY_NAME, and CAPACITY_VALUE) apply specifically to the node identified by the preceding NODE_NAME entry. CLUSTER_LOCK_LUN The pathname of the device file to be used for the lock LUN on each node. The pathname can contain up to 39 characters. See “Setting up a Lock LUN” (page 151) and “Specifying a Lock LUN” (page 166) Can be changed while the cluster is running; see “Updating the Cluster Lock LUN Configuration Online” (page 244).
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (see “Package Configuration Planning” (page 103)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP. Similarly, any subnet that is used by a package for relocatable addresses should be configured into the cluster via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
• two heartbeat subnets; or • one heartbeat subnet using bonding in high availability mode (or mode 1) with two slaves. You cannot configure more than one heartbeat IP address on an interface; only one HEARTBEAT_IP is allowed for each NETWORK_INTERFACE. NOTE: The Serviceguard cmapplyconf, cmcheckconf, and cmquerycl commands check that these minimum requirements are met, and produce a warning if they are not met at the immediate network level.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (see “Package Configuration Planning” (page 103)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP. Similarly, any subnet that is used by a package for relocatable addresses should be configured into the cluster via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
NOTE: cmapplyconf will fail if any node defines a capacity and any package has min_package_node as its failover_policy (page 188) or automatic as its failback_policy (page 189). To specify more than one capacity for a node, repeat these parameters for each capacity. You can specify a maximum of four capacities per cluster, unless you use the reserved CAPACITY_NAME package_limit; in that case, you can use only that capacity throughout the cluster.
NOTE: The failover estimates provided here apply to the Serviceguard component of failover; that is, the package is expected to be up and running on the adoptive node in this time, but the application that the package runs may take more time to start. For most clusters that use a lock LUN, a minimum MEMBER_TIMEOUT of 14 seconds is appropriate. For most clusters that use a MEMBER_TIMEOUT value lower than 14 seconds, a quorum server is more appropriate than a lock LUN.
AUTO_START_TIMEOUT The amount of time a node waits before it stops trying to join a cluster during automatic cluster startup. All nodes wait this amount of time for other nodes to begin startup before the cluster completes the operation. The time should be selected based on the slowest boot time in the cluster. Enter a value equal to the boot time of the slowest booting node minus the boot time of the fastest booting node plus 600 seconds (ten minutes). Default is 600,000,000 microseconds.
Table 4 Failure Recovery Detection Times for an IP Monitored Ethernet Interface Values of Network Polling Failure/Recovery Detection Times (in seconds) Interval (NPI) (in seconds) 1 ~ NPI x 8 - NPI x 9 2 ~ NPI x 4 - NPI x 5 3 ~ NPI x 3 - NPI x 4 4 to 8 ~ NPI x 2 - NPI x 3 >=8 ~ NPI x 1- NPI x 2 IMPORTANT: HP strongly recommends using the default. Changing this value can affect how quickly the link-level and IP-level monitors detect a network failure.
For more information about MBTD, see the white paper Support for NFS as a filesystem type with HP Serviceguard A.11.20 on HP-UX and Linux available at http://www.hp.com/go/linux-serviceguard-docs. • For clusters in which both of the above conditions apply. In this case, set the CONFIGURED_IO_TIMEOUT_EXTENSION to the higher of the two values you get from following the instructions in the preceding two bullets. Default is 0. The value can range from zero to 2147483647.
to use peer polling instead, set IP_MONITOR to ON for this SUBNET, but do not use POLLING_TARGET (comment out or delete any POLLING_TARGET entries that are already there). If a network interface in this subnet fails at the IP level and IP_MONITOR is set to ON, the interface will be marked down. If it is set to OFF, failures that occur only at the IP-level will not be detected. Can be changed while the cluster is running; must be removed if the preceding SUBNET entry is removed.
NOTE: A weight (WEIGHT_NAME, WEIGHT_DEFAULT) has no meaning on a node unless a corresponding capacity (CAPACITY_NAME, CAPACITY_VALUE) is defined for that node. For the reserved weight and capacity package_limit, the default weight is always one. This default cannot be changed in the cluster configuration file, but it can be overridden for an individual package in the package configuration file.
NOTE: To prevent an operator from accidentally activating volume groups on other nodes in the cluster, versions A.11.16.07 and later of Serviceguard for Linux include a type of VG activation protection. This is based on the “hosttags” feature of LVM2. This feature is not mandatory, but HP strongly recommends you implement it as you upgrade existing clusters and create new ones. See “Enabling Volume Group Activation Protection” (page 156) for instructions.
NOTE: Ensure that the NFS module is loaded during boot time for the configurations using NFS file systems as part of the package configuration. The following rules and restrictions apply. • NFS mounts are supported for modular failover packages.
• An NFS file system should not be mounted on more than one mount point at the same time. • Access to an NFS file system used by a package should be restricted to the nodes that can run the package. For more information, see the white paper Using NFS as a file system type with Serviceguard 11.20 on HP-UX and Linux available at http://www.hp.com/go/linux-serviceguard-docs. This paper includes instructions for setting up a sample package that uses an NFS-imported file system.
Table 5 Package Failover Behavior (continued) Switching Behavior Parameters in Configuration File All packages switch following a system reset (an immediate halt without a graceful shutdown) on the node when a specific service fails. Halt scripts are not run. • service_fail_fast_enabled set to yes for a specific service. • auto_run set to yes for all packages. All packages switch following a system reset • service_fail_fast_enabled set to yes for all services. on the node when any service fails.
1. Create a package configuration file that contains the generic resource module: cmmakepkg $SGCONF/pkg1/pkg1.conf Package template is created. This file must be edited before it can be used. NOTE: To generate a configuration file adding the generic resource module to an existing package (enter the command all on one line): cmmakepkg -i $SGCONF/pkg1/pkg1.conf -m sg/generic_resource 2.
pkg1 down halted disabled unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Generic Resource unknown Generic Resource unknown NODE_NAME node1 node2 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled Other_Attributes: ATTRIBUTE_NAME Style Priority NAME sfm_disk sfm_disk NAME node1 node2 ATTRIBUTE_VALUE modular no_priority The cmviewcl -v -f line output (snippet) will be as follo
4.9.6.1.2 Using Serviceguard Command to Set the Status/Value of a Simple/Extended Generic Resource Use the cmsetresource command to set the status of a simple generic resource or the value of an extended generic resource. For example: cmsetresource -r sfm_disk -s up This sets the status of the generic resource sfm_disk to up. This is a simple generic resource and only the status can be set to up/down. cmsetresource -r sfm_lan 10 This sets the current value of the generic resource sfm_lan to 10.
The following operations cannot be performed online: • Changing at once both the mirror halves (xdc/xdc/raid_device_0[] and xdc/xdc/raid_device_1[]) of an existing MD device (xdc/xdc/raid_device[]). • Changing the service_name attribute of "raid_monitor" service when the serviceguard-xdc package is running. • Adding and deleting an MD device simultaneously. • Add an MD device and replacing a mirror half in another existing MD device simultaneously.
• pkg1 will not start on any node unless pkg2 is running on that node. • pkg1’s package_type (page 185) and failover_policy (page 188) constrain the type and characteristics of pkg2, as follows: ◦ If pkg1 is a multi-node package, pkg2 must be a multi-node or system multi-node package. (Note that system multi-node packages are not supported for general use.) ◦ If pkg1 is a failover package and its failover_policy is min_package_node, pkg2 must be a multi-node or system multi-node package.
4.9.7.2.1 Dragging Rules for Simple Dependencies The priority parameter (page 189) gives you a way to influence the startup, failover, and failback behavior of a set of failover packages that have a configured_node failover_policy, when one or more of those packages depend on another or others. The broad rule is that a higher-priority package can drag a lower-priority package, forcing it to start on, or move to, a node that suits the higher-priority package.
If pkg1 depends on pkg2, and pkg1’s priority is lower than or equal to pkg2’s, pkg2’s node order dominates. Assuming pkg2’s node order is node1, node2, node3, then: • On startup: ◦ • pkg2 will start on node1, or node2 if node1 is not available or does not at present meet all of its dependencies, etc. – pkg1 will start on whatever node pkg2 has started on (no matter where that node appears on pkg1’s node_name list) provided all of pkg1’s other dependencies are met there.
Note that the nodes will be tried in the order of pkg1’s node_name list, and pkg2 will be dragged to the first suitable node on that list whether or not it is currently running on another node. • • On failover: ◦ If pkg1 fails on node1, pkg1 will select node2 to fail over to (or node3 if it can run there and node2 is not available or does not meet all of its dependencies; etc.) ◦ pkg2 will be dragged to whatever node pkg1 has selected, and restart there; then pkg1 will restart there.
• You can specify whether the package depended on must be running or must be down. You define this condition by means of the dependency_condition, using one of the literals UP or DOWN (the literals can be upper or lower case). We'll refer to the requirement that another package be down as an exclusionary dependency; see “Rules for Exclusionary Dependencies” (page 116).
lower priority package to halt or (in the case of a same-node exclusion) move to another eligible node, if any. ◦ • When a lower priority package fails over to another node on which the higher priority package with mutually exclusive dependency is running, the lower priority package fails to start. If you start a package manually using cmrunpkg command on a node on which another mutually exclusive package is running, then cmrunpkg command fails with the following message: Unable to execute command.
By default the packages are halted in the reverse of the order in which they were started; and if the halt script for any of the dependent packages hangs, the failed package will wait indefinitely to complete its own halt process. This provides the best chance for all the dependent packages to halt cleanly, but it may not be the behavior you want. You can change it by means of the successor_halt_timeout parameter (page 188). (A successor is a package that depends on another package.
4.9.10.1 Package Weights and Node Capacities You define a capacity, or capacities, for a node (in the cluster configuration file), and corresponding weights for packages (in the package configuration file). Node capacity is consumed by package weights. Serviceguard ensures that the capacity limit you set for a node is never exceeded by the combined weight of packages running on it; if a node's available capacity will be exceeded by a package that wants to run on that node, the package will not run there.
If some packages consume more resources than others, you can use the weight_name and weight_value parameters to override the default value (1) for some or all packages. For example, suppose you have three packages, pkg1, pkg2, and pkg3. pkg2 is about twice as resource-intensive as pkg3 which in turn is about one-and-a-half times as resource-intensive as pkg1.
IMPORTANT: You cannot combine the two methods. If you use the reserved capacity package_limit for any node, Serviceguard will not allow you to define any other type of capacity and weight in this cluster; so you are restricted to the Simple Method in that case. 4.9.10.4.1 Defining Capacities Begin by deciding what capacities you want to define; you can define up to four different capacities for the cluster.
CAPACITY_NAME B CAPACITY_VALUE 50 NODE_NAME node2 CAPACITY_NAME A CAPACITY_VALUE 60 CAPACITY_NAME B CAPACITY_VALUE 70 ... NOTE: You do not have to define capacities for every node in the cluster. If any capacity is not defined for any node, Serviceguard assumes that node has an infinite amount of that capacity.
1. 2. 3. 4. Configure a cluster-wide default weight and let the package use that default. Configure a cluster-wide default weight but override it for this package in its package configuration file. Do not configure a cluster-wide default weight, but assign a weight to this package in its package configuration file. Do not configure a cluster-wide default weight and do not assign a weight for this package in its package configuration file.
• pkg1 will consume all of node2's A capacity; no other package that has A weight can run on this node while pkg1 is running there. But node2 could still run pkg3 while running pkg1, because pkg3 has no A weight, and pkg1 is consuming only 15 units (the default) of node2's B capacity, leaving 35 available to pkg3 (assuming no other package that has B weight is already running there).
• the package configuration file • the cmmakepkg (1m) manpage • the section “Package Parameter Explanations” (page 184) in this manual. For further discussion and use cases, see the white paper Using Serviceguard’s Node Capacity and Package Weight Feature at http://www.hp.com/go/linux-serviceguard-docs. 4.9.10.
4.9.10.8.1 Load Balancing When capacity and weights are configured in a cluster, the packages can either start or failover in one of the following ways: • To the node that has the least used capacity, if the node capacity is INFINITE. • To the node that has the highest capacity remaining, if the node capacity is finite. 4.9.10.8.2 Parameters LOAD_BALANCING You can set the LOAD_BALANCING parameter ON or OFF in the cluster configuration file.
1. 2. Set the LOAD_BALANCING parameter to ON. Configure one capacity for each node in the cluster. The capacity value can either have a INFINITE (for all the nodes in the cluster) or finite value. Package configuration file In the package configuration file: 1. 2. 3. Set the failover_policy parameter to CONFIGURED_NODE. Set the failback_policy parameter to MANUAL. Configure a weight corresponding to the capacity defined in the cluster configuration file. 4.9.10.8.
package:pkg1|weight:test_capacity|name=test_capacity package:pkg1|weight:test_capacity|value=7 package:pkg2|weight:test_capacity|name=test_capacity package:pkg2|weight:test_capacity|value=9 package:pkg3|weight:test_capacity|name=test_capacity package:pkg3|weight:test_capacity|value=10 package:pkg4|weight:test_capacity|name=test_capacity package:pkg4|weight:test_capacity|value=4 package:pkg5|weight:test_capacity|name=test_capacity package:pkg5|weight:test_capacity|value=4 5.
Figure 28 Failover of a Package pkg3 (wt=10) fails over to Node 3 Node 1 pkg2 (wt=9) Used Cap=9 pkg1 (wt=7) Used Cap=17 pkg4 (wt=4) pkg5 (wt=4) Used Cap=8 Node 2 Node 3 Node 4 When the capacity and weights are configured in a cluster, the package fails over to the node that has the least used capacity. Since the Node 3 has the least used capacity of 7, the pkg3 with weight of 10 fails over to Node 3. 7.
Figure 30 (page 130) shows that pkg1 and pkg2 running on Node 2 and pkg4, pkg5, and pkg3 running on Node 4. Figure 30 Load Distributed across the Nodes after Failover pkg2 (wt=9) pkg1 (wt=7) Used Cap=16 pkg4 (wt=4) pkg5 (wt=4) pkg3 (wt=10) Used Cap=18 Node 2 Node 4 Scenario 2: Starting packages with cmrunpkg –a command 1. 2. Create a cluster and packages as described in step 4 of scenario 1.
First, pkg3 is on Node 1, as this is the first node with least used capacity. Then, pkg5 is placed on Node 4. Scenario 3: Packages with dependency are placed in load sensitive placement. 1. Set the LOAD_BALANCING parameter to ON: Parameters in cluster configuration: LOAD_BALANCING ON 2. Create a cluster with four nodes, and CAPACITY_VALUE parameter for each node set to INFINITE: NODE_NAME node1 CAPACITY_NAME test_capacity CAPACITY_VALUE INFINITE 3.
3.
• pkg6 and pkg4 are placed on Node 3 which is of weight 33 • pkg5 which has different node dependency from pkg6 and is placed on the least loaded node out of Node 1, Node 2, and Node 4 since pkg6 is running on Node 3. So, pkg5 is placed on Node 2.
package:pkg3|weight:test_capacity|value=7 package:pkg4|weight:test_capacity|name=test_capacity package:pkg4|weight:test_capacity|value=10 package:pkg5|weight:test_capacity|name=test_capacity package:pkg5|weight:test_capacity|value=6 4. Run the cmrunpkg -a command: cmrunpkg -a pkg1 pkg2 pkg3 pkg4 pkg5 Figure 31 (page 134) shows the load distribution across the nodes with CAPACITY_VALUE parameter set to finite.
Figure 32 Failover of a Package pkg1 (wt=4) fails over to Node 2 pkg3 (wt=7) pkg2 (wt=3) Cap=11 Remaining Cap=1 pkg4 (wt=10) Cap=14 Remaining Cap=0 pkg5 (wt=6) Cap=8 Remaining Cap=2 Node 1 Node 2 Node 3 Node 4 When finite capacity is configured in a cluster, the package fails over to the node that has the highest remaining capacity. Since the Node 2 has the highest remaining capacity of 4, the pkg1 with weight of 4 fails over to Node 2.
• On package startup and shutdown, as essentially the first and last functions the package performs. These scripts are invoked by means of the parameter external_pre_script (page 200); or • During package execution, after volume-groups and file systems are activated, and IP addresses are assigned, and before the service and resource functions are executed; and again, in the reverse order, on package shutdown. These scripts are invoked by means of the parameter external_script (page 200).
SG_UTILS=$SGCONF/scripts/mscripts/utils.sh fi if [[ -f ${SG_UTILS} ]]; then . ${SG_UTILS} if (( $? != 0 )) then echo "ERROR: Unable to source package utility functions file: ${SG_UTILS}" exit 1 fi else echo "ERROR: Unable to find package utility functions file: ${SG_UTILS}" exit 1 fi # Get the environment for this package through utility function # sg_source_pkg_env().
sg_log 0 "Unknown entry point $1" ;; esac exit $exit_val 4.9.11.1 Using Serviceguard Commands in an External Script You can use Serviceguard commands (such as cmmodpkg) in an external script. These commands must not interact with the package itself (that is, the package that runs the external script) but can interact with other packages. But be careful how you code these interactions. If a Serviceguard command interacts with another package, be careful to avoid command loops.
4.9.12 About Cross-Subnet Failover It is possible to configure a cluster that spans subnets joined by a router, with some nodes using one subnet and some another. This is known as a cross-subnet configuration; see “Cross-Subnet Configurations” (page 27). In this context, you can configure packages to fail over from a node on one subnet to a node on another.
NodeA and NodeB use subnet 15.244.65.0, which is not used by NodeC and NodeD; and NodeC and NodeD use subnet 15.244.56.0, which is not used by NodeA and NodeB. (See “Obtaining Cross-Subnet Information” (page 167) for sample cmquerycl output). 4.9.12.2.1 Configuring node_name First you need to make sure that pkg1 will fail over to a node on another subnet only if it has to.
4.10 Planning for Changes in Cluster Size If you intend to add additional nodes to the cluster online (while it is running) ensure that they are connected to the same heartbeat subnets and to the same lock disks as the other cluster nodes. In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes. Remember that while a two-node cluster must use a cluster lock, a cluster of more than four nodes must not use a lock LUN, but can use a quorum server.
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. You carry out these procedures on one node, called the configuration node, and Serviceguard distributes the resulting binary file to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
######################################################################### SGROOT=/opt/cmcluster # SG root directory SGCONF=/opt/cmcluster/conf # configuration files SGSBIN=/opt/cmcluster/bin # binaries SGLBIN=/opt/cmcluster/bin # binaries SGLIB=/opt/cmcluster/lib # libraries SGRUN=/opt/cmcluster/run # location of core dumps from daemons SGAUTOSTART=/opt/cmcluster/conf/cmcluster.rc # SG Autostart file Throughout this document, system filenames are usually given with one of these location prefixes.
the file $SGCONF/cmclnodelist. This is sometimes referred to as a “bootstrap” file because Serviceguard consults it only when configuring a node into a cluster for the first time; it is ignored after that. It does not exist by default, but you will need to create it. You may want to add a comment such as the following at the top of the file: ########################################################### # Do not edit this file! # Serviceguard uses this file only to authorize access to an # unconfigured node.
Serviceguard nodes can communicate over any of the cluster’s shared networks, so the network resolution service you are using (such as DNS, NIS, or LDAP) must be able to resolve each of their primary addresses on each of those networks to the primary hostname of the node in question. In addition, HP recommends that you define name resolution in each node’s /etc/hosts file, rather than rely solely on a service such as DNS.
IMPORTANT: Serviceguard does not support aliases for IPv6 addresses. For information about configuring an IPv6–only cluster, or a cluster that uses a combination of IPv6 and IPv4 addresses for the nodes' hostnames, see “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 86). 5.1.5.
NOTE: HP recommends that you also make the name service itself highly available, either by using multiple name servers or by configuring the name service into a Serviceguard package. 5.1.6 Ensuring Consistency of Kernel Configuration Make sure that the kernel configurations of all cluster nodes are consistent with the expected behavior of the cluster during failover.
DEVICE=bond0 IPADDR=192.168.1.1 NETMASK=255.255.255.0 NETWORK=192.168.1.0 BROADCAST=192.168.1.255 ONBOOT=yes BOOTPROTO=none USERCTL=no For Red Hat 5 and Red Hat 6 only, add the following line to the ifcfg-bond0file: BONDING OPTS=’miimon=100 mode=1’ 2. Create an ifcfg-ethn file for each interface in the bond. All interfaces should have SLAVE and MASTER definitions.
5.1.8.3 Viewing the Configuration You can test the configuration and transmit policy with ifconfig. For the configuration created above, the display should look like this: /sbin/ifconfig bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.
REMOTE_IPADDR='' STARTMODE='onboot' BONDING_MASTER='yes' BONDING_MODULE_OPTS='miimon=100 mode=1' BONDING_SLAVE0='eth0' BONDING_SLAVE1='eth1' The above example configures bond0 with mii monitor equal to 100 and active-backup mode. Adjust the IP, BROADCAST, NETMASK, and NETWORK parameters to correspond to your configuration. As you can see, you are adding the configuration options BONDING_MASTER, BONDING-MODULE_OPTS, and BONDING_SLAVE.
Table 6 Changing Linux Partition Types Prompt Response Action Performed 1. Command (m for help): n Create new partition 2. Partition number (1-4): 1 Partition affected 3.
Creating a Lock LUN on a Whole LUN The lock LUN can be created on a whole LUN of at least 100K starting with the below patches. On Red Hat Enterprise Linux server, you have to install the following patches on Serviceguard Linux Version A.11.20.00: • SGLX_00339 for Red Hat Enterprise Linux 5 (x86_64 architecture) • SGLX_00340 for Red Hat Enterprise Linux 6 (x86_64 architecture) Patches can be downloaded from HP Support Center at http://www.hp.com/go/hpsc.
You can build a cluster (next section) before or after defining volume groups for shared data storage. If you create the cluster first, information about storage can be added to the cluster and package configuration files after the volume groups are created. See “Volume Managers for Data Storage” (page 69) for an overview of volume management in HP Serviceguard for Linux.
Units = cylinders of 2048 * 512 bytes Device Boot Start End Blocks Id System Disk /dev/sdc: 255 heads, 63 sectors, 1106 cylinders Units = cylinders of 16065 * 512 bytesDisk /dev/sdd: 255 heads, 63 sectors, 1106 cylinders Units = cylinders of 16065 * 512 bytes In this example, the disk described by device file /dev/sda has already been partitioned for Linux, into partitions named /dev/sda1 - /dev/sda7.
Command (m for help): w The partition table has been altered! 2. Respond to the prompts as shown in the following table to set a partition type: Prompt Response Action Performed 1. Command (m for help): t Set the partition type 2.
2. Uncomment the line in /etc/lvm/lvm.conf that begins # volume_list =, and edit it to include all of the node's "private" volume groups (those not shared with the other cluster nodes), including the root volume group. For example if the root volume group is vg00 and the node also uses vg01 and vg02 as private volume groups, the line should look like this: volume_list = [ "vg00", "vg01", "vg02" ] 3. 4. Create the file /etc/lvm/lvm_$(uname -n).
3. Check whether there are already volume groups defined on this node. Be sure to give each volume group a unique name. vgdisplay 4. Create separate volume groups for each Serviceguard package you will define.
5.1.12.6 Testing the Shared Configuration When you have finished the shared volume group configuration, you can test that the storage is correctly sharable as follows: 1.
2. On ftsys10, activate the volume group, mount the file system, write a date stamp on to the shared file, and then look at the content of the file: vgchange --addtag $(uname -n) vgpkgB vgchange -a y vgpkgB mount /dev/vgpkgB/lvol1 /extra echo ‘Written by’ ‘hostname‘ ‘on’ ‘date‘ >> /extra/datestamp cat /extra/datestamp You should see something like the following, including the date stamp written by the other node: Written by ftsys9.mydomain on Mon Jan 22 14:23:44 PST 2006 Written by ftsys10.
NOTE: Be careful if you use YAST or YAST2 to configure volume groups, as that may cause all volume groups to be activated. After running YAST or YAST2, check that volume groups for Serviceguard packages not currently running have not been activated, and use LVM commands to deactivate any that have. For example, use the command vgchange -a n /dev/sgvg00 to deactivate the volume group sgvg00. Red Hat It is not necessary to prevent vgscan on Red Hat.
5.1.13.2 Initializing Disks for VxVM You need to initialize the physical disks that are employed in VxVM disk groups. To initialize a disk, log on to one node in the cluster, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: /usr/lib/vxvm/bin/vxdisksetup -i sda 5.1.13.
vxprint -g logdata The output of this command is shown in the following example: TY NAME ASSOC v pl logdata fsgen logdata-01 system KSTATE LENGTH ENABLED ENABLED 1024000 1024000 PLOFFS STATE TUTILO PUTILO ACTIVE ACTIVE NOTE: The specific commands for creating mirrored and multi-path storage using VxVM are described in the Veritas Volume Manager Reference Guide. 5.1.13.6 Creating File Systems If your installation uses file systems, create them next.
When all disk groups have been deported, you must issue the following command on all cluster nodes to allow them to access the disk groups: vxdctl enable 5.1.13.9 Re-Importing Disk Groups After deporting disk groups, they are not available for use on the node until they are imported again either by a package control script or with a vxdg import command.
The manpage for the cmquerycl command further explains the parameters that appear in this file. Many are also described in Chapter 4: “Planning and Documenting an HA Cluster ” (page 77). Modify your /etc/cmcluster/clust1.configfile as needed. 5.2.1 cmquerycl Options 5.2.1.1 Speeding up the Process In a larger or more complex cluster with many nodes, networks or disks, the cmquerycl command may take several minutes to complete.
cmquerycl -v -h ipv6 -C $SGCONF/clust1.conf -n ftsys9 -n ftsys10 • -h ipv4 tells Serviceguard to discover and configure only IPv4 subnets. If it does not find any eligible subnets, the command will fail. • -h ipv6 tells Serviceguard to discover and configure only IPv6 subnets. If it does not find any eligible subnets, the command will fail.
A cluster lock LUN or quorum server, is required for two-node clusters. To obtain a cluster configuration file that includes Quorum Server parameters, use the -q option of the cmquerycl command, specifying a Quorum Server hostname or IP address, for example (all on one line): cmquerycl -q -n ftsys9 -n ftsys10 -C .
15.13.164.0 15.13.172.0 15.13.165.0 15.13.182.0 15.244.65.0 15.244.56.0 lan1 lan1 lan1 lan1 lan2 lan2 lan2 lan2 lan3 lan3 lan4 lan4 (nodeA) (nodeB) (nodeC) (nodeD) (nodeA) (nodeB) (nodeC) (nodeD) (nodeA) (nodeB) (nodeC) (nodeD) lan3 lan3 lan3 lan3 (nodeA) (nodeB) (nodeC) (nodeD) IPv6: 3ffe:1111::/64 3ffe:2222::/64 Possible Heartbeat IPs: 15.13.164.0 15.13.164.1 15.13.164.2 15.13.172.0 15.13.172.158 15.13.172.159 15.13.165.0 15.13.165.1 15.13.165.2 15.13.182.0 15.13.182.158 15.13.182.
The heartbeat can comprise multiple IPv4 subnets joined by a router. In this case at least two heartbeat paths must be configured for each cluster node. See also the discussion of HEARTBEAT_IP (page 94), and “Cross-Subnet Configurations” (page 27). 5.2.6 Specifying Maximum Number of Configured Packages This value must be equal to or greater than the number of packages currently configured in the cluster. The count includes all types of packages: failover, multi-node, and system multi-node.
Figure 34 Access Roles 5.2.8.3 Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 34 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
IMPORTANT: Users on systems outside the cluster can gain Serviceguard root access privileges to configure the cluster only via a secure connection (rsh or ssh). • Non-root access: Other users can be assigned one of four roles: ◦ Full Admin: Allowed to perform cluster administration, package administration, and cluster and package view operations. These users can administer the cluster, but cannot configure or create a cluster. Full Admin includes the privileges of the Package Admin role.
Access control policies are defined by three parameters in the configuration file: • Each USER_NAME can consist either of the literal ANY_USER, or a maximum of 8 login names from the /etc/passwd file on USER_HOST. The names must be separated by spaces or tabs, for example: # Policy 1: USER_NAME john fred patrick USER_HOST bit USER_ROLE PACKAGE_ADMIN • USER_HOST is the node where USER_NAME will issue Serviceguard commands.
USER_HOST bit USER_ROLE PACKAGE_ADMIN If this policy is defined in the cluster configuration file, it grants user john the PACKAGE_ADMIN role for any package on node bit. User john also has the MONITOR role for the entire cluster, because PACKAGE_ADMIN includes MONITOR. If the policy is defined in the package configuration file for PackageA, then user john on node bit has the PACKAGE_ADMIN role only for PackageA. Plan the cluster’s roles and validate them as soon as possible.
5.2.8.5 Package versus Cluster Roles Package configuration will fail if there is any conflict in roles between the package configuration and the cluster configuration, so it is a good idea to have the cluster configuration file in front of you when you create roles for a package; use cmgetconf to get a listing of the cluster configuration file.
# Warning: Neither a quorum server nor a lock lun was specificed. # A Quorum Server or a lock lun is required for clusters of only two nodes. If you attempt to configure both a quorum server and a lock LUN, the following message appears on standard output when issuing the cmcheckconf or cmapplyconf command: Duplicate cluster lock, line 55. Quorum Server already specified. 5.2.
By default, cmruncl will check the networks. Serviceguard will probe the actual network configuration with the network information in the cluster configuration. If you do not need this validation, use cmruncl -v -w none instead, to turn off validation and save time 2. When the cluster has started, make sure that cluster components are operating correctly: cmviewcl -v Make sure that all nodes and networks are functioning as expected.
# # # # # # # to join it; if no cluster is running, the node attempts to form a cluster consisting of all configured nodes. Automatic cluster start is the preferred way to start a cluster. No action is required by the system administrator. If set to 1, the node will attempt to join/form its CM cluster automatically as described above. If set to 0, the node will not attempt to join its CM cluster. AUTOSTART_CMCLD=1 NOTE: The /sbin/init.
5.3.5 Disabling identd Ignore this section unless you have a particular need to disable identd. You can configure Serviceguard not to use identd. CAUTION: This is not recommended. Consult the white paper Securing Serviceguard at http:// www.hp.com/go/linux-serviceguard-docs (Select HP Serviceguard -> White Papers) for more information.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. For more information, see “What is Serviceguard for Linux? ” (page 19), “How the Package Manager Works” (page 41), and“Package Configuration Planning” (page 103).
To generate a package configuration file that creates a multi-node package, include-m sg/multi_node on the cmmakepkg command line. See “Generating the Package Configuration File” (page 201). • System multi-node packages. System multi-node packages are supported only for applications supplied by HP.
You can use cmmakepkg -l (letter “l”) to see a list of all available modules, including non-Serviceguard modules such as those supplied in the HP Toolkits. NOTE: If you are going to create a complex package that contains many modules, you may want to skip the process of selecting modules, and simply create a configuration file that contains all the modules: cmmakepkg -m sg/all $SGCONF/pkg_sg_complex (The output will be written to $SGCONF/pkg_sg_complex.) 6.1.3.
Table 7 Base Modules Module Name Parameters (page) Comments failover package_name (page 185) * module_name (page 185) * module_version (page 185) * package_type (page 185) package_description (page 185) * node_name (page 186) auto_run (page 186) node_fail_fast_enabled (page 186) run_script_timeout (page 187) halt_script_timeout (page 187) successor_halt_script_timeout (page 188) Base module. Use as primary building block for failover packages.
Table 8 Optional Modules Module Name Parameters (page) Comments dependency dependency_name (page 190) * dependency_condition (page 190) dependency_location (page 190) Add to a base module to create a package that depends on one or more other packages. weight weight_name (page 190) * weight value (page 190) * Add to a base module to create a package that has weight that will be counted against a node's capacity.
Table 8 Optional Modules (continued) Module Name Parameters (page) Comments all all parameters Use if you are creating a complex package that requires most or all of the optional parameters; or if you want to see the specifications and comments for all available parameters. multi_node_all all parameters that can be used by a multi-node package; includes multi_node, dependency, monitor_subnet, service, volume_group, filesystem, pev, external_pre, external, and acp modules.
NOTE: For more information, see the comments in the editable configuration file output by the cmmakepkg command, and the cmmakepkg (1m) manpage.
6.1.4.6 node_name The node on which this package can run, or a list of nodes in order of priority, or an asterisk (*) to indicate all nodes. The default is *. For system multi-node packages, you must specify node_name *. If you use a list, specify each node on a new line, preceded by the literal node_name, for example: node_name node_name node_name The order in which you specify the node names is important.
yes means the node on which the package is running will be halted (reboot) if the package fails; no means Serviceguard will not halt the system.
• Switching will be disabled. • The current node will be disabled from running the package. If a halt-script timeout occurs, you may need to perform manual cleanup. See Chapter 10: “Troubleshooting Your Cluster” (page 269). 6.1.4.11 successor_halt_timeout Specifies how long, in seconds, Serviceguard will wait for packages that depend on this package to halt, before halting this package. Can be 0 through 4294, or no_timeout. The default is no_timeout.
• site_preferred means Serviceguard will try all the eligible nodes on the local SITE before failing the package over to a node on another SITE. This policy can be configured only in a Metrocluster with site aware failover configuration; see the documents listed under “Cross-Subnet Configurations” (page 27) for more information. • site_preferred_manual means Serviceguard will try to fail the package over to a node on the local SITE.
IMPORTANT: Because priority is a matter of ranking, a lower number indicates a higher priority (20 is a higher priority than 40). A numerical priority is higher than no_priority. This parameter is new as of A.11.18. See also “About Package Dependencies” (page 111). 6.1.4.18 dependency_name A unique identifier for a particular dependency (see dependency_condition) that must be met in order for this package to run (or keep running). It must be unique among this package's dependency_names.
NOTE: But if weight_name is package_limit, you can use only that one weight and capacity throughout the cluster. package_limit is a reserved value, which, if used, must be entered exactly in that form. It provides the simplest way of managing weights and capacities; see “Simple Method” (page 119) for more information. The rules for forming weight_name are the same as those for forming package_name (page 185). weight_name must exactly match the corresponding CAPACITY_NAME.
6.1.4.24 ip_subnet Specifies an IP subnet used by the package. CAUTION: HP recommends that this subnet be configured into the cluster. You do this in the cluster configuration file by specifying a HEARTBEAT_IP or STATIONARY_IP under a NETWORK_INTERFACE on the same subnet, for each node in this package's NODE_NAME list. For example, an entry such as the following in the cluster configuration file configures subnet 192.10.25.0 (lan1) on node ftsys9: NODE_NAME ftsys9 NETWORK_INTERFACE lan1 HEARTBEAT_IP 192.10.
6.1.4.26 ip_address A relocatable IP address on a specified ip_subnet. For more information about relocatable IP addresses, see “Stationary and Relocatable IP Addresses and Monitored Subnets” (page 60). This parameter can be set for failover packages only. 6.1.4.27 service_name A service is a program or function which Serviceguard monitors as long the package is up. service_name identifies this function and is used by the cmrunserv and cmhaltserv commands.
NOTE: Be careful when defining service run commands. Each run command is executed in the following way: • The cmrunserv command executes the run command. • Serviceguard monitors the process ID (PID) of the process the run command creates. • When the command exits, Serviceguard determines that a failure has occurred and takes appropriate action, which may include transferring the package to an adoptive node.
The following is an example of defining generic resource parameters: generic_resource_name generic_resource_evaluation_type generic_resource_up_criteria cpu_monitor during_package_start <50 See the package configuration file for more examples. 6.1.4.33 generic_resource_evaluation_type Defines when the status of a generic resource is evaluated. Valid values are during_package_start and before_package_start. The default is during_package_start.
• generic_resource_up_criteria is an optional attribute. It determines whether a given generic resource is a simple generic resource or an extended generic resource. It is not specified for a simple resource, but is required for an extended resource. • A single package can contain both simple and extended resources. • A given resource cannot be configured as a simple generic resource in one package and as an extended generic resource in another package.
6.1.4.41 kill_processes_accessing_raw_devices Specifies whether or not to kill processes that are using raw devices (for example, database applications) when the package shuts down. Default is no. See the comments in the package configuration file for more information. 6.1.4.42 File system parameters A package can activate one or more storage groups on startup, and to mount logical volumes to file systems. At halt time, the package script unmounts the file systems and deactivates each storage group.
6.1.4.46 fs_name This parameter, in conjunction with fs_directory, fs_type, fs_mount_opt, fs_umount_opt, and fs_fsck_opt, specifies a filesystem that is to be mounted by the package. fs_name must specify the block devicefile for a logical volume. For an NFS-imported file system, the additional parameters required are fs_server, fs_directory, fs_type, and fs_mount_opt; see fs_server (page 198) for an example.
Table 9 File System Types, Commands, and Platforms File system type fsck command Supported platform ext3 e2fsck Red Hat Enterprise Linux 5 Red Hat Enterprise Linux 6 SUSE Linux Enterprise Server 11 ext4 e4fsck Red Hat Enterprise Linux 51 Red Hat Enterprise Linux 6 XFS xfs_repair Red Hat Enterprise Linux 6 SUSE Linux Enterprise Server 11 Btrfs2 btrfsck Red Hat Enterprise Linux 6 and later SUSE Linux Enterprise Server 11 SP2 VxFS file system fsck Red Hat Enterprise Linux 5 Red Hat Enterprise
IMPORTANT: This parameter is for use only by HP partners, who should follow the instructions in the package configuration file. For information about Serviceguard's implementation of PR, see “About Persistent Reservations” (page 71). 6.1.4.54 pev_ Specifies a package environment variable that can be passed to external_pre_script, external_script, or both, by means of the cmgetpkgenv command. The variable name must be in the form pev_ and contain only alphanumeric characters and underscores.
6.1.4.58 user_name Specifies the name of a user who has permission to administer this package. See also user_host (page 200) and user_role; these three parameters together define the access control policy for this package (see “Controlling Access to the Cluster” (page 169)). These parameters must be defined in this order: user_name, user_host, user_role. Legal values for user_name are any_user or a maximum of eight login names from /etc/ passwd on user_host.
• To generate a configuration file that contains all the optional modules: cmmakepkg $SGCONF/pkg1/pkg1.conf • To create a generic failover package (that could be applied without editing): cmmakepkg -n pkg1 -m sg/failover $SGCONF/pkg1/pkg1.
NOTE: etc. 4. 5. 6. cmcheckconf and cmapplyconf check for missing mount points, volume groups, Halt the package. Configure package IP addresses and application services. Run the package and ensure that applications run as expected and that the package fails over correctly when services are disrupted. See “Testing the Package Manager ” (page 269).
• If this package will depend on another package or packages, enter values for dependency_name, dependency_condition, dependency_location, and optionally priority. See “About Package Dependencies” (page 111) for more information. NOTE: The package(s) this package depends on must already be part of the cluster configuration by the time you validate this package (via cmcheckconf; see “Verifying and Applying the Package Configuration” (page 206)); otherwise validation will fail.
• If the package needs to mount LVM volumes to file systems (see fs_type (page 198)), use the vg parameters to specify the names of the volume groups to be activated, and select the appropriate vgchange_cmd. Use the fs_ parameters (page 198) to specify the characteristics of file systems and how and where to mount them. See the comments in the FILESYSTEMS section of the configuration file for more information and examples.
3. 4. Edit the package configuration file and specify the external_script parameter. Re-apply the package configuration: cmapplyconf -P pkg1_v2.conf To remove a module from an existing package, use the cmmakepkg command to generate a new configuration file excluding the module that you want to remove. Then, copy the remaining package attributes from the old configuration file to the new configuration file and re-apply the package configuration. 6.
NOTE: Ensure that the filesystem is clean before adding any filesystem to the package configuration file. To do so, follow these steps: 1. Activate the volume group on which the filesystem is created: vgchange --addtag vgchange -a y 2. Run the filesystem specific fsck command for ext2, ext3, and ext4 filesystems: fsck_command Where, fsck_command can be different depending on the filesystem type.
Oracle and NFS Toolkits Environment For information about alert notification on Oracle and NFS toolskits environment, see the following documents at http://www.hp.com/go/linux-serviceguard-docs: • HP Serviceguard Toolkit for Oracle version A.05.01.10 on Linux User Guide • HP Serviceguard Toolkit for NFS version A.03.03.10 on Linux User Guide serviceguard-xdc Environment By default, this parameter is commented and is present in the package configuration file for the serviceguard-xdc packages.
The control script imports disk groups using the vxdg command with the -tfC options. The -t option specifies that the disk is imported with the noautoimport flag, which means that the disk will not be automatically re-imported at boot time. Since disk groups included in the package control script are only imported by Serviceguard packages, they should not be auto-imported. The -foption allows the disk group to be imported even if one or more disks (a mirror, for example) is not currently available.
NOTE: The service_cmd entry must include the cmresserviced command. It is also important to set service_restart to none.
7 Cluster and Package Maintenance This chapter describes the cmviewcl command, then shows how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
• starting - The cluster is in the process of determining its active membership. At least one cluster daemon is running. • unknown - The node on which the cmviewcl command is issued cannot communicate with other nodes in the cluster. 7.1.4 Node Status and State The status of a node is either up (active as a member of the cluster) or down (inactive in the cluster), depending on whether its cluster daemon is running or not.
• detached - A package is said to be detached from the cluster or node where it was running, when the cluster or node is halted with —d option. Serviceguard no longer monitors this package. The last known status of the package before it is detached from the cluster was up. • unknown - Serviceguard could not determine the status at the time cmviewcl was run. A system multi-node package is up when it is running on all the active cluster nodes.
7.1.6 Package Switching Attributes cmviewcl shows the following package switching information: • AUTO_RUN: Can be enabled or disabled. For failover packages, enabled means that the package starts when the cluster starts, and Serviceguard can switch the package to another node in the event of failure. For system multi-node packages, enabled means an instance of the package can start on a new node joining the cluster (disabled means it will not).
Failover packages can also be configured with one of two values for the failback_policy parameter (page 189), and these are also displayed in the output of cmviewcl -v: • automatic: Following a failover, a package returns to its primary node when the primary node becomes available again. • manual: Following a failover, a package will run on the adoptive node until moved back to its original node by a system administrator. 7.1.
NOTE: The Script_Parameters section of the PACKAGE output of cmviewcl shows the Subnet status only for the node that the package is running on. In a cross-subnet configuration, in which the package may be able to fail over to a node on another subnet, that other subnet is not shown (see “Cross-Subnet Configurations” (page 27)). 7.1.11.
UNOWNED_PACKAGES PACKAGE pkg2 STATUS down STATE unowned AUTO_RUN disabled NODE unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS NODE_NAME Service down Generic Resource up ftsys9 Subnet up Generic Resource up ftsys10 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NAME service2 sfm_disk1 15.13.168.
Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM Service Service up Subnet Generic Resource up STATUS up 0 up MAX_RESTARTS RESTARTS NAME 0 0 0 sfm_disk_monitor 0 0 sfm_disk Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up NAME ftsys10 ftsys9 service2 15.13.168.0 (current) STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NAME eth0 eth1 7.1.11.
7.1.11.7 Viewing Information about Unowned Packages The following example shows packages that are currently unowned, that is, not running on any configured node.
7.1.12.1 Verifying Cluster and Package Components Table 10 describes how to verify each cluster and package component, the command or tool to use and its description. Table 10 Verifying Cluster and Package Components Component (Context) Tool or Command; More Information Description Volume groups (package) cmcheckconf (1m), cmapplyconf (1m) Verifies for the following: See also “Verifying the Cluster Configuration ” (page 174). • availability across all the nodes where the package is configured to run.
Table 10 Verifying Cluster and Package Components (continued) Component (Context) Tool or Command; More Information Description File consistency (cluster) cmcheckconf (1m), cmcompare (1m). To verify file consistency across all the nodes in the cluster: IMPORTANT: See the manpage for 1. Customize the $SGCONF/ differences in return codes from cmclfiles2check file. cmcheckconf without options versus cmcheckconf -C 2. Distribute it to all the nodes using the cmsync (1m) command. 3.
Table 10 Verifying Cluster and Package Components (continued) Component (Context) Tool or Command; More Information Description NFS server connectivity (package) cmcheckconf (1m), cmapplyconf (1m) If the package configuration file contains NFS file system, it validates the following: • Connectivity to the NFS server from all the package nodes. • Export of share by the NFS server. • The status of the NFS daemons on the NFS server.
• Unreachable DNS server. • Consistency of settings in .rhosts. • Nested mount points. 7.
cmruncl -v -n ftsys9 -n ftsys10 CAUTION: HP Serviceguard cannot guarantee data integrity if you try to start a cluster with the cmruncl -n command while a subset of the cluster's nodes are already running a cluster. If the network connection is down between nodes, using cmruncl -n might result in a second cluster forming, and this second cluster might start up the same applications that are already running on the other cluster. The result could be two applications overwriting each other's data on the disks.
This halts any packages running on the node ftsys9 by executing the halt instructions in each package's master control script. ftsys9 is halted and the packages start on the adoptive node, ftsys10. 7.2.4 Halting the Entire Cluster You can use Serviceguard Manager, or Serviceguard commands as shown below, to halt a running cluster. The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their HP Serviceguard daemons.
• Restart normal package monitoring by restarting the node (cmrunnode) or the cluster (). • You can forcefully halt a detached node (cmhaltnode (1m)) with the -f option. 7.3.2 Rules and Restrictions The following rules and restrictions apply. • All the nodes in the cluster must be running Serviceguard A.11.20.10 or later. • All the configured cluster nodes must be reachable by an available network.
you would need to run cmhaltpkg (1m) to halt the package on the node where it is detached. • You cannot halt a package that is in a transitory state such as STARTING or HALTING. For more information about package states, see “Package Status and State” (page 212). • A package that is in a DETACHED or MAINTENANCE state cannot be moved to a halt_aborted state or vice versa. For more information, see “Handling Failures During Package Halt” (page 230). 7.3.
• • When a node having detached packages is back up after a reboot they can: ◦ Rejoin the cluster and the detached packages can move to "running" or "failed" state. If the detached packages are moved to running state, then they must be halted and rerun as they may have several inconsistencies post reboot. ◦ Not rejoin the cluster and the detached packages remain detached. Such packages must be halted and rerun to avoid any inconsistencies that can be caused due to the reboot.
NOTE: 3. If you do not do this, the cmhaltcl in the next step will fail. Halt the cluster with the -d (detach) option: cmhaltcl -d NOTE: -d and -f are mutually exclusive. See cmhaltcl (1m) for more information. To re-attach the packages, restart cluster: cmrunnode node1 7.3.
You can use Serviceguard Manager to start a package, or Serviceguard commands as shown below. Use the cmrunpkg command to run the package on a particular node, then use the cmmodpkg command to enable switching for the package; for example: cmrunpkg -n ftsys9 pkg1 cmmodpkg -e pkg1 This starts up the package on ftsys9, then enables package switching. This sequence is necessary when a package has previously been halted on some node, since halting the package disables switching. 7.4.1.
NOTE: Non-native Serviceguard modules are those that are not delivered with the Serviceguard product. These are additional modules such as those supplied with HP Serviceguard toolkit modules (for example, HP Serviceguard Contributed Toolkit Suite, Oracle, NFS toolkit, EDB PPAS, Sybase, and so on). This allows errors to be cleaned up manually during the halt process thus minimizing the risk of other follow on errors and reducing package downtime.
After it halts, run the package on the new node using the cmrunpkg command, then re-enable switching as described below. 7.4.4 Changing Package Switching Behavior There are two options to consider: • Whether the package can switch (fail over) or not. • Whether the package can switch to a particular node or not. For failover packages, if package switching is NO the package cannot move to any other node; if node switching is NO, the package cannot move to that particular node.
• Neither maintenance mode nor partial-startup maintenance mode can be used for multi-node packages, or system multi-node packages. • Package maintenance does not alter the configuration of the package, as specified in the package configuration file. For information about reconfiguring a package, see “Reconfiguring a Package” (page 244). NOTE: In order to run a package in partial-startup maintenance mode, you must first put it in maintenance mode.
• The package must have package switching disabled before you can put it in maintenance mode. • You can put a package in maintenance mode only on one node. ◦ The node must be active in the cluster and must be eligible to run the package (on the package's node_name list). ◦ If the package is not running, you must specify the node name when you run cmmodpkg (1m) to put the package in maintenance mode.
7.5.1.2 Dependency Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode You cannot configure new dependencies involving a package running in maintenance mode, and in addition the following rules apply (we'll call the package in maintenance mode pkgA). • The packages that depend on pkgA must be down and disabled when you place pkgA in maintenance mode.
7.5.3 Performing Maintenance Using Partial-Startup Maintenance Mode To put a package in partial-startup maintenance mode, you put it in maintenance mode, then restart it, running only those modules that you will not be working on. 7.5.3.1 Procedure Follow this procedure to perform maintenance on a package. In this example, we'll assume a package pkg1 is running on node1, and that we want to do maintenance on the package's services. 1. Halt the package: cmhaltpkg pkg1 2.
You can also use -e in combination with -m. This has the effect of starting all modules up to and including the module identified by -m, except the module identified by -e. In this case the excluded (-e) module must be earlier in the execution sequence (as listed near the top of the package's configuration file) than the -m module. For example: cmrunpkg -m sg/services -e sg/package_ip pkg1 NOTE: The full execution sequence for starting a package is: 1. The master control script itself 2.
7.6.1 Previewing the Effect of Cluster Changes Many variables affect package placement, including the availability of cluster nodes; the availability of networks and other resources on those nodes; failover and failback policies; and package weights, dependencies, and priorities, if you have configured them. You can preview the effect on packages of certain actions or events before they actually occur.
In the output of cmviewcl -v -f line, you would find the line package:pkg1|autorun=disabled and change it to package:pkg1|autorun=enabled. You should also make sure that the nodes the package is configured to run on are shown as available; for example: package:pkg1|node:node1|available=yes. Then save the file (for example, as newstate.in) and run cmeval: cmeval -v newstate.
7.6.3.1 Adding Nodes to the Configuration While the Cluster is Running Use the following procedure to add a node. For this example, nodes ftsys8 and ftsys9 are already configured in a running cluster named cluster1, and you are adding node ftsys10. NOTE: Before you start, make sure you have configured access to ftsys10 as described under “Configuring Root-Level Access” (page 144). 1.
4. Halt the node you are going to remove (ftsys10in this example): cmhaltnode -f -v ftsys10 5. Verify the new configuration: cmcheckconf -C clconfig.conf 6. From ftsys8 or ftsys9, apply the changes to the configuration and distribute the new binary configuration file to all cluster nodes.: cmapplyconf -C clconfig.
• You cannot change the designation of an interface from STATIONARY_IP to HEARTBEAT_IP unless the subnet is common to all nodes. Remember that the HEARTBEAT_IP must be an IPv4 address, and must be on the same subnet on all nodes, except in cross-subnet configurations; see “Cross-Subnet Configurations” (page 27)). • You cannot delete a subnet or IP address from a node while a package that uses it (as a monitored_subnet, ip_subnet, or ip_address) is configured to run on that node.
2. Edit the file to uncomment the entries for the subnet that is being added (lan0 in this example), and change STATIONARY_IP to HEARTBEAT_IP: NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE 3. ftsys9 lan1 192.3.17.18 lan0 15.13.170.18 lan3 ftsys10 lan1 192.3.17.19 lan0 15.13.170.19 lan3 Verify the new configuration: cmcheckconf -C clconfig.conf 4.
5. Apply the changes to the configuration and distribute the new binary configuration file to all cluster nodes: cmapplyconf -C clconfig.conf 7.6.5 Updating the Cluster Lock LUN Configuration Online Proceed as follows. IMPORTANT: See “What Happens when You Change the Quorum Configuration Online” (page 41) for important information. 1. 2. 3. In the cluster configuration file, modify the value of CLUSTER_LOCK_LUN for each node. Run cmcheckconf to check the configuration.
that it had activated at startup, because it would only see the one volume group in its current configuration file. For more information, see “Allowable Package States During Reconfiguration ” (page 247). 7.7.1 Reconfiguring a Package on a Running Cluster You can reconfigure a package while the cluster is running, and in some cases you can reconfigure the package while the package itself is running; see “Allowable Package States During Reconfiguration ” (page 247).
6. You can now safely delete the original external script on all nodes that are configured to run the package. 7.7.3 Reconfiguring a Package on a Halted Cluster You can also make permanent changes in the package configuration while the cluster is not running. Use the same steps as in “Reconfiguring a Package on a Running Cluster ”. 7.7.4 Adding a Package to a Running Cluster You can create a new package and add it to the cluster configuration while the cluster is up and while other packages are running.
7.7.7 Allowable Package States During Reconfiguration In many cases, you can make changes to a package’s configuration while the package is running. The table that follows shows exceptions — cases in which the package must not be running, or in which the results might not be what you expect. CAUTION: is running.
Table 12 Types of Changes to Packages (continued) Change to the Package Required Package State Add or remove an ip_subnet: Package can be running. See “ip_subnet” (page 192) for important information. Serviceguard will reject the change if you are trying to add an ip_subnet that is not configured on all the nodes on the package's node_name list. Add or remove an ip_address: Package can be running. See “ip_subnet” (page 192) and “ip_address” (page 193) for important information.
Table 12 Types of Changes to Packages (continued) Change to the Package Required Package State Add, change, or delete external scripts and pre-scripts Package can be running. Change package auto_run Package can be either running or halted. Changes take effect when applied, whether or not the package is running. If you add a script, Serviceguard validates it and then (if there are no errors) runs it when you apply the change. If you delete a script, Serviceguard stops it when you apply the change.
Table 12 Types of Changes to Packages (continued) Change to the Package Required Package State Change modular serviceguard-xdc package parameters: Package can be running. See “Online Reconfiguration of serviceguard-xdc Modular Package Parameters” (page 110). xdc/xdc/rpo_target xdc/xdc/raid_monitor_interval xdc/xdc/raid_device xdc/xdc/device_0 xdc/xdc/device_1 Add, change, or delete an email attribute Package can be running. NOTE: Do not include the email module in the modular package.
To handle the failure during online package reconfiguration, see “Handling Failures During Online Package Reconfiguration”. Recommendations • HP recommends that when modifying package parameters online, the modification must be done on one module at a time and also from one package only. • You must consider only one package for online reconfiguration at a time.
reconfiguration of the package. This option is applicable for both failover and multi-node packages. For example, if you enter a wrong fs_type value while adding a new filesystem to the pkg1. ***************************** Package log during this time: ***************************** Nov 28 23:41:19 root@test1.ind.hp.com master_control_script.sh[23516]: ###### reconfiguring package pkg1 ###### Nov 28 23:41:20 root@test1.ind.hp.com pr_util.sh[23621]: New VG vg_dd0 Nov 28 23:41:20 root@test1.ind.hp.com pr_util.
Table 13 Modules affected during online addition Description Modules affected during online addition Extending MD ext/xdc(xdc.sh) (for serviceguard-xdc packages only) Adding an external pre script to the package How to rectify the failures or apply those that are not attempted Example You cannot rectify the failures in XDC module manually and must restart the package. — sg/external_pre_script If an external pre script which is (external.
Table 13 Modules affected during online addition (continued) sg/external_script Adding an external script (external.sh) to the package If an external script which is added to the package configuration failed to start, run the script manually with start option. To halt external pre script: #extern_pre_script.sh stop script_name start Adding a service to the package sg/service (service.
Table 14 Modules affected during online deletion (continued) Removing storage from the package sg/filesystem (filesystem.sh) sg/volume_group (volume_group.sh) sg/pr_cntl (pr_cntl.sh) If storage deleted from the To unmount the mount point mnt1: package has failed or not #umount /mnt1 attempted, ensure the following: To delete the hostags from the • The mount point is vg_dd0 on node test1.ind.hp.com: unmounted. #vgchange --deltag • Delete the hosttagss from the test1.ind.hp.com vg_dd1 disk.
node will probably have applications running on it. As long as the Serviceguard daemon cmcld is active, other nodes can rejoin the cluster. If the Serviceguard daemon fails when the cluster is in single-node operation, it will leave the single node up and your applications running NOTE: This means that Serviceguard itself is no longer running.
8 Simulating a Serviceguard Cluster Cluster simulation allows administrators to simulate different kinds of failures, such as node, network, and workload failures (that is, Serviceguard packages). Cluster simulation is capable of simulating node and network interface failures. Cluster simulation evaluates and analyzes what happens to the package due to simulated failures, whether the packages will failover and start on a specified node.
8.1 Simulating the Cluster This section describes how to create, import, halt, run, and delete the cluster in the simulator view. You can simulate a cluster in two ways: • Create a cluster using cmapplyconf in a simulation environment • Create a cluster by importing the existing cluster into simulation environment Session name is same as the cluster name. For information about how to create cluster configuration file, see Cluster Configuration Parameters and the cmsimulatecl (1m) manpage.
8.1.2.3 Import the Status of the Cluster State Stored in a File You can import the state of the cluster stored in the file using -l option. The state of any cluster can be saved on a file by redirecting the output of the cmviewcl -v -f line command to file. #cmsimulatecl importcluster -l test_cluster_cmviewcl_line Cluster test_cluster is imported. Please use cmviewcl to view the cluster 8.1.2.
NOTE: If no nodes are set by the user, the first node in the cluster configuration file will be chosen to run all the simulator commands. • To return an active node to the cluster, on which all the simulator commands can be executed. For example, to check on which node in "test_cluster" you can run the simulator commands: #cmsimulatecl --session test_cluster getnode test1 is active node for session test_cluster.
Running package PKG1 on node test1 Successfully started package PKG1 on node test1 cmrunpkg: All specified packages are running For more information, see the cmrunpkg (1m) manpage. 8.2.3 Halting a Package You can halt a package using the following command: #cmsimulatecl --session test_cluster cmhaltpkg PKG1 One or more packages or package instances have been halted cmhaltpkg: Completed successfully on all packages specified For more information, see the cmhaltpkg (1m) manpage. 8.2.
NOTE: To verify whether the failure of PKG1 affects any other packages: #cmsimulatecl --session cmviewcl • To recover from the last package failure: #cmsimulatecl --session test_cluster recover -p PKG1 Failure and recovery of an interface • To simulate an interface failure on a node: #cmsimulatecl --session test_cluster fail -n test1 -i eth0 NOTE: To verify whether the packages running on test1 failed over or not, and where the packages are currently running: #cmsimulatecl --session test_cluster cmview
9 Cluster Analytics Serviceguard cluster analytics provides a mechanism to the users to perform "availability audits" on Serviceguard cluster, nodes, and application packages running on the clusters. Key Performance Indicators and Key Events of Cluster Analytics Key Performance Indicators, also known as KPIs, is a metric of Serviceguard cluster object, such as cluster, node, or package for which a value can be obtained from the cluster analytics engine for a specified time period.
Table 15 KPIs of the Cluster, Nodes, and Packages (continued) UNAVAILABILITY The percentage of time duration for which node was unavailable due to technical issues for the total time specified. By default, the total time is considered to be the difference between current time and time at which node is added to the cluster. You can also specify the start and end time for which the KPI value will be computed.
9.1 Installing the Cluster Analytics Software 9.1.1 Pre-requisites Following are the prerequisites for installing serviceguard-analytics version A.12.00.00: • sqlite-3.3 and later for Red Hat Enterprise Linux 5 • sqlite-3.6 and later for Red Hat Enterprise Linux 6 • sqlite3-3.7 and later for SUSE Linux Enterprise Server 11 9.1.2 Installing serviceguard-analytics Software To install serviceguard-analytics software: 1.
start command from one of the nodes on a cluster to start the cluster analytics daemon. Cluster Analytics daemon gets started only in cluster coordinator node.
9.4 Verifying Cluster Analytics Daemon To verify whether cluster analytics daemon is running on any node in a cluster: #cmcaadmin status This command is used to check the state of the cluster analytics daemon. It also provides the information whether the cluster analytics daemon is running or not.
9.6 Limitations • The cluster analytics daemon is not monitored. Therefore, any failure of daemon or node on which it is hosted requires manual restart of the daemon, which results in cluster event database re-initialization. • The internal event database of cluster analytics daemon is stored locally on the node where it is hosted. If the node crashes, all the data related to the cluster analytics daemon is lost. • The data retrieval operation on cluster event database file is not supported.
10 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
1. Obtain the generic resource that is configured in a package by entering cmviewcl -v -p 2. Set the status of generic resource to DOWN using the following command: cmsetresource -r –s down 3. To view the package status, enter cmviewcl -v The package should be running on the specified adoptive node. 4. Move the package back to the primary node (see “Moving a Failover Package ” (page 231)).
Some monitoring can be done through simple physical inspection, but for the most comprehensive monitoring, you should examine the system log file (/var/log/messages) periodically for reports on all configured HA devices. The presence of errors relating to a device will show the need for maintenance. 10.3 Replacing Disks The procedure for replacing a faulty disk mechanism depends on the type of disk configuration you are using. Refer to your Smart Array documentation for issues related to your Smart Array.
Invoke the script as follows, specifying either the device special file (DSF) of a LUN, or a file containing a list of DSF names: pr_cleanup lun -v -k [-f | ] • lun, if used, specifies that a LUN, rather than a volume group, is to be operated on. • -v, if used, specifies verbose output detailing the actions the script performs and their status. • -k , if used, specifies the key to be used in the clear operation.
Now Serviceguard will detect that the MAC address (LLA) of the card has changed from the value stored in the cluster binary configuration file, and it will notify the other nodes in the cluster of the new MAC address. The cluster will operate normally after this. HP recommends that you update the new MAC address in the cluster binary configuration file by re-applying the cluster configuration. Use the following steps for online reconfiguration: 1.
6. The quorum server log file on the new quorum server will show a log message like the following for each cluster that uses the quorum server: Request for lock /sg/ succeeded. New lock owners: N1, N2 7. To check that the quorum server has been correctly configured and to verify the connectivity of a node to the quorum server, you can execute the following command from your cluster nodes as follows: cmquerycl -q -n -n ...
eth1:1 Link encap:Ethernet HWaddr 00:50:DA:64:8A:7C inet addr:192.168.1.200 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:9 Base address:0xda00 eth1:2 Link encap:Ethernet HWaddr 00:50:DA:64:8A:7C inet addr:192.168.1.201 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:9 Base address:0xda00 lo Link encap:Local Loopback inet addr:127.0.0.1 Bcast:192.168.1.255 Mask:255.255.255.
Dec 14 14:39:36 star04 CM-pkg4[2127]: cmrunserv Service4 /vg01/MyPing 127.0.0.1 >>/dev/null Dec 14 14:39:36 star04 cmcld[2098]: Started package pkg4 on node star04. 10.7.3 Reviewing Configuration Files Review the following ASCII configuration files: • Cluster configuration file. • Package configuration files. Ensure that the files are complete and correct according to your configuration planning worksheets. 10.7.
• Package Movement Errors. • Node and Network Failures. • Quorum Server Messages. 10.8.1 Name Resolution Problems Many Serviceguard commands, including cmviewcl, depend on name resolution services to look up the addresses of cluster nodes. When name services are not available (for example, if a name server is down), Serviceguard commands may hang, or may return a network-related error message. If this happens, use the host command on each cluster node to see whether name resolution is correct.
1. Warning: cmcld was unable to run for the last seconds. Consult the Managing Serviceguard manual for guidance on setting MEMBER_TIMEOUT, and information on cmcld. This means that cmcld was unable to get access to a CPU for a significant amount of time. If this occurred while the cluster was re-forming, one or more nodes could have failed.
10.8.5.1 Package Control Script Hangs or Failures When a RUN_SCRIPT_TIMEOUT or HALT_SCRIPT_TIMEOUT value is set, and the control script hangs, causing the timeout to be exceeded, Serviceguard kills the script and marks the package “Halted.” Similarly, when a package control script fails, Serviceguard kills the script and marks the package “Halted.” In both cases, the following also take place: • Control of the package will not be transferred. • The run or halt instructions may not run to completion.
The default Serviceguard control scripts are designed to take the straightforward steps needed to get an application running or stopped. If the package administrator specifies a time limit within which these steps need to occur and that limit is subsequently exceeded for any reason, Serviceguard takes the conservative approach that the control script logic must either be hung or defective in some way.
Unable to set client version at quorum server 192.6.7.2: reply timed out Probe of quorum server 192.6.7.2 timed out These messages could be an indication of an intermittent network problem; or the default quorum server timeout may not be sufficient. You can set the QS_TIMEOUT_EXTENSION to increase the timeout, or you can increase the MEMBER_TIMEOUT value. See “Cluster Configuration Parameters” (page 89)for more information about these parameters.
A Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 284) • Designing Applications to Run on Multiple Systems (page 287) • Restoring Client Connections (page 290) • Handling Application Failures (page 291) • Minimizing Planned Downtime (page 292) Designing for high availability means reducing
• Minimize the reentry of data. • Engineer the system for reserve capacity to minimize the performance degradation experienced by users. A.1.2 Define Application Startup and Shutdown Applications must be restartable without manual intervention. If the application requires a switch to be flipped on a piece of hardware, then automated restart is impossible. Procedures for application startup, shutdown and monitoring must be created so that the HA software can perform these functions automatically.
running the application. After failover, if these data disks are filesystems, they must go through filesystems recovery (fsck) before the data can be accessed. To help reduce this recovery time, the smaller these filesystems are, the faster the recovery will be. Therefore, it is best to keep anything that can be replicated off the data filesystem. For example, there should be a copy of the application executables on each system rather than having one copy of the executables on a shared filesystem.
the beginning. This capability makes the application more robust and reduces the visibility of a failover to the user. A common example is a print job. Printer applications typically schedule jobs. When that job completes, the scheduler goes on to the next job.
A.2.7 Design for Replicated Data Sites Replicated data sites are a benefit for both fast failover and disaster recovery. With replicated data, data disks are not shared between systems. There is no data recovery that has to take place. This makes the recovery time faster. However, there may be performance trade-offs associated with replicating data. There are a number of ways to perform data replication, which should be fully investigated by the application designer.
A.3.1.1 Obtain Enough IP Addresses Each application receives a relocatable IP address that is separate from the stationary IP address assigned to the system itself. Therefore, a single system might have many IP addresses, one for itself and one for each of the applications that it normally runs. Therefore, IP addresses in a given subnet range will be consumed faster than without high availability. It might be necessary to acquire additional IP addresses.
over time if the application migrates. Applications that use gethostname() to determine the name for a call to gethostbyname(3) should also be avoided for the same reason. Also, the gethostbyaddr() call may return different answers over time if called with a stationary IP address. Instead, the application should always refer to the application name and relocatable IP address rather than the hostname and stationary IP address.
With UDP datagram sockets, however, there is a problem. The client may connect to multiple servers utilizing the relocatable IP address and sort out the replies based on the source IP address in the server’s response message. However, the source IP address given in this response will be the stationary IP address rather than the relocatable application IP address.
give up after 2 minutes and go for coffee and don't come back for 28 minutes, the perceived downtime is actually 30 minutes, not 5. Factors to consider are the number of reconnection attempts to make, the frequency of reconnection attempts, and whether or not to notify the user of connection loss. There are a number of strategies to use for client reconnection: • Design clients which continue to try to reconnect to their failed server.
Ideally, if one process fails, the other processes can wait a period of time for that component to come back online. This is true whether the component is on the same system or a remote system. The failed component can be restarted automatically on the same system and rejoin the waiting processing and continue on. This type of failure can be detected and restarted within a few seconds, so the end user would never know a failure occurred.
The trade-off is that the application software must operate with different revisions of the software. In the above example, the database server might be at revision 5.0 while the some of the application servers are at revision 4.0. The application must be designed to handle this type of situation. A.6.1.2 Do Not Change the Data Layout Between Releases Migration of the data to a new format can be very time intensive. It also almost guarantees that rolling upgrade will not be possible.
B Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the appendix “Designing Highly Available Cluster Applications.” 2.
B.1.1 Defining Baseline Application Behavior on a Single System 1. Define a baseline behavior for the application on a standalone system: • Install the application, database, and other required resources on one of the systems. Be sure to follow Serviceguard rules in doing this: ◦ Install all shared data on separate external volume groups. ◦ Use a Journaled filesystem (JFS) as appropriate. • Perform some sort of standard test to ensure the application is running correctly.
# cmhaltpkg pkg1 # cmrunpkg -n node1 pkg1 # cmmodpkg -e pkg1 2. 3. • Fail one of the systems. For example, turn off the power on node 1. Make sure the package starts up on node 2. • Repeat failover from node 2 back to node 1. Be sure to test all combinations of application load during the testing. Repeat the failover processes under different application states such as heavy user load versus no user load, batch jobs versus online transactions, etc.
C Blank Planning Worksheets This appendix reprints blank versions of the planning worksheets described in the “Planning” chapter. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply _______________________ ============================================================================ Tape Backup Power: Tape Unit __________________________ Power Supply _______________________ Tape Unit __________________________
Physical Volume Name: _________________ Physical Volume Name: _________________ ============================================================================= Volume Group Name: ___________________________________ Physical Volume Name: _________________ Physical Volume Name: _________________ Physical Volume Name: _________________ C.
Access Policies: User:_________________ From node:_______ Role:_____________________________ User:_________________ From node:_______ Role:______________________________________________ Log level____ Log file:_______________________________________________________________________________________ Priority_____________ Successor_halt_timeout____________ dependency_name _____ dependency_condition _____ dependency_location _______ ========================================================================== LVM Vo
D IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses, specifically: • IPv6 Address Types • Network Configuration Restrictions (page 306) • Configuring IPv6 on Linux (page 306) D.1 IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces. There are various address formats for IPv6 defined by the RFC 2373.
D.1.2 IPv6 Address Prefix IPv6 Address Prefix is similar to CIDR in IPv4 and is written in CIDR notation. An IPv6 address prefix is represented by the notation: IPv6-address/prefix-length where ipv6-address is an IPv6 address in any notation listed above and prefix-length is a decimal value representing how many of the leftmost contiguous bits of the address comprise the prefix. Example: fec0:0:0:1::1234/64 The first 64-bits of the address fec0:0:0:1 forms the address prefix.
Example: ::ffff:192.168.0.1 D.1.4.3 Aggregatable Global Unicast Addresses The global unicast addresses are globally unique IPv6 addresses. This address format is very well defined in the RFC 2374 (An IPv6 Aggregatable Global Unicast Address Format). The format is: 3 13 8 24 16 64 bits FP TLA ID RES NLA ID SLA ID Interface ID where FP = Format prefix. Value of this is “001” for Aggregatable Global unicast addresses. TLA ID = Top-level Aggregation Identifier. RES = Reserved for future use.
The “scop” field is a 4-bit field which is used to limit the scope of the multicast group. For example, a value of ‘1’ indicates that it is a node-local multicast group. A value of ‘2’ indicates that the scope is link-local. A value of “5” indicates that the scope is site-local. The “group ID” field identifies the multicast group.
NETWORKING_IPV6=yes IPV6FORWARDING=no IPV6_AUTOCONF=no IPV6_AUTOTUNNEL=no # Enable global IPv6 initialization # Disable global IPv6 forwarding # Disable global IPv6 autoconfiguration # Disable automatic IPv6 tunneling D.3.2 Adding persistent IPv6 Addresses on Red Hat Linux This can be done by modifying the system configuration script, for example, /etc/sysconfig/ network-scripts/ifcfg-eth1: DEVICE=eth1BOOTPROTO=static BROADCAST=192.168.1.255 IPADDR=192.168.1.10 NETMASK=255.255.255.0 NETWORK=192.168.1.
BOOTPROTO=static BROADCAST=10.0.2.255 IPADDR=10.0.2.10 NETMASK=255.255.0.0 NETWORK=0.0.2.0 REMOTE_IPADDR="" STARTMODE=onboot IPADDR1=3ffe::f101:10/64IPADDR2=fec0:0:0:1::10/64 BONDING_MASTER=yes BONDING_MODULE_OPTS="mode=active-backup miimon=100" BONDING_SLAVE0=eth1BONDING_SLAVE1=eth2 For each additional IPv6 address, specify an additional parameter with IPADDR in the configuration file.
E Maximum and Minimum Values for Parameters Table 17 shows the range of possible values for cluster configuration parameters. Table 17 Minimum and Maximum Values of Cluster Configuration Parameters Cluster Parameter Minimum Value Maximum Value Default Value Member Timeout See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4. See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4.
F Monitoring Script for Generic Resources Monitoring scripts are the scripts written by an end-user and must contain the core logic to monitor a resource and set the status of a generic resource. These scripts are started as a part of the package start. • You can set the status/value of a simple/extended resource respectively using the cmsetresource(1m) command. • You can define the monitoring interval in the script.
For resources of evaluation_type: before_package_start • Monitoring scripts can also be launched outside of the Serviceguard environment, init, rc scripts, etc. (Serviceguard does not monitor them). • The monitoring scripts for all the resources in a cluster of type before_package_start can be configured in a single multi-node package by using the services functionality and any packages that require the resources can mention the generic resource name in their package configuration file.
generic_resource_evaluation_type before_package_start generic_resource_name generic_resource_evaluation_type lan1 before_package_start dependency_name dependency_condition dependency_location generic_resource_monitors generic_resource_monitors = up same_node Thus, the monitoring scripts for all the generic resources of type before_package_start are configured in one single multi-node package and any package that requires this generic resource can just configure the generic resource name.
# * --------------------------------* # * The following utility functions are sourced in from $SG_UTILS * # * ($SGCONF/scripts/mscripts/utils.sh) and available for use: * # * * # * sg_log * # * * # * By default, only log messages with a log level of 0 will * # * be output to the log file.
{ sg_log 5 "start_command" # ADD your service start steps here return 0 } ######################################################################### # # stop_command # # This function should define actions to take when the package halts # # ######################################################################### function stop_command { sg_log 5 "stop_command" # ADD your halt steps here exit 1 } ################ # main routine ################ sg_log 5 "customer defined monitor script" #####################
G HP Serviceguard Toolkit for Linux The HP Serviceguard Toolkits such as, Contributed Toolkit, NFS, EDB PPAS, Sybase, and Oracle Toolkits are used for the integration of applications such as, Apache, MySQL, NFS, Oracle database, EDB PPAS, Sybase, and so on with the Serviceguard for Linux environment. The Toolkit documentation describes how to customize the package for your needs. For more information, see the Release Notes of these toolkits at http://www.hp.com/go/linux-serviceguard-docs.
Index A Access Control Policies, 169 active node, 20 adding a package to a running cluster, 246 adding cluster nodes advance planning, 141 adding nodes to a running cluster, 224 adding packages on a running cluster, 208 administration adding nodes to a running cluster, 224 halting a package, 230 halting the entire cluster, 225 moving a package, 231 of packages and services, 229 of the cluster, 223 reconfiguring a package while the cluster is running, 245 reconfiguring a package with the cluster offline, 246
Autostart Delay parameter (AUTO_START_TIMEOUT), 99 cluster coordinator defined, 36 cluster lock 4 or more nodes, 40 and cluster reformation, example, 74 and power supplies, 30 identifying in configuration file, 166 no lock, 40 two nodes, 38, 39 use in re-forming a cluster, 38, 39 cluster manager automatic restart of cluster, 37 blank planning worksheet, 301 cluster node parameter, 90, 91, 92 defined, 36 dynamic re-formation, 38 heartbeat subnet parameter, 94 initial configuration of the cluster, 36 main fun
cmpreparestg, 84 cmquerycl, 84 enabling or Disabling switching attributes for a package, 261 enclosure for disks replacing a faulty mechanism, 271 error handling during package halt, 230 Ethernet redundant configuration, 26 exclusive access relinquishing via TOC, 74 expanding the cluster planning ahead, 77 expansion planning for, 106 explanations package parameters, 184 F failback policy used by package manager, 48 FAILBACK_POLICY parameter used by package manager, 48 failover controlling the speed in appl
hardware planning, 80, 81 I/O slots hardware planning, 79, 80 identifying cluster-aware volume groups, 168 Installing Serviceguard, 143 installing software quorum server, 153 integrating HA applications with Serviceguard, 295 introduction Serviceguard at a glance, 19 understanding Serviceguard hardware, 25 understanding Serviceguard software, 31 IP address adding and deleting in packages, 61 for nodes and packages, 60 hardware planning, 80, 83 portable, 60 reviewing for packages, 274 switching, 43, 44, 67 I
network polling interval (NETWORK_POLLING_INTERVAL) parameter in cluster manager configuration, 99, 103 network time protocol (NTP) for clusters, 148 networking redundant subnets, 79 networks binding to IP addresses, 289 binding to port addresses, 289 IP addresses and naming, 287 node and package IP addresses, 60 packages using IP addresses, 288 supported types in Serviceguard, 25 writing network applications as HA services, 284 no cluster lock choosing, 40 node basic concepts, 25 halt (TOC), 74 in Serviceg
hardware configuration, 79 high availability objectives, 77 overview, 77 package configuration, 103 power, 81 quorum server, 82 SPU information, 79 volume groups and physical volumes, 83 worksheets, 81 planning and documenting an HA cluster, 77 planning for cluster expansion, 77 planning worksheets blanks, 299 point of failure in networking, 26 POLLING_TARGET defined, 102 ports dual and single aggregated, 62 power planning power sources, 81 worksheet, 82, 300 power supplies blank planning worksheet, 299 pow
serviceguard WBEM provider, 35 shared disks planning, 80 shutdown and startup defined for applications, 284 simulating a serviceguard cluster, 257 creating, 258 importing the cluster, 258 listing the existing session, 259 simulating failure scenarios, 261 simulating the package creating a simulated package, 260 deleting a package, 261 halting a package, 261 running a package, 260 simulation for the package, 260 single point of failure avoiding, 19 single-node operation, 177, 255 size of cluster preparing fo
W WEIGHT_DEFAULT defined, 102 WEIGHT_NAME defined, 102 What is Serviceguard?, 19 worksheet blanks, 299 cluster configuration, 103, 301 hardware configuration, 81, 299 package configuration, 301 power supply configuration, 82, 299, 300 use in planning, 77 326 Index
