Managing Serviceguard Twentieth Edition HP Part Number: 5900-1869 Published: August 2011
Legal Notices © Copyright 1995-2011 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Publishing History .......................................................................................18 Preface......................................................................................................19 1 Serviceguard at a Glance.........................................................................21 What is Serviceguard? ...........................................................................................................21 Failover........................................
Cluster Object Manager Daemon: cmomd.......................................................................42 Cluster SNMP Agent Daemon: cmsnmpd..........................................................................42 Generic Resource Assistant Daemon: cmresourced.............................................................42 Service Assistant Daemon: cmserviced.............................................................................43 Quorum Server Daemon: qs.........................................
Adding and Deleting Relocatable IP Addresses .....................................................................68 Load Sharing ...............................................................................................................69 Monitoring LAN Interfaces and Detecting Failure: Link Level.....................................................69 Local Switching ............................................................................................................
Disk I/O Information .........................................................................................................96 Hardware Configuration Worksheet ....................................................................................97 Power Supply Planning ...........................................................................................................97 Power Supply Configuration Worksheet ...............................................................................
Parameters for Configuring EMS Resources..........................................................................136 About Package Dependencies...........................................................................................137 Simple Dependencies..................................................................................................137 Rules for Simple Dependencies.................................................................................137 Guidelines for Simple Dependencies....
Configuring Name Resolution............................................................................................168 Safeguarding against Loss of Name Resolution Services...................................................169 Ensuring Consistency of Kernel Configuration .....................................................................170 Enabling the Network Time Protocol ..................................................................................171 Tuning Network and Kernel Parameters......
Levels of Access..........................................................................................................194 Setting up Access-Control Policies..................................................................................194 Role Conflicts.........................................................................................................196 Package versus Cluster Roles.........................................................................................197 Adding Volume Groups..
module_name........................................................................................................234 module_version......................................................................................................234 package_type........................................................................................................234 package_description...............................................................................................234 node_name..........................
fs_name................................................................................................................251 fs_server................................................................................................................251 fs_directory............................................................................................................251 fs_type..................................................................................................................251 fs_mount_opt...
Example................................................................................................................277 Limitations..................................................................................................................277 Managing the Cluster and Nodes ..........................................................................................277 Starting the Cluster When all Nodes are Down ...................................................................
What You Can Do..................................................................................................297 What You Must Keep in Mind..................................................................................298 Example: Adding a Heartbeat LAN..........................................................................299 Example: Deleting a Subnet Used by a Package.........................................................300 Removing a LAN or VLAN Interface from a Node.........................
Replacing Disks....................................................................................................................324 Replacing a Faulty Array Mechanism..................................................................................324 Replacing a Faulty Mechanism in an HA Enclosure..............................................................324 Replacing a Lock Disk.......................................................................................................
Use Restartable Transactions .............................................................................................344 Use Checkpoints .............................................................................................................344 Balance Checkpoint Frequency with Performance ...........................................................344 Design for Multiple Servers ..............................................................................................
Performing a Rolling Upgrade Using DRD................................................................................362 Before You Start...............................................................................................................362 Running the Rolling Upgrade Using DRD.............................................................................362 Example of a Rolling Upgrade ..............................................................................................363 Step 1. ......
I Using Serviceguard Manager....................................................................385 About the Online Help System................................................................................................385 Before Using HP Serviceguard Manager: Setting Up ................................................................385 Accessing Serviceguard Manager..........................................................................................385 Launching Serviceguard Manager.............
Publishing History Table 1 Publishing History Publishing Date Part Number Edition January 1995 B3936-90001 First June 1995 B3936-90003 Second December 1995 B3936-90005 Third August 1997 B3936-90019 Fourth January 1998 B3936-90024 Fifth October 1998 B3936-90026 Sixth December 2000 B3936-90045 Seventh September 2001 B3936-90053 Eighth March 2002 B3936-90065 Ninth June 2003 B3936-90070 Tenth June 2004 B3936-90076 Eleventh June 2005 B3936-90076 Eleventh, First reprint Octo
Preface This twentieth edition of the manual applies to Serviceguard Version A.11.20 for the August 2011 patch release. Earlier versions are available at http://www.hp.com/go/hpux-serviceguard-docs -> HP Serviceguard. The information in this manual has been updated for new features and functionality introduced in the Serviceguard A.11.20 August 2011 patches, or their superseding patches (PHSS_42137 or later). If the Serviceguard A.11.
Related Publications For information about the current version of Serviceguard, and about older versions, see the Serviceguard documents posted at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard. The following documents, which can all be found at http://www.hp.com/go/ hpux-serviceguard-docs, are particularly useful. • The latest edition of the HP Serviceguard Version A.11.20 Release Notes • HP Serviceguard Quorum Server Version A.04.
1 Serviceguard at a Glance This chapter introduces Serviceguard on HP-UX, and shows where to find information in this book. It covers the following: • What is Serviceguard? • Using Serviceguard Manager (page 24) • A Roadmap for Configuring Clusters and Packages (page 25) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4: “Planning and Documenting an HA Cluster ” (page 92).
control of the package to another cluster node, allowing services to remain available with minimal interruption. • There are also packages that run on several cluster nodes at once, and do not fail over. These are called system multi-node packages and multi-node packages. Examples are the packages HP supplies for use with the Veritas Cluster Volume Manager and Veritas Cluster File System from Symantec (on HP-UX releases that support them; see “About Veritas CFS and CVM from Symantec” (page 23)).
Figure 2 Typical Cluster After Failover After this transfer, the failover package typically remains on the adoptive node as long the adoptive node continues running. If you wish, however, you can configure the package to return to its primary node as soon as the primary node comes back online. Alternatively, you may manually transfer control of the package back to the primary node at the appropriate time. Figure 2 does not show the power connections to the cluster, but these are important as well.
Using Serviceguard Manager NOTE: For more-detailed information, see Appendix I (page 385) and the section on Serviceguard Manager in the latest version of the Serviceguard Release Notes. Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for up-to-date information about Serviceguard Manager compatibility. You can find both documents at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard.
on the command line. As of HP-UX 11i v3, SAM offers a Terminal User Interface (TUI) which also acts as a gateway to the web-based System Management Homepage (SMH). • To get to the SMH for any task area, highlight the task area in the SAM TUI and press w. • To go directly to the SMH from the command line, enter /usr/sbin/sam -w For more information, see the HP-UX Systems Administrator’s Guide, at the address given in the preface to this manual.
Figure 3 Tasks in Configuring a Serviceguard Cluster The tasks in Figure 3 are covered in step-by-step detail in chapters 4 through 7. HP recommends you gather all the data that is needed for configuration before you start. See “Planning and Documenting an HA Cluster ” (page 92) for tips on gathering data.
2 Understanding Serviceguard Hardware Configurations This chapter gives a broad overview of how the Serviceguard hardware components work. The following topics are presented: • Redundancy of Cluster Components • Redundant Network Components (page 28) • Redundant Disk Storage (page 32) • Redundant Power Supplies (page 37) • Larger Clusters (page 37) Refer to the next chapter for information about Serviceguard software components.
Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable, and the cables themselves are connected by a component such as a hub or a bridge. This arrangement of physical cables connected to each other via a bridge or concentrator or switch is known as a bridged net .
configured into the Serviceguard cluster, unless those IP addresses themselves will be immediately configured into the cluster as stationary IP addresses. CAUTION: If you configure any address other than a stationary IP address on a Serviceguard network interface, it could collide with a relocatable package IP address assigned by Serviceguard. See “Stationary and Relocatable IP Addresses ” (page 67).
NOTE: You should verify that network traffic is not too heavy on the heartbeat/data LAN. If traffic is too heavy, this LAN might not perform adequately in transmitting heartbeats if the dedicated heartbeat LAN fails. Cross-Subnet Configurations As of Serviceguard A.11.18 it is possible to configure multiple subnets, joined by a router, both for the cluster heartbeat and for data, with some nodes using one subnet and some another.
• Because Veritas Cluster File System from Symantec (CFS) requires link-level traffic communication (LLT) among the nodes, Serviceguard cannot be configured in cross-subnet configurations with CFS alone. But CFS is supported in specific cross-subnet configurations with Serviceguard and HP add-on products; see the documentation listed below. • Each package subnet must be configured with a standby interface on the local bridged net. The standby interface can be shared between subnets.
IMPORTANT: Although cross-subnet topology can be implemented on a single site, it is most commonly used by extended-distance clusters, and specifically site-aware disaster-tolerant clusters, which require Metrocluster (HP add-on software). Design and configuration of such clusters are covered in the disaster-tolerant documentation delivered with Serviceguard. For more information, see the following documents under http:// www.hp.
Data Protection It is required that you provide data protection for your highly available system, using one of two methods: • Disk Mirroring • Disk Arrays using RAID Levels and Multiple Data Paths Disk Mirroring Serviceguard itself does not provide protection for data on your disks, but protection is provided by HP’s Mirrordisk/UX product for LVM storage, and by the Veritas Volume Manager for VxVM and CVM.
Monitoring LVM Disks Through Generic Resources Monitoring Service If you are using LVM, you can configure disk monitoring to detect a failed mechanism by using the disk monitor capabilities of the System Fault Management, available as a separate product, and integrating it in Serviceguard by configuring generic resources in packages.
Sample SCSI Disk Configurations Figure 5 shows a two node cluster. Each node has one root disk which is mirrored and one package for which it is the primary node. Resources have been allocated to each node so that each node may adopt the package from the other node. Each package has one disk volume group assigned to it and the logical volumes in that volume group are mirrored.
Figure 6 Cluster with High Availability Disk Array Details on logical volume configuration for Serviceguard are in the chapter “Building an HA Cluster Configuration.” Sample Fibre Channel Disk Configuration In Figure 7, the root disks are shown with simple mirroring, but the shared storage is now accessed via redundant Fibre Channel switches attached to a disk array. The cabling is set up so that each node is attached to both switches, and both switches are attached to the disk array with redundant links.
Figure 7 Cluster with Fibre Channel Switched Disk Array This type of configuration uses native HP-UX or other multipathing software; see “About Multipathing” (page 33). Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks. HP-supported uninterruptible power supplies (UPS), such as HP PowerTrust, can provide this protection from momentary power loss.
could be set up as an administrative unit, and sub-groupings of four could be set up on different SCSI buses which are attached to different mass storage devices. In the case of non-shared SCSI connections to an XP series or EMC disk array, the four-node limit does not apply. Each node can be connected directly to the XP or EMC by means of two SCSI buses. Packages can be configured to fail over among all sixteen nodes.
Figure 9 Eight-Node Cluster with XP or EMC Disk Array Fibre Channel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP Unix Servers Configuration Guide, available through your HP representative.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
• /usr/lbin/cmfileassistd—Serviceguard File Management daemon • /usr/lbin/cmlogd—Serviceguard Syslog Log Daemon • /usr/lbin/cmlvmd—Cluster Logical Volume Manager Daemon • /opt/cmom/lbin/cmomd—Cluster Object Manager Daemon • /usr/lbin/cmsnmpd—Cluster SNMP subagent (optionally running) • /usr/lbin/cmresourced—Serviceguard Generic Resource Assistant Daemon • /usr/lbin/cmserviced—Serviceguard Service Assistant Daemon • /usr/lbin/qs—Serviceguard Quorum Server Daemon • /usr/lbin/cmnetd—Servicegua
cmcld also manages Serviceguard packages, determining where to run them and when to start them. NOTE: Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 20 on all cluster nodes. It is important that user processes should have a priority lower than 20, otherwise they may prevent Serviceguard from updating the kernel safety timer, causing a system reset.
Generic resources allows custom defined monitors to be integrated. It provides better control, options, and flexibility in terms of getting and setting the status of a resource. This daemon is used by the Serviceguard commands cmgetresource(1m) and cmsetresource(1m) to get or set the status/value of a simple/extended generic resource configured in a package and is local to a node. This daemon runs on every node on which cmcld is running.
You must edit /etc/rc.config.d/cmwbemagt to auto-start cmwbemd. Configure cmwbemd to start before the Serviceguard cluster comes up. You can start and stop cmwbemd with the commands /sbin/init.d/cmwbemagt start and /sbin/init.d/cmwbemagt stop. Proxy Daemon: cmproxyd This daemon is used to proxy or cache Serviceguard configuration data for use by certain Serviceguard commands running on the local node.
Documenting an HA Cluster ” (page 92)). You can set cluster parameters using Serviceguard Manager or by editing the cluster configuration file (see Chapter 5: “Building an HA Cluster Configuration” (page 158)). The parameters you enter are used to build a binary configuration file which is propagated to all nodes in the cluster. This binary cluster configuration file must be the same on all the nodes in the cluster.
Automatic Cluster Startup An automatic cluster startup occurs any time a node reboots and joins the cluster. This can follow the reboot of an individual node, or it may be when all nodes in a cluster have failed, as when there has been an extended power failure and all SPUs went down. Automatic cluster startup will take place if the flag AUTOSTART_CMCLD is set to 1 in /etc/ rc.config.d/cmcluster.
If you have a two-node cluster, you are required to configure a cluster lock. If communications are lost between these two nodes, the node that obtains the cluster lock will take over the cluster and the other node will halt (system reset). Without a cluster lock, a failure of either node in the cluster will cause the other node, and therefore the cluster, to halt. Note also that if the cluster lock fails during an attempt to acquire it, the cluster will halt.
IMPORTANT: A dual lock cannot be implemented on LUNs. This means that the lock LUN mechanism cannot be used in an Extended Distance cluster. Single Lock Disk or LUN A single lock disk or lock LUN should be configured on a power circuit separate from that of any node in the cluster. For example, using three power circuits for a two-node cluster is highly recommended, with a separately powered disk or LUN for the cluster lock.
The operation of the Quorum Server is shown in Figure 12. When there is a loss of communication between node 1 and node 2, the Quorum Server chooses one node (in this example, node 2) to continue running in the cluster. The other node halts. Figure 12 Quorum Server Operation The Quorum Server runs on a separate system, and can provide quorum services for multiple clusters.
IMPORTANT: During Step 1, while the nodes are using a strict majority quorum, node failures can cause the cluster to go down unexpectedly if the cluster has been using a quorum device before the configuration change. For example, suppose you change the Quorum Server polling interval while a two-node cluster is running. If a node fails during Step 1, the cluster will lose quorum and go down, because a strict majority of prior cluster members (two out of two in this case) is required.
Figure 13 Package Moving During Failover Configuring Failover Packages You configure each package separately. You create a failover package by generating and editing a package configuration file template, then adding the package to the cluster configuration database; see Chapter 6: “Configuring Packages and Their Services ” (page 227). For legacy packages (packages created by the method used on versions of Serviceguard earlier than A.11.
A package switch normally involves moving a failover package and its associated IP addresses to a new system on the same subnet. In this case, the new system must have the same subnet configured and working properly; otherwise the package will not be started. NOTE: It is possible to configure a cluster that spans subnets joined by a router, with some nodes using one subnet and some another. This is known as a cross-subnet configuration.
NOTE: For design and configuration information about clusters which span subnets, including site-aware disaster-tolerant clusters, see the documents listed under “Cross-Subnet Configurations” (page 30). Figure 15 After Package Switching Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file.
When the cluster starts, each package starts as shown in Figure 16.
NOTE: Using the min_package_node policy, when node 2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node.
Figure 19 Automatic Failback Configuration before Failover Table 3 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 CONFIGURED_NODE AUTOMATIC pkgB node2, node4 CONFIGURED_NODE AUTOMATIC pkgC node3, node4 CONFIGURED_NODE AUTOMATIC node1 panics, and after the cluster reforms, pkgA starts running on node4: Figure 20 Automatic Failback Configuration After Failover 56 Understanding Serviceguard Software Components
After rebooting, node 1 rejoins the cluster. At that point, pkgA will be automatically stopped on node 4 and restarted on node 1. Figure 21 Automatic Failback Configuration After Restart of Node 1 CAUTION: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period.
mechanisms, such as EMS monitors, SFM/WBEM monitors, Custom monitors, can be used and these can co-exist in a single package. Generic resources has the following advantages: • Custom defined monitors can also be integrated • Provides better control, options, and flexibility in terms of getting and setting the status of a resource Generic resources can be configured into any modular style package, including modular style CFS packages.
• The default current value is 0. • Valid values are positive integer values ranging from 1 to 2147483647. NOTE: You can get or set the status/value of a simple/extended generic resource using the cmgetresource(1m) and cmsetresource(1m) commands respectively. See “Getting and Setting the Status/Value of a Simple/Extended Generic Resource” (page 135) and the manpages for more information.
How Packages Run Packages are the means by which Serviceguard starts and halts configured applications. Failover packages are also units of failover behavior in Serviceguard. A package is a collection of services, disk volumes and IP addresses that are managed by Serviceguard to ensure they are available. There can be a maximum of 300 packages per cluster and a total of 900 services per cluster.
NOTE: This diagram applies specifically to legacy packages. Differences for modular scripts are called out below. Figure 22 Legacy Package Time Line Showing Important Events The following are the most important moments in a package’s life: 1. Before the control script starts. (For modular packages, this is the master control script.) 2. During run script execution. (For modular packages, during control script execution to start the package.) 3. While services are running 4.
During Run Script Execution Once the package manager has determined that the package can start on a particular node, it launches the script that starts the package (that is, a package’s control script or master control script is executed with the start parameter). This script carries out the following steps: 1. 2. 3. 4. 5. 6. 7. 8. Executes any external_pre_scripts (modular packages only (page 252)) Activates volume groups or disk groups. Mounts file systems.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled is set to yes, it will halt the node on which the package is running.
Some failures can result in a local switch. For example, if there is a failure on a specific LAN card and there is a standby LAN configured for that subnet, then the Network Manager will switch to the healthy LAN card. If a service fails but the restart parameter for that service is set to a value greater than 0, the service will restart, up to the configured number of restarts, without halting the package. If there is a generic resource configured and it fails, then the package will be halted.
script is executed with the stop parameter. This script carries out the following steps (also shown in Figure 24): 1. 2. 3. 4. 5. 6. 7. 8. Halts any deferred resources that had been started earlier. Halts all package services. Executes any customer-defined halt commands (legacy packages only) or external_scripts (modular packages only(page 252)). Removes package IP addresses from the LAN card on the node. Unmounts file systems. Deactivates volume groups. Exits with an exit code of zero (0).
NOTE: This diagram applies specifically to legacy packages. Differences for modular scripts are called out above. Normal and Abnormal Exits from the Halt Script The package’s ability to move to other nodes is affected by the exit conditions on leaving the halt script. The following are the possible exit codes: • 0—normal exit. The package halted normally, so all services are down on this node. • 1—abnormal exit, also known as no_restart exit. The package did not halt normally.
Table 4 Error Conditions and Package Movement for Failover Packages (continued) Package Error Condition Results Error or Exit Code Node Failfast Service Enabled Failfast Enabled HP-UX Status on Primary after Error Halt script Package Allowed to Package runs after Run on Primary Allowed to Run Error or Exit Node after Error on Alternate Node Loss of Network YES Either Setting system reset No N/A (system reset) Yes Loss of Network NO Either Setting Running Yes Yes Yes Loss of Generic Resour
The IP addresses associated with a package are called relocatable IP addresses (also known as package IP addresses or floating IP addresses) because the addresses can actually move from one cluster node to another on the same subnet. You can use up to 200 relocatable IP addresses in a cluster. These addresses can be IPv4, IPv6, or a combination of both address families. Because system multi-node and multi-node packages do not fail over, they do not have relocatable IP address.
IP addresses are configured only on each primary network interface card; standby cards are not configured with an IP address. Multiple IPv4 addresses on the same network card must belong to the same IP subnet. CAUTION: HP strongly recommends that you add relocatable addresses to packages only by editing ip_address (page 243) in the package configuration file (or IP [] entries in the control script of a legacy package) and running cmapplyconf (1m).
◦ The primary switch should be directly connected to its standby. ◦ There should be no single point of failure anywhere on all bridged nets. NOTE: You can change the value of the NETWORK_FAILURE_DETECTION parameter while the cluster is up and running. Local Switching A local network switch involves the detection of a local network interface failure and a failover to the local backup LAN card (also known as the standby LAN card). The backup LAN card must not have any IP addresses configured.
Figure 25 Cluster Before Local Network Switching Node 1 and Node 2 are communicating over LAN segment 2. LAN segment 1 is a standby. In Figure 26, we see what would happen if the LAN segment 2 network interface card on Node 1 were to fail. Figure 26 Cluster After Local Network Switching As the standby interface takes over, IP addresses will be switched to the hardware path associated with the standby interface. The switch is transparent at the TCP/IP level.
NOTE: On Ethernet networks, Serviceguard supports local failover between network interfaces configured with “Ethernet protocol” or between network interfaces configured with “SNAP encapsulation within IEEE 802.3 protocol.” You cannot use both protocols on the same interface, nor can you have a local failover between interfaces that are using different protocols. Another example of local switching is shown in Figure 27.
The purpose of the NETWORK_AUTO_FAILBACK parameter is to allow you control Serviceguard's behavior if you find network interfaces are experiencing a “ping-pong” effect, switching back and forth unduly between primary and standby interfaces. If you are not seeing any such problem, leave NETWORK_AUTO_FAILBACK set to YES. You can track switching behavior in the syslog file.
The IP Monitor: • Detects when a network interface fails to send or receive IP messages, even though it is still able to send and/or receive DLPI messages. • Handles the failure, failover, recovery, and failback.
… Route Connectivity (no probing was performed): IPv4: 1 16.89.143.192 16.89.120.0 … Possible IP Monitor Subnets: IPv4: 16.89.112.0 Polling Target 16.89.112.1 IPv6: 3ffe:1000:0:a801:: Polling Target 3ffe:1000:0:a801::254 … The IP Monitor section of the cluster configuration file will look similar to the following for a subnet on which IP monitoring is configured with target polling.
NETWORK_POLLING_INTERVAL, the IP monitor will detect the recovery of an IP address typically within 8–10 seconds for Ethernet and with 16–18 seconds for InfiniBand. IMPORTANT: HP strongly recommends that you do not change the default NETWORK_POLLING_INTERVAL value of 2 seconds. See also “Reporting Link-Level and IP-Level Failures” (page 76). Constraints and Limitations • A subnet must be configured into the cluster in order to be monitored.
But if the IP Monitor detects the failure (and link-level monitoring does not) cmviewcl -v output will look something like this: Network_Parameters: INTERFACE STATUS PRIMARY down (disabled) (IP only) PRIMARY up PATH 0/3/1/0 0/5/1/0 NAME lan2 lan3 cmviewcl -v -f line will report the same failure like this: node:gary|interface:lan2|status=down node:gary|interface:lan2|local_switch_peer=lan1 node:gary|interface:lan2|disabled=true node:gary|interface:lan2|failure_type=ip_only In this case, you would need to
Figure 28 Aggregated Networking Ports Both the Single and Dual ported LANs in the non-aggregated configuration have four LAN cards, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (lan0, lan1, lan2, lan3). When these ports are aggregated all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as lan900, the name by which the aggregate is known on HP-UX 11i.
remote failover of VLAN interfaces when failure is detected. Failure of a VLAN interface is typically the result of the failure of the underlying physical NIC port or aggregated (APA) ports. Configuration Restrictions HP-UX allows up to 1024 VLANs to be created from a physical NIC port. A large pool of system resources is required to accommodate such a configuration; Serviceguard could suffer performance degradation if many network interfaces are configured in each cluster node.
redundant storage in hardware. Two types of mirroring are RAID1 and RAID5. Here are some differences between the two storage methods: • If you are using JBODs, the basic element of storage is an individual disk. This disk must be paired with another disk to create a mirror (RAID1). (Serviceguard configurations usually have separate mirrors on different storage devices).
NOTE: Under agile addressing (see “About Device File Names (Device Special Files)” (page 80)), the storage units in this example would have names such as disk1, disk2, disk3, etc. As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 104). Figure 29 Physical Disks Within Shared Storage Units Figure 30 shows the individual disks combined in a multiple disk mirrored configuration.
Figure 31 Multiple Devices Configured in Volume Groups Examples of Storage on Disk Arrays Figure 32 shows an illustration of storage configured on a disk array. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system. Figure 32 Physical Disks Combined into LUNs NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer.
NOTE: Under agile addressing, the storage units in these examples would have names such as disk1, disk2, disk3, etc. See “About Device File Names (Device Special Files)” (page 80) As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 104). . Figure 33 Multiple Paths to LUNs Finally, the multiple paths are configured into volume groups as shown in Figure 34.
Types of Volume Manager Serviceguard allows a choice of volume managers for data storage: • HP-UX Logical Volume Manager (LVM) and (optionally) Mirrordisk/UX • Veritas Volume Manager for HP-UX (VxVM)—Base and add-on Products • Veritas Cluster Volume Manager for HP-UX Separate sections in Chapters 5 and 6 explain how to configure cluster storage using all of these volume managers.
Veritas Cluster Volume Manager (CVM) NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information on CVM support: http://www.hp.com/go/hpux-serviceguard-docs. You may choose to configure cluster storage with the Veritas Cluster Volume Manager (CVM) instead of the Volume Manager (VxVM).
Redundant Heartbeat Subnets HP recommends that you configure all subnets that connect cluster nodes as heartbeat networks; this increases protection against multiple faults at no additional cost. For CVM 4.1 and later, the minimum recommended configurations are: 1) dual (multiple) heartbeat networks 2) single heartbeat network with standby LAN card(s) Comparison of Volume Managers The following table summarizes the advantages and disadvantages of the volume managers.
Table 5 Pros and Cons of Volume Managers with Serviceguard Product Advantages Tradeoffs Logical Volume Manager (LVM) • Software is provided with all versions of HP-UX. • Lacks flexibility and extended features of some other volume managers • Provides up to 3-way mirroring using optional Mirrordisk/UX software. • Dynamic multipathing (DMP) is active by default as of HP-UX 11i v3.
Table 5 Pros and Cons of Volume Managers with Serviceguard (continued) Product Advantages Tradeoffs • Supports shared activation. • Requires purchase of additional license • Supports exclusive activation. • No support for RAID 5 • Supports activation in different modes on different nodes at the same time • CVM requires all nodes to have connectivity to the shared disk groups • RAID 1+0 mirrored stripes • Not currently supported on all versions of HP-UX • RAID 0+1 striped mirrors • CVM versions 4.
Failure. Only one LAN has been configured for both heartbeat and data traffic. During the course of operations, heavy application traffic monopolizes the bandwidth of the network, preventing heartbeat packets from getting through. Since SystemA does not receive heartbeat messages from SystemB, SystemA attempts to reform as a one-node cluster. Likewise, since SystemB does not receive heartbeat messages from SystemA, SystemB also attempts to reform as a one-node cluster.
Responses to Package and Service Failures In the default case, the failure of a failover package, a generic resource, or of a service within the package, causes the package to shut down by running the control script with the ‘stop’ parameter, and then restarting the package on an alternate node. A package will also fail if it is configured to have a dependency on another package, and that package fails.
NOTE: If a simple resource is down on a particular node, it is down on that node for all the packages using it whereas, in case of an extended resource the resource may be up on a node for a particular package and down for another package, since it is dependent on the generic_resource_up_criteria.
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather information about all the hardware and software components of the configuration.
your cluster without the need to bring it down, careful planning of the initial configuration is required. Use the following guidelines: • Remember the rules for cluster locks when considering expansion. A one-node cluster does not require a cluster lock. A two-node cluster must have a cluster lock. In clusters larger than 3 nodes, a cluster lock is strongly recommended. If you have a cluster with more than 4 nodes, you can use a Quorum Server but a cluster lock disk is not allowed.
slots, and the bus address for each adapter; you can update the details as you do the cluster configuration (described in Chapter 5). SPU Information SPU information includes the basic characteristics of the systems you are using in the cluster. Different models of computers can be mixed in the same cluster. This configuration model also applies to HP Integrity servers. HP-UX workstations are not supported for Serviceguard.
For more details of IPv6 address format, see the Appendix H (page 378). NETWORK_FAILURE_DETECTION When there is a primary and a standby network card, Serviceguard needs to determine when a card has failed, so it knows whether to fail traffic over to the other card. The configuration file specifies one of two ways to decide when the network interface card has failed: • INOUT • INONLY_OR_INOUT The default is INOUT.
Table 6 SCSI Addressing in Cluster Configuration (continued) System or Disk Host Interface SCSI Address Disk #6 14 Others 13 - 8 NOTE: When a boot/root disk is configured with a low-priority address on a shared SCSI bus, a system panic can occur if there is a timeout on accessing the boot/root device. This can happen in a cluster when many nodes and many disks are configured on the same bus.
Hardware Configuration Worksheet You may find a worksheet such as the following useful for organizing and recording your cluster hardware configuration. This worksheet is an example; blank worksheets are in Appendix E. Make as many copies as you need.
units you are using and the power supply to which they will be connected; record the following information: Host Name The host name for each SPU. Disk Unit The disk drive unit number for each disk. Tape Unit The tape unit number for each backup device. Other Unit Tthe number of any other unit. Power Supply The power supply unit number of the UPS to which the host or other device is connected. Be sure to follow UPS and cabinet power limits as well as SPU power limits.
Quorum Server as the cluster lock. In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes. For more information on lock disks, lock LUNs, and the Quorum Server, see “Choosing Cluster Lock Disks” (page 173), “Setting Up a Lock LUN” (page 174), and “Setting Up and Running the Quorum Server” (page 177).
Supported Node Names The name (39 characters or fewer) of each cluster node that will be supported by this quorum server. These entries will be entered into qs_authfile on the system that is running the quorum server process.
You can do this by using the disk monitor capabilities of the System Fault Management, available as a separate product, and integrating it in Serviceguard by configuring generic resources in packages.
IMPORTANT: You must set the IO timeout for all logical volumes within the volume group being monitored to something other than the default of zero (no timeout); otherwise the EMS resource monitor value will never change upon a failure. Suggested IO timeout values are 20 to 60 seconds. See “Setting Logical Volume Timeouts” (page 180) for more information. For more information, see “Using the EMS HA Monitors” (page 59).
NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information on CVM support: http://www.hp.com/go/hpux-serviceguard-docs. When designing a storage configuration using CVM or VxVM disk groups, consider the following: • CVM disk groups are created after the cluster is configured, whereas VxVM disk groups may be created before cluster configuration if desired.
In addition, you must provide consistency across the cluster so that: • User names are the same on all nodes. • UIDs are the same on all nodes. • GIDs are the same on all nodes. • Applications in the system area are the same on all nodes. • System time is consistent across the cluster. • Files that could be used by more than one node, such as files in the /usr directory, must be the same on all nodes.
NOTE: Software that assumes DSFs reside only in /dev/disk and /dev/rdisk will not find cDSFs and may not work properly as a result; as of the date of this manual, this was true of the Veritas Volume Manager, VxVM. Limitations of cDSFs • cDSFs are supported only within a single cluster; you cannot define a cDSF group that crosses cluster boundaries. • A node can belong to only one cDSF group.
Advantages of Easy Deployment • Quick and simple way to create and start a cluster. • Automates security and networking configuration that must always be done before you configure nodes into a cluster. • Simplifies cluster lock configuration. • Simplifies creation of shared storage for packages. Limitations of Easy Deployment • Does not install or verify Serviceguard software • Requires agile addressing for disks. See “About Device File Names (Device Special Files)” (page 80).
IPv4-only means that Serviceguard will try to resolve the hostnames to IPv4 addresses only. IMPORTANT: You can configure an IPv6 heartbeat, or stationary or relocatable IP address, in any mode: IPv4-only, IPv6-only, or mixed. You can configure an IPv4 heartbeat, or stationary or relocatable IP address, in IPv4-only or mixed mode. IPv6-only means that Serviceguard will try to resolve the hostnames to IPv6 addresses only.
• The node's public LAN address (by which it is known to the outside world) must be the last address listed in /etc/hosts. Otherwise there is a possibility of the address being used even when it is not configured into the cluster. • You must use $SGCONF/cmclnodelist, not ~/.rhosts or /etc/hosts.equiv, to provide root access to an unconfigured node. NOTE: This also applies if HOSTNAME_ADDRESS_FAMILY is set to ANY. See “Allowing Root Access to an Unconfigured Node” (page 166) for more information.
IPv6 addresses. Serviceguard will first try to resolve a node's hostname to an IPv4 address, then, if that fails, will try IPv6. Rules and Restrictions for Mixed Mode IMPORTANT: See the latest version of the Serviceguard release notes for the most current information on these and other restrictions.
The cluster name must not contain any of the following characters: space, slash (/), backslash (\), and asterisk (*). NOTE: In addition, the following characters must not be used in the cluster name if you are using the Quorum Server: at-sign (@), equal-sign (=), or-sign (|), semicolon (;). These characters are deprecated, meaning that you should not use them, even if you are not using the Quorum Server, because they will be illegal in a future Serviceguard release. All other characters are legal.
NOTE: Lock volume groups must also be defined in VOLUME_GROUP parameters in the cluster configuration file. Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 49) for important information. QS_HOST The fully-qualified hostname or IP address of a system outside the current cluster that is providing quorum service to the cluster.
IMPORTANT: For special instructions that may apply to your version of Serviceguard and the Quorum Server see “Configuring Serviceguard to Use the Quorum Server” in the latest version HP Serviceguard Quorum Server Version A.04.00 Release Notes, at http://www.hp.com/go/ hpux-serviceguard-docs under HP Serviceguard Quorum Server Software.
NODE_NAME The hostname of each system that will be a node in the cluster. CAUTION: Make sure that the node name is unique within the subnets configured on the cluster nodes; under some circumstances Serviceguard may not be able to detect a duplicate name and unexpected problems may result. Do not use the full domain name. For example, enter ftsys9, not ftsys9.cup.hp.com.
Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 106). NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 125)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
For information about changing the configuration online, see “Changing the Cluster Networking Configuration while the Cluster Is Running” (page 297).
NOTE: Limitations: • Because Veritas Cluster File System from Symantec (CFS) requires link-level traffic communication (LLT) among the nodes, Serviceguard cannot be configured in cross-subnet configurations with CFS alone. But CFS is supported in specific cross-subnet configurations with Serviceguard and HP add-on products; see the documentation listed under “Cross-Subnet Configurations” (page 30) for more information. • IPv6 heartbeat subnets are not supported in a cross-subnet configuration.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 125)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
this list). Used only if a lock disk is used for tie-breaking services. Use FIRST_CLUSTER_LOCK_PV for the first physical lock volume and SECOND_CLUSTER_LOCK_PV for the second physical lock volume, if any. If there is a second physical lock volume, SECOND_CLUSTER_LOCK_PV must be on a separate line. These parameters are used only when you employ a lock disk for tie-breaking services in the cluster.
of four capacities per cluster, unless you use the reserved CAPACITY_NAME package_limit; in that case, you can use only that capacity throughout the cluster. For all capacities other than package_limit, the default weight for all packages is zero, though you can specify a different default weight for any capacity other than package_limit; see the entry for WEIGHT_NAME and WEIGHT_DEFAULT later in this list. See “About Package Weights” (page 144) for more information.
NOTE: • For most clusters that use an LVM cluster lock or lock LUN, a minimum MEMBER_TIMEOUT of 14 seconds is appropriate. • For most clusters that use a MEMBER_TIMEOUT value lower than 14 seconds, a quorum server is more appropriate than a lock disk or lock LUN. The cluster will fail if the time it takes to acquire the disk lock exceeds 0.2 times the MEMBER_TIMEOUT.
See also “What Happens when a Node Times Out” (page 88), “Cluster Daemon: cmcld” (page 41), and the white paper Optimizing Failover Time in a Serviceguard Environment (version A.11.19 and later) on http:// www.hp.com/go/hpux-serviceguard-docs. Can be changed while the cluster is running. AUTO_START_TIMEOUT The amount of time a node waits before it stops trying to join a cluster during automatic cluster startup.
first determine the Maximum Bridge Transit Delay (MBTD) for each switch and router. The value should be in the vendors' documentation. Set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the values for the switches and routers. If there is more than one possible path between the NFS server and the cluster nodes, sum the values for each path and use the largest number. CAUTION: Serviceguard supports NFS-mounted file systems only over switches and routers that support MBTD.
By default, each of the cluster subnets is listed under SUBNET, and, if at least one gateway is detected for that subnet, IP_MONITOR is set to ON and POLLING_TARGET entries are populated with the gateway addresses, enabling target polling; otherwise the subnet is listed with IP_MONITOR set to OFF. See “Monitoring LAN Interfaces and Detecting Failure: IP Level” (page 73) for more information.
NOTE: cmquerycl (1m) detects first-level routers in the cluster (by looking for gateways in each node's routing table) and lists them here as polling targets. If you run cmquerycl with the -w full option (for full network probing) it will also verify that the gateways will work correctly for monitoring purposes. Can be changed while the cluster is running; must be removed if the preceding SUBNET entry is removed.
Access Control Policies (also known as Role Based Access) For each policy, specify USER_NAME, USER_HOST, and USER_ROLE. Policies set in the configuration file of a cluster and its packages must not be conflicting or redundant. For more information, see “Setting up Access-Control Policies” (page 194). VOLUME_GROUP The name of an LVM volume group whose disks are attached to at least two nodes in the cluster. Such disks are considered cluster-aware.
As part of planning, you need to decide the following: • What volume groups are needed? • How much disk space is required, and how should this be allocated in logical volumes? • What file systems need to be mounted for each package? • Which nodes need to import which logical volume configurations? • If a package moves to an adoptive node, what effect will its presence have on performance? • What hardware/software resources need to be monitored as part of the package? You can then configure these
CVM 4.1 and later requires you to configure multiple heartbeat networks, or a single heartbeat with a standby. The following interfaces are supported as heartbeat networks — APA (CVM 5.0.1 and later) and VLAN (CVM 4.1 and later). CVM 4.1 and later with CFS CFS (Veritas Cluster File System) is supported for use with Veritas Cluster Volume Manager Version 4.1 and later. The system multi-node package SG-CFS-pkg manages the cluster’s volumes.
3. The mount point packages should not run unless the disk group packages are running. If the disk groups and mount points are in separate packages, specify the dependencies on the disk group packages in the configuration file. CAUTION: Once you create the modular CVM disk group and CFS mount point packages, you must administer the cluster with cmcheckconf, cmapplyconf, cmrunpkg, cmmodpkg, and cmrunpkg commands.
multi-node package to monitor identically-named root, boot, or swap volumes on cluster nodes. Because the root, boot, and swap volumes are critical to the functioning of the node , the service should be configured with service_fail_fast_enabled (page 244) set to yes. When a monitored volume fails or becomes inaccessible, the monitor service will exit, causing the package to fail on the current node. The package’s failure behavior depends on its configured settings.
never completes, the monitor service will terminate after six such consecutive read attempts (a duration of up to six poll intervals). volume_path The full path to at least one VxVM volume or LVM logical volume device file for monitoring (required). The pathname must identify a block device file. Examples /usr/sbin/cmvolmond -O /log/monlog.
IMPORTANT: Find out the MBTD value for each affected router and switch from the vendors' documentation; determine all of the possible paths; find the worst case sum of the MBTD values on these paths; and use the resulting value to set the Serviceguard CONFIGURED_IO_TIMEOUT_EXTENSION parameter. For instructions, see the discussion of this parameter under “Cluster Configuration Parameters ” (page 109). Switches and routers that do not support MBTD value must not be used in a Serviceguard NFS configuration.
This paper includes instructions for setting up a sample package that uses an NFS-imported file system. See also the description of fs_name (page 251), fs_type (page 251), and the other file system-related package parameters. Planning for Expansion You can add packages to a running cluster. This process is described in “Cluster and Package Maintenance” (page 261).
Parameters for Configuring Generic Resources Serviceguard provides the following parameters for configuring generic resources. Configure each of these parameters in the package configuration file for each resource the package will be dependent on. • generic_resource_name: defines the logical name used to identify a generic resource in a package. • generic_resource_evaluation_type: defines when the status of a generic resource is evaluated. This can be set to during_package_start or before_package_start.
NOTE: Generic resources must be configured to use the monitoring script. It is the monitoring script that contains the logic to monitor the resource and set the status of a generic resource accordingly by using cmsetresource(1m). These scripts must be written by the end-users according to their requirements. The monitoring script must be configured as a service in the package if the monitoring of the resource is required to be started and stopped as a part of the package.
Other_Attributes: ATTRIBUTE_NAME Style Priority ATTRIBUTE_VALUE modular no_priority The cmviewcl -v -f line output will be as follows: cmviewcl -v -f line -p pkg1 | grep generic_resource generic_resource:sfm_disk|name=sfm_disk generic_resource:sfm_disk|evaluation_type=during_package_start generic_resource:sfm_disk|up_criteria=”N/A” generic_resource:sfm_disk|node:node1|status=unknown generic_resource:sfm_disk|node:node1|current_value=0 generic_resource:sfm_disk|node:node2|status=unknown generic_resource:sf
Online Reconfiguration of Generic Resources Online operations such as addition, deletion, and modification of generic resources in packages are supported. The operations that can be performed online are: • Addition of a generic resource of generic_resource_evaluation_type set to during_package_start, if the status of the generic resource is not 'down'. Please ensure that while adding a generic resource, the equivalent monitor is available; if not add the monitor while adding a generic resource.
resource_name /net/interfaces/lan/status/lan0 resource_polling_interval 60 resource_start automatic resource_up_value = up NOTE: For a legacy package, specify the deferred resources again in the package control script, using the DEFERRED_RESOURCE_NAME parameter: DEFERRED_RESOURCE_NAME[0]="/net/interfaces/lan/status/lan0" DEFERRED_RESOURCE_NAME[1]="/net/interfaces/lan/status/lan1" If a resource is configured to be AUTOMATIC in a legacy configuration file, you do not need to define DEFERRED_RESOURCE_NAME in
NOTE: pkg1 can depend on more than one other package, and pkg2 can depend on another package or packages; we are assuming only two packages in order to make the rules as clear as possible. • pkg1 will not start on any node unless pkg2 is running on that node. • pkg1’s package_type (page 234) and failover_policy (page 237) constrain the type and characteristics of pkg2, as follows: ◦ If pkg1 is a multi-node package, pkg2 must be a multi-node or system multi-node package.
on that package; in our example, pkg1 is a successor of pkg2; conversely pkg2 can be referred to as a predecessor of pkg1.) Dragging Rules for Simple Dependencies The priority parameter (page 238) gives you a way to influence the startup, failover, and failback behavior of a set of failover packages that have a configured_node failover_policy, when one or more of those packages depend on another or others.
NOTE: Keep the following in mind when reading the examples that follow, and when actually configuring priorities: 1. auto_run (page 235) should be set to yes for all the packages involved; the examples assume that it is. 2. Priorities express a ranking order, so a lower number means a higher priority (10 is a higher priority than 30, for example).
In a simple dependency, if pkg1 depends on pkg2, and pkg1’s priority is higher than pkg2’s, pkg1’s node order dominates. Assuming pkg1’s node order is node1, node2, node3, then: • On startup: ◦ pkg1 will select node1 to start on, provided pkg2 can run there. ◦ pkg2 will start on node1, provided it can run there (no matter where node1 appears on pkg2’s node_name list). – ◦ If pkg2 is already running on another node, it will be dragged to node1, provided it can run there.
another package be down as an exclusionary dependency; see “Rules for Exclusionary Dependencies” (page 142). • You can specify where the dependency_condition must be satisfied: on the same node, a different node, all nodes, or any node in the cluster. You define this by means of the dependency_location parameter (page 240), using one of the literals same_node, different_node, all_nodes, or any_node. different_node and any_node are allowed only if dependency_condition is UP.
Rules for different_node and any_node Dependencies These rules apply to packages whose dependency_condition is UP and whose dependency_location is different_node or any_node. For same-node dependencies, see Simple Dependencies (page 137); for exclusionary dependencies, see “Rules for Exclusionary Dependencies” (page 142). • Both packages must be failover packages whose failover_policy (page 237) is configured_node.
3. Halts packages the failing package depends on, starting with the package this package immediately depends on. The packages are halted only if: • these are failover packages, and • the failing package can “drag” these packages to a node on which they can all run. Otherwise the failing package halts and the packages it depends on continue to run 4. Starts the packages the failed package depends on (those halted in step 3, if any).
Simple Method Use this method if you simply want to control the number of packages that can run on a given node at any given time. This method works best if all the packages consume about the same amount of computing resources. If you need to make finer distinctions between packages in terms of their resource consumption, use the Comprehensive Method (page 146) instead. To implement the simple method, use the reserved keyword package_limit to define each node's capacity.
you wanted to ensure that the larger packages, pkg2 and pkg3, did not run on node1 at the same time, you could raise the weight_value of one or both so that the combination exceeded 10 (or reduce node1's capacity to 8). Points to Keep in Mind The following points apply specifically to the Simple Method (page 145). Read them in conjunction with the Rules and Guidelines (page 150), which apply to all weights and capacities.
meanings of the names processor and memory; there is no mapping to actual processor and memory usage and you would get exactly the same results if you used the names apples and oranges, for example. Suppose you have the following configuration: • A two node cluster running four packages. These packages contend for resource we'll simply call A and B. • node1 has a capacity of 80 for A and capacity of 50 for B. • node2 has a capacity of 60 for A and capacity of 70 for B.
NOTE: You do not have to define capacities for every node in the cluster. If any capacity is not defined for any node, Serviceguard assumes that node has an infinite amount of that capacity.
NOTE: Option 4 means that the package is “weightless” as far as this particular capacity is concerned, and can run even on a node on which this capacity is completely consumed by other packages. (You can make a package “weightless” for a given capacity even if you have defined a cluster-wide default weight; simply set the corresponding weight to zero in the package's cluster configuration file.
(page 150)). This is true whenever a package has a weight that exceeds the available amount of the corresponding capacity on the node. Rules and Guidelines The following rules and guidelines apply to both the Simple Method (page 145) and the Comprehensive Method (page 146) of configuring capacities and weights. • You can define a maximum of four capacities, and corresponding weights, throughout the cluster.
decide which package to start if it cannot start them both because there is not enough node capacity to support their weight. Example 1 • pkg1 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 10. It is down and has switching disabled. • pkg2 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 20. It is running on node turkey and has switching enabled. • turkey and griffon can run one package each (package_limit is set to 1).
Each external script must have three entry points: start, stop, and validate, and should exit with one of the following values: • 0 - indicating success. • 1 - indicating the package will be halted, and should not be restarted, as a result of failure in this script. • 2 - indicating the package will be restarted on another node, or halted if no other node is available.
elif (( PEV_MONITORING_INTERVAL < 1 )) then sg_log 0 "ERROR: PEV_MONITORING_INTERVAL value ($PEV_MONITORING_INTERVAL) not within legal limits!" ret=1 fi # check monitoring service we are expecting for this package is configured while (( i < ${#SG_SERVICE_NAME[*]} )) do case ${SG_SERVICE_CMD[i]} in *monitor.
commands in their external scripts. If a timeout is not specified and your configuration has a command loop as described above, inconsistent results can occur, including a hung cluster. Determining Why a Package Has Shut Down You can use an external script (or CUSTOMER DEFINED FUNCTIONS area of a legacy package control script) to find out why a package has shut down.
monitored_subnet_access unconfigured for a monitored subnet is equivalent to FULL.) (For legacy packages, see “Configuring Cross-Subnet Failover” (page 309).) • You should not use the wildcard (*) for node_name (page 235) in the package configuration file, as this could allow the package to fail over across subnets when a node on the same subnet is eligible. Failing over across subnets can take longer than failing over on the same subnet. List the nodes in order of preference instead of using the wildcard.
Configuring node_name First you need to make sure that pkg1 will fail over to a node on another subnet only if it has to. For example, if it is running on NodeA and needs to fail over, you want it to try NodeB, on the same subnet, before incurring the cross-subnet overhead of failing over to NodeC or NodeD.
Planning for Changes in Cluster Size If you intend to add additional nodes to the cluster online (while it is running) ensure that they are connected to the same heartbeat subnets and to the same lock disks as the other cluster nodes. For cross-subnet considerations, see “Cross-Subnet Configurations” (page 30). In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes.
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. These procedures are carried out on one node, called the configuration node, and the resulting binary file is distributed by Serviceguard to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
################## cmcluster.conf ############### # Highly Available Cluster file locations # This file must not be edited ################################################# SGCONF=/etc/cmcluster SGSBIN=/usr/sbin SGLBIN=/usr/lbin SGLIB=/usr/lib SGRUN=/var/adm/cmcluster SGAUTOSTART=/etc/rc.config.d/cmcluster SGFFLOC=/opt/cmcluster/cmff CMSNMPD_LOG_FILE=/var/adm/SGsnmpsuba.log NOTE: If these variables are not defined on your system, then source the file /etc/ cmcluster.
1. Identify the nodes for which you want to create cluster-wide device files; this is known as the cDSF group. This should be all the nodes in the cluster (or prospective cluster). IMPORTANT: 2. Remember that: • You cannot create a cDSF group that crosses cluster boundaries; that is, the group must consist of the nodes of a single cluster. • cDSFs use agile addressing; see “About Device File Names (Device Special Files)” (page 80) for information about agile addressing.
“Specifying a Lock LUN” (page 189), and “Creating the Storage Infrastructure and file systems with LVM, VxVM and CVM” (page 177). Adding a Node to a cDSF Group When you add a new node to a cluster that uses cDSFs, you also need to add the node to the cDSF group.
• Make sure all the subnets used by the prospective nodes are accessible to all the nodes. Easy Deployment commands will fail if they detect an asymmetric network configuration, in which only a subset of nodes has access to a given subnet. • Make sure that the LAN or LANs that will be used for the cluster heartbeat meet high-availability requirements. See the requirements for HEARTBEAT_IP listed under “Cluster Configuration Parameters ” (page 109), and the networking “Rules and Restrictions” (page 28).
2. Edit the output file ($SGCONF/mynetwork in this example) to add entries for additional heartbeats, if necessary. NOTE: For information about heartbeat and networking requirements, see the sections listed under “Before You Start” (page 161). If you omit steps 1 and 2, and all the prospective nodes are connected to at least one subnet, cmdeploycl behaves as follows (the configuration is actually done by cmquerycl (1m) which is called by cmdeploycl).
NOTE: You can also accomplish this by running cmpreparecl; cmdeploycl calls cmpreparecl. For more information, see “Configuring Root-Level Access” (page 166) and “Configuring Name Resolution” (page 168). Those sections describe the “manual” (HP-UX command-line) method of accomplishing what cmpreparecl does. • Configures a two-node cluster (cluster1). • Configures the LVM volume group and physical volume for the lock disk. • Starts the cluster on ftsys9 and ftsys10.
5. Create shared storage for a LVM volume group with mirroring for use by cluster packages. cmpreparestg –l –p ... –n –n is the path name of the volume group that is to be created and imported to each specified node, and each is the name of a physical volume on the local node from which the volume group specified by is to be created.
6. Verify the storage using the following commands: • For LVM volume groups: vgdisplay • For VxVM/CVM disk groups: vxdg list The volume group or the disk group can now be used by packages; see Chapter 6 (page 227). NOTE: cmpreparestg requires either -l or -g in all cases. The command must be run on one of the specified nodes (ftsys9 or ftsys10 in this example).
For example: gryf sly bit root root root #cluster1, node1 #cluster1, node2 #cluster1, node3 This example grants root access to the node on which this cmclnodelist file resides to root users on the nodes gryf, sly, and bit. Serviceguard also accepts the use of a “+” in the cmclnodelist file; this indicates that the root user on any Serviceguard node can configure Serviceguard on this node. IMPORTANT: If $SGCONF/cmclnodelist does not exist, Serviceguard will look at ~/.rhosts.
Configuring Name Resolution Serviceguard uses the name resolution services built into HP-UX. NOTE: If you plan to use cmpreparecl (1m) (or cmpdeploycl (1m), which calls cmpreparecl), the /etc/hosts and /etc/nsswitch.conf configuration described in this subsection will be done automatically, but you should still read the entire subsection and make sure you understand the issues. In particular, you still need to make sure that aliases are properly represented in /etc/hosts, as described below.
15.145.162.131 10.8.0.131 10.8.1.131 gryf.uksr.hp.com gryf2.uksr.hp.com gryf3.uksr.hp.com gryf gryf gryf 15.145.162.132 10.8.0.132 10.8.1.132 sly.uksr.hp.com sly2.uksr.hp.com sly3.uksr.hp.com sly sly sly If applications require the use of hostname aliases, the Serviceguard hostname must be one of the aliases in all the entries for that host.
NOTE: If a NIC fails, the affected node will be able to fail over to a standby LAN so long as the node is running in the cluster. But if a NIC that is used by Serviceguard fails when the affected node is not running in the cluster, Serviceguard will not be able to restart the node. (For instructions on replacing a failed NIC, see “Replacing LAN or Fibre Channel Cards” (page 327).) NOTE: If you plan to use cmpreparecl (1m) (orcmpdeploycl (1m), which calls cmpreparecl), the /etc/hosts and /etc/nsswitch.
cluster node, they may also need to be changed on other cluster nodes that can run the same packages. Enabling the Network Time Protocol HP strongly recommends that you enable network time protocol (NTP) services on each node in the cluster. The use of NTP, which runs as a daemon process on each system, ensures that the system time on all nodes is consistent, resulting in consistent timestamps in log files and consistent behavior of message services.
model, see the HP-UX IPSec Version A.03.00 Administrator's Guide which you can find at http://www.hp.com/go/hpux-security-docs —> HP-UX IPSec Software. ip_strong_es_model is disabled (set to 0) by default. Setting it to 1 enables it. CAUTION: HP supports enabling this parameter only if you are not using a cross-subnet configuration (page 30). Otherwise, leave the parameter at its default setting (zero, meaning disabled).
The following is an example of mirroring the root logical volume: lvextend -m 1 /dev/vg00/lvol3 /dev/dsk/c4t6d0 5. Update the boot information contained in the BDRA for the mirror copies of boot, root and primary swap. /usr/sbin/lvlnboot -b /dev/vg00/lvol1 /usr/sbin/lvlnboot -s /dev/vg00/lvol2 /usr/sbin/lvlnboot -r /dev/vg00/lvol3 6. Verify that the mirrors were properly created.
NOTE: You must use the vgcfgbackup and vgcfgrestore commands to back up and restore the lock volume group configuration data regardless of how you create the lock volume group. Setting Up a Lock LUN LUN stands for Logical Unit Number. The term can refer to a single physical disk, but these days is more often used in a SAN (Storage Area Network) or NAS (Network-Attached Storage) context to denote a virtual entity derived from one or more physical disks.
IMPORTANT: On HP 9000 systems, there is no means of partitioning a disk or LUN, so you will need to dedicate an entire small disk or LUN for the lock LUN. This means that in a mixed cluster containing both Integrity and HP-PA systems, you must also use an entire disk or LUN; if you partition the device as described below, the HP-PA nodes will not be able to see the partitions.
Defining the Lock LUN Use cmquerycl -L to create a cluster configuration file that defines the lock LUN. • If the pathname for the lock LUN is the same on all nodes, use a command such as: cmquerycl -C $SGCONF/config.ascii -L /dev/dsk/c0t1d1 -n -n • If the pathname for the lock LUN is different on some nodes, you must specify the path on each node; for example (all on one line): cmquerycl -C $SGCONF/config.
IMPORTANT: The purpose of cmnotdisk.conf is to exclude specific devices, usually CD and DVD drives, that Serviceguard does not recognize and which should not be probed. Make sure you do not add a DEVICE_FILE entry in cmnotdisk.conf for any device that should be probed; that is, disk devices being managed by LVM or LVM2. Excluding any such device will cause cmquerycl to fail.
NOTE: The procedures that follow describe the command-line method of configuring LVM storage. There are two other, more automated methods you can use. • System Management Homepage You can use the System Management Homepage to create or extend volume groups and create logical volumes. From the System Management Homepage, choose Disks and File Systems. Make sure you create mirrored logical volumes with PVG-strict allocation; see“Using Mirrored Individual Data Disks” (page 179).
groups. For more information, see Using High Availability Monitors at the address given in the preface to this manual. Using Mirrored Individual Data Disks The procedures that follow use physical volume groups to allow mirroring of individual disks such that each logical volume is mirrored to a disk on a different I/O bus. This is known as PVG-strict mirroring.
Use the following command to display a list of existing volume groups: ls -l /dev/*/group 3. Create the volume group and add physical volumes to it with the following commands: vgcreate -g bus0 /dev/vgdatabase /dev/dsk/c1t2d0 vgextend -g bus1 /dev/vgdatabase /dev/dsk/c0t2d0 CAUTION: Volume groups used by Serviceguard must have names no longer than 35 characters (that is, the name that follows /dev/, in this example vgdatabase, must be at most 35 characters long).
NOTE: You can create file systems by means of the cmpreparestg (1m) command. See “Using Easy Deployment Commands to Configure the Cluster” (page 162) for more information. If you use cmpreparestg, you can skip the procedure that follows, and proceed to “Making Physical Volume Group Files Consistent” (page 183). Use the following commands to create a file system for mounting on the logical volume just created. 1.
1. On ftsys9, copy the mapping of the volume group to a specified file. vgexport -p -s -m /tmp/vgdatabase.map 2. /dev/vgdatabase Still on ftsys9, copy the map file to ftsys10: rcp /tmp/vgdatabase.map ftsys10:/tmp/vgdatabase.map 3. On ftsys10, create the volume group directory: mkdir /dev/vgdatabase 4. Still on ftsys10, create a control file named group in the directory /dev/vgdatabase, as follows: mknod /dev/vgdatabase/group c 64 0xhh0000 Use the same minor number as on ftsys9.
vgchange -a n /dev/vgdatabase Making Physical Volume Group Files Consistent Skip this section if you do not use physical volume groups for mirrored individual disks in your disk configuration, or if you are using cDSFs; see “About Cluster-wide Device Special Files (cDSFs)” (page 104) for more information about cDSFs. Different volume groups may be activated by different subsets of nodes within a Serviceguard cluster.
make a backup of the data in the volume group. See “Migrating from LVM to VxVM Data Storage ” (page 374) for more information about conversion. Initializing Disks for VxVM You need to initialize the physical disks that will be employed in VxVM disk groups.
or the raw (character) device file /dev/vx/rdsk/logdata/log_files. Verify the configuration with the following command: vxprint -g logdata The output of this command is shown in the following example: TY NAME ASSOC v pl logdata fsgen logdata-01 system KSTATE LENGTH ENABLED ENABLED 1024000 1024000 PLOFFS STATE TUTILO PUTILO ACTIVE ACTIVE NOTE: The specific commands for creating mirrored and multi-path storage using VxVM are described in the Veritas Volume Manager Reference Guide.
NOTE: Unlike LVM volume groups, VxVM disk groups are not entered in the cluster configuration file, nor in the package configuration file. Clearimport at System Reboot Time At system reboot time, the cmcluster RC script does a vxdisk clearimport on all disks formerly imported by the system, provided they have the noautoimport flag set, and provided they are not currently imported by another running node.
cmquerycl Options Speeding up the Process In a larger or more complex cluster with many nodes, networks or disks, the cmquerycl command may take several minutes to complete. To speed up the configuration process, you can direct the command to return selected information only by using the -k and -w options: -k eliminates some disk probing, and does not return information about potential cluster lock volume groups and lock physical volumes.
• If you don't use the -h option, Serviceguard will choose the best available configuration to meet minimum requirements, preferring an IPv4 LAN over IPv6 where both are available. The resulting configuration could be IPv4 only, IPv6 only, or a mix of both. You can override Serviceguard's default choices by means of the HEARTBEAT_IP parameter, discussed under “Cluster Configuration Parameters ” (page 109); that discussion also spells out the heartbeat requirements.
The default FIRST_CLUSTER_LOCK_VG and FIRST_CLUSTER_LOCK_PV supplied in the template created with cmquerycl are the volume group and physical volume name of a disk connected to all cluster nodes; if there is more than one, the disk is chosen on the basis of minimum failover time calculations. You should ensure that this disk meets your power wiring requirements. If necessary, choose a disk powered by a circuit which powers fewer than half the nodes in the cluster.
NETWORK_INTERFACE lan3 CLUSTER_LOCK_LUN /dev/dsk/c0t1d1 Specifying a Quorum Server IMPORTANT: The following are standard instructions. For special instructions that may apply to your version of Serviceguard and the Quorum Server see “Configuring Serviceguard to Use the Quorum Server” in the latest version of the HP Serviceguard Quorum Server Version A.04.00 Release Notes, at http://www.hp.
3 lan2 lan2 (nodeA) (nodeB) 4 lan3 lan4 lan3 lan4 lan1 lan1 (nodeC) (nodeC) (nodeD) (nodeD) (nodeC) (nodeD) lan2 lan2 (nodeC) (nodeD) 5 6 IP subnets: IPv4: 15.13.164.0 lan1 lan1 (nodeA) (nodeB) 15.13.172.0 lan1 lan1 (nodeC) (nodeD) 15.13.165.0 lan2 lan2 (nodeA) (nodeB) 15.13.182.0 lan2 lan2 (nodeC) (nodeD) 15.244.65.0 lan3 lan3 (nodeA) (nodeB) 15.244.56.
3 15.244.65.0 4 15.244.56.0 In the Route connectivity section, the numbers on the left (1-4) identify which subnets are routed to each other (for example 15.13.164.0 and 15.13.172.0). IMPORTANT: Note that in this example subnet 15.244.65.0, used by NodeA and NodeB, is not routed to 15.244.56.0, used by NodeC and NodeD. But subnets 15.13.164.0 and 15.13.165.0, used by NodeA and NodeB, are routed respectively to subnets 15.13.172.0 and15.13.182.0, used by NodeC and NodeD.
A Note about Terminology Although you will also sometimes see the term role-based access (RBA) in the output of Serviceguard commands, the preferred set of terms, always used in this manual, is as follows: • Access-control policies- the set of rules defining user access to the cluster. ◦ • Access-control policy - one of these rules, comprising the three parameters USER_NAME, USER_HOST, USER_ROLE. See “Setting up Access-Control Policies” (page 194).
Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 36 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
NOTE: For more information and advice, see the white paper Securing Serviceguard at http:// www.hp.com/go/hpux-serviceguard-docs. Define access-control policies for a cluster in the cluster configuration file; see “Cluster Configuration Parameters ” (page 109). You can define up to 200 access policies for each cluster. A root user can create or modify access control policies while the cluster is running.
only. These roles are not exclusive; for example, you can configure more than one PACKAGE_ADMIN for the same package. NOTE: You do not have to halt the cluster or package to configure or modify access control policies. Here is an example of an access control policy: USER_NAME john USER_HOST bit USER_ROLE PACKAGE_ADMIN If this policy is defined in the cluster configuration file, it grants user john the PACKAGE_ADMIN role for any package on node bit.
NOTE: Check spelling especially carefully when typing wildcards, such as ANY_USER and ANY_SERVICEGUARD_NODE. If they are misspelled, Serviceguard will assume they are specific users or nodes.
• That each node is connected to each heartbeat network. • That all heartbeat networks are of the same type of LAN. • That network interface device files specified are valid LAN device files. • That VOLUME_GROUP entries are marked as cluster-aware. If the cluster is online, the check also verifies that all the conditions for the specific change in configuration have been met.
NOTE: The apply will not complete unless the cluster lock volume group is activated on exactly one node before applying. There is one exception to this rule: a cluster lock had been previously configured on the same physical volume and volume group. After the configuration is applied, the cluster lock volume group must be deactivated.
Table 8 Differences between Legacy CFS and Modular CFS (continued) Legacy Modular Legacy CFS packages can be created using the Veritas Enterprise Administrator (VEA). using cmmakepkg, cmcheckconf, cmapplyconf, cmdeleteconf, cmrunpkg, cmhaltpkg, or cmmodpkg commands. Modular CFS packages can be created using Serviceguard Manager.
Table 9 Operational commands for Legacy CFS and Modular CFS (continued) Operation Modifying attributes for a disk group in a package Commands used by legacy style cfsdgadm modify Equivalent commands in modular style • Edit parameters in the configuration file • cmapplyconf with modified parameters For information on the parameters that can be added, removed, or modified online, see “Online reconfiguration of modular CFS package parameters” (page 211).
the documentation for HP Serviceguard Storage Management Suite posted at http://www.hp.com/ go/hpux-serviceguard-docs. IMPORTANT: Before you proceed, make sure you have read “Planning Veritas Cluster Volume Manager (CVM) and Cluster File System (CFS)” (page 126), which contains important information and cautions.
NOTE: Because the CVM system multi-node package automatically starts up the Veritas processes, do not edit /etc/llthosts, /etc/llttab, or /etc/gabtab. The cfscluster status command displays the status of the disk groups and mount point packages created only for legacy CFS packages, not for modular CFS packages. To view the status of modular CFS packages, use the cmviewcl –v –f line –p command. Creating the Disk Groups Initialize the disk group from the master node. 1.
ftsys10 sw (sw) MOUNT POINT SHARED VOLUME 5. TYPE To view the name of the package that is monitoring a disk group, use the cfsdgadm show_package command: cfsdgadm show_package logdata SG-CFS-DG-1 Creating Volumes 1. Make log_files volume on the logdata disk group: vxassist -g logdata make log_files 1024m 2.
you to create a package configuration file. The command uses the sg/cfs module that contains attributes for CVM disk groups, CFS mount points, CFS storage checkpoint, and snapshot parameters. A modular CFS package template can be generated using the sg/cfs_all template. The modular CFS packages run as a multi-node package and have a dependency on CVM system multi-node package, SG-CFS-pkg, which must be running before the modular CFS packages can be configured or started.
Package template is created. This file must be edited before it can be used. 2. Edit the following package parameters in the cfspkg1.
Modify the package configuration ([y]/n)? y Completed the cluster update 5. Run the package: cmrunpkg cfspkg1 6. Verify that the modular package of disk group(s) and mount point(s) is running.
IMPORTANT: It is recommended to increment the parameter values in smaller numbers and monitor the effect on performance at each level. As soon as the performance starts to decline, stop increasing and reset it to a lower level. However, you must consider the factors like the number of CPUs, the amount of available memory, the HP-UX kernel settings, and the number and characteristics of other packages that will be running on the node.
4. Check the package configuration file: cmcheckconf -P /etc/cmcluster/ckpt1.ascii cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration 5. Apply the package configuration file: cmapplyconf -P /etc/cmcluster/ckpt1.ascii Modify the package configuration ([y]/n)? y Completed the cluster update 6.
NOTE: By default, the package parameters are commented out. You must uncomment them and specify the values. 4. snapshot_mount_point name of the mount point for the snapshot snapshot_name Diskgroups/Logical volume pair used to store the snapshot image snapshot_cfs_mount_point unique name for CFS mount point; supports nested mount points snapshot_mount_options the CFS mount options for the snapshot. For detailed information about the mount options, see the mount_vxfs (1m) manpage.
/dev/vg00/lvol7 10584064 3269339 6857606 /dev/vg00/lvol4 2048000 18290 1902964 /dev/vg00/lvol6 10076160 5666019 4134547 /dev/vg00/lvol8 3063808 17747 2855686 /dev/odm 0 0 0 /dev/vx/dsk/cvm_dg3_node1/lvol3 167776 3260 154234 32% /usr 1% /tmp 58% /opt 1% /home 0% /dev/odm 2% /snap1 Online reconfiguration of modular CFS package parameters You can reconfigure existing modular CFS packages online, i.e., even when the package is running.
In addition to the above parameters, cvm_concurrent_dg_operations and cfs_concurrent_mount_unmount_operations can also be modified online. IMPORTANT: Modifying the cvm_disk_group, cfs_mount_point, ckpt_mount_point or snapshot_mount_point is an offline operation. When you modify any of these parameters, though cmapplyconf will succeed, it is considered that the existing disk group, mount point, checkpoint mount point, or snapshot mount point are deleted and new ones are added respectively.
cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration Apply the configuration: cmapplyconf -P cfspkg1.ascii One or more of the specified packages are running. Any error in the proposed configuration change could cause these packages to fail. Ensure configuration changes have been tested before applying them.
Figure 37 Legacy Style of Packaging Figure 38 illustrates an example in which all the disk group and mount point packages used by Application 1 are consolidated into a single modular package as compared to four separate packages in the legacy style of packaging. Figure 38 Modular Style of Packaging Use Case 2: Migration of two different applications using checkpoint and snapshot packages Figure 39 illustrates a cluster with two applications independent of each other.
Figure 39 Legacy Style of Packaging Figure 40 illustrates the consolidated modular CFS packages. All the disk group and mount point packages used by Application 1 are consolidated into a single modular package. Similarly, the disk group and mount point packages used by Application 2 are consolidated into a single modular package. The disk group, cvm_dg3, is merged with the snapshot package to enable consolidation of storage within the same package.
Managing Disk Groups and Mount Points Using Legacy Packages To manage disk groups and mount points in a Serviceguard cluster using CVM/CFS, create disk group and mount point packages. These packages are configured for activating disk groups and mounting the volumes. The applications that use CVM/CFS require the disk group and mount point packages to be up and running. Creating a File System and Mount Point Package 1.
SG-CFS-MP-1 7. After creating the mount point packages for the cluster file system, configure your application package to depend on the mount points. In the configuration file, configure a dependency, setting dependency_condition to SG-CFS-MP-pkg-# =UP and dependency_location to SAME_NODE. For more information about these parameters, see “Package Parameter Explanations” (page 233).
SG-CFS-pkg SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-CK-1 up up up up running running running running enabled enabled enabled disabled yes no no no /tmp/check_logfiles now contains a point-in-time view of /tmp/logdata/log_files, and it is persistent.
SG-CFS-MP-1 SG-CFS-SN-1 up up running running enabled disabled no no The snapshot file system /local/snap1 is now mounted and provides a point in time view of /tmp/logdata/log_files.
Initializing the Veritas Volume Manager If you are about to create disk groups for the first time, you need to initialize the Volume Manager. Use the following command after installing VxVM/CVM on each node: vxinstall This displays a menu-driven program that steps you through the VxVM/CVM initialization sequence.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: /usr/lib/vxvm/bin/vxdisksetup -i c4t3d4 Creating Disk Groups Use the following steps to create disk groups. 1. Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: vxdg -s init logdata c0t3d2 2.
packages. Use one cvm_dg for each disk group the package will use, and select the appropriate cvm_activation_cmd. Package configuration is described in detail in Chapter 6 (Chapter 7 for legacy packages).
You can use these commands to test cluster operation, as in the following: 1. If the cluster is not already running, start it. From the Serviceguard Manager menu, choose Run Cluster. From the command line, use cmruncl -v. By default, cmruncl will check the networks. Serviceguard will probe the actual network configuration with the network information in the cluster configuration. If you do not need this validation, use cmruncl -v -w none instead, to turn off validation and save time 2.
There are three cases: • The cluster is not running on any node, all cluster nodes must be reachable, and all must be attempting to start up. In this case, the node attempts to form a cluster consisting of all configured nodes. • The cluster is already running on at least one node. In this case, the node attempts to join that cluster. • Neither is true: the cluster is not running on any node, and not all the nodes are reachable and trying to start.
Single-Node Operation Single-node operation occurs in a single-node cluster or in a multi-node cluster, following a situation where all but one node has failed, or where you have shut down all but one node, which will probably have applications running. As long as the Serviceguard daemon cmcld is active, other nodes can rejoin the cluster at a later time. If the Serviceguard daemon fails when in single-node operation, it will leave the single node up and your applications running.
It is recommended that you do not proceed with the configuration operation unless you are sure these nodes are permanently unavailable.Do you want to continue? Reply Yes to remove the configuration. Later, if the inaccessible node becomes available, you should run the cmdeleteconf command on that node to remove the configuration file.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. See “What is Serviceguard? ” (page 21), “How the Package Manager Works” (page 50), and “Package Configuration Planning ” (page 125) for more information.
NOTE: This is a new process for configuring packages, as of Serviceguard A.11.18. This manual refers to packages created by this method as modular packages, and assumes that you will use it to create new packages; it is simpler and more efficient than the older method, allowing you to build packages from smaller modules, and eliminating the separate package control script and the need to distribute it manually.
resources, or subnets, will cause the package to be halted only on the node on which the failure occurred. The Veritas Cluster File System (CFS) system multi-node packages are examples of multi-node packages; but support for multi-node packages is no longer restricted to CVM/CFS; you can create a multi-node package for any purpose. IMPORTANT: If the package uses volume groups, they must be activated in shared mode: vgchange -a s.
Differences between Failover and Multi-Node Packages Note the following important differences in behavior between multi-node and failover packages: • If a multi-node package has auto_run disabled (set to no in the package configuration file) it will not start when the cluster is started. You can use cmmodpkg to enable package switching and start the package for the first time. But if you then halt the multi-node package via cmhaltpkg, it can be re-started only by means of cmrunpkg, not cmmodpkg.
Table 11 Base Modules Module Name Parameters (page) Comments failover package_name (page 234) * module_name (page 234) * module_version (page 234) * package_type (page 234) package_description (page 234) * node_name (page 235) auto_run (page 235) node_fail_fast_enabled (page 235) run_script_timeout (page 236) halt_script_timeout (page 236) successor_halt_timeout (page 237) script_log_file (page 237) operation_sequence (page 237) * log_level (page 237) * failover_policy (page 237) failback_policy (page 2
Table 12 Optional Modules (continued) Module Name Parameters (page) Comments monitor_subnet local_lan_failover_allowed (page 241) Add to a base module to configure subnet monitoring for the package. monitored_subnet (page 241) * monitored_subnet_access (page 241) * cluster_interconnect_subnet (page 241) * package_ip ip_subnet (page 242) * (S) ip_subnet_node (page 243) ip_address (page 243) * (S) Add to failover module to assign relocatable IP addresses to a failover package.
Table 12 Optional Modules (continued) Module Name Parameters (page) Comments package is starting and after they are deactivated while the package is halting. * external external_script (page 252) Add to a base module to specify additional programs to be run during package start and stopped at package halt time. acp user_name (page 253) user_host (page 253) user_role (page 253) Add to a base module to configure Access Control Policies for the package.
NOTE: For more information, see the comments in the editable configuration file output by the cmmakepkg command, and the cmmakepkg manpage.
node_name The node on which this package can run, or a list of nodes in order of priority, or an asterisk (*) to indicate all nodes. The default is *. For system multi-node packages, you must specify *. If you use a list, specify each node on a new line, preceded by the literal node_name, for example: node_name node_name node_name The order in which you specify the node names is important.
NOTE: If the package halt function fails with “exit 1”, Serviceguard does not halt the node, but sets no_restart for the package, which disables package switching (auto_run), thereby preventing the package from starting on any adoptive node. Setting node_fail_fast_enabled to yes ensures that the package can fail over to another node even if the package cannot halt successfully. Be careful when using node_fail_fast_enabled, as it will cause all packages on the node to halt abruptly.
successor_halt_timeout Specifies how long, in seconds, Serviceguard will wait for packages that depend on this package to halt, before halting this package. Can be 0 through 4294, or no_timeout. The default is no_timeout. • no_timeout means that Serviceguard will wait indefinitely for the dependent packages to halt. • 0 means Serviceguard will not wait for the dependent packages to halt before halting this package. New as of A.11.18 (for both modular and legacy packages).
a site-aware disaster-tolerant cluster, which requires Metrocluster (additional HP software); see the documents listed under “Cross-Subnet Configurations” (page 30) for more information. • site_preferred_manual means Serviceguard will try to fail the package over to a node on the local SITE. If there are no eligible nodes on the local SITE, the package will halt with global switching enabled. You can then restart the package locally, when a local node is available, or start it on another SITE.
IMPORTANT: Restrictions on dependency names in previous Serviceguard releases were less stringent. Packages that specify dependency_names that do not conform to the above rules will continue to run, but if you reconfigure them, you will need to change the dependency_name; cmcheckconf and cmapplyconf will enforce the new rules.
dependency_location Specifies where the dependency_condition must be met. • • If dependency_condition is UP, legal values fordependency_location are same_node, any_node, and different_node. ◦ same_node means that the package depended on must be running on the same node. ◦ different_node means that the package depended on must be running on a different node in this cluster. ◦ any_node means that the package depended on must be running on some node in this cluster.
local_lan_failover_allowed This parameter is used to control whether a package fails or not when a primary LAN card fails on a cluster node and its IP addresses are transferred to a standby LAN card. This parameter requires the corresponding subnet to be monitored. Subnet monitoring can be enabled using the monitored_subnet parameter in the package ascii file or cluster_interconnect_subnet parameter on SGeRAC clusters. Legal values are yes and no. Default is yes.
NOTE: Serviceguard supports both IPv4 and IPv6 for an cluster_interconnect_subnet, but before configuring an IPv6 subnet, make sure that your version of Oracle RAC also supports IPv6 for this subnet. See the latest version of Using Serviceguard Extension for RAC at http://www.hp.com/go/ hpux-serviceguard-docs for more information, including the status of Oracle RAC support for an IPv6 cluster_interconnect_subnet. New for A.11.18 (for both modular and legacy packages).
This parameter can be set for failover packages only. Can be added or deleted while the package is running. ip_subnet_node In a cross-subnet configuration, specifies which nodes an ip_subnet is configured on. If no ip_subnet_node are listed under an ip_subnet, it is assumed to be configured on all nodes in this package’s node_name list (page 235). Can be added or deleted while the package is running, with these restrictions: • The package must not be running on the node that is being added or deleted.
For legacy packages, this parameter is in the package control script as well as the package configuration file. service_cmd The command that runs the application or service for this service_name, for example, /usr/bin/X11/xclock -display 15.244.58.208:0 An absolute pathname is required; neither the PATH variable nor any other environment variable is passed to the command. The default shell is /usr/bin/sh. NOTE: Be careful when defining service run commands.
The length and formal restrictions for the name are the same as for package_name (page 234). Each name must be unique within a package, but a single resource can be specified across multiple packages. For example, a generic resource sfm_disk can be specified in two packages, pkg1 and pkg2. You can configure a maximum of 100 generic resources per cluster.
NOTE: Operators other than the ones mentioned above are not supported. This attribute does not accept more than one up criterion. For e.g., >> 10, << 100 are not valid.
resource_up_value Defines a criterion used to determine whether the resource identified by resource_name is up. Requires an operator and a value. Values can be string or numeric. The legal operators are =, !=, >, <, >=, or<=, depending on the type of value. If the type is string, then only = and != are valid. If the string contains white space, it must be enclosed in quotes. String values are case-sensitive. The maximum length of the entire resource_up_value string is 1024 characters.
IMPORTANT: Make sure you read the configuration file comments for both concurrent_vgchange_operations and enable_threaded_vgchange before configuring these options, as well as the vgchange (1m) manpage. vgchange_cmd Specifies the method of activation for each HP-UX Logical Volume Manager (LVM) volume group identified by a vg entry (see (page 248)). Replaces VGCHANGE which is still supported in the package control script for legacy packages; see “Configuring a Legacy Package” (page 302).
NOTE: A volume group must be configured into the cluster (via the VOLUME_GROUP parameter) before you can configure it into a modular package. For more information about cluster parameters, see “Cluster Configuration Parameters ” (page 109). cvm_dg Specifies a CVM disk group (one per cvm_dg, each on a new line) used by this package. CVM disk groups must be specified whether file systems need to be mounted on them or not.
fs_fsck_opt "-s" fs_type "vxfs" A logical volume can be created in an LVM volume group, a Veritas VxVM disk group, or a Veritas CVM disk group. Logical volumes can be entered in any order, regardless of the type of storage group. The parameter explanations that follow provide more detail. For an NFS-imported file system, see the discussion under fs_name (page 251) and fs_server (page 251).
fs_name • For a local file system, this parameter, in conjunction with fs_directory, fs_type, fs_mount_opt, fs_umount_opt, and fs_fsck_opt, specifies the file system that is to be mounted by the package. fs_name must specify the block devicefile for a logical volume. • For an NFS-imported file system, the additional parameters required are fs_server, fs_directory, fs_type, and fs_mount_opt; see fs_server (page 251) for an example.
fs_mount_opt The mount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. See the mount (1m) manpage for more information. fs_umount_opt The umount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. Using the-s option of umount will improve startup performance if the package uses a large number of file systems. See the mount (1m) manpage for more information.
user_name Specifies the name of a user who has permission to administer this package. See also user_host and user_role; these three parameters together define the access-control policy for this package (see “Controlling Access to the Cluster” (page 192)). These parameters must be defined in this order: user_name, user_host, user_role. Legal values for user_name are any_user or a maximum of eight login names from /etc/ passwd on user_host.
Generating the Package Configuration File When you have chosen the configuration modules your package needs (see “Choosing Package Modules” (page 228)), you are ready to generate a package configuration file that contains those modules. This file will consist of a base module (usually failover, multi-node or system multi-node) plus the modules that contain the additional parameters you have decided to include.
• To generate a configuration file for a failover package that runs an application that requires another package to be up (enter the command all on one line): cmmakepkg -m sg/failover -m sg/dependency -m sg/service $SGCONF/pkg1/pkg1.conf • To generate a configuration file adding the services module to an existing package (enter the command all on one line): cmmakepkg -i $SGCONF/pkg1/pkg1.conf -m sg/service $SGCONF/pkg1/pkg1_v2.conf NOTE: You can add more than one module at a time.
NOTE: Optional parameters are commented out in the configuration file (with a # at the beginning of the line). In some cases these parameters have default values that will take effect unless you uncomment the parameter (remove the #) and enter a valid value different from the default. Read the surrounding comments in the file, and the explanations in this chapter, to make sure you understand the implications both of accepting and of changing a given default.
• local_lan_failover_allowed. Use this parameter control whether a package fails or not when a primary LAN card fails on a cluster node and its IP addresses are transferred to a standby LAN card. Legal values are yes and no. Default is yes. See the parameter description (page 241) for more information. • Use the monitored_subnet parameter to specify a subnet that is to be monitored for this package. If there are multiple subnets, repeat the parameter as many times as needed, on a new line each time.
vgchange_cmd, and use the fs_ options in the FILESYSTEMS portion of the configuration file to specify the options for mounting and unmounting the file systems. Do not use the vxvm_dg or cvm_dg parameters for LVM volume groups.
• To coordinate the startup and shutdown of database software with cluster node startup and shutdown, you can use the database template files provided with the separately purchasable Enterprise Cluster Master Toolkit product. These files are in /opt/cmcluster/toolkit/DB/. Separate toolkits are available for Oracle, Informix, and Sybase. In addition to the standard package script, you use the special script that is provided for the database.
NOTE: For modular packages, you now need to distribute any external scripts identified by the external_pre_script and external_script parameters. But if you are accustomed to configuring legacy packages, note that you do not have to create a separate package control script for a modular package, or distribute it manually. (You do still have to do this for legacy packages; see “Configuring a Legacy Package” (page 302).
7 Cluster and Package Maintenance This chapter describes how to see cluster configuration and status information, how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Viewing CFS Multi-Node Information If you are running Veritas Cluster File System (CFS) cluster, you can use cfs commands to see multi-node package configuration information, status, and dependencies; for example: cfsdgadm show_package cfsmntadm show_package getconf -p | grep dependency Types of Cluster and Package States A cluster or its component nodes may be in several different states at different points in time.
• halt_wait — A cmhaltpkg command is in progress for this package. The package is waiting to be halted, but the halt script cannot start because the package is waiting for packages that depend on it (successors) to halt. The parameter description for successor_halt_timeout (page 237) provides more information. • failing — The package is halting because it, or a package it depends on, has failed.
The following states are possible only for multi-node packages: • blocked — The package has never run on this node, either because a dependency has not been met, or because auto_run is set to no. • changing — The package is in a transient state, different from the status shown, on some nodes. For example, a status of starting with a state of changing would mean that the package was starting on at least one node, but in some other, transitory condition (for example, failing) on at least one other node.
Failover and Failback Policies Failover packages can be configured with one of two values for the failover_policy parameter (page 237), as displayed in the output of cmviewcl -v: • configured_node. The package fails over to the next node in the node_name list in the package configuration file (page 235). • min_package_node. The package fails over to the node in the cluster that has the fewest running packages.
Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM Service Service Subnet Generic Resource STATUS up up up up Node_Switching_Parameters: NODE_TYPE STATUS Primary up Alternate up MAX_RESTARTS 0 0 0 RESTARTS 0 0 0 SWITCHING enabled enabled NAME ftsys10 ftsys9 NAME service2 sfm_disk1_monitor 15.13.168.
SG-CFS-pkg NODE_NAME ftsys9 up running STATUS up yes SWITCHING enabled Script_Parameters: ITEM STATUS MAX_RESTARTS Service up 0 Service up 5 Service up 5 Service up 0 Service up 0 NODE_NAME ftsys10 enabled STATUS up RESTARTS 0 0 0 0 0 NAME SG-CFS-vxconfigd SG-CFS-sgcvmd SG-CFS-vxfsckd SG-CFS-cmvxd SG-CFS-cmvxpingd SWITCHING enabled Script_Parameters: ITEM STATUS MAX_RESTARTS Service up 0 Service up 5 Service up 5 Service up 0 Service up 0 RESTARTS 0 0 0 0 0 NAME SG-CFS-vxconfigd SG-CFS-sgcvmd
PRIMARY STANDBY up up 28.1 32.1 lan0 lan1 STATE halted AUTO_RUN disabled UNOWNED PACKAGES PACKAGE pkg2 STATUS down NODE unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM Resource Subnet Generic Resource Resource Subnet Generic Resource STATUS up up up down up up Node_Switching_Parameters: NODE_TYPE STATUS Primary up Alternate up NODE_NAME ftsys9 ftsys9 ftsys9 ftsys10 ftsys10 ftsys10 NAME /example/float 15.13.168.
NODE_TYPE Primary Alternate PACKAGE pkg2 STATUS up up STATUS up SWITCHING enabled enabled STATE running NAME ftsys9 ftsys10 AUTO_RUN disabled (current) NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM Service Service Subnet Generic Resource STATUS up up up up Node_Switching_Parameters: NODE_TYPE STATUS Primary up Alternate up NODE ftsys10 STATUS up MAX_RESTARTS 0 0 0 RESTARTS 0 0 0 SWITCHING enabled enabled NAME ftsy
CLUSTER example STATUS up NODE ftsys9 STATUS up PACKAGE pkg1 pkg2 NODE ftsys10 STATE running STATUS up up STATUS down STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE halted This output can be seen on both ftsys9 and ftsys10. Viewing Information about Unowned Packages The following example shows packages that are currently unowned, that is, not running on any configured node.
PACKAGE pkg1 NODE tabby STATUS up STATUS up PACKAGE pkg2 STATE running AUTO_RUN enabled NODE manx AUTO_RUN enabled NODE tabby STATE running STATUS up STATE running SYSTEM_MULTI_NODE_PACKAGES: PACKAGE SG-CFS-pkg STATUS up STATE running Checking Status of the Cluster File System (CFS) If the cluster is using the cluster file system, you can check status with the cfscluster command, as in the example that follows.
SG-CFS-pkg up running enabled yes NODE_NAME STATUS SWITCHING soy up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd NODE_NAME STATUS SWITCHING tofu up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service
ftsys10 sw (sw) MOUNT POINT SHARED VOLUME TYPE ... To see which package is monitoring a disk group, use the cfsdgadm show_package command. For example, for the disk group logdata, enter: cfsdgadm show_package logdata SG-CFS-DG-1 Status of Legacy CFS Mount Point Packages To see the status of the legacy CFS mount point package, use the cfsmntadm display command.
NOTE: • All of the checks below are performed when you run cmcheckconf without any arguments (or with only -v, with or without -k or -K). cmcheckconf validates the current cluster and package configuration, including external scripts and pre-scripts for modular packages, and runs cmcompare to check file consistency across nodes. (This new version of the command also performs all of the checks that were done in previous releases.) See “Checking Cluster Components” (page 274) for details.
NOTE: The table includes all the checks available as of A.11.20, not just the new ones. Table 13 Verifying Cluster Components Component (Context) Tool or Command; More Information Comments Volume groups (cluster) cmcheckconf (1m), cmapplyconf (1m) Checked for: See also “Verifying the Cluster Configuration ” (page 197).
Table 13 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Comments File consistency (cluster) cmcheckconf (1m), cmcompare (1m). To check file consistency across all nodes in the cluster, do the IMPORTANT: See the manpage for following: differences in return codes from 1. Customize /etc/cmcluster/ cmcheckconf without options versus cmfiles2check cmcheckconf -C 2. Distribute it to all nodes using cmsysnc (1m) 3.
Setting up Periodic Cluster Verification You can use cron (1m) to run cluster verification at a fixed interval. Specify the commands to run in a crontab file (see crontab (1)). NOTE: The job must run on one of the nodes in the cluster. Because only the root user can run cluster verification, and cron (1m) sets the job’s user and group ID’s to those of the user who submitted the job, you must edit the file /var/spool/cron/crontabs/root as the root user.
• Halting the Entire Cluster • Halting a Node or the Cluster while Keeping Packages Running (page 280) In Serviceguard A.11.16 and later, these tasks can be performed by non-root users with the appropriate privileges, except where specifically noted. See “Controlling Access to the Cluster” (page 192) for more information about configuring access. You can use Serviceguard Manager or the Serviceguard command line to start or stop the cluster, or to add or halt nodes.
Using Serviceguard Commands to Add Previously Configured Nodes to a Running Cluster Use the cmrunnode command to join one or more nodes to an already running cluster. Any node you add must already be a part of the cluster configuration. The following example adds node ftsys8 to the cluster that was just started with only nodes ftsys9 and ftsys10.
Automatically Restarting the Cluster You can configure your cluster to automatically restart after an event, such as a long-term power failure, which brought down all nodes in the cluster. This is done by setting AUTOSTART_CMCLD to 1 in the /etc/rc.config.d/cmcluster file. Halting a Node or the Cluster while Keeping Packages Running There may be circumstances in which you want to do maintenance that involves halting a node, or the entire cluster, without halting or failing over the affected packages.
• Live Application Detach is supported only for modular failover packages and modular multi-node packages. ◦ You cannot detach legacy packages, but you can halt them and then detach the modular packages. ◦ You cannot use Live Application Detach if system multi-node packages (such as SG-CFS-pkg) are configured in the cluster. See Chapter 6 (page 227) for more information about package types.
IMPORTANT: HP does not recommend such configurations. See “ip_subnet” (page 242). • You cannot detach packages when the HP-UX Clustered I/O Congestion Control feature is enabled. • As of the date of this manual, you cannot detach ECMT-based packages. Additional Points To Note Keep the following points in mind. • When packages are detached, they continue to run, but without high availability protection.
• If you halt a package and disable it before running cmhaltcl -d to detach other packages running in the cluster, auto_run will be automatically re-enabled for this package when the cluster is started again, forcing the package to start. To prevent this behavior and keep the package halted and disabled after the cluster restarts, change auto_run to no in the package configuration file (page 235), and re-apply the package, before running cmhaltcl -d.
NOTE: 3. If you do not do this, the cmhaltcl in the next step will fail. Halt the cluster with the -d (detach) option: cmhaltcl -d NOTE: -d and -f are mutually exclusive. See cmhaltcl (1m) for more information.
The cluster must be running, and if the package is dependent on other packages, those packages must be either already running, or started by the same command that starts this package (see the section that follows, and “About Package Dependencies” (page 137).) Starting a Package that Has Dependencies Before starting a package, it is a good idea to use the cmviewcl command to check for package dependencies. You cannot start a package unless all the packages that it depends on are running.
You can use Serviceguard Manager, or Serviceguard commands as shown below, to halt a package. Using Serviceguard Commands to Halt a Package Use the cmhaltpkg command to halt a package, as follows: cmhaltpkg pkg1 This halts pkg1, and, if pkg1 is a failover package, also disables it from switching to another node. You cannot halt a package unless all packages that depend on it are down. If you try, Serviceguard will take no action, except to send a message indicating that not all dependent packages are down.
You can disable package switching to particular nodes by using the -n option of the cmmodpkg command. The following prevents pkg1 from switching to node lptest3: cmmodpkg -d -n lptest3 pkg1 To permanently disable switching so that the next time the cluster restarts, the change you made in package switching is still in effect, change the auto_run flag in the package configuration file, then re-apply the configuration. (See “Reconfiguring a Package on a Running Cluster ” (page 311).
• • Node-wide and cluster-wide events affect the package as follows: ◦ If the node the package is running on is halted or crashes, the package will no longer be in maintenance mode but will not be automatically started. ◦ If the cluster is halted or crashes, the package will not be in maintenance mode when the cluster comes back up. Serviceguard will attempt to start it if auto_run is set to yes in the package configuration file.
Additional Rules for Partial-Startup Maintenance Mode • You must halt the package before taking it out of partial-startup maintenance mode. • To run a package normally after running it in partial-startup maintenance mode, you must take it out of maintenance mode, and then restart it.
NOTE: If you have a package configured with generic resources and you attempt to take it out of the maintenance mode back to the running state, the status of generic resources are evaluated. If any of the generic resources is 'down', the package cannot be taken out of the maintenance mode.
cmrunpkg -e sg/service pkg1 This runs all the package's modules except the services module. You can also use -e in combination with -m. This has the effect of starting all modules up to and including the module identified by -m, except the module identified by -e. In this case the excluded (-e) module must be earlier in the execution sequence (as listed near the top of the package's configuration file) than the -m module.
Table 14 Types of Changes to the Cluster Configuration (continued) Change to the Cluster Configuration Required Cluster State If removing the NIC from the system, see “Removing a LAN or VLAN Interface from a Node” (page 300). Change the designation of an existing interface from Cluster can be running. See “Changing the Cluster Networking HEARTBEAT_IP to STATIONARY_IP, or vice versa Configuration while the Cluster Is Running” (page 297).
What You Can Preview You can preview any of the following, or all of them simultaneously: • Cluster bring-up (cmruncl) • Cluster node state changes (cmrunnode, cmhaltnode) • Package state changes (cmrunpkg, cmhaltpkg) • Package movement from one node to another • Package switching changes (cmmodpkg -e) • Availability of package subnets, EMS resources, and storage • Changes in package priority, node order, dependency, failover and failback policy, node capacity and package weight Using Preview
NOTE: The preview cannot predict run and halt script failures. For more information about package dependencies and priorities, see “About Package Dependencies” (page 137). Using cmeval You can use cmeval to evaluate the effect of cluster changes on Serviceguard packages. You can also use it simply to preview changes you are considering making to the cluster as a whole. NOTE: Preview of generic resources is not yet supported with cmeval.
IMPORTANT: For detailed information and examples, see the cmeval (1m) manpage. Updating the Cluster Lock Configuration Use the procedures that follow whenever you need to change the device file names of the cluster lock physical volumes.
Reconfiguring a Running Cluster This section provides instructions for changing the cluster configuration while the cluster is up and running. Note the following restrictions: • You cannot remove an active node from the cluster. You must halt the node first. • You cannot delete an active volume group from the cluster configuration. You must halt any package that uses the volume group and ensure that the volume is inactive before deleting it.
there are packages that depend on the unreachable node, halt the cluster or use Serviceguard commands as described in the next section. Use the following procedure to delete a node with HP-UX commands. In this example, nodes ftsys8, ftsys9 and ftsys10 are already configured in a running cluster named cluster1, and you are deleting node ftsys10. NOTE: If you want to remove a node from the cluster, run the cmapplyconf command from another node in the same cluster.
• Change IP Monitor parameters: SUBNET, IP_MONITOR, POLLING TARGET; see the entries for these parameters under “Cluster Configuration Parameters ” (page 109) for more information. • A combination of any of these in one transaction (cmapplyconf), given the restrictions below. What You Must Keep in Mind The following restrictions apply: • You must not change the configuration of all heartbeats at one time, or change or delete the only configured heartbeat.
Example: Adding a Heartbeat LAN Suppose that a subnet 15.13.170.0 is shared by nodes ftsys9 and ftsys10 in a two-node cluster cluster1, and you want to add it to the cluster configuration as a heartbeat subnet. Proceed as follows. 1. Run cmquerycl to get a cluster configuration template file that includes networking information for interfaces that are available to be added to the cluster configuration: cmquerycl -c cluster1 -C clconfig.ascii NOTE: As of Serviceguard A.11.
3. 4. In the case of a legacy package, add the new networking information to the package control script if necessary Apply the new package configuration, and redistribute the control script if necessary. For more information, see “Reconfiguring a Package on a Running Cluster ” (page 311). Example: Deleting a Subnet Used by a Package In this example, we are deleting subnet 15.13.170.0 (lan0). This will also mean deleting lan3, which is a standby for lan0 and not shared by any other primary LAN.
3. 4. 5. 6. Edit clconfig.ascii and delete the line(s) specifying the NIC name and its IP address(es) (if any) from the configuration. Run cmcheckconf to verify the new configuration. Run cmapplyconf to apply the changes to the configuration and distribute the new configuration file to all the cluster nodes. Runolrad -d to remove the NIC. See also “Replacing LAN or Fibre Channel Cards” (page 327).
Similarly, you can delete VxVM or CVM disk groups provided they are not being used by a cluster node at the time. CAUTION: Serviceguard manages the Veritas processes, specifically gab and LLT. This means that you should never use administration commands such as gabconfig, llthosts, and lltconfig to administer a cluster. It is safe to use the read-only variants of these commands, such as gabconfig -a. But a Veritas administrative command could potentially crash nodes or the entire cluster.
prioritized list of cluster nodes on which the package can run together with definitions of the acceptable types of failover allowed for the package. You can create a legacy package and its control script in Serviceguard Manager; use the Help for detailed instructions. Otherwise, use the following procedure to create a legacy package. 1. Create a subdirectory for each package you are configuring in the /etc/cmcluster directory: mkdir /etc/cmcluster/pkg1 You can use any directory names you like. 2.
NOTE: For modular packages, the default form for parameter names in the package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameter names are concerned. Because this section is intended to be used primarily when you are reconfiguring an existing legacy package, we are using the legacy parameter names (in upper case) for sake of continuity.
IMPORTANT: A.11.18. • Note that the rules for valid SERVICE_NAMEs are more restrictive as of To configure monitoring for a registered resource, enter values for the following parameters. ◦ RESOURCE_NAME ◦ RESOURCE_POLLING_INTERVAL ◦ RESOURCE_UP_VALUE ◦ RESOURCE_START For more information, see “Parameters for Configuring EMS Resources” (page 136), and the resource_ parameter descriptions (page 246). NOTE: script.
Customizing the Package Control Script You need to customize as follows. See the entries for the corresponding modular-package parameters under “Package Parameter Explanations” (page 233) for more discussion. • Update the PATH statement to reflect any required paths needed to start your services.
Adding Customer Defined Functions to the Package Control Script You can add additional shell commands to the package control script to be executed whenever the package starts or stops. Enter these commands in the CUSTOMER DEFINED FUNCTIONS area of the script. If your package needs to run short-lived processes, such as commands to initialize or halt a packaged application, you can also run these from the CUSTOMER DEFINED FUNCTIONS.
Verifying the Package Configuration Serviceguard checks the configuration you create and reports any errors. For legacy packages, you can do this in Serviceguard Manager: click Check to verify the package configuration you have done under any package configuration tab, or to check changes you have made to the control script. Click Apply to verify the package as a whole. See the local Help for more details.
Distributing the Binary Cluster Configuration File with HP-UX Commands Use the following steps from the node on which you created the cluster and package configuration files: • Verify that the configuration file is correct. Use the following command (all on one line): cmcheckconf -C /etc/cmcluster/cmcl.conf -P /etc/cmcluster/pkg1/pkg1.
Configuring monitored_subnet_access In order to monitor subnet 15.244.65.0 or 15.244.56.0, you would configure monitored_subnet and monitored_subnet_access in pkg1’s package configuration file as follows: monitored_subnet 15.244.65.0 monitored_subnet_access PARTIAL monitored_subnet 15.244.56.
that can be made online to the modular CFS package parameters, see “Online reconfiguration of modular CFS package parameters” (page 211). CAUTION: is running. Be extremely cautious about changing a package's configuration while the package If you reconfigure a package online (by executing cmapplyconf on a package while the package itself is running) it is possible that the package will fail, even if the cmapplyconf succeeds, validating the changes with no errors.
IMPORTANT: Restrictions on package names, dependency names, and service names have become more stringent as of A.11.18. Packages that have or contain names that do not conform to the new rules (spelled out under package_name (page 234)) will continue to run, but if you reconfigure these packages, you will need to change the names that do not conform; cmcheckconf and cmapplyconf will enforce the new rules. 4. Verify your changes as follows: cmcheckconf -v -P pkg1.conf 5.
To create the CVM disk group or CFS mount point multi-node packages on systems that support CFS, see “Creating the Disk Group Cluster Packages” (page 203) and “Creating a File System and Mount Point Package” (page 216). Deleting a Package from a Running Cluster Serviceguard will not allow you to delete a package if any other package is dependent on it. To check for dependencies, use cmviewcl -v -l . System multi-node packages cannot be deleted from a running cluster.
Resetting the Service Restart Counter The service restart counter is the number of times a package service has been automatically restarted. This value is used to determine when the package service has exceeded its maximum number of allowable automatic restarts. When a package service successfully restarts after several attempts, the package manager does not automatically reset the restart count. You can reset the counter online using the cmmodpkg -R -s command.
NOTE: All the nodes in the cluster must be powered up and accessible when you make package configuration changes. Table 15 Types of Changes to Packages Change to the Package Required Package State Delete a package or change package name Package must not be running. Change package type Package must not be running. Add or delete a module: modular package Package can be running. NOTE: it.
Table 15 Types of Changes to Packages (continued) Change to the Package Required Package State Add or remove monitoring for a Package can be running. subnet: monitored_subnet for a Serviceguard will not allow the change if the subnet being added is down, as modular package or SUBNET (in that would cause the running package to fail. the package configuration file) for a legacy package Change Package can be running.
Table 15 Types of Changes to Packages (continued) Change to the Package Required Package State If only fs_umount_opt is being changed, the file system will not be unmounted; the new option will take effect when the package is halted or the file system is unmounted for some other reason. Add a file system: modular package Package can be running. Add or change a file system: legacy Package must not be running. package Remove a file system: modular package Package should not be running.
Table 15 Types of Changes to Packages (continued) Change to the Package Required Package State Change Package can be running. concurrent_vgchange_operations, These changes in themselves will not cause any volume group to be activated or deactivation_retry_count, deactivated. enable_threaded_vgchange: modular package Change cvm_activation_cmd Package must not be running. Add, change, or delete pev_: modular package Package can be running. Change package auto_run Package can be running.
Table 15 Types of Changes to Packages (continued) Change to the Package Required Package State For information on online changes to generic resources, see “Online Reconfiguration of Generic Resources” (page 136). Change the generic_resource_up_criteria Package can be running for resources of evaluation type before_package_start or during_package_start provided the new up criteria does not cause the resource status to evaluate to 'down'.
Single-Node Operation In a multi-node cluster, you could have a situation in which all but one node has failed, or you have shut down all but one node, leaving your cluster in single-node operation. This remaining node will probably have applications running on it. As long as the Serviceguard daemon cmcld is active, other nodes can rejoin the cluster. If the Serviceguard daemon fails when in single-node operation, it will leave the single node up and your applications running.
8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
cmsetresource -r –s down 3. To view the package status, enter cmviewcl -v The package should be running on the specified adoptive node. 4. Move the package back to the primary node (see “Moving a Failover Package ” (page 286)). NOTE: If there was a monitoring script configured for this generic resource, then the monitoring script would also be attempting to set the status of the generic resource.
• LAN cards • Power sources • All cables • Disk interface cards Some monitoring can be done through simple physical inspection, but for the most comprehensive monitoring, you should examine the system log file (/var/adm/syslog/syslog.log) periodically for reports on all configured HA devices. The presence of errors relating to a device will show the need for maintenance. When the proper redundancy has been configured, failures can occur with no external symptoms. Proper monitoring is important.
Using HP ISEE (HP Instant Support Enterprise Edition) In addition to messages reporting actual device failure, the logs may accumulate messages of lesser severity which, over time, can indicate that a failure may happen soon. One product that provides a degree of automation in monitoring is called HP ISEE, which gathers information from the status queues of a monitored system to see what errors are accumulating.
5. On the node from which you issued the lvreduce command, issue the following command to restore the volume group configuration data to the newly inserted disk: vgcfgrestore -n /dev/vg_sg01 /dev/dsk/c2t3d0 6. Issue the following command to extend the logical volume to the newly inserted disk: lvextend -m 1 /dev/vg_sg01 /dev/dsk/c2t3d0 7. Finally, use the lvsync command for each logical volume that has extents on the failed physical volume.
IMPORTANT: If you need to replace a LUN under the HP-UX 11i v3 agile addressing scheme, also used by cDSFs (see “About Device File Names (Device Special Files)” (page 80) and “About Cluster-wide Device Special Files (cDSFs)” (page 104), and you use the same DSF, you may need to use the io_redirect_dsf(1M) command to reassign the existing DSF to the new device, depending on whether the operation changes the WWID of the LUN; see the section on io_redirect_dsf in the white paper The Next Generation Mass Storag
Normally disconnecting any portion of the SCSI bus will leave the SCSI bus in an unterminated state, which will cause I/O errors for other nodes connected to that SCSI bus, so the cluster would need to be halted before disconnecting any portion of the SCSI bus. However, it is not necessary to bring the cluster down to do this if you are using a SCSI configuration that allows disconnection of a portion of the SCSI bus without losing termination.
NOTE: After replacing a Fibre Channel I/O card, it may necessary to reconfigure the SAN to use the World Wide Name (WWN) of the new Fibre Channel card if Fabric Zoning or other SAN security requiring WWN is used.
NOTE: While the old Quorum Server is down and the new one is being set up, these things can happen: • These three commands will not work: cmquerycl -q cmapplyconf -C cmcheckconf -C • If there is a node or network failure that creates a 50-50 membership split, the quorum server will not be available as a tie-breaker, and the cluster will fail. NOTE: Make sure that the old Quorum Server system does not rejoin the network with the old IP address.
directory, by default. You can use a text editor, such as vi, or the more command to view the log file for historical information on your cluster. It is always a good idea to review the syslog.log file on each of the nodes in the cluster when troubleshooting cluster problems. This log provides information on the following: • Commands executed and their outcome. • Major cluster events which may, or may not, be errors. • Cluster status information.
Reviewing Serviceguard Manager Log Files From the System Management Homepage (SMH), click Tools, then select Serviceguard Manager, select the cluster you are interested and then choose View -> Operation Log. Reviewing the System Multi-node Package Files If you are running Veritas Cluster Volume Manager and you have problems starting the cluster, check the log file for the system multi-node package. For CVM 4.1 and later, the file is SG-CFS-pkg.log.
Using the cmviewconf Command cmviewconf allows you to examine the binary cluster configuration file, even when the cluster is not running. The command displays the content of this file on the node where you run the command. Reviewing the LAN Configuration The following networking commands can be used to diagnose problems: • netstat -in can be used to examine the LAN configuration. This command lists all IP addresses assigned to each LAN interface card.
Networking and Security Configuration Errors Many Serviceguard commands, including cmviewcl, depend on name resolution services to look up the addresses of cluster nodes. When name services are not available (for example, if a name server is down), Serviceguard commands may hang, or may return a network-related error message. If this happens, use the nslookup command on each cluster node to see whether name resolution is correct. For example: nslookup ftsys9 Name Server: server1.cup.hp.com Address: 15.13.
largest number says that cmcld was unable to run for the last 1.6 seconds, increase MEMBER_TIMEOUT to more than 16 seconds. 2. This node is at risk of being evicted from the running cluster. Increase MEMBER_TIMEOUT. This means that the hang was long enough for other nodes to have noticed the delay in receiving heartbeats and marked the node “unhealthy”.
“Halted.” Similarly, when a package control script fails, Serviceguard kills the script and marks the package “Halted.” In both cases, the following also take place: • Control of a failover package will not be transferred. • The run or halt instructions may not run to completion. • auto_run (automatic package switching) will be disabled. • The current node will be disabled from running the package.
If after cleaning up the node on which the timeout occurred it is desirable to have that node as an alternate for running the package, remember to re-enable the package to run on the node: cmmodpkg -e -n If the package failed because of a failure in generic resource of evaluation type during_package_start, then the status of the generic resource will be 'down'. The switching parameter will be disabled on the node. To enable switching: • First fix the resource that went down.
You can also check the syslog file for information. CAUTION: Do not use the HP-UX mount and umount commands in a CFS cluster. Use cfsmount and cfsumount for legacy CFS packages; cmhaltpkg and cmrunpkg for modular CFS packages. Commands such as mount -o cluster, dbed_chkptmount, or sfrac_chkptmount could cause conflicts with subsequent command operations on the file system or Serviceguard packages.
Possible node failures can be caused by the following conditions: • HPMC. This is a High Priority Machine Check, a system panic caused by a hardware error. • TOC • Panics • Hangs • Power failures In the event of a TOC, a system dump is performed on the failed node and numerous messages are also displayed on the console.
Attempt to get lock /sg/cluser1 unsuccessful. Reason: request_timedout Messages Serviceguard sometimes sends a request to the Quorum Server to set the lock state. (This is different from a request to obtain the lock in tie-breaking.
A Enterprise Cluster Master Toolkit The Enterprise Cluster Master Toolkit (ECMT) provides a group of example scripts and package configuration files for creating Serviceguard packages for several major database and internet software products. Each toolkit contains a README file that explains how to customize the package for your needs. The ECMT can be installed on HP-UX 11i v2, or 11i v3.
B Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 342) • Designing Applications to Run on Multiple Systems (page 345) • Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY (page 349) • Restoring Client Connections (page 350) • Handling Applicatio
Insulate Users from Outages Wherever possible, insulate your end users from outages. Issues include the following: • Do not require user intervention to reconnect when a connection is lost due to a failed server. • Where possible, warn users of slight delays due to a failover in progress. • Minimize the reentry of data. • Engineer the system for reserve capacity to minimize the performance degradation experienced by users.
Replicate Non-Data File Systems Non-data file systems should be replicated rather than shared. There can only be one copy of the application data itself. It will be located on a set of disks that is accessed by the system that is running the application. After failover, if these data disks are file systems, they must go through file systems recovery (fsck) before the data can be accessed. To help reduce this recovery time, the smaller these file systems are, the faster the recovery will be.
Use Restartable Transactions Transactions need to be restartable so that the client does not need to re-enter or back out of the transaction when a server fails, and the application is restarted on another system. In other words, if a failure occurs in the middle of a transaction, there should be no need to start over again from the beginning. This capability makes the application more robust and reduces the visibility of a failover to the user. A common example is a print job.
Another possibility is having two applications that are both active. An example might be two application servers which feed a database. Half of the clients connect to one application server and half of the clients connect to the second application server. If one server fails, then all the clients connect to the remaining application server. Design for Replicated Data Sites Replicated data sites are a benefit for both fast failover and disaster recovery.
network interface must have a stationary IP address associated with it. This IP address does not move to a remote system in the event of a network failure. Obtain Enough IP Addresses Each application receives a relocatable IP address that is separate from the stationary IP address assigned to the system itself. Therefore, a single system might have many IP addresses, one for itself and one for each of the applications that it normally runs.
One way to look for problems in this area is to look for calls to gethostname(2) in the application. HA services should use gethostname() with caution, since the response may change over time if the application migrates. Applications that use gethostname() to determine the name for a call to gethostbyname(2) should also be avoided for the same reason. Also, the gethostbyaddr() call may return different answers over time if called with a stationary IP address.
With UDP datagram sockets, however, there is a problem. The client may connect to multiple servers utilizing the relocatable IP address and sort out the replies based on the source IP address in the server’s response message. However, the source IP address given in this response will be the stationary IP address rather than the relocatable application IP address.
Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY CAUTION: The procedure in this section depends on setting the HP-UX kernel parameter ip_strong_es_model. HP supports setting this parameter for use with Serviceguard only if you are not using a cross-subnet configuration (page 30). Otherwise, leave the parameter at its default setting (zero, meaning disabled) and do not attempt to use the procedure that follows.
/usr/sbin/route delete net default 128.17.17.1 1 source 128.17.17.17 Once the per-interface default route(s) have been added, netstat –rn would show something like the following, where 128.17.17.17 is the package relocatable address and 128.17.17.19 is the physical address on the same subnet: Destination 127.0.0.1 128.17.17.19 128.17.17.17 192.168.69.82 128.17.17.0 128.17.17.0 192.168.69.0 127.0.0.0 default default Gateway 127.0.0.1 128.17.17.19 128.17.17.17 192.168.69.82 128.17.17.19 128.17.17.17 192.168.
application. If the application can be restarted on the same node after a failure (see “Handling Application Failures ” following), the retry to the current server should continue for the amount of time it takes to restart the server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness.
Minimizing Planned Downtime Planned downtime (as opposed to unplanned downtime) is scheduled; examples include backups, systems upgrades to new operating system revisions, or hardware replacements. For planned downtime, application designers should consider: • Reducing the time needed for application upgrades/patches.
Providing Online Application Reconfiguration Most applications have some sort of configuration information that is read when the application is started. If to make a change to the configuration, the application must be halted and a new configuration file read, downtime is incurred. To avoid this downtime use configuration tools that interact with an application and make dynamic changes online. The ideal solution is to have a configuration tool which interacts with the application.
C Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the Appendix “Designing Highly Available Cluster Applications.” 2.
1. 2. 3. 4. 5. Install the application, database, and other required resources on one of the systems. Be sure to follow Serviceguard rules in doing this: • Install all shared data on separate external volume groups. • Use JFS/VxFS file systems as appropriate. Perform some sort of standard test to ensure the application is running correctly. This test can be used later in testing with Serviceguard. If possible, try to connect to the application through a client.
NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information about support for CVM and CFS: http://www.hp.com/go/hpux-serviceguard-docs. Testing the Cluster 1. Test the cluster: • Have clients connect. • Provide a normal system load. • Halt the package on the first node and move it to the second node: cmhaltpkg pkg1 cmrunpkg -n node2 pkg1 cmmodpkg -e pkg1 • Move it back.
D Software Upgrades There are five types of upgrade you can do: • rolling upgrade • rolling upgrade using DRD • non-rolling upgrade • non-rolling upgrade using DRD • migration with cold install Each of these is discussed below. Special Considerations for Upgrade to Serviceguard A.11.20 • Before you proceed, read the sections Upgrading from an Earlier Serviceguard Release and Rolling Upgrade in the latest version of the release notes for A.11.20 • Serviceguard A.11.
CAUTION: From the time when the old cluster manager is shut down until the new cluster manager forms its first cluster, a node failure will cause the entire cluster to fail. HP strongly recommends that you use no Serviceguard commands other than cmviewcl (1m) until the new cluster manager successfully completes its first cluster re-formation.
Restrictions for DRD Upgrades • Upgrade using DRD is supported only on HP-UX 11i v3. • You must use the DRD software released with the September 2009 release of HP-UX 11i v3 or later. You can obtain the DRD software free from HP (see “Rolling Upgrade Using DRD” ); you do not have to upgrade the operating system itself, so long as you are running 11i v3. • The cluster must meet both general and release-specific requirements for a rolling upgrade; see “Guidelines for Rolling Upgrade” (page 359).
IMPORTANT: Do not proceed without reading and understanding the following points: • The above are general guidelines. The requirements for any particular Serviceguard release may be more restrictive. Rolling upgrade is supported only for the HP-UX/Serviceguard combinations listed in the Release Notes for the target version of Serviceguard.
Before You Start Make sure you plan sufficient system capacity to allow moving the packages from node to node during the process without an unacceptable loss of performance. CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 357). Running the Rolling Upgrade 1. 2. Halt the node you want to upgrade. You can do this in Serviceguard Manager, or use the cmhaltnode command.
Performing a Rolling Upgrade Using DRD IMPORTANT: All the limitations listed under “Guidelines for Rolling Upgrade” (page 359) and “Limitations of Rolling Upgrades ” (page 360) also apply to a rolling upgrade with DRD. You should read the entire section on “Performing a Rolling Upgrade” (page 360) before you proceed. Before You Start CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 357).
4. Restart the cluster on the node, using Serviceguard Manager or cmrunnode (1m). • If the applications do not function properly and this is the last node to be upgraded, you cannot revert to the previous release on just this node. You must either solve the problems with this release on this node, or revert the entire cluster to the previous release by halting the cluster (cmhaltcl), rebooting each node from its original (pre-upgrade) root disk, and restarting the cluster (cmruncl).
Figure 42 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 to the next operating system release (“HP-UX (new)”), and install the next version of Serviceguard (“SG (new)”). Figure 43 Node 1 Upgraded to new HP-UX version Step 3. When upgrading is finished, enter the following command on node 1 to restart the cluster on node 1. # cmrunnode -n node1 At this point, different versions of the Serviceguard daemon (cmcld) are running on the two nodes, as shown in Figure 44.
Figure 44 Node 1 Rejoining the Cluster Step 4. Repeat the process on node 2. Halt the node, as follows: # cmhaltnode -f node2 This causes both packages to move to node 1. Then upgrade node 2 to the new versions of HP-UX and Serviceguard. Figure 45 Running Cluster with Packages Moved to Node 1 Step 5. Move pkg2 back to its original node.
Figure 46 Running Cluster After Upgrades Guidelines for Non-Rolling Upgrade Do a non-rolling upgrade if: • Your cluster does not meet the requirements for rolling upgrade as specified in the Release Notes for the target version of Serviceguard; or • The limitations imposed by rolling upgrades make it impractical for you to do a rolling upgrade (see “Limitations of Rolling Upgrades ” (page 360)); or • For some other reason you need or prefer to bring the cluster down before performing the upgrade.
1. Halt all nodes in the cluster: cmhaltcl -f 2. If necessary, upgrade all the nodes in the cluster to the new HP-UX release. (See Step 3 under “Running the Rolling Upgrade” (page 361) for more information.) Upgrade all the nodes in the cluster to the new Serviceguard release. (See Step 3 under “Running the Rolling Upgrade” (page 361) for more information.) Restart the cluster: cmruncl 3. 4.
CAUTION: You must reboot all the nodes from their original disks before restarting the cluster; do not try to restart the cluster with some nodes booted from the upgraded disks and some booted from the pre-upgrade disks. Guidelines for Migrating a Cluster with Cold Install There may be circumstances when you prefer to do a cold install of the HP-UX operating system rather than an upgrade.
E Blank Planning Worksheets This appendix contains blank versions of the planning worksheets mentioned in“Planning and Documenting an HA Cluster ” (page 92). You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
=============================================================================== Tape Backup Power: Tape Unit __________________________ Power Supply _______________________ Tape Unit __________________________ Power Supply _______________________ =============================================================================== Other Power: Unit Name __________________________ Power Supply _______________________ Unit Name __________________________ Power Supply _______________________ .
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ VxVM Disk Group and Disk Worksheet DISK GROUP WORKSHEET Page ___ of ____ ==========================================================
Heartbeat Subnet: _________________________________ Monitored Non-heartbeat Subnet: ___________________ Monitored Non-heartbeat Subnet: ___________________ =====================================----====================================== Cluster Lock: Volume Groups, LUN, or Quorum Server =============================================================================== Quorum Server: QS_HOST___________________________QS_ADDR_________________ QS_POLLING_INTERVAL________________________________ QS_TIMEOUT_EXTENSIO
fs_umount_opt_____________ fs_fsck_opt_________________fs_type_________________ fs_mount_retry_count: ____________fs_umount_retry_count:___________________ Concurrent vgchange operations: ___________________________________________ Concurrent mount/umount operations: ______________________________________ Concurrent fsck operations: ______________________________________________ Kill processes accessing raw devices?_____ =============================================================================== Network
F Migrating from LVM to VxVM Data Storage This appendix describes how to migrate LVM volume groups to VxVM disk groups for use with the Veritas Volume Manager (VxVM), or with the Cluster Volume Manager (CVM) on systems that support it.
separation.) The same mirroring pattern should be followed in creating the VxVM plexes, with different plexes configured on disks that are attached to different buses. As an alternative to defining the VxVM disk groups on a new set of disks, it is possible to convert existing LVM volume groups into VxVM disk groups in line using the vxvmconvert(1M) utility.
4. 5. 6. 7. Be sure to copy from the old script any user-specific code that may have been added, including environment variables and customer defined functions. Distribute the new package control scripts to all nodes in the cluster. Test to make sure the disk group and data are intact. Deport the disk group: vxdg deport DiskGroupName 8. Make the disk group visible to the other nodes in the cluster by issuing the following command on all other nodes: vxdctl enable 9. Restart the package.
G Migrating from Legacy CFS Packages to Modular CFS Packages To migrate from Legacy CFS packages to Modular CFS packages for use with the CVM or CFS on systems that support it, follow these steps: 1. Halt the application package: cmhaltpkg 2. Unconfigure the CFS legacy disk group and the mount point package: cfsumount cfsdgadm deactivate cfsmntadm delete cfsdgadm delete 3.
H IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses. Topics: • IPv6 Address Types • Network Configuration Restrictions • Local Primary/Standby LAN Patterns • IPv6 Relocatable Address and Duplicate Address Detection Feature (page 381) IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces.
IPv6 Address Prefix IPv6 Address Prefix is similar to CIDR in IPv4 and is written in CIDR notation. An IPv6 address prefix is represented by the notation: IPv6-address/prefix-length where ipv6-address is an IPv6 address in any notation listed above and prefix-length is a decimal value representing how many of the leftmost contiguous bits of the address comprise the prefix. Example: fec0:0:0:1::1234/64 The first 64-bits of the address fec0:0:0:1 forms the address prefix.
::ffff:192.168.0.1 Aggregatable Global Unicast Addresses The global unicast addresses are globally unique IPv6 addresses. This address format is very well defined in the RFC 2374 (An IPv6 Aggregatable Global Unicast Address Format). The format is: 3 13 8 24 16 64 bits FP TLA ID RES NLA ID SLA ID Interface ID where: FP = Format prefix. Value of this is “001” for Aggregatable Global unicast addresses. TLA ID = Top-level Aggregation Identifier. RES = Reserved for future use.
All Node Addresses = FF02:0:0:0:0:0:0:1 (link-local) All Router Addresses = FF02:0:0:0:0:0:0:2 (link-local) All Router Addresses = FF05:0:0:0:0:0:0:2 (site-local) Network Configuration Restrictions Serviceguard supports IPv6 for data and heartbeat IP. To configure IPv6, the system should be set up in what is called a dual-stack configuration, which requires the IPv6 product bundle. The restrictions for supporting IPv6 in Serviceguard are listed below.
ndd -get /dev/ip6 ip6_nd_dad_solicit_count If the result is 1, the feature is turned on. If the result is 0, the feature is turned off. To temporarily change the state of DAD on your computer, use the ndd -set command to change the kernel parameter. ndd -set /dev/ip6 ip6_nd_dad_solicit_countn where n is a number: either 1 to turn the feature on, or 0 to turn it off.
Following the loss of lan0 or lan2, lan1 can adopt either address, as shown below. The same LAN card can be configured with both IPv4 and IPv6 addresses, as shown in below. This type of configuration allows failover of both addresses to the standby. This is shown in below.
IPv6 Network Support
I Using Serviceguard Manager HP Serviceguard Manager is a web-based, HP System Management Homepage (HP SMH) tool, that replaces the functionality of the earlier Serviceguard management tools. HP Serviceguard Manager allows you to monitor, administer and configure a Serviceguard cluster from any system with a supported web browser. Serviceguard Manager does not require additional software installation.
Table 17 Accessing Serviceguard Manager Scenario Scenario Use Case Number of Clusters Serviceguard Version Serviceguard Manager Version 1 Single cluster management 1 A.11.20 B.03.10 2 Multi-cluster management more than 1 cluster B.03.00 A.11.19 B.02.00 A.11.18 B.01.01 A.11.17.01 B.01.00 A.11.17.01 and later HP SIM or multiple browser sessions of B.02.00 or later Launching Serviceguard Manager This section provides information about three common scenarios.
NOTE: If a cluster is not yet configured, then you will not see the Serviceguard Cluster section on this screen. To create a cluster, from the SMH Tools menu, you must click Serviceguard Manager link in the Serviceguard box first, then click Create Cluster. The figure below shows a browser session at the HP Serviceguard Manager Main Page.
NOTE: Serviceguard Manager can be launched by HP Systems Insight Manager version 5.10 or later if Serviceguard Manager is installed on an HP Systems Insight Manager Central Management Server. For a Serviceguard A.11.17 cluster or earlier, Serviceguard Manager A.05.01 will be launched via Java Web Start. You must ensure that the hostname for each Serviceguard node can be resolved by DNS. For more information about this older version of Serviceguard Manager, see the Serviceguard Manager Version A.05.
J Maximum and Minimum Values for Parameters Table 18 shows the range of possible values for cluster configuration parameters. Table 18 Minimum and Maximum Values of Cluster Configuration Parameters Cluster Parameter Minimum Value Maximum Value Default Value Member Timeout See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4. See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4.
K Monitoring Script for Generic Resources The monitoring scripts are the scripts written by an end-user and must contain the core logic to monitor a resource and set the status of a generic resource. These scripts are started as a part of the package start. • You can set the status/value of a simple/extended resource respectively using the cmsetresource(1m) command. • You can define the monitoring interval in the script.
• It is not mandatory to have the same name for a generic resource and its monitoring script, i.e., service_name and generic_resource_name. However, it is good practice to have the same name, so that it is easier to identify the monitor. • A common resource specified across multiple packages can be monitored using one monitoring script. For resources of evaluation_type: before_package_start • Monitoring scripts can also be launched outside of the Serviceguard environment, init, rc scripts, etc.
Similarly, consider another package pkg2 that requires the 'CPU' to be configured as before_package_start.
# * "sg/generic_resource" modules need to be specified in the package * # * configuraton file in order to configure these parameters. * # * * # * * # * --------------------------------* # * U T I L I T Y F U N C T I O N S * # * --------------------------------* # * The following utility functions are sourced in from $SG_UTILS * # * ($SGCONF/scripts/mscripts/utils.
# # ######################################################################### function stop_command { sg_log 5 "stop_command" # ADD your halt steps here exit 1 } ################ # main routine ################ sg_log 5 "customer defined monitor script" ######################################################################### # # Customer defined monitor script should be doing following # functionality.
Sample script that demonstrates how to monitor disk space of a file system using extended generic resources can be found at: /etc/cmcluster/examples/sample_generic_resource_disk_space_monitor.sh Before using this script, copy it to the package specific directory. This script expects the following parameters when configured as a service: • $1 : Generic resource name: The name of the generic resource configured in the package that corresponds to this monitoring script.
L Migrating EMS Resources to Generic Resources To migrate from EMS resource to generic resource for monitoring resources, follow these steps: 1. Identify the EMS resource that is configured, resource_start, and polling interval. 2. If this is a standard EMS resource then identify the equivalent SFM style resource monitor.
19. Validate the changes done by running: cmcheckconf -v -p 20. Edit the package configuration file to add the generic resource parameters. 21. Validate the changes done by running: cmcheckconf -v -p 22. Apply the changes by running: cmapplyconf -v -p 23. Start the package. Example of Migration of an EMS Resource to a Generic Resource Let us consider an example of an EMS resource /vg/vgpkg/pv_summary that monitors the status of all PV links for the volume group vgpkg.
Index A Access Control Policies, 192 Access Control Policy, 125 Access roles, 125 active node, 22 adding a package to a running cluster, 312 adding cluster nodes advance planning, 157 adding nodes to a running cluster, 278 adding packages on a running cluster , 260 additional package resources monitoring, 59 addressing, SCSI, 95 administration adding nodes to a ruuning cluster, 278 cluster and package states, 262 halting a package, 285 halting the entire cluster, 279 moving a package, 286 of packages and se
understanding components, 27 cluster administration, 277 solving problems, 332 cluster and package maintenance , 261 cluster configuration creating with SAM or Commands, 186 file on all nodes, 45 identifying cluster lock volume group, 188 identifying cluster-aware volume groups, 192 planning, 103 planning worksheet, 125 sample diagram, 93 verifying the cluster configuration, 197 cluster configuration file Autostart Delay parameter (AUTO_START_TIMEOUT), 121 cluster coordinator defined, 44 cluster lock, 46 4
support for additional productss, 307 troubleshooting, 331 controlling the speed of application failover, 342 creating the package configuration, 302 Critical Resource Analysis (CRA) LAN or VLAN, 300 customer defined functions adding to the control script, 307 CVM, 85 creating a storage infrastructure, 219 not supported on all HP-UX versions, 23 planning, 102 CVM planning Version 4.1 with CFS, 127 Version 4.
used by package manager, 55 FAILBACK_POLICY parameter used by package manager, 55 failover controlling the speed in applications, 342 defined, 22 failover behavior in packages, 132 failover package, 50 failover policy used by package manager, 53 FAILOVER_POLICY parameter used by package manager, 53 failure kinds of responses, 88 network communication, 91 package, service, node, 88 response to hardware failures, 89 responses to package and service failures, 90 restarting a service after failure, 91 failures
hardware planning, 94 HOSTNAME_ADDRESS_FAMILY defined, 110 discussion and restrictions, 106 how the cluster manager works, 44 how the network manager works, 67 HP Predictive monitoring in troubleshooting, 324 kernel hang, and TOC, 88 safety timer, 41 kernel consistency in cluster configuration, 170 kernel interrupts and possible TOC, 120 Serviceguard behavior, 27 LAN interfaces monitoring with network manager, 69 primary and secondary, 28 LAN planning host IP address, 94, 99, 100 traffic type, 95 larger c
and cluster re-formation, 88 and safety timer, 41 configuring, 120 defined, 119 maximum and minimum values , 119 modifying, 192 membership change reasons for, 46 memory capacity hardware planning, 94 memory requirements lockable memory for Serviceguard, 92 Migrating EMS Resources to Generic Resources, 396 minimizing planned down time, 352 mirror copies of data protection against disk failure, 22 MirrorDisk/UX, 33 mirrored disks connected for high availability figure, 35 mirroring disks, 33 mirroring disks,
by means of in-line SCSI terminators, 326 OTS/9000 support, 389 outages insulating users from, 342 P package adding and deleting package IP addresses, 68 base modules, 230 basic concepts, 27 changes while the cluster is running, 315 configuring legacy, 302 failure, 88 halting, 285 legacy, 302 local interface switching, 70 modular, 230 modular and legacy, 227 modules, 230 moving, 286 optional modules, 231 parameters, 233 reconfiguring while the cluster is running, 311 reconfiguring with the cluster offline,
power supply and cluster lock, 37 blank planning worksheet, 369 UPS for OPS on HP-UX, 37 Predictive monitoring, 324 primary LAN interfaces defined, 28 primary network interface, 28 primary node, 22 pvcreate creating a root mirror with, 172 PVG-strict mirroring creating volume groups with, 179 Q qs daemon, 41 QS_ADDR parameter in cluster manager configuration, 112 QS_HOST parameter in cluster manager configuration, 111 QS_POLLING_INTERVAL parameter in cluster manager configuration, 112 QS_TIMEOUT_EXTENSION
step by step, 227 service failures responses, 90 service restarts, 91 SERVICE_CMD array variable in package control script, 253 in sample package control script, 306 SERVICE_FAIL_FAST_ENABLED and node TOC, 90 SERVICE_NAME in sample package control script, 306 SERVICE_RESTART in sample package control script, 306 Serviceguard install, 158 introduction, 21 Serviceguard at a glance, 21 Serviceguard behavior after monitored resource failure, 27 Serviceguard behavior in LAN failure, 27 Serviceguard behavior in s
reviewing system log file, 329 using cmquerycl and cmcheckconf, 331 troubleshooting your cluster, 321 typical cluster after failover figure, 23 typical cluster configuration figure, 21 vxfend for CVM and CFS, 44 VxVM, 84 creating a storage infrastructure, 183 migrating from LVM to VxVM, 374 planning, 102 VXVM_DG in package control script, 306 U W uname(2), 347 UPS in power planning, 97 power supply for OPS on HP-UX, 37 use of the cluster lock, 47, 48 USER_HOST, 125 USER_NAME, 125 USER_ROLE, 125 WEIGHT_
