Managing Serviceguard A.11.
Legal Notices © Copyright 1995-2012 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Publishing History .......................................................................................18 Preface......................................................................................................19 1 Serviceguard at a Glance.........................................................................21 What is Serviceguard? ...........................................................................................................21 Failover........................................
Cluster Logical Volume Manager Daemon: cmlvmd...........................................................42 Persistent Reservation Daemon: cmprd..............................................................................42 Cluster Object Manager Daemon: cmomd.......................................................................42 Cluster SNMP Agent Daemon: cmsnmpd..........................................................................42 Generic Resource Assistant Daemon: cmresourced.....................
Stationary and Relocatable IP Addresses ..............................................................................68 Types of IP Addresses....................................................................................................69 Adding and Deleting Relocatable IP Addresses .....................................................................69 Load Sharing ...............................................................................................................
4 Planning and Documenting an HA Cluster ..................................................97 General Planning ..................................................................................................................97 Serviceguard Memory Requirements.....................................................................................97 Planning for Expansion ......................................................................................................97 Hardware Planning ..................
Planning for Expansion.....................................................................................................137 Choosing Switching and Failover Behavior..........................................................................137 Parameters for Configuring Generic Resources.....................................................................138 Configuring a Generic Resource........................................................................................
Removing a Node from a cDSF Group...........................................................................166 Displaying the cDSF Configuration................................................................................166 Migrating Existing LVM Cluster Storage to cDSFs.............................................................166 Using Easy Deployment....................................................................................................166 Before You Start..............................
Specifying a Lock Disk......................................................................................................193 Specifying a Lock LUN......................................................................................................194 Specifying a Quorum Server.............................................................................................194 Obtaining Cross-Subnet Information...................................................................................
6 Configuring Packages and Their Services ..................................................232 Choosing Package Modules...................................................................................................232 Types of Package: Failover, Multi-Node, System Multi-Node...................................................233 Differences between Failover and Multi-Node Packages........................................................234 Package Modules and Parameters...................................
cvm_dg.................................................................................................................252 vxvm_dg...............................................................................................................253 vxvm_dg_retry.......................................................................................................253 deactivation_retry_count..........................................................................................
Viewing Information about Unowned Packages...........................................................274 Viewing Information about System Multi-Node Packages..............................................274 Checking Status of the Cluster File System (CFS)..........................................................275 Status of the Packages in a Cluster File System............................................................276 Status of CFS Modular Disk Group and Mount Point Packages...........................
What You Can Preview................................................................................................298 Using Preview mode for Commands and in Serviceguard Manager...................................298 Using cmeval..............................................................................................................299 Updating the Cluster Lock Configuration.............................................................................
8 Troubleshooting Your Cluster....................................................................327 Testing Cluster Operation ......................................................................................................327 Start the Cluster using Serviceguard Manager.....................................................................327 Testing the Package Manager ...........................................................................................327 Testing the Cluster Manager ..........
A Enterprise Cluster Master Toolkit ..............................................................346 B Designing Highly Available Cluster Applications ........................................347 Automating Application Operation ........................................................................................347 Insulate Users from Outages .............................................................................................348 Define Application Startup and Shutdown ..........................
Types of Upgrade.................................................................................................................364 Rolling Upgrade..............................................................................................................364 Rolling Upgrade Using DRD..............................................................................................364 Restrictions for DRD Upgrades...........................................................................................
Unicast Addresses............................................................................................................385 IPv4 and IPv6 Compatibility..............................................................................................385 IPv4 Compatible IPv6 Addresses...................................................................................385 IPv4 Mapped IPv6 Address..........................................................................................
Publishing History Table 1 Publishing History Publishing Date Part Number Edition January 1995 B3936-90001 First June 1995 B3936-90003 Second December 1995 B3936-90005 Third August 1997 B3936-90019 Fourth January 1998 B3936-90024 Fifth October 1998 B3936-90026 Sixth December 2000 B3936-90045 Seventh September 2001 B3936-90053 Eighth March 2002 B3936-90065 Ninth June 2003 B3936-90070 Tenth June 2004 B3936-90076 Eleventh June 2005 B3936-90076 Eleventh, First reprint Octo
Preface This edition of the manual applies to Serviceguard Version A.11.20 for the March 2012 patch release. Earlier versions are available at http://www.hp.com/go/hpux-serviceguard-docs -> HP Serviceguard. The information in this manual has been updated for new features and functionality introduced in the Serviceguard A.11.20 March 2012 patch (PHSS_42558). If the Serviceguard A.11.
Related Publications For information about the current version of Serviceguard, and about older versions, see the Serviceguard documents posted at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard. The following documents, which can all be found at http://www.hp.com/go/ hpux-serviceguard-docs, are particularly useful. • The latest edition of the HP Serviceguard Version A.11.20 Release Notes • HP Serviceguard Quorum Server Version A.04.
1 Serviceguard at a Glance This chapter introduces Serviceguard on HP-UX, and shows where to find information in this book. It covers the following: • What is Serviceguard? • Using Serviceguard Manager (page 24) • A Roadmap for Configuring Clusters and Packages (page 25) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4: “Planning and Documenting an HA Cluster ” (page 97).
control of the package to another cluster node, allowing services to remain available with minimal interruption. • There are also packages that run on several cluster nodes at once, and do not fail over. These are called system multi-node packages and multi-node packages. Examples are the packages HP supplies for use with the Veritas Cluster Volume Manager and Veritas Cluster File System from Symantec (on HP-UX releases that support them; see “About Veritas CFS and CVM from Symantec” (page 23)).
Figure 2 Typical Cluster After Failover After this transfer, the failover package typically remains on the adoptive node as long the adoptive node continues running. If you wish, however, you can configure the package to return to its primary node as soon as the primary node comes back online. Alternatively, you may manually transfer control of the package back to the primary node at the appropriate time. Figure 2 does not show the power connections to the cluster, but these are important as well.
Using Serviceguard Manager NOTE: For more-detailed information, see Appendix I (page 391) and the section on Serviceguard Manager in the latest version of the Serviceguard Release Notes. Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for up-to-date information about Serviceguard Manager compatibility. You can find both documents at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard.
on the command line. As of HP-UX 11i v3, SAM offers a Terminal User Interface (TUI) which also acts as a gateway to the web-based System Management Homepage (SMH). • To get to the SMH for any task area, highlight the task area in the SAM TUI and press w. • To go directly to the SMH from the command line, enter /usr/sbin/sam -w For more information, see the HP-UX Systems Administrator’s Guide, at the address given in the preface to this manual.
Figure 3 Tasks in Configuring a Serviceguard Cluster The tasks in Figure 3 are covered in step-by-step detail in chapters 4 through 7. HP recommends you gather all the data that is needed for configuration before you start. See “Planning and Documenting an HA Cluster ” (page 97) for tips on gathering data.
2 Understanding Serviceguard Hardware Configurations This chapter gives a broad overview of how the Serviceguard hardware components work. The following topics are presented: • Redundancy of Cluster Components • Redundant Network Components (page 28) • Redundant Disk Storage (page 32) • Redundant Power Supplies (page 37) • Larger Clusters (page 37) Refer to the next chapter for information about Serviceguard software components.
Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable, and the cables themselves are connected by a component such as a hub or a bridge. This arrangement of physical cables connected to each other via a bridge or concentrator or switch is known as a bridged net .
(Oracle VIPs are an exception to this rule; such configurations require the HP add-on product Serviceguard Extension for Oracle RAC). ◦ Similarly, Serviceguard does not support using networking tools to move or reconfigure any IP addresses configured into the cluster. Doing so leads to unpredictable results because the Serviceguard view of the configuration is different from the reality.
A cross-subnet configuration allows: • Automatic package failover from a node on one subnet to a node on another • A cluster heartbeat that spans subnets. Configuration Tasks Cluster and package configuration tasks are affected as follows: • You must use the -w full option to cmquerycl to discover actual or potential nodes and subnets across routers.
• You must not set the HP-UX network parameter ip_strong_es_model in a cross-subnet configuration. Leave it set to the default (0, meaning disabled); Serviceguard does not support enabling it for cross-subnet configurations. For more information about this parameter, see “Tuning Network and Kernel Parameters” (page 176) and “Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY” (page 355).
Redundant Disk Storage Each node in a cluster has its own root disk, but each node is also physically connected to several other disks in such a way that more than one node can obtain access to the data and programs associated with a package it is configured for. This access is provided by a Storage Manager, such as Logical Volume Manager (LVM), or Veritas Volume Manager (VxVM) (or Veritas Cluster Volume Manager (CVM).
Data Protection It is required that you provide data protection for your highly available system, using one of two methods: • Disk Mirroring • Disk Arrays using RAID Levels and Multiple Data Paths Disk Mirroring Serviceguard itself does not provide protection for data on your disks, but protection is provided by HP’s Mirrordisk/UX product for LVM storage, and by the Veritas Volume Manager for VxVM and CVM.
Monitoring LVM Disks Through Generic Resources Monitoring Service If you are using LVM, you can configure disk monitoring to detect a failed mechanism by using the disk monitor capabilities of the System Fault Management, available as a separate product, and integrating it in Serviceguard by configuring generic resources in packages.
Sample SCSI Disk Configurations Figure 5 shows a two node cluster. Each node has one root disk which is mirrored and one package for which it is the primary node. Resources have been allocated to each node so that each node may adopt the package from the other node. Each package has one disk volume group assigned to it and the logical volumes in that volume group are mirrored.
Figure 6 Cluster with High Availability Disk Array Details on logical volume configuration for Serviceguard are in the chapter “Building an HA Cluster Configuration.” Sample Fibre Channel Disk Configuration In Figure 7, the root disks are shown with simple mirroring, but the shared storage is now accessed via redundant Fibre Channel switches attached to a disk array. The cabling is set up so that each node is attached to both switches, and both switches are attached to the disk array with redundant links.
Figure 7 Cluster with Fibre Channel Switched Disk Array This type of configuration uses native HP-UX or other multipathing software; see “About Multipathing” (page 33). Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks. HP-supported uninterruptible power supplies (UPS), such as HP PowerTrust, can provide this protection from momentary power loss.
could be set up as an administrative unit, and sub-groupings of four could be set up on different SCSI buses which are attached to different mass storage devices. In the case of non-shared SCSI connections to an XP series or EMC disk array, the four-node limit does not apply. Each node can be connected directly to the XP or EMC by means of two SCSI buses. Packages can be configured to fail over among all sixteen nodes.
Figure 9 Eight-Node Cluster with XP or EMC Disk Array Fibre Channel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP Unix Servers Configuration Guide, available through your HP representative.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
• /usr/lbin/cmfileassistd—Serviceguard File Management daemon • /usr/lbin/cmlogd—Serviceguard Syslog Log Daemon • /usr/lbin/cmlvmd—Cluster Logical Volume Manager Daemon • /usr/lbin/cmprd—Persistent Reservation Daemon • /opt/cmom/lbin/cmomd—Cluster Object Manager Daemon • /usr/lbin/cmsnmpd—Cluster SNMP subagent (optionally running) • /usr/lbin/cmresourced—Serviceguard Generic Resource Assistant Daemon • /usr/lbin/cmserviced—Serviceguard Service Assistant Daemon • /usr/lbin/qs—Serviceguard Qu
cmcld also manages Serviceguard packages, determining where to run them and when to start them. NOTE: Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 20 on all cluster nodes. It is important that user processes should have a priority lower than 20, otherwise they may prevent Serviceguard from updating the kernel safety timer, causing a system reset.
Generic Resource Assistant Daemon: cmresourced This daemon is responsible to set and get the status/value of generic resources configured as part of the package and influence the availability of the package based on the availability of the resource. Generic resources allows custom defined monitors to be integrated. It provides better control, options, and flexibility in terms of getting and setting the status of a resource.
Cluster WBEM Agent Daemon: cmwbemd This daemon collaborates with the Serviceguard WBEM provider plug-in module (SGProviders) and WBEM services cimserver to provide notification (WBEM Indications) of Serviceguard cluster events to Serviceguard WBEM Indication subscribers that have registered a subscription with the cimserver. For example, an Indication is sent when the cluster configuration changes, or when a Serviceguard package has failed. You must edit /etc/rc.config.d/cmwbemagt to auto-start cmwbemd.
. Although all nodes perform some cluster management functions, the cluster coordinator is the central point for inter-node communication. Configuring the Cluster The system administrator sets up cluster configuration parameters and does an initial cluster startup; thereafter, the cluster regulates itself without manual intervention in normal operation.
Manual Startup of Entire Cluster A manual startup forms a cluster out of all the nodes in the cluster configuration. Manual startup is normally done the first time you bring up the cluster, after cluster-wide maintenance or upgrade, or after reconfiguration. Before startup, the same binary cluster configuration file must exist on all nodes in the cluster. The system administrator starts the cluster in Serviceguard Manager or with the cmruncl command issued from one node.
Cluster Lock Although a cluster quorum of more than 50% is generally required, exactly 50% of the previously running nodes may re-form as a new cluster provided that the other 50% of the previously running nodes do not also re-form. This is guaranteed by the use of a tie-breaker to choose between the two equal-sized node groups, allowing one group to form the cluster and forcing the other group to shut down. This tie-breaker is known as a cluster lock.
Figure 11 Lock Disk or Lock LUN Operation Serviceguard periodically checks the health of the lock disk or LUN and writes messages to the syslog file if the device fails the health check. This file should be monitored for early detection of lock disk problems. If you are using a lock disk, you can choose between two lock disk options—a single or dual lock disk—based on the kind of high availability configuration you are building. A single lock disk is recommended where possible.
to both lock disks. In the event of a failure of one of the data centers, the nodes in the remaining data center will be able to acquire their local lock disk, allowing them to successfully reform a new cluster. NOTE: A dual lock disk does not provide a redundant cluster lock. In fact, the dual lock is a compound lock. This means that two disks must be available at cluster formation time rather than the one that is needed for a single lock disk.
No Cluster Lock Normally, you should not configure a cluster of three or fewer nodes without a cluster lock. In two-node clusters, a cluster lock is required. You may consider using no cluster lock with configurations of three or more nodes, although the decision should be affected by the fact that any cluster may require tie-breaking. For example, if one node in a three-node cluster is removed for maintenance, the cluster re-forms as a two-node cluster.
Package Types Three different types of packages can run in the cluster; the most common is the failover package. There are also special-purpose packages that run on more than one node at a time, and so do not failover. They are typically used to manage resources of certain failover packages.
control script that is installed with Serviceguard; see Chapter 6: “Configuring Packages and Their Services ” (page 232), for instructions for creating modular packages. Deciding When and Where to Run and Halt Failover Packages The package configuration file assigns a name to the package and includes a list of the nodes on which the package can run. Failover packages list the nodes in order of priority (i.e., the first node in the list is the highest priority node).
Figure 14 Before Package Switching Figure 15 shows the condition where Node 1 has failed and Package 1 has been transferred to Node 2 on the same subnet. Package 1’s IP address was transferred to Node 2 along with the package. Package 1 continues to be available and is now running on Node 2. Also note that Node 2 can now access both Package1’s disk and Package2’s disk.
Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file. The failover policy governs how the package manager selects which node to run a package on when a specific node has not been identified and the package needs to be started. This applies not only to failovers but also to startup for the package, including the initial startup.
Figure 17 Rotating Standby Configuration after Failover NOTE: Using the min_package_node policy, when node 2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node.
If you use configured_node as the failover policy, the package will start up on the highest priority node in the node list, assuming that the node is running as a member of the cluster. When a failover occurs, the package will move to the next highest priority node in the list that is available.
Figure 20 Automatic Failback Configuration After Failover After rebooting, node 1 rejoins the cluster. At that point, pkgA will be automatically stopped on node 4 and restarted on node 1.
Figure 21 Automatic Failback Configuration After Restart of Node 1 CAUTION: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period. If you are using automatic failback, you may want to wait to add the package’s primary node back into the cluster until you can allow the package to be taken out of service temporarily while it switches back to the primary node.
Generic resources has the following advantages: • Custom defined monitors can also be integrated • Provides better control, options, and flexibility in terms of getting and setting the status of a resource Generic resources can be configured into any modular style package, including modular style CFS packages. They can be configured for failover or multi-node packages and are included in modular failover packages by default. A single resource can be specified across multiple packages.
NOTE: You can get or set the status/value of a simple/extended generic resource using the cmgetresource(1m) and cmsetresource(1m) commands respectively. See “Getting and Setting the Status/Value of a Simple/Extended Generic Resource” (page 141) and the manpages for more information. A single package can have a combination of simple and extended resources, but a given generic resource cannot be configured as a simple resource in one package and as an extended resource in another package. For e.g.
How Packages Run Packages are the means by which Serviceguard starts and halts configured applications. Failover packages are also units of failover behavior in Serviceguard. A package is a collection of services, disk volumes and IP addresses that are managed by Serviceguard to ensure they are available. There can be a maximum of 300 packages per cluster and a total of 900 services per cluster.
NOTE: This diagram applies specifically to legacy packages. Differences for modular scripts are called out below. Figure 22 Legacy Package Time Line Showing Important Events The following are the most important moments in a package’s life: 1. Before the control script starts. (For modular packages, this is the master control script.) 2. During run script execution. (For modular packages, during control script execution to start the package.) 3. While services are running 4.
During Run Script Execution Once the package manager has determined that the package can start on a particular node, it launches the script that starts the package (that is, a package’s control script or master control script is executed with the start parameter). This script carries out the following steps: 1. 2. 3. 4. 5. 6. 7. 8. Executes any external_pre_scripts (modular packages only (page 256)) Activates volume groups or disk groups. Mounts file systems.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled is set to yes, it will halt the node on which the package is running.
Some failures can result in a local switch. For example, if there is a failure on a specific NIC (network interface card) and there are one or more standby NICs configured for that subnet, then the Network Manager will move the network stack to the first healthy NIC. If the first standby NIC fails, Serviceguard will move the traffic to the second standby NIC (if configured).
script is executed with the stop parameter. This script carries out the following steps (also shown in Figure 24): 1. 2. 3. 4. 5. 6. 7. 8. Halts any deferred resources that had been started earlier. Halts all package services. Executes any customer-defined halt commands (legacy packages only) or external_scripts (modular packages only (page 256)). Removes package IP addresses from the NIC on the node. Unmounts file systems. Deactivates volume groups. Exits with an exit code of zero (0).
Normal and Abnormal Exits from the Halt Script The package’s ability to move to other nodes is affected by the exit conditions on leaving the halt script. The following are the possible exit codes: • 0—normal exit. The package halted normally, so all services are down on this node. • 1—abnormal exit, also known as no_restart exit. The package did not halt normally. Services are killed, and the package is disabled globally. It is not disabled on the current node, however.
Table 4 Error Conditions and Package Movement for Failover Packages (continued) Package Error Condition Results Error or Exit Code Node Failfast Service Enabled Failfast Enabled HP-UX Status on Primary after Error Halt script Package Allowed to Package runs after Run on Primary Allowed to Run Error or Exit Node after Error on Alternate Node Halt Script Timeout NO Either Setting Running N/A Yes No Service Failure Either Setting YES system reset No N/A (system reset) Yes Service Failure Ei
IMPORTANT: Every subnet configured as a monitored_subnet in a package configuration file must be configured into the cluster via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP in the cluster configuration file. See “Cluster Configuration Parameters ” (page 114) and “Package Parameter Explanations” (page 237) for more information. In addition to the stationary IP address, you normally assign one or more unique IP addresses to each failover package.
IP addresses are configured only on each primary network interface card; standby cards are not configured with an IP address. Multiple IPv4 addresses on the same network card must belong to the same IP subnet. CAUTION: HP strongly recommends that you add relocatable addresses to packages only by editing ip_address (page 247) in the package configuration file (or IP [] entries in the control script of a legacy package) and running cmapplyconf (1m).
◦ The primary switch should be directly connected to its standby. ◦ There should be no single point of failure anywhere on all bridged nets. NOTE: You can change the value of the NETWORK_FAILURE_DETECTION parameter while the cluster is up and running. Local Switching A local network switch involves the detection of a local network interface failure and a failover to the local backup NIC (also known as the standby NIC). The backup NIC must not have any IP addresses configured.
Figure 25 Cluster Before Local Network Switching Node 1 and Node 2 are communicating over LAN segment 2. LAN segment 1 is a standby. In Figure 26, we see what would happen if the LAN segment 2 network interface card on Node 1 were to fail. Figure 26 Cluster After Local Network Switching As the standby interface takes over, IP addresses will be switched to the hardware path associated with the standby interface. The switch is transparent at the TCP/IP level.
Another example of local switching is shown in Figure 27. In this case a failure affecting segment 2 causes both nodes to switch to the NICs attached to segment 1. Figure 27 Local Switching After Cable Failure Local network switching will work with a cluster containing one or more nodes. You may wish to design a single-node cluster in order to take advantage of this local network switching feature in situations where you need only one node and do not wish to set up a more complex cluster.
NOTE: The NETWORK_AUTO_FAILBACK setting applies only to link-level failures, not to failures at the IP level; see “Monitoring LAN Interfaces and Detecting Failure: IP Level” (page 74) for more information about such failures. For more information about the cluster configuration file, see “Cluster Configuration Parameters ” (page 114). Remote Switching A remote switch (that is, a package switch) involves moving packages to a new system.
Reasons To Use IP Monitoring Beyond the capabilities already provided by link-level monitoring, IP monitoring can: • Monitor network status beyond the first level of switches; see “How the IP Monitor Works” (page 75) • Detect and handle errors such as: ◦ IP packet corruption on the router or switch ◦ Link failure between switches and a first-level router ◦ Inbound failures even when the cluster configuration parameter NETWORK_FAILURE_DETECTION is not set to INONLY_OR_INOUT NOTE: This applies only t
16.89.120.0 … Possible IP Monitor Subnets: IPv4: 16.89.112.0 Polling Target 16.89.112.1 IPv6: 3ffe:1000:0:a801:: Polling Target 3ffe:1000:0:a801::254 … The IP Monitor section of the cluster configuration file will look similar to the following for a subnet on which IP monitoring is configured with target polling. NOTE: This is the default if cmquerycl detects a gateway for the subnet in question; see SUBNET under “Cluster Configuration Parameters ” (page 114) for more information.
Constraints and Limitations • A subnet must be configured into the cluster in order to be monitored. • Polling targets are not detected beyond the first-level router. • Polling targets must accept and respond to ICMP (or ICMPv6) ECHO messages. • A peer IP on the same subnet should not be a polling target because a node can always ping itself. • On an HP-UX system, an IP-level failure on the standby interface will not be detected until the IP address fails over to the standby.
But if the IP Monitor detects the failure (and link-level monitoring does not) cmviewcl -v output will look something like this: Network_Parameters: INTERFACE STATUS PRIMARY down (disabled) (IP only) PRIMARY up PATH 0/3/1/0 0/5/1/0 NAME lan2 lan3 cmviewcl -v -f line will report the same failure like this: node:gary|interface:lan2|status=down node:gary|interface:lan2|local_switch_peer=lan1 node:gary|interface:lan2|disabled=true node:gary|interface:lan2|failure_type=ip_only In this case, you would need to
Figure 28 Aggregated Networking Ports Both the Single and Dual ported LANs in the non-aggregated configuration have four NICs, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (lan0, lan1, lan2, lan3). When these ports are aggregated all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as lan900, the name by which the aggregate is known on HP-UX 11i.
remote failover of VLAN interfaces when failure is detected. Failure of a VLAN interface is typically the result of the failure of the underlying physical NIC port or aggregated (APA) ports. Configuration Restrictions HP-UX allows up to 1024 VLANs to be created from a physical NIC port. A large pool of system resources is required to accommodate such a configuration; Serviceguard could suffer performance degradation if many network interfaces are configured in each cluster node.
redundant storage in hardware. Two types of mirroring are RAID1 and RAID5. Here are some differences between the two storage methods: • If you are using JBODs, the basic element of storage is an individual disk. This disk must be paired with another disk to create a mirror (RAID1). (Serviceguard configurations usually have separate mirrors on different storage devices).
NOTE: Under agile addressing (see “About Device File Names (Device Special Files)” (page 81)), the storage units in this example would have names such as disk1, disk2, disk3, etc. As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 109). Figure 29 Physical Disks Within Shared Storage Units Figure 30 shows the individual disks combined in a multiple disk mirrored configuration.
Figure 31 Multiple Devices Configured in Volume Groups Examples of Storage on Disk Arrays Figure 32 shows an illustration of storage configured on a disk array. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system. Figure 32 Physical Disks Combined into LUNs NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer.
NOTE: Under agile addressing, the storage units in these examples would have names such as disk1, disk2, disk3, etc. See “About Device File Names (Device Special Files)” (page 81) As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 109). . Figure 33 Multiple Paths to LUNs Finally, the multiple paths are configured into volume groups as shown in Figure 34.
Types of Volume Manager Serviceguard allows a choice of volume managers for data storage: • HP-UX Logical Volume Manager (LVM) and (optionally) Mirrordisk/UX • Veritas Volume Manager for HP-UX (VxVM)—Base and add-on Products • Veritas Cluster Volume Manager for HP-UX Separate sections in Chapters 5 and 6 explain how to configure cluster storage using all of these volume managers.
Veritas Cluster Volume Manager (CVM) NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information on CVM support: http://www.hp.com/go/hpux-serviceguard-docs. You may choose to configure cluster storage with the Veritas Cluster Volume Manager (CVM) instead of the Volume Manager (VxVM).
Redundant Heartbeat Subnets HP recommends that you configure all subnets that connect cluster nodes as heartbeat networks; this increases protection against multiple faults at no additional cost. For CVM 4.1 and later, the minimum recommended configurations are: 1) dual (multiple) heartbeat networks 2) single heartbeat network with standby NIC(s) Comparison of Volume Managers The following table summarizes the advantages and disadvantages of the volume managers.
Table 5 Pros and Cons of Volume Managers with Serviceguard Product Advantages Tradeoffs Logical Volume Manager (LVM) • Software is provided with all versions of HP-UX. • Lacks flexibility and extended features of some other volume managers • Provides up to 3-way mirroring using optional Mirrordisk/UX software. • Dynamic multipathing (DMP) is active by default as of HP-UX 11i v3.
Table 5 Pros and Cons of Volume Managers with Serviceguard (continued) Product Advantages Tradeoffs • • No support for RAID 5 • Supports activation in different modes on different nodes at the same time • CVM requires all nodes to have connectivity to the shared disk groups • RAID 1+0 mirrored stripes • Not currently supported on all versions of HP-UX • RAID 0+1 striped mirrors • CVM versions 4.
Rules and Limitations Serviceguard automatically implements PR for packages using volume groups configured on a LUN storage, subject to the following guidelines: • The LUN device must support PR and be consistent with the SPC-3 specification. • PR is enabled for packages with iSCSI disks, but disabled for packages with non-iSCSI disks. • Packages with a mix of iSCSI and non-iSCSI disks are not supported. • iSCSI is only supported with modular packages.
When you run cmapplyconf (1m) to configure a new cluster, or add a new node, Serviceguard sets the variable cluster_pr_mode to either pr_enabled or pr_disabled. • pr_enabled means that packages can in principle use PR, but in practice will do so only if they fulfill the conditions described in the “Rules and Limitations” section. • pr_disabled means that no packages can use PR. You can view the setting of cluster_pr_mode in the output of cmviewcl -f line; for example, ...
Running a Package For a failover package, when a volume group with an iSCSI disk is added as a part of a package startup, the disk underlying the volume is registered and reserved by the node where the package is running. When the package is halted, all the registration and reservation keys are cleared. In multi-node packages, when a package with a volume group is applied, reservation is made to the disk by one of the nodes where the package is up and running.
RESERVED KEYS : NONE RESERVED FROM NODE : NONE The physical volume /dev/rdsk/c11t0d0 contains the registered and reserved key which is the node_pr_key (See cmviewcl(5) for parameter explanation).
1. 2. 3. The node contacts the other nodes and tries to re-form the cluster without the failed node. If the remaining nodes are a majority or can obtain the cluster lock, they form a new cluster without the failed node. If the remaining nodes are not a majority or cannot get the cluster lock, they halt (system reset). Example Situation. Assume a two-node cluster, with Package1 running on SystemA and Package2 running on SystemB.
Disk protection is provided by separate products, such as Mirrordisk/UX in LVM or Veritas mirroring in VxVM and related products. In addition, separately available EMS disk monitors allow you to notify operations personnel when a specific failure, such as a lock disk failure, takes place. Refer to the manual Using High Availability Monitors, which you can find at the address given in the preface to this manual.
If the value that is set (current_value) does not satisfy the generic_resource_up_criteria, then the generic resource is marked as 'down' on that node. NOTE: If a simple resource is down on a particular node, it is down on that node for all the packages using it whereas, in case of an extended resource the resource may be up on a node for a particular package and down for another package, since it is dependent on the generic_resource_up_criteria.
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather information about all the hardware and software components of the configuration.
your cluster without the need to bring it down, careful planning of the initial configuration is required. Use the following guidelines: • Remember the rules for cluster locks when considering expansion. A one-node cluster does not require a cluster lock. A two-node cluster must have a cluster lock. In clusters larger than 3 nodes, a cluster lock is strongly recommended. If you have a cluster with more than 4 nodes, you can use a Quorum Server but a cluster lock disk is not allowed.
slots, and the bus address for each adapter; you can update the details as you do the cluster configuration (described in Chapter 5). SPU Information SPU information includes the basic characteristics of the systems you are using in the cluster. Different models of computers can be mixed in the same cluster. This configuration model also applies to HP Integrity servers. HP-UX workstations are not supported for Serviceguard.
For more details of IPv6 address format, see the Appendix H (page 384). NETWORK_FAILURE_DETECTION When there is a primary and a standby network card, Serviceguard needs to determine when a card has failed, so it knows whether to fail traffic over to the other card. The configuration file specifies one of two ways to decide when the network interface card has failed: • INOUT • INONLY_OR_INOUT The default is INOUT.
Table 6 SCSI Addressing in Cluster Configuration (continued) System or Disk Host Interface SCSI Address Disk #6 14 Others 13 - 8 NOTE: When a boot/root disk is configured with a low-priority address on a shared SCSI bus, a system panic can occur if there is a timeout on accessing the boot/root device. This can happen in a cluster when many nodes and many disks are configured on the same bus.
Hardware Configuration Worksheet You may find a worksheet such as the following useful for organizing and recording your cluster hardware configuration. This worksheet is an example; blank worksheets are in Appendix E. Make as many copies as you need.
units you are using and the power supply to which they will be connected; record the following information: Host Name The host name for each SPU. Disk Unit The disk drive unit number for each disk. Tape Unit The tape unit number for each backup device. Other Unit Tthe number of any other unit. Power Supply The power supply unit number of the UPS to which the host or other device is connected. Be sure to follow UPS and cabinet power limits as well as SPU power limits.
Quorum Server as the cluster lock. In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes. For more information on lock disks, lock LUNs, and the Quorum Server, see “Choosing Cluster Lock Disks” (page 178), “Setting Up a Lock LUN” (page 178), and “Setting Up and Running the Quorum Server” (page 181).
Supported Node Names The name (39 characters or fewer) of each cluster node that will be supported by this quorum server. These entries will be entered into qs_authfile on the system that is running the quorum server process.
Using Generic Resources to Monitor Volume Groups You can monitor a particular disk that is a part of an LVM volume group used by packages. You can do this by using the disk monitor capabilities of the System Fault Management, available as a separate product, and integrating it in Serviceguard by configuring generic resources in packages.
IMPORTANT: You must set the IO timeout for all logical volumes within the volume group being monitored to something other than the default of zero (no timeout); otherwise the EMS resource monitor value will never change upon a failure. Suggested IO timeout values are 20 to 60 seconds. See “Setting Logical Volume Timeouts” (page 184) for more information. For more information, see “Using the EMS HA Monitors” (page 60).
When designing a storage configuration using CVM or VxVM disk groups, consider the following: • CVM disk groups are created after the cluster is configured, whereas VxVM disk groups may be created before cluster configuration if desired. • High availability applications, services, and data should be placed in separate disk groups from non-high availability applications, services, and data.
• GIDs are the same on all nodes. • Applications in the system area are the same on all nodes. • System time is consistent across the cluster. • Files that could be used by more than one node, such as files in the /usr directory, must be the same on all nodes. About Cluster-wide Device Special Files (cDSFs) Under agile addressing on HP-UX 11i v3, each device has a unique identifier as seen from a given host; this identifier is reflected in the name of the Device Special File (DSF).
Limitations of cDSFs • cDSFs are supported only within a single cluster; you cannot define a cDSF group that crosses cluster boundaries. • A node can belong to only one cDSF group. • cDSFs are not supported by VxVM, CVM, CFS, or any other application that assumes DSFs reside only in /dev/disk and /dev/rdisk. • cDSFs do not support disk partitions. Such partitions can be addressed by a device file using the agile addressing scheme, but not by a cDSF.
• Simplifies cluster lock configuration. • Simplifies creation of shared storage for packages. Limitations of Easy Deployment • Does not install or verify Serviceguard software • Requires agile addressing for disks. See “About Device File Names (Device Special Files)” (page 81). • cmpreparestg (1m) will fail if cDSFs and persistent DSFs are mixed in a volume group. See “About Cluster-wide Device Special Files (cDSFs)” (page 109) for more information about cDSFs.
IPv6-only means that Serviceguard will try to resolve the hostnames to IPv6 addresses only. Mixed means that when resolving the hostnames, Serviceguard will try both IPv4 and IPv6 address families. You specify the address family the cluster will use in the cluster configuration file (by setting HOSTNAME_ADDRESS_FAMILY to IPV4, IPV6, or ANY), or by means of the -a option of cmquerycl (1m); see “Specifying the Address Family for the Cluster Hostnames” (page 191). The default is IPV4.
• You must use $SGCONF/cmclnodelist, not ~/.rhosts or /etc/hosts.equiv, to provide root access to an unconfigured node. NOTE: This also applies if HOSTNAME_ADDRESS_FAMILY is set to ANY. See “Allowing Root Access to an Unconfigured Node” (page 171) for more information. • If you use a Quorum Server, you must make sure that the Quorum Server hostname (and the alternate Quorum Server address specified by QS_ADDR, if any) resolve to IPv6 addresses, and you must use Quorum Server version A.04.00 or later.
Rules and Restrictions for Mixed Mode IMPORTANT: See the latest version of the Serviceguard release notes for the most current information on these and other restrictions. • The hostname resolution file on each node (for example, /etc/hosts) must contain entries for all the IPv4 and IPv6 addresses used throughout the cluster, including all STATIONARY_IP and HEARTBEAT_IP addresses as well any private addresses.
NOTE: In addition, the following characters must not be used in the cluster name if you are using the Quorum Server: at-sign (@), equal-sign (=), or-sign (|), semicolon (;). These characters are deprecated, meaning that you should not use them, even if you are not using the Quorum Server, because they will be illegal in a future Serviceguard release. All other characters are legal. The cluster name can contain up to 39 characters (bytes).
QS_HOST The fully-qualified hostname or IP address of a system outside the current cluster that is providing quorum service to the cluster. Can be (or resolve to) either an IPv4 or an IPv6 address if HOSTNAME_ADDRESS_FAMILY is set to ANY, but otherwise must match the setting of HOSTNAME_ADDRESS_FAMILY. This parameter is used only when you employ a Quorum Server for tie-breaking services in the cluster.
QS_POLLING_INTERVAL The time (in microseconds) between attempts to contact the Quorum Server to make sure it is running. Default is 300,000,000 microseconds (5 minutes). Minimum is 10,000,000 (10 seconds). Maximum is 2,147,483,647 (approximately 35 minutes). Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 50) for important information.
IMPORTANT: Node names must be 39 characters (bytes) or less, and are case-sensitive; for each node, the NODE_NAME in the cluster configuration file must exactly match the corresponding node_name in the package configuration file (see Chapter 6: “Configuring Packages and Their Services ” (page 232)) and these in turn must exactly match the hostname portion of the name specified in the node’s networking configuration.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 131)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
A minimum Serviceguard configuration on HP-UX 11i v2 or 11i v3 needs two network interface cards for the heartbeat in all cases, using one of the following configurations: • Two heartbeat subnets; or • One heartbeat subnet with a standby; or • One heartbeat subnet using APA with two physical ports in hot standby mode or LAN monitor mode. You cannot configure more than one heartbeat IP address on an interface; only one HEARTBEAT_IP is allowed for each NETWORK_INTERFACE.
NOTE: Limitations: • Because Veritas Cluster File System from Symantec (CFS) requires link-level traffic communication (LLT) among the nodes, Serviceguard cannot be configured in cross-subnet configurations with CFS alone. But CFS is supported in specific cross-subnet configurations with Serviceguard and HP add-on products; see the documentation listed under “Cross-Subnet Configurations” (page 29) for more information. • IPv6 heartbeat subnets are not supported in a cross-subnet configuration.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 131)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
this list). Used only if a lock disk is used for tie-breaking services. Use FIRST_CLUSTER_LOCK_PV for the first physical lock volume and SECOND_CLUSTER_LOCK_PV for the second physical lock volume, if any. If there is a second physical lock volume, SECOND_CLUSTER_LOCK_PV must be on a separate line. These parameters are used only when you employ a lock disk for tie-breaking services in the cluster.
of four capacities per cluster, unless you use the reserved CAPACITY_NAME package_limit; in that case, you can use only that capacity throughout the cluster. For all capacities other than package_limit, the default weight for all packages is zero, though you can specify a different default weight for any capacity other than package_limit; see the entry for WEIGHT_NAME and WEIGHT_DEFAULT later in this list. See “About Package Weights” (page 150) for more information.
NOTE: • For most clusters that use an LVM cluster lock or lock LUN, a minimum MEMBER_TIMEOUT of 14 seconds is appropriate. • For most clusters that use a MEMBER_TIMEOUT value lower than 14 seconds, a quorum server is more appropriate than a lock disk or lock LUN. The cluster will fail if the time it takes to acquire the disk lock exceeds 0.2 times the MEMBER_TIMEOUT.
See also “What Happens when a Node Times Out” (page 93), “Cluster Daemon: cmcld” (page 41), and the white paper Optimizing Failover Time in a Serviceguard Environment (version A.11.19 and later) on http:// www.hp.com/go/hpux-serviceguard-docs. Can be changed while the cluster is running. AUTO_START_TIMEOUT The amount of time a node waits before it stops trying to join a cluster during automatic cluster startup.
Table 7 Failure Recovery Detection Times for an IP Monitored Ethernet Interface Values of Network Polling Failure/Recovery Detection Times (in seconds) Interval (NPI) (in seconds) 1 ~ NPI x 8 - NPI x 9 2 ~ NPI x 4 - NPI x 5 3 ~ NPI x 3 - NPI x 4 >=4 ~ NPI x 2 - NPI x 3 WARNING! If the values of NETWORK_POLLING_INTERVAL is greater than or equal to 8, a single lost link-level packet will cause a LAN to be marked as DOWN. IMPORTANT: HP strongly recommends using the default.
and the cluster nodes, sum the values for each path and use the largest number. CAUTION: Serviceguard supports NFS-mounted file systems only over switches and routers that support MBTD. If you are using NFS-mounted file systems, you must set CONFIGURED_IO_TIMEOUT_EXTENSION as described here. For a fuller discussion of MBTD, see the white paper Support for NFS as a file system type with Serviceguard 11.20 on HP-UX 11i v3 which you can find at http:// www.hp.com/go/hpux-serviceguard-docs.
target polling; otherwise the subnet is listed with IP_MONITOR set to OFF. See “Monitoring LAN Interfaces and Detecting Failure: IP Level” (page 74) for more information. Can be changed while the cluster is running; must be removed, with its accompanying IP_MONITOR and POLLING_TARGET entries, if the subnet in question is removed from the cluster configuration. IP_MONITOR Specifies whether or not the subnet specified in the preceding SUBNET entry will be monitored at the IP layer.
MAX_CONFIGURED_PACKAGES This parameter sets the maximum number of packages that can be configured in the cluster. The minimum value is 0, and the maximum value is 300. The default value is 300, and you can change it without halting the cluster. WEIGHT_NAME, WEIGHT_DEFAULT Default value for this weight for all packages that can have weight; see “Rules and Guidelines” (page 155) under“About Package Weights” (page 150).
Cluster Configuration: Next Step When you are ready to configure the cluster, proceed to “Configuring the Cluster ” (page 190). If you find it useful to record your configuration ahead of time, use the worksheet in Appendix E. Package Configuration Planning Planning for packages involves assembling information about each group of highly available services. NOTE: As of Serviceguard A.11.18, there is a new and simpler way to configure packages.
NOTE: Generic resources influences the package based on the status. The actual monitoring of the resource should be done in a script and this must be configured as a service. The script sets the status of the resource based on the availability of the resource. See “Monitoring Script for Generic Resources” (page 396). Create a list by package of volume groups, logical volumes, and file systems. Indicate which nodes need to have access to common file systems at different times.
To create modular CVM disk group and CFS mount point packages, use the cmmakepkg command and edit the parameters in the package configuration file. You can choose to name the packages. You can also use the Serviceguard Manager to create the modular CFS packages. See the Serviceguard Manager online help for more information. Alternatively, you can create CFS legacy disk group and mount point packages with the cfs family of commands.
About the Volume Monitor Simply monitoring each physical disk in a Serviceguard cluster does not provide adequate monitoring for volumes managed by Veritas Volume Manager from Symantec (VxVM) or logical volumes managed by HP-UX Logical Volume Manager (LVM), because a physical volume failure is not always a critical failure that triggers failover (for example, the failure of a single physical disk within a mirrored volume is not considered critical).
[-t, --poll-interval [ ...] A brief description of each parameter follows: -h or --help Displays the usage, as listed above, and exits. NOTE: Do not include the help or version parameters in your service command; this will result in immediate package failure at runtime. -v or --version Displays the monitor version and exits. -O or --log-file Specifies a file for logging (log messages are printed to the console by default). -D or --log-level Specifies the log level.
The Volume Monitor does not detect the following failures: • Failure of a redundant link to a storage device or set of devices where a working link remains • Failure of a mirror or mirrored plex within a volume (assuming at least one mirror or plex is functional) • Corruption of data on a monitored volume. Planning for NFS-mounted File Systems As of Serviceguard A.11.20, you can use NFS-mounted (imported) file systems as shared storage in packages.
◦ The nodes should not mount the file system on boot; it should be mounted only as part of the startup for the package that uses it. ◦ The NFS file system should be used by only one package. ◦ While the package is running, the file system should be used exclusively by the package. ◦ If the package fails, do not attempt to restart it manually until you have verified that the file system has been unmounted properly. In addition, you should observe the following guidelines.
Table 8 Package Failover Behavior Switching Behavior Parameters in Configuration File Package switches normally after detection of • node_fail_fast_enabled set to no. (Default) service, network, generic resource failure, or EMS failure, or when a configured resource • service_fail_fast_enabled set to NO for all services. (Default) dependency is not met. Halt script runs before • auto_run set to yes for the package. (Default) switch takes place.
The following is an example of how to configure simple and extended resources. Simple generic resource: generic_resource_name generic_resource_evaluation_type sfm_disk before_package_start Extended generic resource: generic_resource_name generic_resource_evaluation_type generic_resource_up_criteria cpu_lan during_package_start <50 For more information on the generic resource parameters, see “Package Parameter Explanations” (page 237).
cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration 4. When verification has completed without errors, apply the package configuration file. This adds the package configuration information (along with generic resources) to the binary cluster configuration file in the $SGCONF directory (usually /etc/cmcluster directory) and distributes it to all the cluster nodes. cmapplyconf -P $SGCONF/pkg1/pkg1.
Getting and Setting the Status/Value of a Simple/Extended Generic Resource You can use the Serviceguard commands cmgetresource(1m) and cmsetresource(1m), respectively, to get or set the status of a simple generic resource or the value of an extended generic resource. These commands can also be used in the monitoring script or executed from the CLI. You must be a root user (UID=0) to execute these commands. Non-root users cannot run these commands.
Parameters for Configuring EMS Resources NOTE: The default form for parameter names and literal values in the modular package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameters are concerned. This manual uses lower case, unless the parameter in question is used only in legacy packages, or the context refers exclusively to such a package.
are discussed later in this section under “Extended Dependencies” (page 147). You should read the next section, “Simple Dependencies” (page 143), first. Simple Dependencies A simple dependency occurs when one package requires another to be running on the same node. You define these conditions by means of the parameters dependency_condition and dependency_location, using the literal values UP and same_node, respectively.
• A package cannot depend on itself, directly or indirectly. That is, not only must pkg1 not specify itself in the dependency_condition (page 243), but pkg1 must not specify a dependency on pkg2 if pkg2 depends on pkg1, or if pkg2 depends on pkg3 which depends on pkg1, etc.
NOTE: Keep the following in mind when reading the examples that follow, and when actually configuring priorities: 1. auto_run (page 239) should be set to yes for all the packages involved; the examples assume that it is. 2. Priorities express a ranking order, so a lower number means a higher priority (10 is a higher priority than 30, for example).
In a simple dependency, if pkg1 depends on pkg2, and pkg1’s priority is lower than or equal to pkg2’s, pkg2’s node order dominates. Assuming pkg2’s node order is node1, node2, node3, then: • On startup: ◦ • pkg2 will start on node1, or node2 if node1 is not available or does not at present meet all of its dependencies, etc. – pkg1 will start on whatever node pkg2 has started on (no matter where that node appears on pkg1’s node_name list) provided all of pkg1’s other dependencies are met there.
Note that the nodes will be tried in the order of pkg1’s node_name list, and pkg2 will be dragged to the first suitable node on that list whether or not it is currently running on another node. • • On failover: ◦ If pkg1 fails on node1, pkg1 will select node2 to fail over to (or node3 if it can run there and node2 is not available or does not meet all of its dependencies; etc.) ◦ pkg2 will be dragged to whatever node pkg1 has selected, and restart there; then pkg1 will restart there.
IMPORTANT: If you have not already done so, read the discussion of Simple Dependencies (page 143) before you go on. The interaction of the legal values of dependency_location and dependency_condition creates the following possibilities: • Same-node dependency: a package can require that another package be UP on the same node. This is the case covered in the section on Simple Dependencies (page 143). • Different-node dependency: a package can require that another package be UP on a different node.
see Simple Dependencies (page 143); for exclusionary dependencies, see “Rules for Exclusionary Dependencies” (page 148). • Both packages must be failover packages whose failover_policy (page 241) is configured_node. • The priority (page 242) of the package depended on must be higher than or equal to the priority of the dependent package and the priorities of that package's dependents.
4. Starts the packages the failed package depends on (those halted in step 3, if any). If the failed package has been able to drag the packages it depends on to the adoptive node, Serviceguard starts them in the reverse of the order it halted them in the previous step (that is, the package that does not depend on any other package is started first). 5. 6. Starts the failed package. Starts the packages that depend on the failed package (those halted in step 1).
To implement the simple method, use the reserved keyword package_limit to define each node's capacity. In this case, Serviceguard will allow you to define only this single type of capacity, and corresponding package weight, in this cluster. Defining package weight is optional; for package_limit it will default to 1 for all packages, unless you change it in the package configuration file.
Points to Keep in Mind The following points apply specifically to the Simple Method (page 150). Read them in conjunction with the Rules and Guidelines (page 155), which apply to all weights and capacities. • If you use the reserved CAPACITY_NAME package_limit, then this is the only type of capacity and weight you can define in this cluster. • If you use the reserved CAPACITY_NAME package_limit, the default weight for all packages is 1.
Suppose you have the following configuration: • A two node cluster running four packages. These packages contend for resource we'll simply call A and B. • node1 has a capacity of 80 for A and capacity of 50 for B. • node2 has a capacity of 60 for A and capacity of 70 for B. • pkg1 uses 60 of the A capacity and 15 of the B capacity. • pkg2 uses 40 of the A capacity and 15 of the B capacity. • pkg3 uses insignificant amount (zero) of the A capacity and 35 of the B capacity.
You define weights for individual packages in the package configuration file, but you can also define a cluster-wide default value for a given weight, and, if you do, this default will specify the weight of all packages that do not explicitly override it in their package configuration file. NOTE: There is one exception: system multi-node packages cannot have weight, so a cluster-wide default weight does not apply to them.
weight_name A weight_value 40 In pkg3's package configuration file: weight_name B weight_value 35 weight_name A weight_value 0 In pkg4's package configuration file: weight_name B weight_value 40 IMPORTANT: weight_name in the package configuration file must exactly match the corresponding CAPACITY_NAME in the cluster configuration file. This applies to case as well as spelling: weight_name a would not match CAPACITY_NAME A.
• Package weight can be defined in cluster configuration file, via the WEIGHT_NAME and WEIGHT_DEFAULT parameters, or in the package configuration file, via the weight_name and weight_value parameters, or both. • Weights can be assigned (and WEIGHT_DEFAULTs, apply) only to multi-node packages and to failover packages whose failover_policy (page 241) is configured_node and whose failback_policy (page 242) is manual.
Example 2 • pkg1 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 10. It is running on node turkey and has switching enabled. • pkg2 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 20. It is running on node turkey and has switching enabled. • pkg3 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 30. It is down and has switching disabled.
NOTE: Some variables, including SG_PACKAGE, and SG_NODE, are available only at package run and halt time, not when the package is validated. You can use SG_PACKAGE_NAME at validation time as a substitute for SG_PACKAGE. IMPORTANT: For more information, see the template in $SGCONF/examples/ external_script.template. A sample script follows. It assumes there is another script called monitor.sh, which will be configured as a Serviceguard service to monitor some application. The monitor.
{ sg_log 5 "start_command" # log current PEV_MONITORING_INTERVAL value, PEV_ attribute can be changed # while the package is running sg_log 0 "PEV_MONITORING_INTERVAL for $SG_PACKAGE_NAME is $PEV_MONITORING_INTERVAL" return 0 } function stop_command { sg_log 5 "stop_command" # log current PEV_MONITORING_INTERVAL value, PEV_ attribute can be changed # while the package is running sg_log 0 "PEV_MONITORING_INTERVAL for $SG_PACKAGE_NAME is $PEV_MONITORING_INTERVAL" return 0 } typeset -i exit_val=0 case ${1} in
You can add custom code to the package to interrogate this variable, determine why the package halted, and take appropriate action. For legacy packages, put the code in the customer_defined_halt_cmds() function in the CUSTOMER DEFINED FUNCTIONS area of the package control script; see “Adding Customer Defined Functions to the Package Control Script ” (page 312). For modular packages, put the code in the package’s external script; see “About External Scripts” (page 157).
• Deploying applications in this environment requires careful consideration; see “Implications for Application Deployment” (page 161). • If a monitored_subnet (page 245) is configured for PARTIAL monitored_subnet_access in a package’s configuration file, it must be configured on at least one of the nodes on the node_name list for that package.
Configuring monitored_subnet_access In order to monitor subnet 15.244.65.0 or 15.244.56.0, depending on where pkg1 is running, you would configure monitored_subnet and monitored_subnet_access in pkg1’s package configuration file as follows: monitored_subnet 15.244.65.0 monitored_subnet_access PARTIAL monitored_subnet 15.244.56.
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. These procedures are carried out on one node, called the configuration node, and the resulting binary file is distributed by Serviceguard to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
################## cmcluster.conf ############### # Highly Available Cluster file locations # This file must not be edited ################################################# SGCONF=/etc/cmcluster SGSBIN=/usr/sbin SGLBIN=/usr/lbin SGLIB=/usr/lib SGRUN=/var/adm/cmcluster SGAUTOSTART=/etc/rc.config.d/cmcluster SGFFLOC=/opt/cmcluster/cmff CMSNMPD_LOG_FILE=/var/adm/SGsnmpsuba.log NOTE: If these variables are not defined on your system, then source the file /etc/ cmcluster.
2. If the cluster does not yet exist, set up root access among the prospective nodes: a. If you have not already done so, set up ssh public/private key pairs on each node. This will allow the necessary commands to operate on all the prospective nodes before a cluster is formed.
Removing a Node from a cDSF Group When you remove a node from a cluster that uses cDSFs, you should also remove the node from the cDSF group.
For a discussion of cluster networking topology, see “Redundant Ethernet Configuration ” (page 29). NOTE: You cannot use the Easy Deployment commands to create a cross-subnet configuration, as described under “Cross-Subnet Configurations” (page 29). • If you have not already done so, set up ssh public/private key pairs on each node. This will allow the Easy Deployment commands to operate on all the prospective nodes before cluster is formed.
NOTE: For information about heartbeat and networking requirements, see the sections listed under “Before You Start” (page 166). If you omit steps 1 and 2, and all the prospective nodes are connected to at least one subnet, cmdeploycl behaves as follows (the configuration is actually done by cmquerycl (1m) which is called by cmdeploycl). • If multiple subnets are configured among the nodes, cmdeploycl chooses a subnet with standby interfaces as the heartbeat.
• Configures a two-node cluster (cluster1). • Configures the LVM volume group and physical volume for the lock disk. • Starts the cluster on ftsys9 and ftsys10. NOTE: The cluster does not yet have shared storage for packages. Other forms of cmdeploycl allow you to create the cluster with a quorum server or lock LUN instead of a lock disk; see the manpage for more information.
5. Create shared storage for a LVM volume group with mirroring for use by cluster packages. cmpreparestg –l –p ... –n –n is the path name of the volume group that is to be created and imported to each specified node, and each is the name of a physical volume on the local node from which the volume group specified by is to be created.
6. Verify the storage using the following commands: • For LVM volume groups: vgdisplay • For VxVM/CVM disk groups: vxdg list The volume group or the disk group can now be used by packages; see Chapter 6 (page 232). NOTE: cmpreparestg requires either -l or -g in all cases. The command must be run on one of the specified nodes (ftsys9 or ftsys10 in this example).
For example: gryf sly bit root root root #cluster1, node1 #cluster1, node2 #cluster1, node3 This example grants root access to the node on which this cmclnodelist file resides to root users on the nodes gryf, sly, and bit. Serviceguard also accepts the use of a “+” in the cmclnodelist file; this indicates that the root user on any Serviceguard node can configure Serviceguard on this node. IMPORTANT: If $SGCONF/cmclnodelist does not exist, Serviceguard will look at ~/.rhosts.
Configuring Name Resolution Serviceguard uses the name resolution services built into HP-UX. NOTE: If you plan to use cmpreparecl (1m) (or cmpdeploycl (1m), which calls cmpreparecl), the /etc/hosts and /etc/nsswitch.conf configuration described in this subsection will be done automatically, but you should still read the entire subsection and make sure you understand the issues. In particular, you still need to make sure that aliases are properly represented in /etc/hosts, as described below.
15.145.162.131 10.8.0.131 10.8.1.131 gryf.uksr.hp.com gryf2.uksr.hp.com gryf3.uksr.hp.com gryf gryf gryf 15.145.162.132 10.8.0.132 10.8.1.132 sly.uksr.hp.com sly2.uksr.hp.com sly3.uksr.hp.com sly sly sly If applications require the use of hostname aliases, the Serviceguard hostname must be one of the aliases in all the entries for that host.
1. Edit the /etc/hosts file on all nodes in the cluster. Add name resolution for all heartbeat IP addresses, and other IP addresses from all the cluster nodes; see “Configuring Name Resolution” (page 173) for discussion and examples. NOTE: For each cluster node, the public-network IP address must be the first address listed. This enables other applications to talk to other nodes on public networks. 2. If you are using DNS, make sure your name servers are configured in /etc/resolv.
For information about configuring NTP services, refer to the HP-UX manual HP-UX Internet Services Administrator’s Guide posted at http://www.hp.com/go/hpux-networking-docs. Tuning Network and Kernel Parameters Serviceguard and its extension products, such as SGeSAP and SGeRAC, have been tested with default values of the network and kernel parameters supported by the ndd and kmtune utilities. You may need to adjust these parameters for larger cluster configurations and applications.
is enabled. For more information see “Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY” (page 355). Creating Mirrors of Root Logical Volumes HP strongly recommends that you use mirrored root volumes on all cluster nodes. The following procedure assumes that you are using separate boot and root volumes; you create a mirror of the boot volume (/dev/vg00/lvol1), primary swap (/dev/vg00/lvol2), and root volume (/dev/vg00/lvol3).
Root: lvol3 on: Swap: lvol2 on: Dump: lvol2 on: /dev/dsk/c4t6d0 /dev/dsk/c4t5d0 /dev/dsk/c4t6d0 /dev/dsk/c4t5d0 /dev/dsk/c4t6d0 /dev/dsk/c4t6d0, 0 Choosing Cluster Lock Disks The following guidelines apply if you are using a lock disk. See “Cluster Lock ” (page 47) and “Cluster Lock Planning” (page 103) for discussion of cluster lock options. The cluster lock disk is configured on an LVM volume group that is physically connected to all cluster nodes.
• A lock LUN cannot be used in a dual-lock configuration. • You do not need to back up the lock LUN data, and in fact there is no way to do so. A lock LUN needs only a small amount of storage, about 100 KB. • If you are using a disk array, create the smallest LUN the array will allow, or, on an HP Integrity server, you can partition a LUN; see “Creating a Disk Partition on an HP Integrity System”. • If you are using individual disks, use either a small disk, or a portion of a disk.
NOTE: The first partition, identified by the device file /dev/dsk/c1t4d0s1 or /dev/ disk/disk12_p1 in this example, is reserved by EFI and cannot be used for any other purpose. 4. Create the device files on the other cluster nodes. Use the command insf -e on each node. This will create device files corresponding to the three partitions, though the names themselves may differ from node to node depending on each node’s I/O configuration. 5. Define the lock LUN; see “Defining the Lock LUN”.
cmquerycl -C test.conf -n pig Note: Disks were discovered which are not in use by either LVM or VxVM. Use pvcreate(1M) to initialize a disk for LVM or, use vxdiskadm(1M) to initialize a disk for VxVM. IMPORTANT: The purpose of cmnotdisk.conf is to exclude specific devices, usually CD and DVD drives, that Serviceguard does not recognize and which should not be probed. Make sure you do not add a DEVICE_FILE entry in cmnotdisk.
• “Distributing Volume Groups to Other Nodes” (page 185) • “Making Physical Volume Group Files Consistent” (page 187) NOTE: The procedures that follow describe the command-line method of configuring LVM storage. There are two other, more automated methods you can use. • System Management Homepage You can use the System Management Homepage to create or extend volume groups and create logical volumes. From the System Management Homepage, choose Disks and File Systems.
Using the EMS Disk Monitor The Event Monitoring Service HA Disk Monitor allows you to monitor the health of LVM disks. If you intend to use this monitor for your mirrored disks, you should configure them in physical volume groups. For more information, see Using High Availability Monitors at the address given in the preface to this manual.
where hh must be unique to the volume group you are creating. Use a unique minor number that is available across all the nodes for the mknod command above. (This will avoid further reconfiguration later, when NFS-mounted logical volumes are created in the volume group.) Use the following command to display a list of existing volume groups: ls -l /dev/*/group 3.
sets the timeout value in seconds for a logical volume. For example, to set the timeout for /dev/ vg01/lvol1 to one minute, enter the following command: lvchange -t 60 /dev/vg01/lvol1 TIP: Set the logical volume timeout to an integral multiple of any timeout assigned to the underlying physical volumes. Otherwise, the actual duration of the I/O request can exceed the logical volume timeout. For details on how to change the I/O timeout value on a physical volume, see the manpage for pvchange (1m).
NOTE: Do this during this setup process only, so that activation and mounting can be done by the package control script at run time. You do not need to deactivate and unmount a volume simply in order to create a map file (as in step 1 of the procedure that follows). Distributing the Volume Group NOTE: If you use cmpreparestg, you can skip the procedure that follows and proceed to “Making Physical Volume Group Files Consistent” (page 187).
NOTE: When you use PVG-strict mirroring, the physical volume group configuration is recorded in the /etc/lvmpvg file on the configuration node. This file defines the physical volume groups which are the basis of mirroring and indicate which physical volumes belong to each physical volume group. Note that on each cluster node, the /etc/lvmpvg file must contain the correct physical volume names for the physical volume groups’s disks as they are known on that node.
substituting other volume group names, logical volume names, and physical volume names. Pay close attention to the disk device names, which can vary from one node to another. Creating a Storage Infrastructure with VxVM In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM), Veritas Volume Manager (VxVM), or Veritas Cluster Volume Manager (CVM).
Creating Disk Groups NOTE: You can use cmpreparestg (1m) to create a VxVM/CVM disk group. See “Using Easy Deployment Commands to Configure the Cluster” (page 167) for more information. If you use cmpreparestg, you do not need to perform the procedures that follow, but it is a good idea to read them so that you understand what cmpreparestg does for you.
3. Mount the volume: mount /dev/vx/dsk/logdata/log_files /logs 4.
NOTE: You can use Serviceguard Manager to configure a cluster: open the System Management Homepage (SMH) and choose Tools-> Serviceguard Manager. See “Using Serviceguard Manager” (page 24) for more information. You can also use Easy Deployment commands; see“About Easy Deployment” (page 110) and “Using Easy Deployment Commands to Configure the Cluster” (page 167). To use traditional Serviceguard commands to configure the cluster, follow directions in the remainder of this section.
If you use the -a option, Serviceguard will ignore the value of the HOSTNAME_ADDRESS_FAMILY parameter in the existing cluster configuration, if any, and attempt to resolve the cluster and Quorum Server hostnames as specified by the -a option: • If you specify -a ipv4, each of the hostnames must resolve to at least one IPv4 address; otherwise the command will fail. • Similarly, if you specify -a ipv6, each of the hostnames must resolve to at least one IPv6 address; otherwise the command will fail.
You can now edit mynetwork to add IP address and subnet information for new interfaces which are not yet configured, and then use cmapplyconf (1m) to configure the new interfaces into the cluster; for example: cmapplyconf -N mynetwork IMPORTANT: • You cannot use cmapplyconf -N if the cluster already exists; in that case, follow instructions under “Changing the Cluster Networking Configuration while the Cluster Is Running” (page 302).
NOTE: You should not configure a second lock volume group or physical volume unless your configuration specifically requires it. See “Dual Lock Disk” (page 48).
setting, which defaults to IPv4. See the parameter descriptions under “Cluster Configuration Parameters ” (page 114) for more information. IMPORTANT: For important information, see also “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 111); and “What Happens when You Change the Quorum Configuration Online” (page 50) Obtaining Cross-Subnet Information As of Serviceguard A.11.
15.13.182.0 lan2 lan2 (nodeC) (nodeD) 15.244.65.0 lan3 lan3 (nodeA) (nodeB) 15.244.56.0 lan4 lan4 (nodeC) (nodeD) 3ffe:1111::/64 lan3 lan3 (nodeA) (nodeB) 3ffe:2222::/64 lan3 lan3 (nodeC) (nodeD) IPv6: Possible Heartbeat IPs: 15.13.164.0 15.13.164.1 15.13.164.2 (nodeA) (nodeB) 15.13.172.0 15.13.172.158 15.13.172.159 (nodeC) (nodeD) 15.13.165.0 15.13.165.1 15.13.165.2 (nodeA) (nodeB) 15.13.182.0 15.13.182.158 15.13.182.
The heartbeat can comprise multiple subnets joined by a router. In this case at least two heartbeat paths must be configured for each cluster node. See also the discussion of HEARTBEAT_IP under “Cluster Configuration Parameters ” (page 114), and “Cross-Subnet Configurations” (page 29). Specifying Maximum Number of Configured Packages This specifies the most packages that can be configured in the cluster.
Figure 36 Access Roles Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 36 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
IMPORTANT: Users on systems outside the cluster can gain Serviceguard root access privileges to configure the cluster only via a secure connection (rsh or ssh). • Non-root access: Other users can be assigned one of four roles: ◦ Full Admin: Allowed to perform cluster administration, package administration, and cluster and package view operations. These users can administer the cluster, but cannot configure or create a cluster. Full Admin includes the privileges of the Package Admin role.
Access control policies are defined by three parameters in the configuration file: • Each USER_NAME can consist either of the literal ANY_USER, or a maximum of 8 login names from the /etc/passwd file on USER_HOST. The names must be separated by spaces or tabs, for example: # Policy 1: USER_NAME john fred patrick USER_HOST bit USER_ROLE PACKAGE_ADMIN • USER_HOST is the node where USER_NAME will issue Serviceguard commands.
If this policy is defined in the cluster configuration file, it grants user john the PACKAGE_ADMIN role for any package on node bit. User john also has the MONITOR role for the entire cluster, because PACKAGE_ADMIN includes MONITOR. If the policy is defined in the package configuration file for PackageA, then user john on node bit has the PACKAGE_ADMIN role only for PackageA. Plan the cluster’s roles and validate them as soon as possible.
If a role is configured for a username/hostname in the cluster configuration file, do not specify a role for the same username/hostname in the package configuration file; and note that there is no point in assigning a package administration role to a user who is root on any node in the cluster; this user already has complete control over the administration of the cluster and its packages.
NOTE: Using the -k option means that cmcheckconf only checks disk connectivity to the LVM disks that are identified in the ASCII file. Omitting the -k option (the default behavior) means that cmcheckconf tests the connectivity of all LVM disks on all nodes. Using -k can result in significantly faster operation of the command. For more information, see the manpage for cmcheckconf (1m) and “Checking Cluster Components” (page 278).
NOTE: You must use the vgcfgbackup command to store a copy of the cluster lock disk's configuration data whether you created the volume group using the System Management Homepage (SMH), SAM, or HP-UX commands. If the cluster lock disk ever needs to be replaced while the cluster is running, you must use the vgcfgrestore command to restore lock information to the replacement disk.
Table 9 Differences between Legacy CFS and Modular CFS (continued) Legacy Modular You must not manually edit the packages, but use the cfs commands to edit the package. The disadvantage is that if you have multiple packages, you must edit each of them using the commands. Provides improved manageability of the package by allowing you to edit the parameters in the package configuration file. Also, allows online addition, deletion, and/or modification of CFS parameters.
Table 10 Operational commands for Legacy CFS and Modular CFS (continued) Operation Commands used by legacy style Equivalent commands in modular style For information on the parameters that can be added, removed, or modified online, see “Online reconfiguration of modular CFS package parameters” (page 215).
NOTE: Do not edit the configuration file SG-CFS-pkg.conf. Create and modify configuration using the cfs administration commands. 1. First, make sure the cluster is running: cmviewcl 2. If it is not, start it: cmruncl 3. 4. If you have not initialized your disk groups, or if you have an old install that needs to be re-initialized, use the vxinstall command to initialize VxVM/CVM disk groups. See “Initializing the Veritas Volume Manager ” (page 225).
1. Find the master node using vxdctl or cfscluster status . 2. Initialize a new disk group, or import an existing disk group, in shared mode, using the vxdg command. • For a new disk group use the init option: vxdg -s init logdata c4t0d6 • For an existing disk group, use the import option: vxdg -C -s import logdata 3. Verify the disk group. The state should be enabled and shared: vxdg list NAME logdata STATE enabled, shared, cds ID 11192287592.39.ftsys9 Creating the Disk Group Cluster Packages 1.
Creating Volumes 1. Make log_files volume on the logdata disk group: vxassist -g logdata make log_files 1024m 2.
modular CFS package configuration file created using the sg/cfs_all template. CVM must be up and running to create, modify, or run modular CFS packages. Refer to the Serviceguard man pages for more information about the commands cmmakepkg (1m), cmapplyconf (1m), cfsdgadm (1m), cfsmntadm (1m), cmrunpkg (1m), cmhaltpkg (1m), and cmdeleteconf (1m).
cvm_disk_group cvm_activation_mode cvm_dg1_node1 "node1=sw node2=sw node3=sw node4=sw" cfs_mount_point cfs_volume cfs_mount_options cfs_primary_policy /mnt2 cvm_dg1_node1/lvol1 "node1=cluster node2=cluster node3=cluster node4=cluster" "" NOTE: By default, the package parameters are commented. You must uncomment them and specify the values.
NODE node1 node2 node3 node4 STATUS up up up up STATE running running running running MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg cfspkg1 STATUS up up STATE running running AUTO_RUN enabled enabled SYSTEM yes no bdf Filesystem kbytes used avail %used Mounted on /dev/vg00/lvol3 24215552 20954099 3058098 87% / /dev/vg00/lvol1 2031616 365773 1561780 19% /stand /dev/odm 0 0 0 0% /dev/odm /dev/vx/dsk/cvm_dg1_node1/lvol1 167808 5313 152346 3% /mnt2 Parallel Activation of Disk Groups and Parallel Mounting of M
1. Create a package configuration file for the storage checkpoint: cmmakepkg -m sg/cfs_all /etc/cmcluster/ckpt1.ascii Package template is created. This file must be edited before it can be used. 2. Edit the following package parameters in the ckpt1.
Successfully started package ckpt1 on node node1 Successfully started package ckpt1 on node node2 cmrunpkg: All specified packages are running 7. Verify the modular package of mount point for the checkpoint package(s) running.
dependency_name dependency_condition dependency_location 5. dg2pkg dg2pkg = UP same_node Check the package configuration file: cmcheckconf -P snap1.ascii cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration 6. Apply the packages configuration file: cmapplyconf -P snap1.ascii Modify the package configuration ([y]/n)? y Completed the cluster update 7.
The following table lists the parameters that can be reconfigured online for the various modular CFS packages: Table 11 Package parameters that can be added, removed, or modified online Packages Addition Removal Modification Disk group cvm_disk_group cvm_disk_group cvm_activation_mode cvm_activation_mode cvm_activation_mode NOTE: Modifying this parameter typically works as an offline operation.
NOTE: Some of the CVM/CFS parameters cannot be modified online and requires the package to be halted. This is due to the limitation that changing CVM/CFS parameters while they are in use is not supported by Symantec. So, it is recommended to do this only when applications that depend on these packages are offline. Example of online addition and deletion of a disk group and a mount point from a modular CFS package For online changes, the package must be up and running.
cvm_disk_group:cvm_dg0_node1|cvm_activation_mode=all=sw cvm_disk_group:cvm_dg2_node1|cvm_disk_group=cvm_dg2_node1 cvm_disk_group:cvm_dg2_node1|cvm_activation_mode=all=sw cfs_mount_point:/mnt1|cfs_volume=cvm_dg1_node1/lvol1 cfs_mount_point:/mnt3|cfs_volume=cvm_dg0_node1/lvol3 Guidelines for Migrating from Legacy CFS Package to Modular CFS Package • You must migrate the legacy CFS packages to modular CFS package only when the package is offline.
Figure 37 Legacy Style of Packaging Figure 38 illustrates an example in which all the disk group and mount point packages used by Application 1 are consolidated into a single modular package as compared to four separate packages in the legacy style of packaging. Figure 38 Modular Style of Packaging Use Case 2: Migration of two different applications using checkpoint and snapshot packages Figure 39 illustrates a cluster with two applications independent of each other.
Figure 39 Legacy Style of Packaging Figure 40 illustrates the consolidated modular CFS packages. All the disk group and mount point packages used by Application 1 are consolidated into a single modular package. Similarly, the disk group and mount point packages used by Application 2 are consolidated into a single modular package. The disk group, cvm_dg3, is merged with the snapshot package to enable consolidation of storage within the same package.
Managing Disk Groups and Mount Points Using Legacy Packages To manage disk groups and mount points in a Serviceguard cluster using CVM/CFS, create disk group and mount point packages. These packages are configured for activating disk groups and mounting the volumes. The applications that use CVM/CFS require the disk group and mount point packages to be up and running. Creating a File System and Mount Point Package 1.
SG-CFS-MP-1 7. After creating the mount point packages for the cluster file system, configure your application package to depend on the mount points. In the configuration file, configure a dependency, setting dependency_condition to SG-CFS-MP-pkg-# =UP and dependency_location to SAME_NODE. For more information about these parameters, see “Package Parameter Explanations” (page 237).
SG-CFS-pkg SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-CK-1 up up up up running running running running enabled enabled enabled disabled yes no no no /tmp/check_logfiles now contains a point-in-time view of /tmp/logdata/log_files, and it is persistent.
SG-CFS-MP-1 SG-CFS-SN-1 up up running running enabled disabled no no The snapshot file system /local/snap1 is now mounted and provides a point in time view of /tmp/logdata/log_files.
Initializing the Veritas Volume Manager If you are about to create disk groups for the first time, you need to initialize the Volume Manager. Use the following command after installing VxVM/CVM on each node: vxinstall This displays a menu-driven program that steps you through the VxVM/CVM initialization sequence.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: /usr/lib/vxvm/bin/vxdisksetup -i c4t3d4 Creating Disk Groups Use the following steps to create disk groups. 1. Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: vxdg -s init logdata c0t3d2 2.
packages. Use one cvm_dg for each disk group the package will use, and select the appropriate cvm_activation_cmd. Package configuration is described in detail in Chapter 6 (Chapter 7 for legacy packages).
You can use these commands to test cluster operation, as in the following: 1. If the cluster is not already running, start it. From the Serviceguard Manager menu, choose Run Cluster. From the command line, use cmruncl -v. By default, cmruncl will check the networks. Serviceguard will probe the actual network configuration with the network information in the cluster configuration. If you do not need this validation, use cmruncl -v -w none instead, to turn off validation and save time 2.
There are three cases: • The cluster is not running on any node, all cluster nodes must be reachable, and all must be attempting to start up. In this case, the node attempts to form a cluster consisting of all configured nodes. • The cluster is already running on at least one node. In this case, the node attempts to join that cluster. • Neither is true: the cluster is not running on any node, and not all the nodes are reachable and trying to start.
Single-Node Operation Single-node operation occurs in a single-node cluster or in a multi-node cluster, following a situation where all but one node has failed, or where you have shut down all but one node, which will probably have applications running. As long as the Serviceguard daemon cmcld is active, other nodes can rejoin the cluster at a later time. If the Serviceguard daemon fails when in single-node operation, it will leave the single node up and your applications running.
It is recommended that you do not proceed with the configuration operation unless you are sure these nodes are permanently unavailable.Do you want to continue? Reply Yes to remove the configuration. Later, if the inaccessible node becomes available, you should run the cmdeleteconf command on that node to remove the configuration file.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. See “What is Serviceguard? ” (page 21), “How the Package Manager Works” (page 50), and “Package Configuration Planning ” (page 131) for more information.
To choose the right package modules, you need to decide the following things about the package you are creating: • What type of package it is; see “Types of Package: Failover, Multi-Node, System Multi-Node” (page 233). • Which parameters need to be specified for the package (beyond those included in the base type, which is normally failover, multi-node, or system-multi-node). See “Package Modules and Parameters” (page 234).
NOTE: On systems that support CFS, you configure the legacy CFS system multi-node package by means of the cfscluster command, not by editing a package configuration file. For modular CFS packages, you cannot edit using the cfs commands, but must edit the parameters in the package configuration files; see “Creating a Storage Infrastructure with Veritas Cluster File System (CFS)” (page 204).
you are used to creating legacy packages, you will notice that parameters from the package control script (or their equivalents) are now in the package configuration file; these parameters are marked (S) in the table. You can use cmmakepkg -l (letter “l”) to see a list of all available modules, including non-Serviceguard modules such as those supplied in the HP Toolkits.
Optional Package Modules Add optional modules to a base module if you need to configure the functions in question. Parameters marked with an asterisk (*) are new or changed as of Serviceguard A.11.18/19. (S) indicates that the parameter (or its equivalent) has moved from the package control script to the package configuration file for modular packages. See the “Package Parameter Explanations” (page 237) for more information.
Table 13 Optional Modules (continued) Module Name Parameters (page) Comments fs_mount_retry_count (page 254) (S) fs_umount_retry_count (page 254) * (S) fs_name (page 255) * (S) fs_directory (page 255) * (S) fs_type (page 255) (S) fs_mount_opt (page 256) (S) fs_umount_opt (page 256) (S) fs_fsck_opt (page 256) (S) * pev pev_ (page 256) Add to a base module to configure environment variables to be passed to an external script.
NOTE: For more information, see the comments in the editable configuration file output by the cmmakepkg command, and the cmmakepkg manpage.
node_name The node on which this package can run, or a list of nodes in order of priority, or an asterisk (*) to indicate all nodes. The default is *. For system multi-node packages, you must specify *. If you use a list, specify each node on a new line, preceded by the literal node_name, for example: node_name node_name node_name The order in which you specify the node names is important.
NOTE: If the package halt function fails with “exit 1”, Serviceguard does not halt the node, but sets no_restart for the package, which disables package switching (auto_run), thereby preventing the package from starting on any adoptive node. Setting node_fail_fast_enabled to yes ensures that the package can fail over to another node even if the package cannot halt successfully. Be careful when using node_fail_fast_enabled, as it will cause all packages on the node to halt abruptly.
successor_halt_timeout Specifies how long, in seconds, Serviceguard will wait for packages that depend on this package to halt, before halting this package. Can be 0 through 4294, or no_timeout. The default is no_timeout. • no_timeout means that Serviceguard will wait indefinitely for the dependent packages to halt. • 0 means Serviceguard will not wait for the dependent packages to halt before halting this package. New as of A.11.18 (for both modular and legacy packages).
a site-aware disaster-tolerant cluster, which requires Metrocluster (additional HP software); see the documents listed under “Cross-Subnet Configurations” (page 29) for more information. • site_preferred_manual means Serviceguard will try to fail the package over to a node on the local SITE. If there are no eligible nodes on the local SITE, the package will halt with global switching enabled. You can then restart the package locally, when a local node is available, or start it on another SITE.
IMPORTANT: Restrictions on dependency names in previous Serviceguard releases were less stringent. Packages that specify dependency_names that do not conform to the above rules will continue to run, but if you reconfigure them, you will need to change the dependency_name; cmcheckconf and cmapplyconf will enforce the new rules.
dependency_location Specifies where the dependency_condition must be met. • • If dependency_condition is UP, legal values fordependency_location are same_node, any_node, and different_node. ◦ same_node means that the package depended on must be running on the same node. ◦ different_node means that the package depended on must be running on a different node in this cluster. ◦ any_node means that the package depended on must be running on some node in this cluster.
local_lan_failover_allowed This parameter is used to control whether a package fails or not when a primary NIC fails on a cluster node and its IP addresses are transferred to a standby NIC. This parameter requires the corresponding subnet to be monitored. Subnet monitoring can be enabled using the monitored_subnet parameter in the package ascii file or cluster_interconnect_subnet parameter on SGeRAC clusters. Legal values are yes and no. Default is yes.
See the latest version of Using Serviceguard Extension for RAC at http://www.hp.com/go/ hpux-serviceguard-docs for more information, including the status of Oracle RAC support for an IPv6 cluster_interconnect_subnet. New for A.11.18 (for both modular and legacy packages). ip_subnet Specifies an IP subnet used by the package for relocatable addresses; see also ip_address (page 247) and “Stationary and Relocatable IP Addresses ” (page 68).
ip_subnet_node In a cross-subnet configuration, specifies which nodes an ip_subnet is configured on. If no ip_subnet_node are listed under an ip_subnet, it is assumed to be configured on all nodes in this package’s node_name list (page 239). Can be added or deleted while the package is running, with these restrictions: • The package must not be running on the node that is being added or deleted.
service_cmd The command that runs the application or service for this service_name, for example, /usr/bin/X11/xclock -display 15.244.58.208:0 An absolute pathname is required; neither the PATH variable nor any other environment variable is passed to the command. The default shell is /usr/bin/sh. NOTE: Be careful when defining service run commands. Each run command is executed in the following way: • The cmrunserv command executes the run command.
Each name must be unique within a package, but a single resource can be specified across multiple packages. For example, a generic resource sfm_disk can be specified in two packages, pkg1 and pkg2. You can configure a maximum of 100 generic resources per cluster. Each generic resource is defined by three parameters: • generic_resource_name • generic_resource_evaluation_type • generic_resource_up_criteria See the descriptions that follow.
NOTE: Operators other than the ones mentioned above are not supported. This attribute does not accept more than one up criterion. For e.g., >> 10, << 100 are not valid.
resource_up_value Defines a criterion used to determine whether the resource identified by resource_name is up. Requires an operator and a value. Values can be string or numeric. The legal operators are =, !=, >, <, >=, or<=, depending on the type of value. If the type is string, then only = and != are valid. If the string contains white space, it must be enclosed in quotes. String values are case-sensitive. The maximum length of the entire resource_up_value string is 1024 characters.
vgchange_cmd Specifies the method of activation for each HP-UX Logical Volume Manager (LVM) volume group identified by a vg entry (see (page 252)). Replaces VGCHANGE which is still supported in the package control script for legacy packages; see “Configuring a Legacy Package” (page 307). The default is vgchange -a e. The configuration file contains several other vgchange command variants; either uncomment one of these and comment out the default, or use the default.
Any package using a CVM disk group must declare a dependency (see the entries for dependency_name and related parameters starting on (page 242)) on the CVM system multi-node package. See “Preparing the Cluster for Use with CVM ” (page 225). vxvm_dg Specifies a VxVM disk group (one per vxvm_dg, each on a new line) on which a file system needs to be mounted.
concurrent_fsck_operations The number of concurrent fsck operations allowed on file systems being mounted during package startup. Legal value is any number greater than zero. The default is 1. If the package needs to run fsck on a large number of file systems, you can improve performance by carefully tuning this parameter during testing (increase it a little at time and monitor performance each time).
fs_name • For a local file system, this parameter, in conjunction with fs_directory, fs_type, fs_mount_opt, fs_umount_opt, and fs_fsck_opt, specifies the file system that is to be mounted by the package. fs_name must specify the block devicefile for a logical volume. • For an NFS-imported file system, the additional parameters required are fs_server, fs_directory, fs_type, and fs_mount_opt; see fs_server (page 255) for an example.
fs_mount_opt The mount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. See the mount (1m) manpage for more information. fs_umount_opt The umount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. Using the-s option of umount will improve startup performance if the package uses a large number of file systems. See the mount (1m) manpage for more information.
user_name Specifies the name of a user who has permission to administer this package. See also user_host and user_role; these three parameters together define the access-control policy for this package (see “Controlling Access to the Cluster” (page 197)). These parameters must be defined in this order: user_name, user_host, user_role. Legal values for user_name are any_user or a maximum of eight login names from /etc/ passwd on user_host.
Generating the Package Configuration File When you have chosen the configuration modules your package needs (see “Choosing Package Modules” (page 232)), you are ready to generate a package configuration file that contains those modules. This file will consist of a base module (usually failover, multi-node or system multi-node) plus the modules that contain the additional parameters you have decided to include.
• To generate a configuration file adding the services module to an existing package (enter the command all on one line): cmmakepkg -i $SGCONF/pkg1/pkg1.conf -m sg/service $SGCONF/pkg1/pkg1_v2.conf • To generate a configuration file adding the Persistent Reservation module to an existing package: cmmakepkg -i $SGCONF/pkg1/pkg1.conf -m sg/pr_cntl NOTE: You can add more than one module at a time.
NOTE: Optional parameters are commented out in the configuration file (with a # at the beginning of the line). In some cases these parameters have default values that will take effect unless you uncomment the parameter (remove the #) and enter a valid value different from the default. Read the surrounding comments in the file, and the explanations in this chapter, to make sure you understand the implications both of accepting and of changing a given default.
• local_lan_failover_allowed. Use this parameter control whether a package fails or not when a primary NIC fails on a cluster node and its IP addresses are transferred to a standby NIC. Legal values are yes and no. Default is yes. See the parameter description (page 245) for more information. • Use the monitored_subnet parameter to specify a subnet that is to be monitored for this package. If there are multiple subnets, repeat the parameter as many times as needed, on a new line each time.
• If the package needs to mount LVM volumes to file systems, use the vg parameters (page 252) to specify the names of the volume groups to be activated, select the appropriate vgchange_cmd, and use the fs_ options in the FILESYSTEMS portion of the configuration file to specify the options for mounting and unmounting the file systems. Do not use the vxvm_dg or cvm_dg parameters for LVM volume groups.
• To coordinate the startup and shutdown of database software with cluster node startup and shutdown, you can use the database template files provided with the separately purchasable Enterprise Cluster Master Toolkit product. These files are in /opt/cmcluster/toolkit/DB/. Separate toolkits are available for Oracle, Informix, and Sybase. In addition to the standard package script, you use the special script that is provided for the database.
Adding the Package to the Cluster You can add the new package to the cluster while the cluster is running, subject to the value of MAX_CONFIGURED_PACKAGES in the cluster configuration file. See “Adding a Package to a Running Cluster” (page 318). How Control Scripts Manage VxVM Disk Groups VxVM disk groups (other than those managed by CVM) are outside the control of the Serviceguard cluster. The package control script uses standard VxVM commands to import and deport these disk groups.
7 Cluster and Package Maintenance This chapter describes how to see cluster configuration and status information, how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Viewing CFS Multi-Node Information If you are running Veritas Cluster File System (CFS) cluster, you can use cfs commands to see multi-node package configuration information, status, and dependencies; for example: cfsdgadm show_package cfsmntadm show_package getconf -p | grep dependency Types of Cluster and Package States A cluster or its component nodes may be in several different states at different points in time.
• start_wait — A cmrunpkg command is in progress for this package. The package is waiting for packages it depends on (predecessors) to start before it can start. • starting — The package is starting. The package master control script is running. • halting — A cmhaltpkg command is in progress for this package and the halt script is running. • halt_wait — A cmhaltpkg command is in progress for this package.
• reconfiguring — The node where this package is running is adjusting the package configuration to reflect the latest changes that have been applied. • reconfigure_wait — The node where this package is running is waiting to adjust the package configuration to reflect the latest changes that have been applied. • unknown — Serviceguard could not determine the state at the time cmviewcl was run.
Failover and Failback Policies Failover packages can be configured with one of two values for the failover_policy parameter (page 241), as displayed in the output of cmviewcl -v: • configured_node. The package fails over to the next node in the node_name list in the package configuration file (page 239). • min_package_node. The package fails over to the node in the cluster that has the fewest running packages.
Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM Service Service Subnet Generic Resource STATUS up up up up Node_Switching_Parameters: NODE_TYPE STATUS Primary up Alternate up MAX_RESTARTS 0 0 0 RESTARTS 0 0 0 SWITCHING enabled enabled NAME ftsys10 ftsys9 NAME service2 sfm_disk1_monitor 15.13.168.
SG-CFS-pkg NODE_NAME ftsys9 up running STATUS up yes SWITCHING enabled Script_Parameters: ITEM STATUS MAX_RESTARTS Service up 0 Service up 5 Service up 5 Service up 0 Service up 0 NODE_NAME ftsys10 enabled STATUS up RESTARTS 0 0 0 0 0 NAME SG-CFS-vxconfigd SG-CFS-sgcvmd SG-CFS-vxfsckd SG-CFS-cmvxd SG-CFS-cmvxpingd SWITCHING enabled Script_Parameters: ITEM STATUS MAX_RESTARTS Service up 0 Service up 5 Service up 5 Service up 0 Service up 0 RESTARTS 0 0 0 0 0 NAME SG-CFS-vxconfigd SG-CFS-sgcvmd
PRIMARY STANDBY up up 28.1 32.1 lan0 lan1 STATE halted AUTO_RUN disabled UNOWNED PACKAGES PACKAGE pkg2 STATUS down NODE unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM Resource Subnet Generic Resource Resource Subnet Generic Resource STATUS up up up down up up Node_Switching_Parameters: NODE_TYPE STATUS Primary up Alternate up NODE_NAME ftsys9 ftsys9 ftsys9 ftsys10 ftsys10 ftsys10 NAME /example/float 15.13.168.
Service Subnet Resource Generic Resource up up up up 0 0 Node_Switching_Parameters: NODE_TYPE STATUS Primary up Alternate up PACKAGE pkg2 STATUS up SWITCHING enabled enabled STATE running 0 0 NAME ftsys9 ftsys10 AUTO_RUN disabled sfm_disk_monitor 15.13.168.
Status After Halting a Node After we halt ftsys10 with the following command cmhaltnode ftsys10 we’ll see the following output from cmviewcl: CLUSTER example STATUS up NODE ftsys9 STATUS up PACKAGE pkg1 pkg2 NODE ftsys10 STATE running STATUS up up STATUS down STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE halted This output can be seen on both ftsys9 and ftsys10.
CLUSTER cats STATUS up NODE manx STATUS up PACKAGE pkg1 NODE tabby STATE running STATUS up STATUS up PACKAGE pkg2 STATE running AUTO_RUN enabled NODE manx AUTO_RUN enabled NODE tabby STATE running STATUS up STATE running SYSTEM_MULTI_NODE_PACKAGES: PACKAGE SG-CFS-pkg STATUS up STATE running Checking Status of the Cluster File System (CFS) If the cluster is using the cluster file system, you can check status with the cfscluster command, as in the example that follows.
Status of the Packages in a Cluster File System You can use cmviewcl to see the status of the package and the cluster file system on all nodes: cmviewcl -v -p SG-CFS-pkg MULTI_NODE_PACKAGES PACKAGE STATUS STATE AUTO_RUN SYSTEM SG-CFS-pkg up running enabled yes NODE_NAME STATUS SWITCHING soy up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd
Status of Legacy CVM Disk Group Packages To see the status of the legacy CVM disk group, use the cfsdgadm display command. For example, for the disk group logdata, enter: cfsdgadm display -v logdata NODE NAME ACTIVATION MODE ftsys9 MOUNT POINT sw (sw) SHARED VOLUME TYPE ftsys10 sw (sw) MOUNT POINT SHARED VOLUME TYPE ... To see which package is monitoring a disk group, use the cfsdgadm show_package command.
NOTE: • All of the checks below are performed when you run cmcheckconf without any arguments (or with only -v, with or without -k or -K). cmcheckconf validates the current cluster and package configuration, including external scripts and pre-scripts for modular packages, and runs cmcompare to check file consistency across nodes. (This new version of the command also performs all of the checks that were done in previous releases.) See “Checking Cluster Components” (page 278) for details.
Table 14 Verifying Cluster Components Component (Context) Tool or Command; More Information Comments Volume groups (cluster) cmcheckconf (1m), cmapplyconf (1m) Checked for: See also “Verifying the Cluster Configuration ” (page 202). • availability on each node • existence • same physical volumes on each node • physical volumes connected on each node Volume groups (package) cmcheckconf (1m), cmapplyconf (1m) Checked only on nodes configured to run the package.
Table 14 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Comments as ownership, content, etc., use cmcompare (1m). VxVM disk groups (package) cmcheckconf (1m), cmapplyconf (1m) See also “Verifying and Applying the Package Configuration” (page 263). Mount points (package) cmcheckconf (1m), cmapplyconf (1m) See also “Verifying and Applying the Package Configuration” (page 263).
Example The short script that follows runs cluster verification and sends an email to admin@hp.com when verification fails. #!/bin/sh cmcheckconf -v >/tmp/cmcheckconf.output if (( $? != 0 )) then mailx -s "Cluster verification failed" admin@hp.com 2>&1
on whether all nodes are currently down (that is, no cluster daemons are running), or whether you are starting the cluster daemon on an individual node. Note the distinction that is made in this chapter between adding an already configured node to the cluster and adding a new node to the cluster configuration. An already configured node is one that is already entered in the cluster configuration file; a new node is added to the cluster by modifying the cluster configuration file.
network and find the check takes a very long time, you can use the -w none option to bypass the validation. Since the node’s cluster is already running, the node joins the cluster. Packages may be started, depending on the package configuration; see “node_name” (page 239)). If the node does not find its cluster running, or the node is not part of the cluster configuration, the command fails.
New command options in Serviceguard A.11.20 (collectively known as Live Application Detach (LAD)) allow you to do this kind of maintenance while keeping the packages running. The packages are no longer monitored by Serviceguard, but the applications continue to run. Packages in this state are called detached packages. When you have done the necessary maintenance, you can restart the node or cluster, and normal monitoring will resume on the packages.
• You cannot detach a package that is in maintenance mode, and you cannot place a package into maintenance mode if any of its dependent packages are detached. For more information about maintenance mode, see“Maintaining a Package: Maintenance Mode” (page 292). For more information about dependencies, see “About Package Dependencies” (page 142). • You cannot make configuration changes to a cluster in which any packages are detached. cmapplyconf (1m) will fail.
Additional Points To Note Keep the following points in mind. • When packages are detached, they continue to run, but without high availability protection. Serviceguard does not detect failures of components of detached packages, and packages are not failed over. Serviceguard will not provide local LAN failover of IP addresses if the node has been halted in detach mode.
if the IP address is switched to the standby LAN and NETWORK_AUTO_FAILBACK cluster parameter is set to FALSE. If the primary LAN recovers while the node is halted and you want the IP address to fail back to the primary LAN, run cmmodnet –e to re-enable the primary LAN interface and trigger the failback. Halting a Node and Detaching its Packages To halt a node and detach its packages, proceed as follows. 1. Make sure that the conditions spelled out under “Rules and Restrictions” (page 284) are met. 2.
Example: Halting the Cluster for Maintenance on the Heartbeat Subnets Suppose that you need to do networking maintenance that will disrupt all the cluster's heartbeat subnets, but it is essential that the packages continue to run while you do it. In this example we'll assume that packages pkg1 through pkg5 are unsupported for Live Application Detach, and pkg6 through pkgn are supported. Proceed as follows: 1. Halt all the unsupported packages: cmhaltpkg pkg1 pkg2 pkg3 pkg4 pkg5 2.
If this happens, you can repeat the run command, this time including the package(s) this package depends on; Serviceguard will start all the packages in the correct order. Using Serviceguard Commands to Start a Package Use the cmrunpkg command to run the package on a particular node, then use the cmmodpkg command to enable switching for the package. For example, to start a failover package: cmrunpkg -n ftsys9 pkg1 cmmodpkg -e pkg1 This starts up the package on ftsys9, then enables package switching.
You cannot halt a package unless all packages that depend on it are down. If you try, Serviceguard will take no action, except to send a message indicating that not all dependent packages are down. Before you halt a system multi-node package, or halt all instances of a multi-node package, halt any packages that depend on them Handling Failures During Package Halt When you halt a package using cmhaltpkg, sometimes errors may occur for various reasons resulting in the failure of the command.
The following operations cannot be performed on a package which is in the partially_down status: • Reconfigure a package • Run a package • Halt a node (however, you can forcefully halt a node using cmhaltnode -f option.) • Halt a cluster (however, you can forcefully halt a cluster using cmhaltcl -f option.) • Delete a package • Failover of a package automatically. You must halt the package completely and manually failover the package.
You can disable package switching to particular nodes by using the -n option of the cmmodpkg command. The following prevents pkg1 from switching to node lptest3: cmmodpkg -d -n lptest3 pkg1 To permanently disable switching so that the next time the cluster restarts, the change you made in package switching is still in effect, change the auto_run flag in the package configuration file, then re-apply the configuration. (See “Reconfiguring a Package on a Running Cluster ” (page 316).
• • Node-wide and cluster-wide events affect the package as follows: ◦ If the node the package is running on is halted or crashes, the package will no longer be in maintenance mode but will not be automatically started. ◦ If the cluster is halted or crashes, the package will not be in maintenance mode when the cluster comes back up. Serviceguard will attempt to start it if auto_run is set to yes in the package configuration file.
• You cannot configure new dependencies involving this package; that is, you cannot make it dependent on another package, or make another package depend on it. See also “Dependency Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode ” (page 294). • You cannot use the -t option of any command that operates on a package that is in maintenance mode; see “Previewing the Effect of Cluster Changes” (page 297) for information about the -t option.
NOTE: If you now run cmviewcl, you'll see that the STATUS of pkg1 is up and its STATE is maintenance. 3. If everything is working as expected, take the package out of maintenance mode: cmmodpkg -m off pkg1 NOTE: If you have a package configured with generic resources and you attempt to take it out of the maintenance mode back to the running state, the status of generic resources are evaluated. If any of the generic resources is 'down', the package cannot be taken out of the maintenance mode.
8. Restart the package: cmrunpkg pkg1 Excluding Modules in Partial-Startup Maintenance Mode In the example above, we used cmrunpkg -m to run all the modules up to and including package_ip, but none of those after it. But you might want to run the entire package apart from the module whose components you are going to work on. In this case you can use the -e option: cmrunpkg -e sg/service pkg1 This runs all the package's modules except the services module. You can also use -e in combination with -m.
Table 15 Types of Changes to the Cluster Configuration (continued) Change to the Cluster Configuration Required Cluster State Change the Quorum Configuration Online” (page 50), and the NOTE below this table. Add NICs and their IP addresses, if any, to the cluster Cluster can be running. See “Changing the Cluster Networking configuration Configuration while the Cluster Is Running” (page 302). Delete NICs and their IP addresses, if any, from the cluster configuration Cluster can be running.
if a currently disabled package has package switching enabled, or if a package halts and cannot restart because none of the nodes on its node_list is available. Serviceguard provides two ways to do this: you can use the preview mode of Serviceguard commands, or you can use the cmeval (1m) command to simulate different cluster states. Alternatively, you might want to model changes to the cluster as a whole; cmeval allows you to do this; see “Using cmeval” (page 299).
This shows that pkg1, when enabled, will “drag” pkg2 and pkg3 to its primary node, node1. It can do this because of its higher priority; see “Dragging Rules for Simple Dependencies” (page 144). Running the preview confirms that all three packages will successfully start on node2 (assuming conditions do not change between now and when you actually enable pkg1, and there are no failures in the run scripts). NOTE: The preview cannot predict run and halt script failures.
IMPORTANT: For detailed information and examples, see the cmeval (1m) manpage. Updating the Cluster Lock Configuration Use the procedures that follow whenever you need to change the device file names of the cluster lock physical volumes.
Reconfiguring a Running Cluster This section provides instructions for changing the cluster configuration while the cluster is up and running. Note the following restrictions: • You cannot remove an active node from the cluster. You must halt the node first. • You cannot delete an active volume group from the cluster configuration. You must halt any package that uses the volume group and ensure that the volume is inactive before deleting it.
there are packages that depend on the unreachable node, halt the cluster or use Serviceguard commands as described in the next section. Use the following procedure to delete a node with HP-UX commands. In this example, nodes ftsys8, ftsys9 and ftsys10 are already configured in a running cluster named cluster1, and you are deleting node ftsys10. NOTE: If you want to remove a node from the cluster, run the cmapplyconf command from another node in the same cluster.
• Change IP Monitor parameters: SUBNET, IP_MONITOR, POLLING TARGET; see the entries for these parameters under “Cluster Configuration Parameters ” (page 114) for more information. • A combination of any of these in one transaction (cmapplyconf), given the restrictions below. What You Must Keep in Mind The following restrictions apply: • You must not change the configuration of all heartbeats at one time, or change or delete the only configured heartbeat.
Example: Adding a Heartbeat LAN Suppose that a subnet 15.13.170.0 is shared by nodes ftsys9 and ftsys10 in a two-node cluster cluster1, and you want to add it to the cluster configuration as a heartbeat subnet. Proceed as follows. 1. Run cmquerycl to get a cluster configuration template file that includes networking information for interfaces that are available to be added to the cluster configuration: cmquerycl -c cluster1 -C clconfig.ascii NOTE: As of Serviceguard A.11.
3. 4. In the case of a legacy package, add the new networking information to the package control script if necessary Apply the new package configuration, and redistribute the control script if necessary. For more information, see “Reconfiguring a Package on a Running Cluster ” (page 316). Example: Deleting a Subnet Used by a Package In this example, we are deleting subnet 15.13.170.0 (lan0). This will also mean deleting lan3, which is a standby for lan0 and not shared by any other primary LAN.
3. 4. 5. 6. Edit clconfig.ascii and delete the line(s) specifying the NIC name and its IP address(es) (if any) from the configuration. Run cmcheckconf to verify the new configuration. Run cmapplyconf to apply the changes to the configuration and distribute the new configuration file to all the cluster nodes. Runolrad -d to remove the NIC. See also “Replacing LAN or Fibre Channel Cards” (page 333).
Similarly, you can delete VxVM or CVM disk groups provided they are not being used by a cluster node at the time. CAUTION: Serviceguard manages the Veritas processes, specifically gab and LLT. This means that you should never use administration commands such as gabconfig, llthosts, and lltconfig to administer a cluster. It is safe to use the read-only variants of these commands, such as gabconfig -a. But a Veritas administrative command could potentially crash nodes or the entire cluster.
prioritized list of cluster nodes on which the package can run together with definitions of the acceptable types of failover allowed for the package. You can create a legacy package and its control script in Serviceguard Manager; use the Help for detailed instructions. Otherwise, use the following procedure to create a legacy package. 1. Create a subdirectory for each package you are configuring in the /etc/cmcluster directory: mkdir /etc/cmcluster/pkg1 You can use any directory names you like. 2.
NOTE: For modular packages, the default form for parameter names in the package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameter names are concerned. Because this section is intended to be used primarily when you are reconfiguring an existing legacy package, we are using the legacy parameter names (in upper case) for sake of continuity.
• To configure monitoring for a registered resource, enter values for the following parameters. ◦ RESOURCE_NAME ◦ RESOURCE_POLLING_INTERVAL ◦ RESOURCE_UP_VALUE ◦ RESOURCE_START For more information, see “Parameters for Configuring EMS Resources” (page 142), and the resource_ parameter descriptions (page 250). NOTE: script. • For legacy packages, DEFERRED resources must be specified in the package control ACCESS_CONTROL_POLICY.
Customizing the Package Control Script You need to customize as follows. See the entries for the corresponding modular-package parameters under “Package Parameter Explanations” (page 237) for more discussion. • Update the PATH statement to reflect any required paths needed to start your services.
Adding Customer Defined Functions to the Package Control Script You can add additional shell commands to the package control script to be executed whenever the package starts or stops. Enter these commands in the CUSTOMER DEFINED FUNCTIONS area of the script. If your package needs to run short-lived processes, such as commands to initialize or halt a packaged application, you can also run these from the CUSTOMER DEFINED FUNCTIONS.
Verifying the Package Configuration Serviceguard checks the configuration you create and reports any errors. For legacy packages, you can do this in Serviceguard Manager: click Check to verify the package configuration you have done under any package configuration tab, or to check changes you have made to the control script. Click Apply to verify the package as a whole. See the local Help for more details.
Distributing the Binary Cluster Configuration File with HP-UX Commands Use the following steps from the node on which you created the cluster and package configuration files: • Verify that the configuration file is correct. Use the following command (all on one line): cmcheckconf -C /etc/cmcluster/cmcl.conf -P /etc/cmcluster/pkg1/pkg1.
Configuring monitored_subnet_access In order to monitor subnet 15.244.65.0 or 15.244.56.0, you would configure monitored_subnet and monitored_subnet_access in pkg1’s package configuration file as follows: monitored_subnet 15.244.65.0 monitored_subnet_access PARTIAL monitored_subnet 15.244.56.
that can be made online to the modular CFS package parameters, see “Online reconfiguration of modular CFS package parameters” (page 215). CAUTION: is running. Be extremely cautious about changing a package's configuration while the package If you reconfigure a package online (by executing cmapplyconf on a package while the package itself is running) it is possible that the package will fail, even if the cmapplyconf succeeds, validating the changes with no errors.
1. Halt the package if necessary: cmhaltpkg pkg1 CAUTION: Make sure you read and understand the information and caveats under“Allowable Package States During Reconfiguration ” (page 319) before you decide to reconfigure a running package. 2. If it is not already available, obtain a copy of the package's configuration file by using the cmgetconf command, specifying the package name. cmgetconf -p pkg1 pkg1.conf 3. Edit the package configuration file.
Adding a Package to a Running Cluster You can create a new package and add it to the cluster configuration while the cluster is up and while other packages are running. The number of packages you can add is subject to the value of MAX_CONFIGURED_PACKAGES in the cluster configuration file. To create the package, follow the steps in the chapter “Configuring Packages and Their Services ” (page 232).
1. Remove any dependencies on the package being deleted. Delete dependency_ parameters from the failover application package configuration file, then apply the modified configuration file: cmapplyconf -v -P app1.conf 2. Unmount the shared file system cfsumount 3. Remove the mount point package from the cluster cfsmntadm delete This disassociates the mount point from the cluster.
which the results might not be what you expect — as well as differences between modular and legacy packages. CAUTION: is running. Be extremely cautious about changing a package's configuration while the package If you reconfigure a package online (by executing cmapplyconf on a package while the package itself is running) it is possible that the package will fail, even if the cmapplyconf succeeds, validating the changes with no errors.
Table 16 Types of Changes to Packages (continued) Change to the Package Required Package State Change run script contents: legacy Package can be running, but should not be starting. package Timing problems may occur if the script is changed while the package is starting. Change halt script contents: legacy Package can be running, but should not be halting. package Timing problems may occur if the script is changed while the package is halting.
Table 16 Types of Changes to Packages (continued) Change to the Package Required Package State incrementally; resources that are very slow to come up may need to be configured offline. Serviceguard treats a change to resource_name as deleting the existing resource and adding a new one, meaning that the existing resource is stopped. Change Package can be running. resource_polling_interval, Serviceguard will not allow a change to resource_up_value if it would cause resource_start, the package to fail.
Table 16 Types of Changes to Packages (continued) Change to the Package Required Package State Remove a file system: legacy package Package must not be running. Change Package can be running. concurrent_fsck_operations, These changes in themselves will not cause any file system to be unmounted. concurrent_mount_and_umount_operations, fs_mount_retry_count, fs_umount_retry_count: modular package Change Package must not be running.
Table 16 Types of Changes to Packages (continued) Change to the Package Required Package State Add or delete a configured dependency Both packages can be either running or halted. Special rules apply to packages in maintenance mode; see “Dependency Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode ” (page 294). For dependency purposes, a package being reconfigured is considered to be UP.
NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information on CVM and CFS support: http://www.hp.com/go/hpux-serviceguard-docs. Changes that Will Trigger Warnings Changes to the following will trigger warnings, giving you a chance to cancel, if the change would cause the package to fail. NOTE: You will not be able to cancel if you use cmapplyconf -f.
Instead of restarting the cluster, choose an appropriate time to shut down the applications and reboot the node; this will allow Serviceguard to restart the cluster after the reboot. Disabling Serviceguard If for some reason you want to disable Serviceguard on a system, you can do so by commenting out the following entries in /etc/inetd.conf: hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c Then force inetd to re-read inetd.
8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
cmsetresource -r –s down 3. To view the package status, enter cmviewcl -v The package should be running on the specified adoptive node. 4. Move the package back to the primary node (see “Moving a Failover Package ” (page 291)). NOTE: If there was a monitoring script configured for this generic resource, then the monitoring script would also be attempting to set the status of the generic resource.
• NICs • Power sources • All cables • Disk interface cards Some monitoring can be done through simple physical inspection, but for the most comprehensive monitoring, you should examine the system log file (/var/adm/syslog/syslog.log) periodically for reports on all configured HA devices. The presence of errors relating to a device will show the need for maintenance. When the proper redundancy has been configured, failures can occur with no external symptoms. Proper monitoring is important.
Using HP ISEE (HP Instant Support Enterprise Edition) In addition to messages reporting actual device failure, the logs may accumulate messages of lesser severity which, over time, can indicate that a failure may happen soon. One product that provides a degree of automation in monitoring is called HP ISEE, which gathers information from the status queues of a monitored system to see what errors are accumulating.
5. On the node from which you issued the lvreduce command, issue the following command to restore the volume group configuration data to the newly inserted disk: vgcfgrestore -n /dev/vg_sg01 /dev/dsk/c2t3d0 6. Issue the following command to extend the logical volume to the newly inserted disk: lvextend -m 1 /dev/vg_sg01 /dev/dsk/c2t3d0 7. Finally, use the lvsync command for each logical volume that has extents on the failed physical volume.
IMPORTANT: If you need to replace a LUN under the HP-UX 11i v3 agile addressing scheme, also used by cDSFs (see “About Device File Names (Device Special Files)” (page 81) and “About Cluster-wide Device Special Files (cDSFs)” (page 109), and you use the same DSF, you may need to use the io_redirect_dsf(1M) command to reassign the existing DSF to the new device, depending on whether the operation changes the WWID of the LUN; see the section on io_redirect_dsf in the white paper The Next Generation Mass Storag
Normally disconnecting any portion of the SCSI bus will leave the SCSI bus in an unterminated state, which will cause I/O errors for other nodes connected to that SCSI bus, so the cluster would need to be halted before disconnecting any portion of the SCSI bus. However, it is not necessary to bring the cluster down to do this if you are using a SCSI configuration that allows disconnection of a portion of the SCSI bus without losing termination.
Offline Replacement Follow these steps to replace an I/O card off-line. 1. 2. 3. 4. 5. 6. Halt the node by using the cmhaltnode command. Shut down the system using /usr/sbin/shutdown, then power down the system. Remove the defective I/O card. Install the new I/O card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed. Power up the system. If necessary, add the node back into the cluster by using the cmrunnode command.
3. 4. Install and configure the Quorum Server software on the new system. Be sure to include in the new QS authorization file (/etc/cmcluster/qs_authfile) all of the nodes that were configured for the old quorum server. Refer to the qs(1) manpage for details about configuring the QS authorization file. Start the Quorum Server as follows: • 5. 6. Edit the /etc/inittab file to add the Quorum Server entries. • Use the init q command to run the Quorum Server. See the qs(1) manpage for more details.
NOTE: HP recommends you use Serviceguard Manager as a convenient way to observe the status of a cluster and the properties of cluster objects: from the System Management Homepage (SMH), select the cluster you need to troubleshoot. Reviewing Package IP Addresses The netstat -in command can be used to examine the LAN configuration. The command, if executed on ftsys9 after ftsys10 has been halted, shows that the package IP addresses are assigned to lan0 on ftsys9 along with the primary LANIP address.
NO_RESTART. Dec 14 14:34:45 star04 cmcld[2048]: Examine the file /etc/cmcluster/pkg5/pkg5_run.log for more details. The following is an example of a successful package starting: Dec Dec Dec Dec Dec Dec 14 14:39:27 star04 CM-CMD[2096]: cmruncl 14 14:39:27 star04 cmcld[2098]: Starting cluster management protocols.
Using the cmcheckconf Command In addition, cmcheckconf can be used to troubleshoot your cluster just as it was used to verify the configuration. The following example shows the commands used to verify the existing cluster configuration on ftsys9 and ftsys10: cmquerycl -v -C /etc/cmcluster/verify.ascii -n ftsys9 -n ftsys10 cmcheckconf -v -C /etc/cmcluster/verify.ascii The cmcheckconf command checks: • The network addresses and connections. • The cluster lock disk connectivity.
• Package control script hangs • Problems with VxVM disk groups • Package movement errors • Node and network failures • Quorum Server problems The first two categories of problems occur with the incorrect configuration of Serviceguard. The last category contains “normal” failures to which Serviceguard is designed to react and ensure the availability of your applications.
Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low If you have set the MEMBER_TIMEOUT parameter too low, the cluster demon, cmcld, will write warnings to syslog that indicate the problem. There are three in particular that you should watch for: 1. Warning: cmcld was unable to run for the last seconds. Consult the Managing Serviceguard manual for guidance on setting MEMBER_TIMEOUT, and information on cmcld.
You can use the following commands to check the status of your disks: • bdf - to see if your package's volume group is mounted. • vgdisplay -v - to see if all volumes are present. • lvdisplay -v - to see if the mirrors are synchronized. • strings /etc/lvmtab - to ensure that the configuration is correct. • ioscan -fnC disk - to see physical disks. • diskinfo -v /dev/rdsk/cxtydz - to display information about a disk. • lssf /dev/d*/* - to check logical volumes and paths.
cmmodnet -r -i where is the address in the Address or the Address column and is the corresponding entry in the Network column for IPv4, or the prefix (which can be derived from the IPV6 address) for IPv6. 3. Ensure that package volume groups are deactivated. First unmount any package logical volumes which are being used for file systems. This is determined by inspecting the output resulting from running the command bdf -l.
Problems with Cluster File System (CFS) NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information about support for CFS (http://www.hp.com/go/hpux-serviceguard-docs). If you have a system multi-node package for Veritas CFS, you may not be able to start the cluster until SG-CFS-pkg starts. Check SG-CFS-pkg.log for errors.
CAUTION: This force import procedure should only be used when you are certain the disk is not currently being accessed by another node. If you force import a disk that is already being accessed on another node, data corruption can result. Package Movement Errors These errors are similar to the system administration errors, but they are caused specifically by errors in the control script for legacy packages.
The reason may be that you have not updated the authorization file. Verify that the node is included in the file, and try using /usr/lbin/qs -update to re-read the Quorum Server authorization file. Timeout Problems The following kinds of message in a Serviceguard node’s syslog file may indicate timeout problems: Unable to set client version at quorum server 192.6.7.2: reply timed out Probe of quorum server 192.6.7.
A Enterprise Cluster Master Toolkit The Enterprise Cluster Master Toolkit (ECMT) provides a group of example scripts and package configuration files for creating Serviceguard packages for several major database and internet software products. Each toolkit contains a README file that explains how to customize the package for your needs. The ECMT can be installed on HP-UX 11i v2, or 11i v3.
B Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 348) • Designing Applications to Run on Multiple Systems (page 351) • Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY (page 355) • Restoring Client Connections (page 356) • Handling Applicatio
Insulate Users from Outages Wherever possible, insulate your end users from outages. Issues include the following: • Do not require user intervention to reconnect when a connection is lost due to a failed server. • Where possible, warn users of slight delays due to a failover in progress. • Minimize the reentry of data. • Engineer the system for reserve capacity to minimize the performance degradation experienced by users.
Replicate Non-Data File Systems Non-data file systems should be replicated rather than shared. There can only be one copy of the application data itself. It will be located on a set of disks that is accessed by the system that is running the application. After failover, if these data disks are file systems, they must go through file systems recovery (fsck) before the data can be accessed. To help reduce this recovery time, the smaller these file systems are, the faster the recovery will be.
Use Restartable Transactions Transactions need to be restartable so that the client does not need to re-enter or back out of the transaction when a server fails, and the application is restarted on another system. In other words, if a failure occurs in the middle of a transaction, there should be no need to start over again from the beginning. This capability makes the application more robust and reduces the visibility of a failover to the user. A common example is a print job.
Another possibility is having two applications that are both active. An example might be two application servers which feed a database. Half of the clients connect to one application server and half of the clients connect to the second application server. If one server fails, then all the clients connect to the remaining application server. Design for Replicated Data Sites Replicated data sites are a benefit for both fast failover and disaster recovery.
network interface must have a stationary IP address associated with it. This IP address does not move to a remote system in the event of a network failure. Obtain Enough IP Addresses Each application receives a relocatable IP address that is separate from the stationary IP address assigned to the system itself. Therefore, a single system might have many IP addresses, one for itself and one for each of the applications that it normally runs.
One way to look for problems in this area is to look for calls to gethostname(2) in the application. HA services should use gethostname() with caution, since the response may change over time if the application migrates. Applications that use gethostname() to determine the name for a call to gethostbyname(2) should also be avoided for the same reason. Also, the gethostbyaddr() call may return different answers over time if called with a stationary IP address.
With UDP datagram sockets, however, there is a problem. The client may connect to multiple servers utilizing the relocatable IP address and sort out the replies based on the source IP address in the server’s response message. However, the source IP address given in this response will be the stationary IP address rather than the relocatable application IP address.
Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY CAUTION: The procedure in this section depends on setting the HP-UX kernel parameter ip_strong_es_model. HP supports setting this parameter for use with Serviceguard only if you are not using a cross-subnet configuration (page 29). Otherwise, leave the parameter at its default setting (zero, meaning disabled) and do not attempt to use the procedure that follows.
/usr/sbin/route delete net default 128.17.17.1 1 source 128.17.17.17 Once the per-interface default route(s) have been added, netstat –rn would show something like the following, where 128.17.17.17 is the package relocatable address and 128.17.17.19 is the physical address on the same subnet: Destination 127.0.0.1 128.17.17.19 128.17.17.17 192.168.69.82 128.17.17.0 128.17.17.0 192.168.69.0 127.0.0.0 default default Gateway 127.0.0.1 128.17.17.19 128.17.17.17 192.168.69.82 128.17.17.19 128.17.17.17 192.168.
application. If the application can be restarted on the same node after a failure (see “Handling Application Failures ” following), the retry to the current server should continue for the amount of time it takes to restart the server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness.
Minimizing Planned Downtime Planned downtime (as opposed to unplanned downtime) is scheduled; examples include backups, systems upgrades to new operating system revisions, or hardware replacements. For planned downtime, application designers should consider: • Reducing the time needed for application upgrades/patches.
Providing Online Application Reconfiguration Most applications have some sort of configuration information that is read when the application is started. If to make a change to the configuration, the application must be halted and a new configuration file read, downtime is incurred. To avoid this downtime use configuration tools that interact with an application and make dynamic changes online. The ideal solution is to have a configuration tool which interacts with the application.
C Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the Appendix “Designing Highly Available Cluster Applications.” 2.
1. 2. 3. 4. 5. Install the application, database, and other required resources on one of the systems. Be sure to follow Serviceguard rules in doing this: • Install all shared data on separate external volume groups. • Use JFS/VxFS file systems as appropriate. Perform some sort of standard test to ensure the application is running correctly. This test can be used later in testing with Serviceguard. If possible, try to connect to the application through a client.
Testing the Cluster 1. Test the cluster: • Have clients connect. • Provide a normal system load. • Halt the package on the first node and move it to the second node: cmhaltpkg pkg1 cmrunpkg -n node2 pkg1 cmmodpkg -e pkg1 • Move it back. cmhaltpkg pkg1 cmrunpkg -n node1 pkg1 cmmodpkg -e pkg1 2. 3. • Fail one of the systems. For example, turn off the power on node 1. Make sure the package starts up on node 2. • Repeat failover from node2 back to node1.
D Software Upgrades There are five types of upgrade you can do: • rolling upgrade • rolling upgrade using DRD • non-rolling upgrade • non-rolling upgrade using DRD • migration with cold install Each of these is discussed below. Special Considerations for Upgrade to Serviceguard A.11.20 • Before you proceed, read the sections Upgrading from an Earlier Serviceguard Release and Rolling Upgrade in the latest version of the release notes for A.11.20 • Serviceguard A.11.
CAUTION: From the time when the old cluster manager is shut down until the new cluster manager forms its first cluster, a node failure will cause the entire cluster to fail. HP strongly recommends that you use no Serviceguard commands other than cmviewcl (1m) until the new cluster manager successfully completes its first cluster re-formation.
Restrictions for DRD Upgrades • Upgrade using DRD is supported only on HP-UX 11i v3. • You must use the DRD software released with the September 2009 release of HP-UX 11i v3 or later. You can obtain the DRD software free from HP (see “Rolling Upgrade Using DRD” ); you do not have to upgrade the operating system itself, so long as you are running 11i v3. • The cluster must meet both general and release-specific requirements for a rolling upgrade; see “Guidelines for Rolling Upgrade” (page 365).
IMPORTANT: Do not proceed without reading and understanding the following points: • The above are general guidelines. The requirements for any particular Serviceguard release may be more restrictive. Rolling upgrade is supported only for the HP-UX/Serviceguard combinations listed in the Release Notes for the target version of Serviceguard.
Before You Start Make sure you plan sufficient system capacity to allow moving the packages from node to node during the process without an unacceptable loss of performance. CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 363). Running the Rolling Upgrade 1. 2. Halt the node you want to upgrade. You can do this in Serviceguard Manager, or use the cmhaltnode command.
Performing a Rolling Upgrade Using DRD IMPORTANT: All the limitations listed under “Guidelines for Rolling Upgrade” (page 365) and “Limitations of Rolling Upgrades ” (page 366) also apply to a rolling upgrade with DRD. You should read the entire section on “Performing a Rolling Upgrade” (page 366) before you proceed. Before You Start CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 363).
4. Restart the cluster on the node, using Serviceguard Manager or cmrunnode (1m). • If the applications do not function properly and this is the last node to be upgraded, you cannot revert to the previous release on just this node. You must either solve the problems with this release on this node, or revert the entire cluster to the previous release by halting the cluster (cmhaltcl), rebooting each node from its original (pre-upgrade) root disk, and restarting the cluster (cmruncl).
Figure 42 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 to the next operating system release (“HP-UX (new)”), and install the next version of Serviceguard (“SG (new)”). Figure 43 Node 1 Upgraded to new HP-UX version Step 3. When upgrading is finished, enter the following command on node 1 to restart the cluster on node 1. # cmrunnode -n node1 At this point, different versions of the Serviceguard daemon (cmcld) are running on the two nodes, as shown in Figure 44.
Figure 44 Node 1 Rejoining the Cluster Step 4. Repeat the process on node 2. Halt the node, as follows: # cmhaltnode -f node2 This causes both packages to move to node 1. Then upgrade node 2 to the new versions of HP-UX and Serviceguard. Figure 45 Running Cluster with Packages Moved to Node 1 Step 5. Move pkg2 back to its original node.
Figure 46 Running Cluster After Upgrades Guidelines for Non-Rolling Upgrade Do a non-rolling upgrade if: • Your cluster does not meet the requirements for rolling upgrade as specified in the Release Notes for the target version of Serviceguard; or • The limitations imposed by rolling upgrades make it impractical for you to do a rolling upgrade (see “Limitations of Rolling Upgrades ” (page 366)); or • For some other reason you need or prefer to bring the cluster down before performing the upgrade.
1. Halt all nodes in the cluster: cmhaltcl -f 2. If necessary, upgrade all the nodes in the cluster to the new HP-UX release. (See Step 3 under “Running the Rolling Upgrade” (page 367) for more information.) Upgrade all the nodes in the cluster to the new Serviceguard release. (See Step 3 under “Running the Rolling Upgrade” (page 367) for more information.) Restart the cluster: cmruncl 3. 4.
Guidelines for Migrating a Cluster with Cold Install There may be circumstances when you prefer to do a cold install of the HP-UX operating system rather than an upgrade. A cold install erases the existing operating system and data and then installs the new operating system and software; you must then restore the data. CAUTION: The cold install process erases the existing software, operating system, and data. If you want to retain any existing software, make sure you back up that software before migrating.
E Blank Planning Worksheets This appendix contains blank versions of the planning worksheets mentioned in“Planning and Documenting an HA Cluster ” (page 97). You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
=============================================================================== Tape Backup Power: Tape Unit __________________________ Power Supply _______________________ Tape Unit __________________________ Power Supply _______________________ =============================================================================== Other Power: Unit Name __________________________ Power Supply _______________________ Unit Name __________________________ Power Supply _______________________ .
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ VxVM Disk Group and Disk Worksheet DISK GROUP WORKSHEET Page ___ of ____ ==========================================================
Heartbeat Subnet: _________________________________ Monitored Non-heartbeat Subnet: ___________________ Monitored Non-heartbeat Subnet: ___________________ =====================================----====================================== Cluster Lock: Volume Groups, LUN, or Quorum Server =============================================================================== Quorum Server: QS_HOST___________________________QS_ADDR_________________ QS_POLLING_INTERVAL________________________________ QS_TIMEOUT_EXTENSIO
fs_umount_opt_____________ fs_fsck_opt_________________fs_type_________________ fs_mount_retry_count: ____________fs_umount_retry_count:___________________ Concurrent vgchange operations: ___________________________________________ Concurrent mount/umount operations: ______________________________________ Concurrent fsck operations: ______________________________________________ Kill processes accessing raw devices?_____ =============================================================================== Network
F Migrating from LVM to VxVM Data Storage This appendix describes how to migrate LVM volume groups to VxVM disk groups for use with the Veritas Volume Manager (VxVM), or with the Cluster Volume Manager (CVM) on systems that support it.
separation.) The same mirroring pattern should be followed in creating the VxVM plexes, with different plexes configured on disks that are attached to different buses. As an alternative to defining the VxVM disk groups on a new set of disks, it is possible to convert existing LVM volume groups into VxVM disk groups in line using the vxvmconvert(1M) utility.
4. 5. 6. 7. Be sure to copy from the old script any user-specific code that may have been added, including environment variables and customer defined functions. Distribute the new package control scripts to all nodes in the cluster. Test to make sure the disk group and data are intact. Deport the disk group: vxdg deport DiskGroupName 8. Make the disk group visible to the other nodes in the cluster by issuing the following command on all other nodes: vxdctl enable 9. Restart the package.
G Migrating from Legacy CFS Packages to Modular CFS Packages To migrate from Legacy CFS packages to Modular CFS packages for use with the CVM or CFS on systems that support it, follow these steps: 1. Halt the application package: cmhaltpkg 2. Unconfigure the CFS legacy disk group and the mount point package: cfsumount cfsdgadm deactivate cfsmntadm delete cfsdgadm delete 3.
H IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses. Topics: • IPv6 Address Types • Network Configuration Restrictions • Local Primary/Standby LAN Patterns • IPv6 Relocatable Address and Duplicate Address Detection Feature (page 387) IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces.
IPv6 Address Prefix IPv6 Address Prefix is similar to CIDR in IPv4 and is written in CIDR notation. An IPv6 address prefix is represented by the notation: IPv6-address/prefix-length where ipv6-address is an IPv6 address in any notation listed above and prefix-length is a decimal value representing how many of the leftmost contiguous bits of the address comprise the prefix. Example: fec0:0:0:1::1234/64 The first 64-bits of the address fec0:0:0:1 forms the address prefix.
::ffff:192.168.0.1 Aggregatable Global Unicast Addresses The global unicast addresses are globally unique IPv6 addresses. This address format is very well defined in the RFC 2374 (An IPv6 Aggregatable Global Unicast Address Format). The format is: 3 13 8 24 16 64 bits FP TLA ID RES NLA ID SLA ID Interface ID where: FP = Format prefix. Value of this is “001” for Aggregatable Global unicast addresses. TLA ID = Top-level Aggregation Identifier. RES = Reserved for future use.
All Node Addresses = FF02:0:0:0:0:0:0:1 (link-local) All Router Addresses = FF02:0:0:0:0:0:0:2 (link-local) All Router Addresses = FF05:0:0:0:0:0:0:2 (site-local) Network Configuration Restrictions Serviceguard supports IPv6 for data and heartbeat IP. To configure IPv6, the system should be set up in what is called a dual-stack configuration, which requires the IPv6 product bundle. The restrictions for supporting IPv6 in Serviceguard are listed below.
ndd -get /dev/ip6 ip6_nd_dad_solicit_count If the result is 1, the feature is turned on. If the result is 0, the feature is turned off. To temporarily change the state of DAD on your computer, use the ndd -set command to change the kernel parameter. ndd -set /dev/ip6 ip6_nd_dad_solicit_countn where n is a number: either 1 to turn the feature on, or 0 to turn it off.
Following the loss of lan0 or lan2, lan1 can adopt either address, as shown below. The same NIC can be configured with both IPv4 and IPv6 addresses, as shown in below. This type of configuration allows failover of both addresses to the standby. This is shown in below.
IPv6 Network Support
I Using Serviceguard Manager HP Serviceguard Manager is a web-based, HP System Management Homepage (HP SMH) tool, that replaces the functionality of the earlier Serviceguard management tools. HP Serviceguard Manager allows you to monitor, administer and configure a Serviceguard cluster from any system with a supported web browser. Serviceguard Manager does not require additional software installation.
Table 18 Accessing Serviceguard Manager Scenario Scenario Use Case Number of Clusters Serviceguard Version Serviceguard Manager Version 1 Single cluster management 1 A.11.20 B.03.10 2 Multi-cluster management more than 1 cluster B.03.00 A.11.19 B.02.00 A.11.18 B.01.01 A.11.17.01 B.01.00 A.11.17.01 and later HP SIM or multiple browser sessions of B.02.00 or later Launching Serviceguard Manager This section provides information about three common scenarios.
NOTE: If a cluster is not yet configured, then you will not see the Serviceguard Cluster section on this screen. To create a cluster, from the SMH Tools menu, you must click Serviceguard Manager link in the Serviceguard box first, then click Create Cluster. The figure below shows a browser session at the HP Serviceguard Manager Main Page.
NOTE: Serviceguard Manager can be launched by HP Systems Insight Manager version 5.10 or later if Serviceguard Manager is installed on an HP Systems Insight Manager Central Management Server. For a Serviceguard A.11.17 cluster or earlier, Serviceguard Manager A.05.01 will be launched via Java Web Start. You must ensure that the hostname for each Serviceguard node can be resolved by DNS. For more information about this older version of Serviceguard Manager, see the Serviceguard Manager Version A.05.
J Maximum and Minimum Values for Parameters Table 19 shows the range of possible values for cluster configuration parameters. Table 19 Minimum and Maximum Values of Cluster Configuration Parameters Cluster Parameter Minimum Value Maximum Value Default Value Member Timeout See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4. See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4.
K Monitoring Script for Generic Resources The monitoring scripts are the scripts written by an end-user and must contain the core logic to monitor a resource and set the status of a generic resource. These scripts are started as a part of the package start. • You can set the status/value of a simple/extended resource respectively using the cmsetresource(1m) command. • You can define the monitoring interval in the script.
• It is not mandatory to have the same name for a generic resource and its monitoring script, i.e., service_name and generic_resource_name. However, it is good practice to have the same name, so that it is easier to identify the monitor. • A common resource specified across multiple packages can be monitored using one monitoring script. For resources of evaluation_type: before_package_start • Monitoring scripts can also be launched outside of the Serviceguard environment, init, rc scripts, etc.
Similarly, consider another package pkg2 that requires the 'CPU' to be configured as before_package_start.
# * "sg/generic_resource" modules need to be specified in the package * # * configuraton file in order to configure these parameters. * # * * # * * # * --------------------------------* # * U T I L I T Y F U N C T I O N S * # * --------------------------------* # * The following utility functions are sourced in from $SG_UTILS * # * ($SGCONF/scripts/mscripts/utils.
# # ######################################################################### function stop_command { sg_log 5 "stop_command" # ADD your halt steps here exit 1 } ################ # main routine ################ sg_log 5 "customer defined monitor script" ######################################################################### # # Customer defined monitor script should be doing following # functionality.
Sample script that demonstrates how to monitor disk space of a file system using extended generic resources can be found at: /etc/cmcluster/examples/sample_generic_resource_disk_space_monitor.sh Before using this script, copy it to the package specific directory. This script expects the following parameters when configured as a service: • $1 : Generic resource name: The name of the generic resource configured in the package that corresponds to this monitoring script.
L Migrating EMS Resources to Generic Resources To migrate from EMS resource to generic resource for monitoring resources, follow these steps: 1. Identify the EMS resource that is configured, resource_start, and polling interval. 2. If this is a standard EMS resource then identify the equivalent SFM style resource monitor.
19. Validate the changes done by running: cmcheckconf -v -P 20. Edit the package configuration file to add the generic resource parameters. 21. Validate the changes done by running: cmcheckconf -v -P 22. Apply the changes by running: cmapplyconf -v -P 23. Start the package.
Index A Access Control Policies, 197 Access Control Policy, 130 Access roles, 130 active node, 22 adding a package to a running cluster, 318 adding cluster nodes advance planning, 162 adding nodes to a running cluster, 282 adding packages on a running cluster , 264 additional package resources monitoring, 60 addressing, SCSI, 100 administration adding nodes to a ruuning cluster, 282 cluster and package states, 266 halting a package, 289 halting the entire cluster, 283 moving a package, 291 of packages and s
understanding components, 27 cluster administration, 281 solving problems, 338 cluster and package maintenance , 265 cluster configuration creating with SAM or Commands, 190 file on all nodes, 45 identifying cluster lock volume group, 193 identifying cluster-aware volume groups, 196 planning, 108 planning worksheet, 131 sample diagram, 98 verifying the cluster configuration, 202 cluster configuration file Autostart Delay parameter (AUTO_START_TIMEOUT), 126 cluster coordinator defined, 44 cluster lock, 47 4
pathname parameter in package configuration, 257 support for additional productss, 312 troubleshooting, 337 controlling the speed of application failover, 348 creating the package configuration, 307 Critical Resource Analysis (CRA) LAN or VLAN, 305 customer defined functions adding to the control script, 312 CVM, 86 creating a storage infrastructure, 224 not supported on all HP-UX versions, 23 planning, 107 CVM planning Version 4.1 with CFS, 132 Version 4.
F failback policy used by package manager, 56 FAILBACK_POLICY parameter used by package manager, 56 failover controlling the speed in applications, 348 defined, 22 failover behavior in packages, 137 failover package, 51 failover policy used by package manager, 54 FAILOVER_POLICY parameter used by package manager, 54 failure kinds of responses, 93 network communication, 96 package, service, node, 93 response to hardware failures, 94 responses to package and service failures, 95 restarting a service after fai
hardware planning, 99, 104, 105 host name hardware planning, 99 HOSTNAME_ADDRESS_FAMILY defined, 115 discussion and restrictions, 111 how the cluster manager works, 44 how the network manager works, 68 HP Predictive monitoring in troubleshooting, 330 kernel hang, and TOC, 93 safety timer, 41 kernel consistency in cluster configuration, 175 kernel interrupts and possible TOC, 125 interface name, 99, 104 planning information, 99 LAN CRA (Critical Resource Analysis), 305 LAN failure Serviceguard behavior, 27
MAX_CONFIGURED_PACKAGES defined, 130 maximum number of nodes, 27 MEMBER_TIMEOUT and cluster re-formation, 93 and safety timer, 41 configuring, 125 defined, 124 maximum and minimum values , 124 modifying, 197 membership change reasons for, 46 memory capacity hardware planning, 99 memory requirements lockable memory for Serviceguard, 97 Migrating EMS Resources to Generic Resources, 402 minimizing planned down time, 358 mirror copies of data protection against disk failure, 22 MirrorDisk/UX, 33 mirrored disks
O olrad command removing a LAN or VLAN interface, 305 online hardware maintenance by means of in-line SCSI terminators, 332 OTS/9000 support, 395 outages insulating users from, 348 P package adding and deleting package IP addresses, 69 base modules, 235 basic concepts, 27 changes while the cluster is running, 320 configuring legacy, 307 error handling, 290 failure, 93 halting, 289 legacy, 307 local interface switching, 71 modular, 234 modular and legacy, 232 modules, 234 moving, 291 optional modules, 236 p
planning for cluster expansion, 97 planning worksheets blanks, 375 point of failure in networking, 29 point to point connections to storage devices, 38 POLLING_TARGET defined, 129 ports dual and single aggregated, 79 power planning power sources, 102 worksheet, 103 power supplies blank planning worksheet, 375 power supply and cluster lock, 37 blank planning worksheet, 375 UPS for OPS on HP-UX, 37 PR keys Distribution, 92 Predictive monitoring, 330 primary LAN interfaces defined, 28 primary network interface
S safety timer and node TOC, 41 and syslog.
and MEMBER_TIMEOUT, 93 and package availability, 94 and safety timer, 125 and the safety timer, 41 defined, 41 when a node fails, 93 toolkits for databases, 346 traffic type LAN hardware planning, 100 troubleshooting approaches, 335 monitoring hardware, 328 replacing disks, 330 reviewing control scripts, 337 reviewing package IP addresses, 336 reviewing system log file, 336 using cmquerycl and cmcheckconf, 338 troubleshooting your cluster, 327 typical cluster after failover figure, 23 typical cluster config
