Managing Serviceguard Eighteenth Edition HP Part Number: B3936-90147 Published: September 2010
Legal Notices © Copyright 1995-2010 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents Printing History ...........................................................................................................................25 Preface.......................................................................................................................................27 1 Serviceguard at a Glance.........................................................................................................29 What is Serviceguard? .................................................
Redundant Power Supplies ...............................................................................................49 Larger Clusters ...................................................................................................................50 Active/Standby Model ..................................................................................................50 Point to Point Connections to Storage Devices ............................................................
Deciding When and Where to Run and Halt Failover Packages .......................69 Failover Packages’ Switching Behavior..............................................................70 Failover Policy....................................................................................................72 Automatic Rotating Standby..............................................................................73 Failback Policy......................................................................................
Additional Heartbeat Requirements......................................................................106 Volume Managers for Data Storage..................................................................................106 Types of Redundant Storage.......................................................................................106 About Device File Names (Device Special Files).........................................................106 Examples of Mirrored Storage...................................
LVM Planning ..................................................................................................................132 Using EMS to Monitor Volume Groups......................................................................132 LVM Worksheet ..........................................................................................................133 CVM and VxVM Planning ...............................................................................................134 CVM and VxVM Worksheet .....
Rules for different_node and any_node Dependencies...................................186 What Happens when a Package Fails....................................................................186 For More Information.............................................................................................187 About Package Weights...............................................................................................187 Package Weights and Node Capacities..............................................
Using Easy Deployment Commands to Configure the Cluster.............................211 Configuring Root-Level Access...................................................................................216 Allowing Root Access to an Unconfigured Node..................................................216 Ensuring that the Root User on Another Node Is Recognized..............................217 About identd.....................................................................................................
Specifying the Address Family for the Cluster Hostnames...................................244 Specifying the Address Family for the Heartbeat .................................................244 Specifying the Cluster Lock...................................................................................245 Generating a Network Template File.....................................................................245 Full Network Probing............................................................................
Checking Cluster Operation with Serviceguard Commands.....................................273 Preventing Automatic Activation of LVM Volume Groups .......................................274 Setting up Autostart Features .....................................................................................274 Changing the System Message ...................................................................................275 Managing a Single-Node Cluster..........................................................
ip_subnet_node ..................................................................................................299 ip_address...........................................................................................................299 service_name.......................................................................................................299 service_cmd.........................................................................................................300 service_restart....................
Next Step.....................................................................................................................313 Editing the Configuration File..........................................................................................313 Verifying and Applying the Package Configuration........................................................317 Adding the Package to the Cluster...................................................................................
Removing Nodes from Participation in a Running Cluster........................................343 Halting the Entire Cluster ...........................................................................................344 Automatically Restarting the Cluster .........................................................................344 Halting a Node or the Cluster while Keeping Packages Running..............................344 What You Can Do.....................................................................
Reconfiguring a Running Cluster................................................................................365 Adding Nodes to the Cluster While the Cluster is Running ................................365 Removing Nodes from the Cluster while the Cluster Is Running ........................366 Changing the Cluster Networking Configuration while the Cluster Is Running..................................................................................................................367 What You Can Do........
Responding to Cluster Events ..........................................................................................397 Single-Node Operation ....................................................................................................397 Disabling Serviceguard.....................................................................................................398 Removing Serviceguard from a System...........................................................................
Networking and Security Configuration Errors.........................................................414 Cluster Re-formations Caused by Temporary Conditions..........................................414 Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low.............415 System Administration Errors ....................................................................................416 Package Control Script Hangs or Failures ............................................................
Bind to Relocatable IP Addresses ...............................................................................433 Call bind() before connect() ...................................................................................434 Give Each Application its Own Volume Group .........................................................434 Use Multiple Destinations for SNA Applications ......................................................435 Avoid File Locking ....................................................
Before You Start...........................................................................................................454 Running the Rolling Upgrade Using DRD..................................................................454 Example of a Rolling Upgrade .........................................................................................455 Step 1. ..........................................................................................................................456 Step 2. ..............
IPv4 Compatible IPv6 Addresses...........................................................................477 IPv4 Mapped IPv6 Address...................................................................................477 Aggregatable Global Unicast Addresses...............................................................477 Link-Local Addresses.............................................................................................478 Site-Local Addresses...................................................
List of Figures 1-1 1-2 1-3 2-1 2-2 2-3 2-4 2-5 2-6 3-1 3-2 3-3 3-4 3-5 3-6 3-7 3-8 3-9 3-10 3-11 3-12 3-13 3-14 3-15 3-16 3-17 3-18 3-19 3-20 3-21 3-22 3-23 3-24 3-25 4-1 5-1 D-1 D-2 D-3 D-4 D-5 Typical Cluster Configuration ....................................................................................29 Typical Cluster After Failover ....................................................................................31 Tasks in Configuring a Serviceguard Cluster ....................................
D-6 H-1 H-2 22 Running Cluster After Upgrades .............................................................................458 System Management Homepage with Serviceguard Manager................................488 Cluster by Type.........................................................................................................
List of Tables 1 3-1 3-2 3-3 3-4 4-1 4-2 6-1 6-2 7-1 7-2 7-3 G-1 G-2 G-3 G-4 G-5 G-6 G-7 G-8 I-1 I-2 Printing History..........................................................................................................25 Package Configuration Data.......................................................................................73 Node Lists in Sample Cluster......................................................................................
Printing History Table 1 Printing History Printing Date Part Number Edition January 1995 B3936-90001 First June 1995 B3936-90003 Second December 1995 B3936-90005 Third August 1997 B3936-90019 Fourth January 1998 B3936-90024 Fifth October 1998 B3936-90026 Sixth December 2000 B3936-90045 Seventh September 2001 B3936-90053 Eighth March 2002 B3936-90065 Ninth June 2003 B3936-90070 Tenth June 2004 B3936-90076 Eleventh June 2005 B3936-90076 Eleventh, First reprint October 20
Preface This eighteenth edition of the manual applies to Serviceguard Version A.11.20. Earlier versions are available at www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard. This guide describes how to configure Serviceguard to run on HP 9000 or HP Integrity servers under the HP-UX operating system. The contents are as follows: • • • • • • • • • • • • • • • “Serviceguard at a Glance” (page 29), describes a Serviceguard cluster and provides a roadmap for using this guide.
• • Appendix H (page 485) describes the Serviceguard Manager GUI. “Maximum and Minimum Values for Parameters” (page 491) provides a reference to the supported ranges for Serviceguard parameters. Related Publications For information about the current version of Serviceguard, and about older versions, see the Serviceguard documents posted at www.hp.com/go/ hpux-serviceguard-docs —> HP Serviceguard. The following documents, which can all be found at www.hp.
1 Serviceguard at a Glance This chapter introduces Serviceguard on HP-UX, and shows where to find information in this book. It covers the following: • What is Serviceguard? • Using Serviceguard Manager (page 32) • A Roadmap for Configuring Clusters and Packages (page 34) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4: “Planning and Documenting an HA Cluster ” (page 121). Specific steps for setup are given in Chapter 5: “Building an HA Cluster Configuration” (page 205).
network (LAN) component. In the event that one component fails, the redundant component takes over. Serviceguard and other high availability subsystems coordinate the transfer between components. A Serviceguard cluster is a networked grouping of HP 9000 or HP Integrity servers (or both), known as nodes, having sufficient redundancy of software and hardware that a single point of failure will not significantly disrupt service. A package groups application services (individual HP-UX processes) together.
services also are used for other types of inter-node communication. (The heartbeat is explained in more detail in the chapter “Understanding Serviceguard Software.”) Failover Any host system running in a Serviceguard cluster is called an active node. Under normal conditions, a fully operating Serviceguard cluster monitors the health of the cluster's components on all its active nodes. Most Serviceguard packages are failover packages.
provide as many separate power circuits as needed to prevent a single point of failure of your nodes, disks and disk mirrors. Each power circuit should be protected by an uninterruptible power source. For more details, refer to the section on “Power Supply Planning” in Chapter 4, “Planning and Documenting an HA Cluster.
You can use Serviceguard Manager to monitor, administer, and configure Serviceguard clusters. • • • You can see properties, status, and alerts of clusters, nodes, and packages. You can do administrative tasks such as run or halt clusters, cluster nodes, and packages. Yyou can create or modify a cluster and its packages. Monitoring Clusters with Serviceguard Manager From the main page of Serviceguard Manager, you can see status and alerts for the cluster, nodes, and packages.
on the command line. As of HP-UX 11i v3, SAM offers a Terminal User Interface (TUI) which also acts as a gateway to the web-based System Management Homepage (SMH). • • To get to the SMH for any task area, highlight the task area in the SAM TUI and press w. To go directly to the SMH from the command line, enter /usr/sbin/sam -w For more information, see the HP-UX Systems Administrator’s Guide, at the address given in the preface to this manual.
Figure 1-3 Tasks in Configuring a Serviceguard Cluster The tasks in Figure 1-3 are covered in step-by-step detail in chapters 4 through 7. HP recommends you gather all the data that is needed for configuration before you start. See “Planning and Documenting an HA Cluster ” (page 121) for tips on gathering data.
2 Understanding Serviceguard Hardware Configurations This chapter gives a broad overview of how the Serviceguard hardware components work. The following topics are presented: • Redundancy of Cluster Components • Redundant Network Components (page 38) • Redundant Disk Storage (page 44) • Redundant Power Supplies (page 49) • Larger Clusters (page 50) Refer to the next chapter for information about Serviceguard software components.
Fibre Channel or HP StorageWorks XP or EMC Symmetrix disk technology can be configured for failover among 16 nodes. Note that a package that does not access data from a disk on a shared bus can be configured to fail over to as many nodes as you have configured in the cluster (regardless of disk technology).
NOTE: Serial (RS232) lines are no longer supported for the cluster heartbeat. Fibre Channel, Token Ring and FDDI networks are no longer supported as heartbeat or data LANs. Rules and Restrictions • • • • • A single subnet cannot be configured on different network interfaces (NICs) on the same node. In the case of subnets that can be used for communication between cluster nodes, the same network interface must not be used to route more than one subnet configured on the same node.
addresses themselves will be immediately configured into the cluster as stationary IP addresses. CAUTION: If you configure any address other than a stationary IP address on a Serviceguard network interface, it could collide with a relocatable package IP address assigned by Serviceguard. See “Stationary and Relocatable IP Addresses ” (page 90). (Oracle VIPs are an exception to this rule; such configurations require the HP add-on product Serviceguard Extension for Oracle RAC).
In the figure, a two-node Serviceguard cluster has one bridged net configured with both a primary and a standby LAN card for the data/heartbeat subnet (subnetA). Another LAN card provides an optional dedicated heartbeat LAN. Note that the primary and standby LAN segments are connected by a hub to provide a redundant data/heartbeat subnet. Each node has its own IP address for this subnet.
• You should not use the wildcard (*) for node_name in the package configuration file, as this could allow the package to fail over across subnets when a node on the same subnet is eligible, and failing over across subnets can take longer. List the nodes in order of preference rather than using the wildcard. • You should configure IP monitoring for each subnet; see “Monitoring LAN Interfaces and Detecting Failure: IP Level” (page 98).
• • cmrunnode will fail if the “hostname LAN” is down on the node in question. (“Hostname LAN” refers to the public LAN on which the IP address that the node’s hostname resolves to is configured). If a monitored_subnet is configured for PARTIAL monitored_subnet_access in a package’s configuration file, it must be configured on at least one of the nodes on the node_name list for that package.
cluster is running; see “Changing the Cluster Networking Configuration while the Cluster Is Running” (page 367). Redundant Disk Storage Each node in a cluster has its own root disk, but each node is also physically connected to several other disks in such a way that more than one node can obtain access to the data and programs associated with a package it is configured for.
Disk Mirroring Serviceguard itself does not provide protection for data on your disks, but protection is provided by HP’s Mirrordisk/UX product for LVM storage, and by the Veritas Volume Manager for VxVM and CVM. The logical volumes used for Serviceguard packages should be mirrored; so should the cluster nodes’ root disks. When you configure logical volumes using software mirroring, the members of each mirrored set contain exactly the same data.
NOTE: 4.1 and later versions of Veritas Volume Manager (VxVM) and Dynamic Multipathing (DMP) from Symantec are supported on HP-UX 11i v3, but do not provide multipathing and load balancing; DMP acts as a pass-through driver, allowing multipathing and load balancing to be controlled by the HP-UX I/O subsystem instead.
Replacing Failed I/O Cards Depending on the system configuration, it is possible to replace failed disk I/O cards while the system remains online. The process is described under “Replacing I/O Cards” (page 406). Sample SCSI Disk Configurations Figure 2-2 shows a two node cluster. Each node has one root disk which is mirrored and one package for which it is the primary node. Resources have been allocated to each node so that each node may adopt the package from the other node.
Figure 2-3 Cluster with High Availability Disk Array Details on logical volume configuration for Serviceguard are in the chapter “Building an HA Cluster Configuration.” Sample Fibre Channel Disk Configuration In Figure 2-4, the root disks are shown with simple mirroring, but the shared storage is now accessed via redundant Fibre Channel switches attached to a disk array.
Figure 2-4 Cluster with Fibre Channel Switched Disk Array This type of configuration uses native HP-UX or other multipathing software; see “About Multipathing” (page 45). Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks. HP-supported uninterruptible power supplies (UPS), such as HP PowerTrust, can provide this protection from momentary power loss.
Therefore, if all of the hardware in the cluster has 2 or 3 power inputs, then at least three separate power circuits will be required to ensure that there is no single point of failure in the power circuit design for the cluster. Larger Clusters You can create clusters of up to 16 nodes with Serviceguard. Clusters of up to 16 nodes may be built by connecting individual SPUs via Ethernet.
Figure 2-5 Eight-Node Active/Standby Cluster Point to Point Connections to Storage Devices Some storage devices allow point-to-point connection to a large number of host nodes without using a shared SCSI bus. An example is shown in Figure 2-11, a cluster consisting of eight nodes with a SCSI interconnect. The nodes access shared data on an XP or EMC disk array configured with 16 SCSI I/O ports. Each node is connected to the array using two separate SCSI channels.
Figure 2-6 Eight-Node Cluster with XP or EMC Disk Array Fibre Channel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP Unix Servers Configuration Guide, available through your HP representative.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
NOTE: Veritas CFS may not yet be supported on the version of HP-UX you are running; see “About Veritas CFS and CVM from Symantec” (page 32).
• • • • • /usr/lbin/cmvxpingd—Serviceguard-to-Veritas Activation daemon. (Only present if Veritas CFS is installed.) /usr/lbin/cmdisklockd— Lock LUN daemon /usr/lbin/cmlockd—Utility daemon /opt/sgproviders/bin/cmwbemd—WBEM daemon /usr/lbin/cmproxyd—Proxy daemon Each of these daemons logs to the /var/adm/syslog/syslog.logfile except for /opt/cmom/lbin/cmomd, which logs to /var/opt/cmom/cmomd.log. The Quorum Server runs outside the cluster.
NOTE: Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 20 on all cluster nodes. It is important that user processes should have a priority lower than 20, otherwise they may prevent Serviceguard from updating the kernel safety timer, causing a system reset. File Management Daemon: cmfileassistd The cmfileassistd daemon is used by cmcld to manage the files that it needs to read from, and write to, disk.
The SNMP Master Agent and the cmsnmpd provide notification (traps) for cluster-related events. For example, a trap is sent when the cluster configuration changes, or when a Serviceguard package has failed. You must edit /etc/SnmpAgent.d/ snmpd.conf to tell cmsnmpd where to send this information. You must also edit /etc/rc.config.d/cmsnmpagt to auto-start cmsnmpd. Configure cmsnmpd to start before the Serviceguard cluster comes up. For more information, see the cmsnmpd (1m) manpage.
Lock LUN Daemon: cmdisklockd If a lock LUN is being used, cmdisklockd runs on each node in the cluster and is started by cmcld when the node joins the cluster. Utility Daemon: cmlockd Runs on every node on which cmcld is running (though currently not actually used by Serviceguard on HP-UX systems).
deployed as part of the Serviceguard Storage Management Suite bundles, the file /etc/gabtab is automatically configured and maintained by Serviceguard. GAB provides membership and messaging for CVM and the CFS. GAB membership also provides orderly startup and shutdown of the cluster file system.
Heartbeat Messages Central to the operation of the cluster manager is the sending and receiving of heartbeat messages among the nodes in the cluster. Each node in the cluster exchanges UDP heartbeat messages with every other node over each monitored IP network configured as a heartbeat device. (LAN monitoring is discussed later, in the section “Monitoring LAN Interfaces and Detecting Failure: Link Level” (page 92).
IMPORTANT: When multiple heartbeats are configured, heartbeats are sent in parallel; Serviceguard must receive at least one heartbeat to establish the health of a node. HP recommends that you configure all subnets that connect cluster nodes as heartbeat networks; this increases protection against multiple faults at no additional cost. Heartbeat IP addresses are usually on the same subnet on each node, but it is possible to configure a cluster that spans subnets; see “Cross-Subnet Configurations” (page 41).
which is a permanent modification of the configuration files. Re-formation of the cluster occurs under the following conditions (not a complete list): • • • • • • • • An SPU or network failure was detected on an active node. An inactive node wants to join the cluster. The cluster manager daemon has been started on that node. A node has been added to or deleted from the cluster configuration. The system administrator halted a node. A node halts because of a package failure.
If you have a two-node cluster, you are required to configure a cluster lock. If communications are lost between these two nodes, the node that obtains the cluster lock will take over the cluster and the other node will halt (system reset). Without a cluster lock, a failure of either node in the cluster will cause the other node, and therefore the cluster, to halt. Note also that if the cluster lock fails during an attempt to acquire it, the cluster will halt.
Figure 3-2 Lock Disk or Lock LUN Operation Serviceguard periodically checks the health of the lock disk or LUN and writes messages to the syslog file if the device fails the health check. This file should be monitored for early detection of lock disk problems. If you are using a lock disk, you can choose between two lock disk options—a single or dual lock disk—based on the kind of high availability configuration you are building. A single lock disk is recommended where possible.
in two separate data centers, a single lock disk would be a single point of failure should the data center it resides in suffer a catastrophic failure. In these two cases only, a dual cluster lock, with two separately powered cluster disks, should be used to eliminate the lock disk as a single point of failure. NOTE: You must use Fibre Channel connections for a dual cluster lock; you can no longer implement it in a parallel SCSI configuration.
Figure 3-3 Quorum Server Operation The Quorum Server runs on a separate system, and can provide quorum services for multiple clusters. IMPORTANT: For more information about the Quorum Server, see the latest version of the HP Serviceguard Quorum Server release notes at www.hp.com/go/ hpux-serviceguard-docs -> HP Serviceguard Quorum Server Software. No Cluster Lock Normally, you should not configure a cluster of three or fewer nodes without a cluster lock. In two-node clusters, a cluster lock is required.
Server), the quorum device (for example from one quorum server to another), and the parameters that govern them (for example the Quorum Server polling interval). For more information about the Quorum Server and lock parameters, see “Cluster Configuration Parameters ” (page 143). NOTE: If you are using the Veritas Cluster Volume Manager (CVM) you cannot change the quorum configuration while SG-CFS-pkg is running. For more information about CVM, see “CVM and VxVM Planning ” (page 134).
Package Types Three different types of packages can run in the cluster; the most common is the failover package. There are also special-purpose packages that run on more than one node at a time, and so do not failover. They are typically used to manage resources of certain failover packages.
Figure 3-4 Package Moving During Failover Configuring Failover Packages You configure each package separately. You create a failover package by generating and editing a package configuration file template, then adding the package to the cluster configuration database; see Chapter 6: “Configuring Packages and Their Services ” (page 279). For legacy packages (packages created by the method used on versions of Serviceguard earlier than A.11.
that determine failover behavior. These are the auto_run parameter, the failover_policy parameter, and the failback_policy parameter.
Figure 3-5 Before Package Switching Figure 3-6 shows the condition where Node 1 has failed and Package 1 has been transferred to Node 2 on the same subnet. Package 1’s IP address was transferred to Node 2 along with the package. Package 1 continues to be available and is now running on Node 2. Also note that Node 2 can now access both Package1’s disk and Package2’s disk.
NOTE: For design and configuration information about clusters which span subnets, including site-aware disaster-tolerant clusters, see the documents listed under “Cross-Subnet Configurations” (page 41). Figure 3-6 After Package Switching Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file.
Automatic Rotating Standby Using the min_package_node failover policy, it is possible to configure a cluster that lets you use one node as an automatic rotating standby node for the cluster. Consider the following package configuration for a four node cluster. Note that all packages can run on all nodes and have the same node_name lists. Although the example shows the node names in a different order for each package, this is not required.
Figure 3-8 Rotating Standby Configuration after Failover NOTE: Using the min_package_node policy, when node 2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node.
Figure 3-9 CONFIGURED_NODE Policy Packages after Failover If you use configured_node as the failover policy, the package will start up on the highest priority node in the node list, assuming that the node is running as a member of the cluster. When a failover occurs, the package will move to the next highest priority node in the list that is available.
Figure 3-10 Automatic Failback Configuration before Failover Table 3-2 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 CONFIGURED_NODE AUTOMATIC pkgB node2, node4 CONFIGURED_NODE AUTOMATIC pkgC node3, node4 CONFIGURED_NODE AUTOMATIC node1 panics, and after the cluster reforms, pkgA starts running on node4: 76 Understanding Serviceguard Software Components
Figure 3-11 Automatic Failback Configuration After Failover After rebooting, node 1 rejoins the cluster. At that point, pkgA will be automatically stopped on node 4 and restarted on node 1.
Figure 3-12 Automatic Failback Configuration After Restart of Node 1 NOTE: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period. If you are using automatic failback, you may want to wait to add the package’s primary node back into the cluster until you can allow the package to be taken out of service temporarily while it switches back to the primary node.
Using the Event Monitoring Service Basic package resources include cluster nodes, LAN interfaces, and services, which are the individual processes within an application. All of these are monitored by Serviceguard directly. In addition, you can use the Event Monitoring Service registry through which add-on monitors can be configured. This registry allows other software components to supply monitoring of their resources for Serviceguard.
How Packages Run Packages are the means by which Serviceguard starts and halts configured applications. Failover packages are also units of failover behavior in Serviceguard. A package is a collection of services, disk volumes and IP addresses that are managed by Serviceguard to ensure they are available. There can be a maximum of 300 packages per cluster and a total of 900 services per cluster.
packages and failover packages can name some subset of the cluster’s nodes or all of them. If the auto_run parameter is set to yes in a package’s configuration file Serviceguard automatically starts the package when the cluster starts. System multi-node packages are required to have auto_run set to yes. If a failover package has auto_run set to no, Serviceguard cannot start it automatically at cluster startup time; you must explicitly enable this kind of package using the cmmodpkg command.
1. 2. 3. 4. 5. 6. 7. Before the control script starts. (For modular packages, this is the master control script.) During run script execution. (For modular packages, during control script execution to start the package.) While services are running When a service, subnet, or monitored resource fails, or a dependency is not met. During halt script execution. (For modular packages, during control script execution to halt the package.
7. 8. Starts up any EMS (Event Monitoring Service) resources needed by the package that were specially marked for deferred startup. Exits with an exit code of zero (0). Figure 3-14 Package Time Line (Legacy Package) At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). For example, if a package service is unable to be started, the control script will exit with an error. NOTE: This diagram is specific to legacy packages.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled is set to yes, it will halt the node on which the package is running.
SERVICE_RESTART[0]=" " SERVICE_RESTART[0]="-r " SERVICE_RESTART[0]="-R" ; do not restart ; restart as many as times ; restart indefinitely NOTE: If you set restarts and also set service_fail_fast_enabled to yes, the failfast will take place after restart attempts have failed. It does not make sense to set service_restart to “-R” for a service and also set service_fail_fast_enabled to yes.
NOTE: If a package is dependent on a subnet, and the subnet fails on the node where the package is running, the package will start to shut down. If the subnet recovers immediately (before the package is restarted on an adoptive node), the package manager restarts the package on the same node; no package switch occurs.
Figure 3-15 Legacy Package Time Line for Halt Script Execution At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). Also, if the halt script execution is not complete before the time specified in the HALT_SCRIPT_TIMEOUT, the package manager will kill the script. During halt script execution, messages are written to a log file. For legacy packages, this is in the same directory as the run script and has the same name as the run script and the extension.
• • • 0—normal exit. The package halted normally, so all services are down on this node. 1—abnormal exit, also known as no_restart exit. The package did not halt normally. Services are killed, and the package is disabled globally. It is not disabled on the current node, however. Timeout—Another type of exit occurs when the halt_script_timeout is exceeded. In this scenario, the package is killed and disabled globally. It is not disabled on the current node, however.
Table 3-3 Error Conditions and Package Movement for Failover Packages (continued) Package Error Condition Results Error or Exit Code Node Failfast Enabled Service Failfast Enabled HP-UX Status on Primary after Error Halt script runs after Error or Exit Package Allowed Package to Run on Primary Allowed to Node after Error Run on Alternate Node Halt Script Timeout YES Either Setting system reset N/A N/A (system reset) Yes, unless the timeout happened after the cmhaltpkg command was executed.
where the package is running and monitoring the health of all interfaces, switching them when necessary. NOTE: Serviceguard monitors the health of the network interfaces (NICs) and can monitor the IP level (layer 3) network. Stationary and Relocatable IP Addresses Each node (host system) should have at least one IP address for each active network interface. This address, known as a stationary IP address, is configured in the node's /etc/rc.config.d/netconf file or in the node’s /etc/rc.config.
In addition, relocatable addresses (but not stationary addresses) can be taken over by an adoptive node on the same subnet if control of the package is transferred. This means that applications can access the package via its relocatable address without knowing which node the package currently resides on.
IP addresses are configured only on each primary network interface card; standby cards are not configured with an IP address. Multiple IPv4 addresses on the same network card must belong to the same IP subnet. CAUTION: HP strongly recommends that you add relocatable addresses to packages only by editing ip_address (page 299) in the package configuration file (or IP [] entries in the control script of a legacy package) and running cmapplyconf (1m).
• calculates the time depending on the type of LAN card.) Serviceguard will not declare the card as bad if only the inbound or only the outbound count stops incrementing. Both must stop. This is the default. INONLY_OR_INOUT: This option will also declare the card as bad if both inbound and outbound counts stop incrementing. However, it will also declare it as bad if only the inbound count stops. This option is not suitable for all environments.
Jumbo Frames. Additionally, network interface cards running 1000Base-T or 1000Base-SX cannot do local failover to 10BaseT. During the transfer, IP packets will be lost, but TCP (Transmission Control Protocol) will retransmit the packets. In the case of UDP (User Datagram Protocol), the packets will not be retransmitted automatically by the protocol. However, since UDP is an unreliable service, UDP applications should be prepared to handle the case of lost network packets and recover appropriately.
Figure 3-17 Cluster After Local Network Switching As the standby interface takes over, IP addresses will be switched to the hardware path associated with the standby interface. The switch is transparent at the TCP/IP level. All applications continue to run on their original nodes. During this time, IP traffic on Node 1 will be delayed as the transfer occurs. However, the TCP/IP connections will continue to be maintained and applications will continue to run.
Figure 3-18 Local Switching After Cable Failure Local network switching will work with a cluster containing one or more nodes. You may wish to design a single-node cluster in order to take advantage of this local network switching feature in situations where you need only one node and do not wish to set up a more complex cluster. Switching Back to Primary LAN Interfaces after Local Switching If a primary interface fails, the IP address will be switched to a standby.
cmhaltnode command is issued or on all nodes in the cluster if a cmhaltcl command is issued. • Configurable behavior: NETWORK_AUTO_FAILBACK = NO. Serviceguard will detect and log the recovery of the interface, but will not switch the IP address back from the standby to the primary interface. You can tell Serviceguard to switch the IP address back to the primary interface by means of the cmmodnet command: cmmodnet -e where is the primary interface.
configured on that node, and identified as monitored subnets in the package configuration file, must be available.) Note that remote switching is supported only between LANs of the same type. For example, a remote switchover between an Ethernet interface on one machine and an IPoIB interface on the failover machine is not supported. The remote switching of relocatable IP addresses is shown in Figure 3-5 and Figure 3-6.
NOTE: This applies only to subnets for which the cluster configuration parameter IP_MONITOR is set to ON. See “Cluster Configuration Parameters ” (page 143) for more information. — Errors that prevent packets from being received but do not affect the link-level health of an interface IMPORTANT: You should configure the IP Monitor in a cross-subnet configuration, because IP monitoring will detect some errors that link-level monitoring will not. See also “Cross-Subnet Configurations” (page 41).
IPv4: 1 16.89.143.192 16.89.120.0 … Possible IP Monitor Subnets: IPv4: 16.89.112.0 Polling Target 16.89.112.1 IPv6: 3ffe:1000:0:a801:: Polling Target 3ffe:1000:0:a801::254 … The IP Monitor section of the cluster configuration file will look similar to the following for a subnet on which IP monitoring is configured with target polling.
NOTE: This is the default if cmquerycl does not detect a gateway for the subnet in question; it is equivalent to having no SUBNET entry for the subnet. See SUBNET under “Cluster Configuration Parameters ” (page 143) for more information. Failure and Recovery Detection Times With the default NETWORK_POLLING_INTERVAL of 2 seconds (see “Cluster Configuration Parameters ” (page 143)) the IP monitor will detect IP failures typically within 8–10 seconds for Ethernet and within 16–18 seconds for InfiniBand.
local switching may occur, that is, a switch to a standby LAN card if one has been configured; see “Local Switching ” (page 93). The following examples show when and how a link-level failure is differentiated from an IP-level failure in the output of cmviewcl (1m). As you can see, if local switching is configured, the difference is the keyword disabled, which appears in the tabular output, and is set to true in the line output, if the IP Monitor detects the failure.
cmviewcl -v -f line will report the same failure like this: node:gary|interface:lan2|status=down node:gary|interface:lan2|disabled=false node:gary|interface:lan2|failure_type=link+ip If local switching is not configured and a failure is detected by IP monitoring, output from cmviewcl -v will look like something like this: Network_Parameters: INTERFACE STATUS PRIMARY down (IP only) PRIMARY up PATH 0/3/1/0 0/5/1/0 NAME lan2 lan3 cmviewcl -v -f line will report the same failure like this: node:gary|interfa
Figure 3-19 Aggregated Networking Ports Both the Single and Dual ported LANs in the non-aggregated configuration have four LAN cards, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (lan0, lan1, lan2, lan3). When these ports are aggregated all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as lan900, the name by which the aggregate is known on HP-UX 11i.
What is VLAN? Virtual LAN (or VLAN) is a technology that allows logical grouping of network nodes, regardless of their physical locations. VLAN can be used to divide a physical LAN into multiple logical LAN segments or broadcast domains, helping to reduce broadcast traffic, increase network performance and security, and improve manageability.
Additional Heartbeat Requirements VLAN technology allows great flexibility in network configuration. To maintain Serviceguard’s reliability and availability in such an environment, the heartbeat rules are tightened as follows when the cluster is using VLANs: 1. VLAN heartbeat networks must be configured on separate physical NICs or APA aggregates, to avoid single points of failure. 2. Heartbeats are still recommended on all cluster networks, including VLANs. 3.
to agile addressing when you upgrade to 11i v3, though you should seriously consider its advantages. For instructions on migrating a system to agile addressing, see the white paper LVM Migration from Legacy to Agile Naming Model at www.hp.com/go/hpux-core-docs.
NOTE: Under agile addressing (see “About Device File Names (Device Special Files)” (page 106)), the storage units in this example would have names such as disk1, disk2, disk3, etc. As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 135). Figure 3-20 Physical Disks Within Shared Storage Units Figure 3-21 shows the individual disks combined in a multiple disk mirrored configuration.
Figure 3-21 Mirrored Physical Disks Figure 3-22 shows the mirrors configured into LVM volume groups, shown in the figure as /dev/vgpkgA and /dev/vgpkgB. The volume groups are activated by Serviceguard packages for use by highly available applications.
Examples of Storage on Disk Arrays Figure 3-23 shows an illustration of storage configured on a disk array. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system. Figure 3-23 Physical Disks Combined into LUNs NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer. Since arrays vary considerably, you should refer to the documentation that accompanies your storage unit.
Figure 3-24 Multiple Paths to LUNs Finally, the multiple paths are configured into volume groups as shown in Figure 3-25.
Types of Volume Manager Serviceguard allows a choice of volume managers for data storage: • HP-UX Logical Volume Manager (LVM) and (optionally) Mirrordisk/UX • Veritas Volume Manager for HP-UX (VxVM)—Base and add-on Products • Veritas Cluster Volume Manager for HP-UX Separate sections in Chapters 5 and 6 explain how to configure cluster storage using all of these volume managers.
• • need to use software RAID mirroring or striped mirroring. have multiple heartbeat subnets configured. Propagation of Disk Groups in VxVM A VxVM disk group can be created on any node, whether the cluster is up or not. You must validate the disk group by trying to import it on each node. Package Startup Time with VxVM With VxVM, each disk group is imported by the package control script that uses the disk group.
CVM 4.1 and later can be used with Veritas Cluster File System (CFS) in Serviceguard. Several of the HP Serviceguard Storage Management Suite bundles include features to enable both CVM and CFS. CVM can be used in clusters that: • • • run applications that require fast disk group activation after package failover; require storage activation on more than one node at a time, for example to perform a backup from one node while a package using the volume is active on another node.
Table 3-4 Pros and Cons of Volume Managers with Serviceguard Product Advantages Tradeoffs Logical Volume Manager (LVM) • Software is provided with all versions • Lacks flexibility and extended of HP-UX. features of some other volume managers • Provides up to 3-way mirroring using optional Mirrordisk/UX software. • Dynamic multipathing (DMP) is active by default as of HP-UX 11i v3.
Table 3-4 Pros and Cons of Volume Managers with Serviceguard (continued) Product Advantages Tradeoffs Veritas Volume Manager— Full VxVM product • Disk group configuration from any node. • DMP for active/active storage devices. • Supports exclusive activation.
The case is covered in more detail under “What Happens when a Node Times Out” (page 117). See also “Cluster Daemon: cmcld” (page 55). A system reset is also initiated by Serviceguard itself under specific circumstances; see “Responses to Package and Service Failures ” (page 119). What Happens when a Node Times Out Each node sends a heartbeat message to all other nodes at an interval equal to one-fourth of the value of the configured MEMBER_TIMEOUT or 1 second, whichever is less.
SystemB recognizes that it has failed to get the cluster lock and so cannot re-form the cluster. To release all resources related to Package2 (such as exclusive access to volume group vg02 and the Package2 IP address) as quickly as possible, SystemB halts (system reset). NOTE: If AUTOSTART_CMCLD in /etc/rc.config.d/cmcluster ($SGAUTOSTART) is set to zero, the node will not attempt to join the cluster when it comes back up.
component, and will result in the appropriate switching behavior. Power protection is provided by HP-supported uninterruptible power supplies (UPS). Responses to Package and Service Failures In the default case, the failure of a failover package, or of a service within the package, causes the package to shut down by running the control script with the ‘stop’ parameter, and then restarting the package on an alternate node.
Network Communication Failure An important element in the cluster is the health of the network itself. As it continuously monitors the cluster, each node listens for heartbeat messages from the other nodes confirming that all nodes are able to communicate with each other. If a node does not hear these messages within the configured amount of time, a node timeout occurs; see “What Happens when a Node Times Out” (page 117).
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather information about all the hardware and software components of the configuration.
• • electrical points of failure. application points of failure. Serviceguard Memory Requirements Serviceguard requires approximately 15.5 MB of lockable memory. Planning for Expansion When you first set up the cluster, you indicate a set of nodes and define a group of packages for the initial configuration. At a later time, you may wish to add additional nodes and packages, or you may wish to use additional disk hardware for shared data storage.
NOTE: Under agile addressing, the storage units in this example would have names such as disk1, disk2, disk3, etc. See “About Device File Names (Device Special Files)” (page 106). As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 135). Figure 4-1 Sample Cluster Configuration Create a similar sketch for your own cluster.
Server Number The series number; for example, rp8400 or rx8620-32. Host Name The name to be used on the system as the host name. Memory Capacity The memory in MB. Number of I/O slots The number of slots. Network Information Serviceguard monitors LAN interfaces. NOTE: Serviceguard supports communication across routers between nodes in the same cluster; for more information, see the documents listed under “Cross-Subnet Configurations” (page 41).
IP Address This node’s host IP address(es), to be used on this interface. If the interface is a standby and does not have an IP address, enter 'Standby.' An IPv4 address is a string of 4 digits separated with decimals, in this form: nnn.nnn.nnn.nnn An IPV6 address is a string of 8 hexadecimal values separated with colons, in this form: xxx:xxx:xxx:xxx:xxx:xxx:xxx:xxx. For more details of IPv6 address format, see the Appendix G (page 475).
priorities. Therefore, when configuring a highly available cluster, you should give nodes the highest priority SCSI addresses, and give disks addresses of lesser priority. For SCSI, high priority starts at seven, goes down to zero, and then goes from 15 to eight. Therefore, seven is the highest priority and eight is the lowest priority.
Slot Number Indicate the slot number in which the interface card is inserted in the backplane of the computer. Address Enter the bus hardware path number, which will be seen on the system later when you use ioscan to display hardware. Disk Device File Enter the disk device file name. To display the name use the ioscan -fnC disk command (for legacy DSFs) or ioscan -fnNC disk (for agile addressing). This information is used in creating the mirrored disk configuration using Logical Volume Manager.
Name of Subnet __Blue___ Name of Node IP Interface ___lan2___ Addr_______________ Traffic Type _standby_ Name of Subnet __Red____ Name of Node IP Interface ___lan1___ Addr___35.12.15.
Host Name The host name for each SPU. Disk Unit The disk drive unit number for each disk. Tape Unit The tape unit number for each backup device. Other Unit Tthe number of any other unit. Power Supply The power supply unit number of the UPS to which the host or other device is connected. Be sure to follow UPS and cabinet power limits as well as SPU power limits.
specified in its packages. You can specify an LVM lock disk, a lock LUN, or a Quorum Server as the cluster lock. For more information about the cluster lock, and requirements and recommendations, see “Cluster Lock ” (page 62). NOTE: You cannot use more than one type of lock in the same cluster. A one-node cluster does not require a lock. Two-node clusters require the use of a cluster lock, and a lock is recommended for larger clusters as well.
IMPORTANT: If you plan to use a Quorum Server, make sure you read the HP Serviceguard Quorum Server Version A.04.00 Release Notes before you proceed. You can find them at: www.hp.com/go/hpux-serviceguard-docs -> HP Serviceguard Quorum Server Software. You should also consult the Quorum Server white papers at the same location. A quorum server: • • • Can be used with up to 150 clusters, not exceeding 300 nodes total. Can support a cluster with any supported number of nodes.
Host Names ____________________________________________ LVM Planning You can create storage groups using the HP-UX Logical Volume Manager (LVM), or using Veritas VxVM and CVM software as described in the next section. When designing your disk layout using LVM, you should consider the following: • • • • • • • • The root disk should belong to its own volume group.
resource_start AUTOMATIC resource_up_value = UP The example above will monitor all PV links for the volume group vgpkg. As long as all devices within the vgpkg volume group are functional, the resource will remain in UP status. When the last path to the storage for the volume group fails, or any device within the volume group fails, the resource value will change, and at the next polling interval the package will fail because the resource no longer meets the package requirements.
Physical Volume Name: ______________/dev/dsk/c4t2d0________________________ Physical Volume Name: ______________/dev/dsk/c5t2d0________________________ Physical Volume Name: ______________/dev/dsk/c6t2d0________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________
========================================================================= Disk Group Name: ______dg01_____________________ Disk Name: _____________________c1t2d0__________________________ Disk Name: _____________________c2t2d0__________________________ Disk Name: _____________________c3t2d0__________________________ Disk Name: _____________________________________________________ CFS DG Package Name: ___________SG-CFS-DG_1_____ CFS Volume, MP, and MP Pkg: _logdata_/mnt/lvol1_ SG-CFS-MP_1______ CFS Volume, M
Because DSF names may be duplicated between one host and other, it is possible for different storage devices to have the same name on different nodes in a cluster, and for the same piece of storage to be addressed by different names. Cluster-wide device files (cDSFs), available as of the September 2010 HP-UX Fusion Release, ensure that each storage device used by the cluster has a unique device file name.
NOTE: Software that assumes DSFs reside only in /dev/disk and /dev/rdisk will not find cDSFs and may not work properly as a result; as of the date of this manual, this was true of the Veritas Volume Manager, VxVM. Limitations of cDSFs • • • • • cDSFs are supported only within a single cluster; you cannot define a cDSF group that crosses cluster boundaries. A node can belong to only one cDSF group.
• • • Configure networking and security (cmpreparecl) Create and start the cluster with a cluster lock device (cmdeploycl) Create and import volume groups as additional shared storage for use by cluster packages (cmpreparestg) Advantages of Easy Deployment • • • • Quick and simple way to create and start a cluster. Automates security and networking configuration that must always be done before you configure nodes into a cluster. Simplifies cluster lock configuration.
NOTE: For heartbeat configuration requirements, see the discussion of the HEARTBEAT_IP parameter later in this chapter. For more information about managing the speed of cluster re-formation, see the discussion of the MEMBER_TIMEOUT parameter, and further discussion under “What Happens when a Node Times Out” (page 117) ,“Modifying the MEMBER_TIMEOUT Parameter” (page 251), and, for troubleshooting, “Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low” (page 415).
NOTE: This applies only to hostname resolution. You can have IPv6 heartbeat and data LANs no matter what the HOSTNAME_ADDRESS_FAMILY parameter is set to. (IPv4 heartbeat and data LANs are allowed in IPv4 and mixed mode.
NOTE: This also applies if HOSTNAME_ADDRESS_FAMILY is set to ANY. See “Allowing Root Access to an Unconfigured Node” (page 216) for more information. • If you use a Quorum Server, you must make sure that the Quorum Server hostname (and the alternate Quorum Server address specified by QS_ADDR, if any) resolve to IPv6 addresses, and you must use Quorum Server version A.04.00 or later. See the latest Quorum Server release notes for more information; you can find them at www.hp.
Recommendations for IPv6-Only Mode IMPORTANT: Check the current Serviceguard release notes for the latest instructions and recommendations. • If you decide to migrate the cluster to IPv6-only mode, you should plan to do so while the cluster is down. What Is Mixed Mode? If you configure mixed mode (HOSTNAME_ADDRESS_FAMILY set to ANY), then the addresses used by the cluster, including the heartbeat, and Quorum Server addresses if any, can be IPv4 or IPv6 addresses.
• CFS, CVM, VxVM, and VxFS are not supported. NOTE: • This also applies if HOSTNAME_ADDRESS_FAMILY is set to IPV6. HPVM is not supported. You cannot have a virtual machine that is either a node or a package if HOSTNAME_ADDRESS_FAMILY is set to ANY or IPV6. Cluster Configuration Parameters You need to define a set of cluster parameters. These are stored in the binary cluster configuration file, which is distributed to each node in the cluster.
NOTE: In addition, the following characters must not be used in the cluster name if you are using the Quorum Server: at-sign (@), equal-sign (=), or-sign (|), semicolon (;). These characters are deprecated, meaning that you should not use them, even if you are not using the Quorum Server, because they will be illegal in a future Serviceguard release. All other characters are legal. The cluster name can contain up to 39 characters (bytes).
IMPORTANT: See “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 139) for important information. See also the latest Serviceguard release notes at www.hp.com/go/ hpux-serviceguard-docs. FIRST_CLUSTER_LOCK_VG, SECOND_CLUSTER_LOCK_VG The volume group containing the physical disk volume on which a cluster lock is written. This parameter is used only when you employ a lock disk for tie-breaking services in the cluster.
Server” (page 248). See also “Configuring Serviceguard to Use the Quorum Server” in the latest version HP Serviceguard Quorum Server Version A.04.00 Release Notes, at www.hp.com/ go/hpux-serviceguard-docs -> HP Serviceguard Quorum Server Software. IMPORTANT: See also“About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 139) for important information about requirements and restrictions in an IPv6–only cluster.
IMPORTANT: For special instructions that may apply to your version of Serviceguard and the Quorum Server see “Configuring Serviceguard to Use the Quorum Server” in the latest version HP Serviceguard Quorum Server Version A.04.00 Release Notes, at www.hp.com/go/ hpux-serviceguard-docs -> HP Serviceguard Quorum Server Software. See also “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 139) for important information about requirements and restrictions in an IPv6–only cluster.
so until you have read the HP Serviceguard Quorum Server Version A.04.00 Release Notes, and in particular the following sections in that document: “About the QS Polling Interval and Timeout Extension”, “Network Recommendations”, and “Setting Quorum Server Parameters in the Cluster Configuration File”. Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 66) for important information.
NODE_NAME The hostname of each system that will be a node in the cluster. CAUTION: Make sure that the node name is unique within the subnets configured on the cluster nodes; under some circumstances Serviceguard may not be able to detect a duplicate name and unexpected problems may result. Do not use the full domain name. For example, enter ftsys9, not ftsys9.cup.hp.com.
which the node identified by the preceding NODE_NAME entry belongs. Can be used only in a site-aware disaster-tolerant cluster, which requires Metrocluster (additional HP software); see the documents listed under “Cross-Subnet Configurations” (page 41) for more information. If SITE is used, it must be used for each node in the cluster (that is, all the nodes must be associated with some defined site, though not necessarily the same one).
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 168)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 168)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
• • • Two heartbeat subnets; or One heartbeat subnet with a standby; or One heartbeat subnet using APA with two physical ports in hot standby mode or LAN monitor mode. You cannot configure more than one heartbeat IP address on an interface; only one HEARTBEAT_IP is allowed for each NETWORK_INTERFACE. Do not mix APA LAN Monitor or hot standby modes on the same network interfaces that are configured for Serviceguard local LAN failover.
IP addresses for a given heartbeat path are usually on the same subnet on each node, but it is possible to configure the heartbeat on multiple subnets such that the heartbeat is carried on one subnet for one set of nodes and another subnet for others, with the subnets joined by a router. This is called a cross-subnet configuration, and in this case at least two heartbeat paths must be configured for each cluster node.
NOTE: The use of a private heartbeat network is not advisable if you plan to use Remote Procedure Call (RPC) protocols and services. RPC assumes that each network adapter device or I/O card is connected to a route-able network. An isolated or private heartbeat LAN is not route-able, and could cause an RPC request-reply, directed to that LAN, to risk timeout without being serviced. NFS, NIS and NIS+, and CDE are examples of RPC based applications that are frequently used on HP-UX.
monitored non-heartbeat subnets here. You can identify any number of subnets to be monitored. IMPORTANT: In a cross-subnet configuration, each package subnet configured on an interface (NIC) must have a standby interface connected to the local bridged network. See “Cross-Subnet Configurations” (page 41) for important information about requirements for such configurations. If HOSTNAME_ADDRESS_FAMILY is set to IPV4 or ANY, a stationary IP address can be either an IPv4 or an IPv6 address.
FIRST_CLUSTER_LOCK_PV, SECOND_CLUSTER_LOCK_PV The path on this node for the physical volume within the cluster-lock Volume Group that will have the cluster lock written on it (see the entry for FIRST_CLUSTER_LOCK_VG and SECOND_CLUSTER_LOCK_VG near the beginning of this list). Used only if a lock disk is used for tie-breaking services. Use FIRST_CLUSTER_LOCK_PV for the first physical lock volume and SECOND_CLUSTER_LOCK_PV for the second physical lock volume, if any.
CAPACITY_NAME name can be any string that starts and ends with an alphanumeric character, and otherwise contains only alphanumeric characters, dot (.), dash (-), or underscore (_). Maximum length is 39 characters. CAPACITY_NAME must be unique in the cluster. CAPACITY_VALUE specifies a value for the CAPACITY_NAME that precedes it. It must be a floating-point value between 0 and 1000000.
MEMBER_TIMEOUT The amount of time, in microseconds, after which Serviceguard declares that the node has failed and begins re-forming the cluster without this node. Default value: 14 seconds (14,000,000 microseconds). This value leads to a failover time of between approximately 18 and 22 seconds, if you are using a Quorum Server, or a Fiber Channel cluster lock, or no cluster lock. Increasing the value to 25 seconds increases the failover time to between approximately 29 and 39 seconds.
NOTE: • For most clusters that use an LVM cluster lock or lock LUN, a minimum MEMBER_TIMEOUT of 14 seconds is appropriate. • For most clusters that use a MEMBER_TIMEOUT value lower than 14 seconds, a quorum server is more appropriate than a lock disk or lock LUN. The cluster will fail if the time it takes to acquire the disk lock exceeds 0.2 times the MEMBER_TIMEOUT.
spike prevents the node from sending a heartbeat signal within the MEMBER_TIMEOUT value. More than one node could be affected if, for example, a network event such as a broadcast storm caused kernel interrupts to be turned off on some or all nodes while the packets are being processed, preventing the nodes from sending and processing heartbeat messages. See “Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low” (page 415) for troubleshooting information.
Default is 2,000,000 microseconds (2 seconds). This means that the network manager will poll each network interface every 2 seconds, to make sure it can still send and receive information. The minimum value is 1,000,000 (1 second) and the maximum value supported is 30 seconds. IMPORTANT: HP strongly recommends using the default. Changing this value can affect how quickly the link-level and IP-level monitors detect a network failure.
switch and router. The value should be in the vendors' documentation. Set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the values for the switches and routers. If there is more than one possible path between the NFS server and the cluster nodes, sum the values for each path and use the largest number. CAUTION: Serviceguard supports NFS-mounted file systems only over switches and routers that support MBTD.
NETWORK_FAILURE_DETECTION The configuration file specifies one of two ways to decide when a network interface card has failed: • INOUT • INONLY_OR_INOUT The default is INOUT. See “Monitoring LAN Interfaces and Detecting Failure: Link Level” (page 92) for more information. Can be changed while the cluster is running. NETWORK_AUTO_FAILBACK How Serviceguard will handle the recovery of the primary LAN interface after it has failed over to the standby interface because of a link level failure.
IP_MONITOR and POLLING_TARGET entries, if the subnet in question is removed from the cluster configuration. IP_MONITOR Specifies whether or not the subnet specified in the preceding SUBNET entry will be monitored at the IP layer. To enable IP monitoring for the subnet, set IP_MONITOR to ON; to disable it, set it to OFF. By default IP_MONITOR is set to ON if a gateway is detected for the SUBNET in question, and POLLING_TARGET entries are populated with the gateway addresses, enabling target polling.
target polling; see “How the IP Monitor Works” (page 99) for more information. NOTE: cmquerycl (1m) detects first-level routers in the cluster (by looking for gateways in each node's routing table) and lists them here as polling targets. If you run cmquerycl with the -w full option (for full network probing) it will also verify that the gateways will work correctly for monitoring purposes. Can be changed while the cluster is running; must be removed if the preceding SUBNET entry is removed.
IMPORTANT: CAPACITY_NAME, WEIGHT_NAME, and weight_value must all match exactly. NOTE: A weight (WEIGHT_NAME, WEIGHT_DEFAULT) has no meaning on a node unless a corresponding capacity (CAPACITY_NAME, CAPACITY_VALUE) is defined for that node. For the reserved weight and capacity package_limit, the default weight is always one. This default cannot be changed in the cluster configuration file, but it can be overridden for an individual package in the package configuration file.
Package Configuration Planning Planning for packages involves assembling information about each group of highly available services. NOTE: As of Serviceguard A.11.18, there is a new and simpler way to configure packages. This method allows you to build packages out of smaller modules, and eliminates the separate package control script and the need to distribute it manually; see Chapter 6: “Configuring Packages and Their Services ” (page 279) for complete instructions.
As part of planning, you need to decide the following: • • • • • What volume groups are needed? How much disk space is required, and how should this be allocated in logical volumes? What file systems need to be mounted for each package? Which nodes need to import which logical volume configurations? If a package moves to an adoptive node, what effect will its presence have on performance? Create a list by package of volume groups, logical volumes, and file systems.
NOTE: Do not use /etc/fstab to mount file systems that are used by Serviceguard packages. Planning Veritas Cluster Volume Manager (CVM) and Cluster File System (CFS) NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information on support for CVM and CFS: www.hp.com/go/ hpux-serviceguard-docs.
1. The failover package’s applications should not run on a node unless the mount point packages are already running. In the package’s configuration file, you fill out the dependency parameter to specify the requirement SG-CFS-MP-id# =UP on the SAME_NODE. 2. The mount point packages should not run unless the disk group packages are running. Create the mount point packages using the cfsmntadm and cfsmount commands.
About the Volume Monitor Simply monitoring each physical disk in a Serviceguard cluster does not provide adequate monitoring for volumes managed by Veritas Volume Manager from Symantec (VxVM) or logical volumes managed by HP-UX Logical Volume Manager (LVM), because a physical volume failure is not always a critical failure that triggers failover (for example, the failure of a single physical disk within a mirrored volume is not considered critical).
dependencies operate in this fashion – a package will not be run unless the specified EMS resources conditions are met. NOTE: When using the volume monitor to monitor LVM logical volumes, you need to make sure that the logical volume timeout value is properly configured. This value should be configured to be at least one second less than the poll-interval specified in the monitor service command. I/O requests to logical volumes with no timeout set may block indefinitely.
service will terminate after six such consecutive read attempts (a duration of up to six poll intervals). volume_path The full path to at least one VxVM volume or LVM logical volume device file for monitoring (required). The pathname must identify a block device file. Examples /usr/sbin/cmvolmond -O /log/monlog.
The following rules and restrictions apply. • NFS mounts are supported only for modular, failover packages. See Chapter 6 (page 279) for a discussion of types of packages. • So that Serviceguard can ensure that all I/O from a node on which a package has failed is flushed before the package restarts on an adoptive node, all the network switches and routers between the NFS server and client must support a worst-case timeout, after which packets and frames are dropped.
In addition, you should observe the following guidelines. • CacheFS and AutoFS should be disabled on all nodes configured to run a package that uses NFS mounts. For more information, see the NFS Services Administrator's Guide HP-UX 11i version 3 at www.hp.com/go/hpux-networking-docs. • HP recommends that you avoid a single point of failure by ensuring that the NFS server is highly available.
The following table describes different types of failover behavior and the settings in the package configuration file that determine each behavior. See “Package Parameter Explanations” (page 287) for more information. Table 4-2 Package Failover Behavior Switching Behavior Parameters in Configuration File Package switches normally after • node_fail_fast_enabled set to no. (Default) detection of service, network, or EMS • service_fail_fast_enabled set to NO for all services.
Parameters for Configuring EMS Resources NOTE: The default form for parameter names and literal values in the modular package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameters are concerned. This manual uses lower case, unless the parameter in question is used only in legacy packages, or the context refers exclusively to such a package.
NOTE: For a legacy package, specify the deferred resources again in the package control script, using the DEFERRED_RESOURCE_NAME parameter: DEFERRED_RESOURCE_NAME[0]="/net/interfaces/lan/status/lan0" DEFERRED_RESOURCE_NAME[1]="/net/interfaces/lan/status/lan1" If a resource is configured to be AUTOMATIC in a legacy configuration file, you do not need to define DEFERRED_RESOURCE_NAME in the package control script. About Package Dependencies Starting in Serviceguard A.11.
Rules for Simple Dependencies Assume that we want to make pkg1 depend on pkg2. NOTE: pkg1 can depend on more than one other package, and pkg2 can depend on another package or packages; we are assuming only two packages in order to make the rules as clear as possible. • • pkg1 will not start on any node unless pkg2 is running on that node.
• node. This is called dragging and is determined by each package’s priority (page 293). See “Dragging Rules for Simple Dependencies” (page 181). If pkg2 fails, Serviceguard will halt pkg1 and any other packages that depend directly or indirectly on pkg2. By default, Serviceguard halts packages in dependency order, the dependent package(s) first, then the package depended on. In our example, pkg1 would be halted first, then pkg2.
NOTE: Keep the following in mind when reading the examples that follow, and when actually configuring priorities: 1. auto_run (page 289) should be set to yes for all the packages involved; the examples assume that it is. 2. Priorities express a ranking order, so a lower number means a higher priority (10 is a higher priority than 30, for example).
— If both packages have moved from node1 to node2 and node1 becomes available, pkg2 will fail back to node1 only if pkg2’s priority is higher than pkg1’s: ◦ If the priorities are equal, neither package will fail back (unless pkg1 is not running; in that case pkg2 can fail back).
because that provides the best chance for a successful failover (and failback) if pkg1 fails. But you also need to weigh the relative importance of the packages. If pkg2 runs a database that is central to your business, you probably want it to run undisturbed, no matter what happens to application packages that depend on it. In this case, the database package should have the highest priority.
IMPORTANT: If you have not already done so, read the discussion of Simple Dependencies (page 179) before you go on. The interaction of the legal values of dependency_location and dependency_condition creates the following possibilities: • Same-node dependency: a package can require that another package be UP on the same node. This is the case covered in the section on Simple Dependencies (page 179). • • • • Different-node dependency: a package can require that another package be UP on a different node.
Rules for different_node and any_node Dependencies These rules apply to packages whose dependency_condition is UP and whose dependency_location is different_node or any_node. For same-node dependencies, see Simple Dependencies (page 179); for exclusionary dependencies, see “Rules for Exclusionary Dependencies” (page 185). • • • Both packages must be failover packages whose failover_policy (page 292) is configured_node.
will allow the failed package to halt after the successor_halt_timeout number of seconds whether or not the dependent packages have completed their halt scripts. 2. Halts the failing package. After the successor halt timer has expired or the dependent packages have all halted, Serviceguard starts the halt script of the failing package, regardless of whether the dependents' halts succeeded, failed, or timed out. 3.
Package Weights and Node Capacities You define a capacity, or capacities, for a node (in the cluster configuration file), and corresponding weights for packages (in the package configuration file). Node capacity is consumed by package weights. Serviceguard ensures that the capacity limit you set for a node is never exceeded by the combined weight of packages running on it; if a node's available capacity will be exceeded by a package that wants to run on that node, the package will not run there.
CAPACITY_VALUE 10 Now all packages will be considered equal in terms of their resource consumption, and this node will never run more than ten packages at one time. (You can change this behavior if you need to by modifying the weight for some or all packages, as the next example shows.) Next, define the CAPACITY_NAME and CAPACITY_VALUE parameters for the remaining nodes, setting CAPACITY_NAME to package_limit in each case. You may want to set CAPACITY_VALUE to different values for different nodes.
Points to Keep in Mind The following points apply specifically to the Simple Method (page 188). Read them in conjunction with the Rules and Guidelines (page 195), which apply to all weights and capacities. • If you use the reserved CAPACITY_NAME package_limit, then this is the only type of capacity and weight you can define in this cluster. • If you use the reserved CAPACITY_NAME package_limit, the default weight for all packages is 1.
could be misleading to identify single resources, such as “processor”, if packages really contend for sets of interacting resources that are hard to characterize with a single name. In any case, the real-world meanings of the names you assign to node capacities and package weights are outside the scope of Serviceguard. Serviceguard simply ensures that for each capacity configured for a node, the combined weight of packages currently running on that node does not exceed that capacity.
CAPACITY_NAME A CAPACITY_VALUE 80 CAPACITY_NAME B CAPACITY_VALUE 50 NODE_NAME node2 CAPACITY_NAME A CAPACITY_VALUE 60 CAPACITY_NAME B CAPACITY_VALUE 70 ... NOTE: You do not have to define capacities for every node in the cluster. If any capacity is not defined for any node, Serviceguard assumes that node has an infinite amount of that capacity.
Example 3 WEIGHT_NAME A WEIGHT_DEFAULT 20 WEIGHT_NAME B WEIGHT_DEFAULT 15 This means that any package for which weight A is not defined in its package configuration file will have a weight A of 20, and any package for which weight B is not defined in its package configuration file will have a weight B of 15. Given the capacities we defined in the cluster configuration file (see “Defining Capacities”), node1 can run any three packages that use the default for both A and B.
weight_name A weight_value 40 In pkg3's package configuration file: weight_name B weight_value 35 weight_name A weight_value 0 In pkg4's package configuration file: weight_name B weight_value 40 IMPORTANT: weight_name in the package configuration file must exactly match the corresponding CAPACITY_NAME in the cluster configuration file. This applies to case as well as spelling: weight_name a would not match CAPACITY_NAME A.
Rules and Guidelines The following rules and guidelines apply to both the Simple Method (page 188) and the Comprehensive Method (page 190) of configuring capacities and weights. • You can define a maximum of four capacities, and corresponding weights, throughout the cluster. NOTE: But if you use the reserved CAPACITY_NAME package_limit, you can define only that single capacity and corresponding weight. See “Simple Method” (page 188).
For further discussion and use cases, see the white paper Using Serviceguard’s Node Capacity and Package Weight Feature at www.hp.com/go/hpux-serviceguard-docs. How Package Weights Interact with Package Priorities and Dependencies If necessary, Serviceguard will halt a running lower-priority package that has weight to make room for a higher-priority package that has weight.
About External Scripts The package configuration template for modular scripts explicitly provides for external scripts. These replace the CUSTOMER DEFINED FUNCTIONS in legacy scripts, and can be run either: • • On package startup and shutdown, as essentially the first and last functions the package performs.
NOTE: Some variables, including SG_PACKAGE, and SG_NODE, are available only at package run and halt time, not when the package is validated. You can use SG_PACKAGE_NAME at validation time as a substitute for SG_PACKAGE. IMPORTANT: For more information, see the template in $SGCONF/examples/external_script.template. A sample script follows. It assumes there is another script called monitor.sh, which will be configured as a Serviceguard service to monitor some application. The monitor.
do case ${SG_SERVICE_CMD[i]} in *monitor.
Suppose a script run by pkg1 does a cmmodpkg -d of pkg2, and a script run by pkg2 does a cmmodpkg -d of pkg1. If both pkg1 and pkg2 start at the same time, the pkg1 script now tries to cmmodpkg pkg2. But that cmmodpkg command has to wait for pkg2 startup to complete. The pkg2 script tries to cmmodpkg pkg1, but pkg2 has to wait for pkg1 startup to complete, thereby causing a command loop.
NOTE: last_halt_failed appears only in the line output of cmviewcl, not the default tabular format; you must use the -v and -f line options to see it. The value of last_halt_failed is no if the halt script ran successfully, or was not run since the node joined the cluster, or was not run since the package was configured to run on the node; otherwise it is yes.
• • Deploying applications in this environment requires careful consideration; see “Implications for Application Deployment” (page 202). If a monitored_subnet (page 296) is configured for PARTIAL monitored_subnet_access in a package’s configuration file, it must be configured on at least one of the nodes on the node_name list for that package.
Configuring node_name First you need to make sure that pkg1 will fail over to a node on another subnet only if it has to. For example, if it is running on NodeA and needs to fail over, you want it to try NodeB, on the same subnet, before incurring the cross-subnet overhead of failing over to NodeC or NodeD.
ip_address 15.244.56.100 ip_address 15.244.56.101 Configuring a Package: Next Steps When you are ready to start configuring a package, proceed to Chapter 6: “Configuring Packages and Their Services ” (page 279); start with “Choosing Package Modules” (page 280). (If you find it helpful, you can assemble your package configuration data ahead of time on a separate worksheet for each package; blank worksheets are in Appendix E.
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. These procedures are carried out on one node, called the configuration node, and the resulting binary file is distributed by Serviceguard to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
Appendix D (page 447) provides instructions for upgrading Serviceguard without halting the cluster. Make sure you read the entire Appendix, and the corresponding section in the Release Notes, before you begin. Where Serviceguard Files Are Kept Serviceguard uses a special file, /etc/cmcluster.conf, to define the locations for configuration and log files within the HP-UX filesystem. The following locations are defined in the file: ################## cmcluster.
Before You Start Before you start, make sure you have read and understood the restrictions and limitations spelled out under “About Cluster-wide Device Special Files (cDSFs)” (page 135). You should also read the cmsetdsfgroup (1m) manpage. If the cluster does not yet exist, make sure Serviceguard is installed on each prospective node. Creating cDSFs for a Group of Nodes Proceed as follows.
1. Identify the nodes for which you want to create cluster-wide device files; this is known as the cDSF group. This should be all the nodes in the cluster (or prospective cluster). IMPORTANT: Remember that: • You cannot create a cDSF group that crosses cluster boundaries; that is, the group must consist of the nodes of a single cluster. • cDSFs use agile addressing; see “About Device File Names (Device Special Files)” (page 106) for information about agile addressing. 2.
3. Create the cDSFs. NOTE: cDSFs apply only to shared storage; they will not be generated for local storage, such as root, boot, and swap devices. • If the cluster does not exist yet, specify the name of each prospective node, for example: cmsetdsfgroup -n node1 -n node2 -n node3 -n node4 • If the cluster does exist, you can simply run: cmsetdsfgroup -c NOTE: You must be logged in as superuser on one of the cluster nodes. You do not need to provide the cluster name.
Displaying the cDSF Configuration • • To see the members of a cDSF group, log in on one of the members and run cmsetdsfgroup -q To display all the information about the configuration stored in the configuration file /etc/cmcluster/cdsf/cdsf_config, log in on one of the cDSF group members and run cmsetdsfgroup -v -q CAUTION: • • Do not modify cdsf_config. To report information (in line output format only) about DSFs, cDSFs, and volume groups for one or more nodes, use cmquerystg (1m).
• Make sure all the subnets used by the prospective nodes are accessible to all the nodes. Easy Deployment commands will fail if they detect an asymmetric network configuration, in which only a subset of nodes has access to a given subnet. • Make sure that the LAN or LANs that will be used for the cluster heartbeat meet high-availability requirements. See the requirements for HEARTBEAT_IP listed under “Cluster Configuration Parameters ” (page 143), and the networking“Rules and Restrictions” (page 39).
NOTE: The actions taken by the commands you will be using are logged to /var/ adm/cmcluster/sgeasy/easy_deployment.log, providing a useful record of the changes to system files. 1. Obtain networking information for the cluster heartbeat: cmquerycl –N For example: cmquerycl –N $SGCONF/mynetwork 2. Edit the output file ($SGCONF/mynetwork in this example) to add entries for additional heartbeats, if necessary.
fully-qualified domain name (see “Configuring Name Resolution” (page 218) for more information about node names and hostnames). :, if used, must be, respectively, a volume group name that does not already exist, and the name of a physical volume that is unused and is not configured into any volume manager. HP recommends you use cluster-wide device special files (cDSFs); see “About Cluster-wide Device Special Files (cDSFs)” (page 135).
See cmdeploycl (1m) for more information.
5. Create shared storage with PVG-strict mirroring for use by cluster packages. cmpreparestg –l –p ... –n –n is the path name of the volume group that is to be created and imported to each specified node, and each is the name of a physical volume on the local node from which the volume group specified by is to be created.
This command also builds a VxFS file system on the logical volume, and mounts it to /newdir. If the logical volume will be used for raw storage (for example for a database) omit the fsopts and –m options; for example: cmpreparestg –l /dev/vgdatabase –L –o lv_opts=”-L 120 –s g” –n node1 –n node2 The volume group can now be used by packages; see Chapter 6 (page 279). NOTE: cmpreparestg requires -l in all cases. The command must be run on one of the specified nodes (ftsys9 or ftsys10 in this example).
NOTE: cmclnodelist is created automatically by cmpreparecl (1m) (orcmpdeploycl (1m), which calls cmpreparecl). You can skip the rest of this subsection if you have used, or plan to use, either of these commands. See “Using Easy Deployment Commands to Configure the Cluster” (page 211).
on each node you intend to configure into the cluster, and ensure that the entry for the root user comes before any other entry with a UID of 0. NOTE: You need to do this even if you plan to use cmpreparecl (1m) orcmpdeploycl (1m), which calls cmpreparecl. For more information about these commands, see “Using Easy Deployment Commands to Configure the Cluster” (page 211).
NOTE: If you plan to use cmpreparecl (1m) (or cmpdeploycl (1m), which calls cmpreparecl), the /etc/hosts and /etc/nsswitch.conf configuration described in this subsection will be done automatically, but you should still read the entire subsection and make sure you understand the issues. In particular, you still need to make sure that aliases are properly represented in /etc/ hosts, as described below.
15.145.162.131 gryf or : — the first (or only) alias. Examples: 15.145.162.131 gryf.uksr.hp.com gryf 10.8.0.131 gryf.uksr.hp.com gryf 10.8.1.131 gryf.uksr.hp.com gryf 15.145.162.132 sly.uksr.hp.com sly 10.8.0.132 sly.uksr.hp.com sly 10.8.1.132 sly.uksr.hp.com sly 15.145.162.131 gryf.uksr.hp.com 10.8.0.131 gryf2.uksr.hp.com 10.8.1.131 gryf3.uksr.hp.com gryf gryf gryf 15.145.162.132 10.8.0.132 10.8.1.132 sly sly sly sly.uksr.hp.com sly2.uksr.hp.com sly3.uksr.hp.
addresses of all the cluster nodes. If the name service is not available, the command could hang or return an unexpected networking error message. NOTE: If such a hang or error occurs, Serviceguard and all protected applications will continue working even though the command you issued does not. That is, only the Serviceguard configuration commands (and corresponding Serviceguard Manager functions) are affected, not the cluster daemon or package services.
3. Edit or create the /etc/nsswitch.
Tuning Network and Kernel Parameters Serviceguard and its extension products, such as SGeSAP and SGeRAC, have been tested with default values of the network and kernel parameters supported by the ndd and kmtune utilities. You may need to adjust these parameters for larger cluster configurations and applications. • • ndd is the network tuning utility. For more information, see the man page for ndd (1m) kmtune is the system tuning utility. For more information, see the man page for kmtune (1m).
information about this model, see the HP-UX IPSec Version A.03.00 Administrator's Guide which you can find on www.docs.hp.com/en/internet.html.. ip_strong_es_model is disabled (set to 0) by default. Setting it to 1 enables it. CAUTION: HP supports enabling this parameter only if you are not using a cross-subnet configuration (page 41). Otherwise, leave the parameter at its default setting (zero, meaning disabled).
2. Add this disk to the current root volume group. vgextend /dev/vg00 /dev/dsk/c4t6d0 3. Make the new disk a boot disk. mkboot -l /dev/rdsk/c4t6d0 4. Mirror the boot, primary swap, and root logical volumes to the new bootable disk. Ensure that all devices in vg00, such as those for /usr, /swap, are mirrored.
The cluster lock disk is configured on an LVM volume group that is physically connected to all cluster nodes. This volume group may also contain data that is used by packages. When you are using dual cluster lock disks, it is required that the default I/O timeout values are used for the cluster lock physical volumes.
• • • • A lock LUN cannot also be used in an LVM physical volume or VxVM or CVM disk group. A lock LUN cannot be shared by more than one cluster. A lock LUN cannot be used in a dual-lock configuration. You do not need to back up the lock LUN data, and in fact there is no way to do so. A lock LUN needs only a small amount of storage, about 100 KB.
3. Create the partition; for example (using partition.txt as input): /usr/sbin/idisk -w -p -f partition.txt /dev/rdsk/c1t4d0 Or, on an HP-UX 11i v3 system using agile addressing (see “About Device File Names (Device Special Files)” (page 106): /usr/sbin/idisk -w -p -f partition.txt /dev/rdisk/disk12 NOTE: Device files for partitions cannot be cluster-wide DSFs (cDSFs). For more information about cDSFs, see “About Cluster-wide Device Special Files (cDSFs)” (page 135).
These commands create a configuration file which you can apply to the cluster configuration when you are ready to do so; see “Distributing the Binary Configuration File ” (page 259). See also “Specifying a Lock LUN” (page 247). CAUTION: Once you have specified the lock LUN in the cluster configuration file, running cmapplyconf (1m) or cmdeploycl (1m) will destroy any data on the LUN.
IMPORTANT: The purpose of cmnotdisk.conf is to exclude specific devices, usually CD and DVD drives, that Serviceguard does not recognize and which should not be probed. Make sure you do not add a DEVICE_FILE entry in cmnotdisk.conf for any device that should be probed; that is, disk devices being managed by LVM or LVM2. Excluding any such device will cause cmquerycl to fail.
NOTE: If you are configuring volume groups that use mass storage on HP’s HA disk arrays, you should use redundant I/O channels from each node, connecting them to separate ports on the array. As of HP-UX 11i v3, the I/O subsystem performs load balancing and multipathing automatically. Creating a Storage Infrastructure with LVM This section describes how to configure disk storage using HP's Logical Volume Manager (LVM).
NOTE: The procedures that follow assume you are using LVM rather than LVM 2.0. For information about LVM 2.0 specifically, see the white paper LVM 2.0 Volume Groups in HP-UX 11i v3, which you can find under www.hp.com/go/hpux-core-docs. For more information on using LVM in general, including LVM 2.0, see the Logical Volume Management volume of the HP-UX System Administrator’s Guide at www.hp.com/go/hpux-core-docs.
NOTE: Under agile addressing, the physical devices in these examples would have names such as /dev/rdisk/disk1 and /dev/rdisk/disk2. See “About Device File Names (Device Special Files)” (page 106). If you are using cDSFs, the device files would have names such as /dev/rcdisk/ disk1 and /dev/rcdisk/disk2. See “About Cluster-wide Device Special Files (cDSFs)” (page 135). On the configuration node (ftsys9), use the pvcreate(1m) command to define disks as physical volumes.
vgcreate -g bus0 /dev/vgdatabase /dev/dsk/c1t2d0 vgextend -g bus1 /dev/vgdatabase /dev/dsk/c0t2d0 CAUTION: Volume groups used by Serviceguard must have names no longer than 35 characters (that is, the name that follows /dev/, in this example vgdatabase, must be at most 35 characters long). NOTE: If you are using cDSFs, you should be using them exclusively. The first command creates the volume group and adds a physical volume to it in a physical volume group called bus0.
TIP: Set the logical volume timeout to an integral multiple of any timeout assigned to the underlying physical volumes. Otherwise, the actual duration of the I/O request can exceed the logical volume timeout. For details on how to change the I/O timeout value on a physical volume, see the manpage for pvchange (1m). Creating File Systems If your installation uses filesystems, create them next. NOTE: You can create filesystems by means of the cmpreparestg (1m) command.
NOTE: You can distribute the volume groups by means of the cmpreparestg (1m) command. See “Using Easy Deployment Commands to Configure the Cluster” (page 211) for more information. If you use cmpreparestg, you can skip to “Making Physical Volume Group Files Consistent” (page 238). Deactivating the Volume Group NOTE: If you plan to use cmpreparestg, you can skip this step and proceed to “Making Physical Volume Group Files Consistent” (page 238).
4. Still on ftsys10, create a control file named group in the directory /dev/ vgdatabase, as follows: mknod /dev/vgdatabase/group c 64 0xhh0000 Use the same minor number as on ftsys9. Use the following command to display a list of existing volume groups: ls -l /dev/*/group 5. Import the volume group data using the map file from node ftsys9. On node ftsys10, enter: vgimport -s -m /tmp/vgdatabase.map /dev/vgdatabase Note that the disk device names onftsys10 may be different from their names on ftsys9.
mount /dev/vgdatabase/lvol1 /mnt1 10. Unmount the volume group on ftsys10: umount /mnt1 11. Deactivate the volume group on ftsys10: vgchange -a n /dev/vgdatabase Making Physical Volume Group Files Consistent Skip this section if you do not use physical volume groups for mirrored individual disks in your disk configuration, or if you are using cDSFs; see “About Cluster-wide Device Special Files (cDSFs)” (page 135) for more information about cDSFs.
Creating a Storage Infrastructure with VxVM In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM), Veritas Volume Manager (VxVM), or Veritas Cluster Volume Manager (CVM). You can also use a mixture of volume types, depending on your needs. LVM and VxVM configuration are done before cluster configuration, and CVM configuration is done after cluster configuration.
NOTE: These commands make the disk and its data unusable by LVM, and allow it to be initialized by VxVM. (The commands should only be used if you have previously used the disk with LVM and do not want to save the data on it.
Creating Volumes Use the vxassist command to create logical volumes. The following is an example: vxassist -g logdata make log_files 1024m This command creates a 1024 MB volume named log_files in a disk group named logdata. The volume can be referenced with the block device file /dev/vx/dsk/logdata/log_files or the raw (character) device file /dev/vx/rdsk/logdata/log_files.
When all disk groups have been deported, you must issue the following command on all cluster nodes to allow them to access the disk groups: vxdctl enable Re-Importing Disk Groups After deporting disk groups, they are not available for use on the node until they are imported again either by a package control script or with a vxdg import command.
NOTE: You can use Serviceguard Manager to configure a cluster: open the System Management Homepage (SMH) and choose Tools-> Serviceguard Manager. See “Using Serviceguard Manager” (page 32) for more information. You can also use Easy Deployment commands; see“About Easy Deployment” (page 137) and “Using Easy Deployment Commands to Configure the Cluster” (page 211). To use traditional Serviceguard commands to configure the cluster, follow directions in the remainder of this section.
(Do not use -w local if you need to discover nodes and subnets for a cross-subnet configuration; see “Full Network Probing” below.) -w none skips network querying. If you have recently checked the networks, this option will save time. Specifying the Address Family for the Cluster Hostnames You can use the -a option to tell Serviceguard to resolve cluster node names (as well as Quorum Server hostnames, if any) to IPv4 addresses only (-a ipv4) IPv6 addresses only (-a ipv6), or both (-a any).
cmquerycl -v -h ipv6 -C $SGCONF/clust1.conf -n ftsys9 -n ftsys10 • • • • -h ipv4 tells Serviceguard to discover and configure only IPv4 subnets. If it does not find any eligible subnets, the command will fail. -h ipv6 tells Serviceguard to discover and configure only IPv6 subnets. If it does not find any eligible subnets, the command will fail.
Full Network Probing -w full lets you specify full network probing, in which actual connectivity is verified among all LAN interfaces on all nodes in the cluster, whether or not they are all on the same subnet. NOTE: This option must be used to discover actual or potential nodes and subnets in a cross-subnet configuration. See “Obtaining Cross-Subnet Information” (page 248).
NOTE: You should not configure a second lock volume group or physical volume unless your configuration specifically requires it. See “Dual Lock Disk” (page 64).
Specifying a Quorum Server IMPORTANT: The following are standard instructions. For special instructions that may apply to your version of Serviceguard and the Quorum Server see “Configuring Serviceguard to Use the Quorum Server” in the latest version of the HP Serviceguard Quorum Server Version A.04.00 Release Notes, at www.hp.com/go/ hpux-serviceguard-docs -> HP Quorum Server Software. A cluster lock LUN or Quorum Server is required for two-node clusters.
Node Names: nodeA nodeB nodeC nodeD Bridged networks (full probing performed): 1 lan3 lan4 lan3 lan4 (nodeA) (nodeA) (nodeB) (nodeB) 2 lan1 lan1 (nodeA) (nodeB) 3 lan2 lan2 (nodeA) (nodeB) 4 lan3 lan4 lan3 lan4 lan1 lan1 (nodeC) (nodeC) (nodeD) (nodeD) (nodeC) (nodeD) lan2 lan2 (nodeC) (nodeD) 5 6 IP subnets: IPv4: 15.13.164.0 lan1 lan1 (nodeA) (nodeB) 15.13.172.0 lan1 lan1 (nodeC) (nodeD) 15.13.165.0 lan2 lan2 (nodeA) (nodeB) 15.13.182.0 lan2 lan2 (nodeC) (nodeD) 15.244.65.
3ffe:1111::/64 lan3 lan3 (nodeA) (nodeB) 3ffe:2222::/64 lan3 lan3 (nodeC) (nodeD) Possible Heartbeat IPs: 15.13.164.0 15.13.164.1 15.13.164.2 (nodeA) (nodeB) 15.13.172.0 15.13.172.158 15.13.172.159 (nodeC) (nodeD) 15.13.165.0 15.13.165.1 15.13.165.2 (nodeA) (nodeB) 15.13.182.0 15.13.182.158 15.13.182.159 (nodeC) (nodeD) Route connectivity(full probing performed): 1 15.13.164.0 15.13.172.0 2 15.13.165.0 15.13.182.0 3 15.244.65.0 4 15.244.56.
Identifying Heartbeat Subnets The cluster configuration file includes entries for IP addresses on the heartbeat subnet. HP recommends that you use a dedicated heartbeat subnet, and configure heartbeat on other subnets as well, including the data subnet. The heartbeat can be configured on an IPv4 or IPv6 subnet; see “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 139). The heartbeat can comprise multiple subnets joined by a router.
A Note about Terminology Although you will also sometimes see the term role-based access (RBA) in the output of Serviceguard commands, the preferred set of terms, always used in this manual, is as follows: • Access-control policies- the set of rules defining user access to the cluster. — Access-control policy - one of these rules, comprising the three parameters USER_NAME, USER_HOST, USER_ROLE. See “Setting up Access-Control Policies” (page 254).
Figure 5-1 Access Roles Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 5-1 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
IMPORTANT: Users on systems outside the cluster can gain Serviceguard root access privileges to configure the cluster only via a secure connection (rsh or ssh). • Non-root access: Other users can be assigned one of four roles: — Full Admin: Allowed to perform cluster administration, package administration, and cluster and package view operations. These users can administer the cluster, but cannot configure or create a cluster. Full Admin includes the privileges of the Package Admin role.
NOTE: For more information and advice, see the white paper Securing Serviceguard at www.hp.com/go/hpux-serviceguard-docs. Define access-control policies for a cluster in the cluster configuration file; see “Cluster Configuration Parameters ” (page 143). You can define up to 200 access policies for each cluster. A root user can create or modify access control policies while the cluster is running.
NOTE: If you set USER_HOST to ANY_SERVICEGUARD_NODE, set USER_ROLE to MONITOR; users connecting from outside the cluster cannot have any higher privileges (unless they are connecting via rsh or ssh; this is treated as a local connection). Depending on your network configuration, ANY_SERVICEGUARD_NODE can provide wide-ranging read-only access to the cluster.
Plan the cluster’s roles and validate them as soon as possible. If your organization’s security policies allow it, you may find it easiest to create group logins. For example, you could create a MONITOR role for user operator1 from ANY_CLUSTER_NODE. Then you could give this login name and password to everyone who will need to monitor your clusters.
NOTE: Check spelling especially carefully when typing wildcards, such as ANY_USER and ANY_SERVICEGUARD_NODE. If they are misspelled, Serviceguard will assume they are specific users or nodes.
• • • • • • • • • • • • That all nodes specified are in the same heartbeat subnet, except in cross-subnet configurations(page 41) . That the configuration filename is correcy. That all nodes are reachable. That no more than one CLUSTER_NAME and AUTO_START_TIMEOUT are specified. That the value for package run and halt script timeouts is less than 4294 seconds. That the value for AUTO_START_TIMEOUT variables is >=0. That the heartbeat network minimum requirement is met.
cmapplyconf -k -v -C /etc/cmcluster/clust1.conf or cmapplyconf -k -v -C /etc/cmcluster/clust1.ascii NOTE: Using the -k option means that cmapplyconf only checks disk connectivity to the LVM disks that are identified in the ASCII file. Omitting the -k option (the default behavior) means that cmapplyconf tests the connectivity of all LVM disks on all nodes. Using -k can result in significantly faster operation of the command. • Deactivate the cluster lock volume group.
NOTE: You must use the vgcfgbackup command to store a copy of the cluster lock disk's configuration data whether you created the volume group using the System Management Homepage (SMH), SAM, or HP-UX commands. If the cluster lock disk ever needs to be replaced while the cluster is running, you must use the vgcfgrestore command to restore lock information to the replacement disk.
The package for CVM 4.1 and later has the following responsibilities: • • • Maintain Veritas configuration files /etc/llttab, /etc/llthosts, /etc/ gabtab Launch required services: cmvxd, cmvxpingd, vxfsckd Start/halt Veritas processes in the proper order: llt, gab, vxfen, odm, cvm, cfs NOTE: Do not edit the configuration file SG-CFS-pkg.conf. Create and modify configuration using the cfs administration commands. 1. First, make sure the cluster is running: cmviewcl 2. If it is not, start it: cmruncl 3.
NOTE: Because the CVM system multi-node package automatically starts up the Veritas processes, do not edit /etc/llthosts, /etc/llttab, or /etc/gabtab. Creating the Disk Groups Initialize the disk group from the master node. 1. 2. Find the master node using vxdctl or cfscluster status Initialize a new disk group, or import an existing disk group, in shared mode, using the vxdg command.
4. To verify, you can use cfsdgadm or cmviewcl. This example shows the cfsdgadm output: cfsdgadm display -v logdata NODE NAME ACTIVATION MODE ftsys9 sw (sw) MOUNT POINT SHARED VOLUME ftsys10 sw (sw) MOUNT POINT SHARED VOLUME 5. TYPE TYPE To view the name of the package that is monitoring a disk group, use the cfsdgadm show_package command: cfsdgadm show_package logdata SG-CFS-DG-1 Creating Volumes 1. Make log_files volume on the logdata disk group: vxassist -g logdata make log_files 1024m 2.
Cluster Configuration for Node: ftsys10 MOUNT POINT TYPE SHARED VOLUME /tmp/logdata/log_files regular log_files 4. DISK GROUP logdata STATUS NOT MOUNTED Mount the filesystem: cfsmount /tmp/logdata/log_files This starts up the multi-node package and mounts a cluster-wide filesystem. 5.
significantly reduce I/O overhead by identifying and maintaining only the filesystem blocks that have changed since the last storage checkpoint or backup. This is done by a copy-on-write technique. Unlike a disk-based mirroring technology, which requires a separate storage space, this Veritas technology minimizes the use of disk space by creating a storage checkpoint within the same free space available to the filesystem.
SG-CFS-MP-1 SG-CFS-CK-1 up up running running enabled disabled no no /tmp/check_logfiles now contains a point-in-time view of /tmp/logdata/ log_files, and it is persistent.
CLUSTER cfs-cluster STATUS up NODE STATUS ftsys9 up ftsys10 up MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-SN-1 STATUS up up up up STATE running running STATE running running running running AUTO_RUN enabled enabled enabled disabled SYSTEM yes no no no The snapshot file system /local/snap1 is now mounted and provides a point in time view of /tmp/logdata/log_files.
NOTE: You must use CFS if you want to configure file systems on your CVM disk groups; see “Creating a Storage Infrastructure with Veritas Cluster File System (CFS)” (page 261). The two sets of procedures — with and without CFS — use many of the same commands, but in a slightly different order. Before you start, make sure the directory in which VxVM commands are stored, /usr/ lib/vxvm/bin, is in your path.
NOTE: Cluster configuration is described under “Configuring the Cluster ” (page 242). Check the heartbeat configuration. CVM 4.1 and later versions require that the cluster have either multiple heartbeats or a single heartbeat with a standby, and do not allow the use of Auto Port Aggregation, Infiniband, or VLAN interfaces as a heartbeat subnet. The CVM cluster volumes are managed by a Serviceguard-supplied system multi-node package which runs on all nodes at once, and cannot fail over. For CVM 4.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: /usr/lib/vxvm/bin/vxdisksetup -i c4t3d4 Creating Disk Groups Use the following steps to create disk groups. 1. Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: vxdg -s init logdata c0t3d2 2.
This command creates a 1024 MB volume named log_files in a disk group named logdata. The volume can be referenced with the block device file /dev/vx/dsk/ logdata/log_files or the raw (character) device file /dev/vx/rdsk/logdata/ log_files. Verify the configuration with the following command: vxdg list Now deport the disk groups: vxdg deport Adding Disk Groups to the Package Configuration Next you need to specify the CVM disk groups for each package that uses them.
Checking Cluster Operation with Serviceguard Commands Serviceguard also provides several commands for control of the cluster: • • • • • • cmviewcl checks status of the cluster and many of its components. A non-root user with the role of Monitor can run this command from a cluster node or see status information in Serviceguard Manager.
• • 4. Start the node. You can use Serviceguard Manager or use the cmrunnode command. To verify that the node has returned to operation, check in Serviceguard Manager, or use the cmviewcl command again. Bring down the cluster. You can do this in Serviceguard Manager, or use the cmhaltcl -v -f command. Additional cluster testing is described in See “Troubleshooting Your Cluster” (page 399). Refer to Appendix A for a complete list of Serviceguard commands.
AUTO_START_TIMEOUT period. If neither of the other two cases becomes true in that time, startup will fail. To enable automatic cluster start, set the flag AUTOSTART_CMCLD to 1 in /etc/ rc.config.d/cmclusteron each node in the cluster; the nodes will then join the cluster at boot time. Here is an example of the /etc/rc.config.d/cmcluster file: #************************ CMCLUSTER ************************ # Highly Available Cluster configuration # # @(#) $Revision: 72.
You still need to have redundant networks, but you do not need to specify any heartbeat LANs, since there is no other node to send heartbeats to. In the cluster configuration file, specify all the LANs that you want Serviceguard to monitor. Use the STATIONARY_IP parameter, rather than HEARTBEAT_IP, to specify LANs that already have IP addresses. For standby LANs, all that is required is the NETWORK_INTERFACE parameter with the LAN device name.
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmomd -i -f /var/opt/cmom/cmomd.log -r /var/opt/cmom 3. Restart inetd: /etc/init.d/inetd restart Deleting the Cluster Configuration As root user, you can delete a cluster configuration from all cluster nodes by using Serviceguard Manager or the command line. The cmdeleteconf command prompts for a verification before deleting the files unless you use the -f option. You can delete the configuration only when the cluster is down.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. See “What is Serviceguard? ” (page 29), “How the Package Manager Works” (page 67), and “Package Configuration Planning ” (page 168) for more information.
NOTE: This is a new process for configuring packages, as of Serviceguard A.11.18. This manual refers to packages created by this method as modular packages, and assumes that you will use it to create new packages; it is simpler and more efficient than the older method, allowing you to build packages from smaller modules, and eliminating the separate package control script and the need to distribute it manually. Packages created using Serviceguard A.11.17 or earlier are referred to as legacy packages.
them, and then start them up on another node selected from the package’s configuration list; see “node_name” (page 289). To generate a package configuration file that creates a failover package, include-m sg/failover on the cmmakepkg command line. See “Generating the Package Configuration File” (page 311). • Multi-node packages. These packages run simultaneously on more than one node in the cluster.
NOTE: On systems that support CFS, you configure the CFS system multi-node package by means of the cfscluster command, not by editing a package configuration file; see “Creating a Storage Infrastructure with Veritas Cluster File System (CFS)” (page 261). NOTE: The following parameters cannot be configured for multi-node or system multi-node packages: • failover_policy • failback_policy • ip_subnet • ip_address Volume groups configured for packages of these types must be activated in shared mode.
start on the first eligible node on which an instance of the multi-node package comes up; this may not be the dependent packages’ primary node. To ensure that dependent failover packages restart on their primary node if the multi-node packages they depend on need to be restarted, make sure the dependent packages’ package switching is not re-enabled before the multi-node packages are restarted.
Table 6-1 Base Modules Module Name Parameters (page) Comments failover package_name (page 288) * module_name (page 288) * module_version (page 288) * package_type (page 289) package_description (page 289) * node_name (page 289) auto_run (page 289) node_fail_fast_enabled (page 290) run_script_timeout (page 290) halt_script_timeout (page 291) successor_halt_timeout (page 291) * script_log_file (page 292) operation_sequence (page 292) * log_level (page 292) * failover_policy (page 292) failback_policy (pag
Table 6-2 Optional Modules Module Name Parameters (page) Comments dependency dependency_name (page 294) * dependency_condition (page 294) * dependency_location (page 295) * Add to a base module to create a package that depends on one or more other packages. weight weight_name (page 295) * weight value (page 295) * Add to a base module to create a package that has weight that will be counted against a node's capacity.
Table 6-2 Optional Modules (continued) 286 Module Name Parameters (page) Comments filesystem concurrent_fsck_operations (page 306) (S) concurrent_mount_and_umount_operations (page 306) (S) fs_mount_retry_count (page 307) (S) fs_umount_retry_count (page 307) * (S) fs_name (page 307) * (S) fs_directory (page 308) * (S) fs_type (page 308) (S) fs_mount_opt (page 308) (S) fs_umount_opt (page 309) (S) fs_fsck_opt (page 309) (S) Add to a base module to configure filesystem options for the package.
Table 6-2 Optional Modules (continued) Module Name Parameters (page) Comments multi_node_all all parameters that can be used by a multi-node package; includes multi_node, dependency, monitor_subnet, service, resource, volume_group, filesystem, pev, external_pre, external, and acp modules. Use if you are creating a multi-node package that requires most or all of the optional parameters that are available for this type of package.
NOTE: For more information, see the comments in the editable configuration file output by the cmmakepkg command, and the cmmakepkg manpage.
package_type The type can be failover, multi_node, or system_multi_node. Default is failover. You can configure only failover or multi-node packages; see “Types of Package: Failover, Multi-Node, System Multi-Node” (page 280). Packages of one type cannot include the base module for another; for example, if package_type is failover, the package cannot include the multi_node, or system_multi_node module. package_description The application that the package runs.
For failover packages, yes allows Serviceguard to start the package (on the first available node listed under node_name) on cluster start-up, and to automatically restart it on an adoptive node if it fails. no prevents Serviceguard from automatically starting the package, and from restarting it on another node. This is also referred to as package switching, and can be enabled or disabled while the package is running, by means of the cmmodpkg (1m) command.
If no timeout is specified (no_timeout), Serviceguard will wait indefinitely for the package to start. If a timeout occurs: • • Switching will be disabled. The current node will be disabled from running the package. NOTE: VxVM disk groups are imported at package run time and exported at package halt time. If a package uses a large number of VxVM disk, the timeout value must be large enough to allow all of them to finish the import or export.
New as of A.11.18 (for both modular and legacy packages). See also “About Package Dependencies” (page 179). script_log_file The full pathname of the package’s log file. The default is $SGRUN/log/.log. (See “Where Serviceguard Files Are Kept” (page 206) for more information about Serviceguard pathnames.) See also log_level (page 292). operation_sequence Defines the order in which the scripts defined by the package’s component modules will start up.
• Metrocluster (additional HP software); see the documents listed under “Cross-Subnet Configurations” (page 41) for more information. site_preferred_manual means Serviceguard will try to fail the package over to a node on the local SITE. If there are no eligible nodes on the local SITE, the package will halt with global switching enabled. You can then restart the package locally, when a local node is available, or start it on another SITE.
dependency_name An identifier for a particular dependency that must be met in order for this package to run (or keep running). It must be unique among this package's dependency_names. The length and formal restrictions for the name are the same as for package_name (page 288). IMPORTANT: Restrictions on dependency names in previous Serviceguard releases were less stringent.
multi-node package, or a failover package whose failover_policy is configured_node. • DOWN means that this package requires the package identified by to be down (that is, the status reported by cmviewcl is UP). This is known as an exclusionary dependency (or exclusion dependency). If you specify DOWN: — The exclusionary dependency must be mutual: only one of the packages can be running at any given time.
Both parameters are optional, but if weight_value is specified, weight_name must also be specified, and must come first. You can define up to four weights, corresponding to four different capacities, per cluster. To specify more than one weight for this package, repeat weight_name and weight_value. NOTE: But if weight_name is package_limit, you can use only that one weight and capacity throughout the cluster. package_limit is a reserved value, which, if used, must be entered exactly in that form.
will not run. (For cross-subnet configurations, in which a subnet may be configured on some nodes and not on others, see monitored_subnet_access below, ip_subnet_node (page 299), and “About Cross-Subnet Failover” (page 201).
ip_subnet Specifies an IP subnet used by the package for relocatable addresses; see also ip_address (page 299) and “Stationary and Relocatable IP Addresses ” (page 90). Replaces SUBNET, which is still supported in the package control script for legacy packages. CAUTION: HP recommends that this subnet be configured into the cluster.
In a cross-subnet configuration, you also need to specify which nodes the subnet is configured on; see ip_subnet_node below. See also monitored_subnet_access (page 297) and “About Cross-Subnet Failover” (page 201). This parameter can be set for failover packages only. Can be added or deleted while the package is running. ip_subnet_node In a cross-subnet configuration, specifies which nodes an ip_subnet is configured on.
IMPORTANT: Restrictions on service names in previous Serviceguard releases were less stringent. Packages that specify services whose names do not conform to the above rules will continue to run, but if you reconfigure them, you will need to change the name; cmcheckconf and cmapplyconf will enforce the new rules. Each service is defined by five parameters: service_name, service_cmd, service_restart, service_fail_fast_enabled, and service_halt_timeout. See the descriptions that follow.
service_restart The number of times Serviceguard will attempt to re-run the service_cmd. Valid values are unlimited, none or any positive integer value. Default is none. If the value is unlimited, the service will be restarted an infinite number of times. If the value is none, the service will not be restarted. This parameter is in the package control script for legacy packages.
resource_polling_interval How often, in seconds, the resource identified by resource_name (page 301) is to be monitored. Default is 60 seconds. The minimum value is 1. (There is no practical maximum.) resource_start Specifies when Serviceguard will begin monitoring the resource identified by resource_name. Valid values are automatic and deferred. automatic means Serviceguard will begin monitoring the resource as soon as the node joins the cluster.
NOTE: If you set concurrent_vgchange_operations to a value greater than 1, you may see messages such as this in the package log file: Cannot lock “/etc/lvmconf//lvm_lock” still trying...” This is an informational message that can be safely ignored. enable_threaded_vgchange Indicates whether multi-threaded activation of volume groups (vgchange -T)is enabled. New for modular packages. Available on HP-UX 11i v3 only. Legal values are zero (disabled) or 1 (enabled). The default is zero.
IMPORTANT: Volume groups for multi-node and system multi-node packages must be activated in shared mode: vgchange -a s, which is only available if the add-on product Serviceguard Extension for Real Application Cluster (SGeRAC) is installed. (See the latest version of Using Serviceguard Extension for RAC at www.hp.com/go/hpux-serviceguard-docs for more information.) Shared LVM volume groups must not contain a file system.
NOTE: A volume group must be configured into the cluster (via the VOLUME_GROUP parameter) before you can configure it into a modular package. For more information about cluster parameters, see “Cluster Configuration Parameters ” (page 143). cvm_dg Specifies a CVM disk group (one per cvm_dg, each on a new line) used by this package. CVM disk groups must be specified whether file systems need to be mounted on them or not.
File system parameters A package can activate one or more storage groups on startup, and mount logical volumes to file systems. At halt time, the package script unmounts the file systems and deactivates each storage group. All storage groups must be accessible on each target node. (CVM disk groups must be accessible on all nodes in the cluster).
fs_mount_retry_count The number of mount retries for each file system. Legal value is zero or any greater number. The default is zero. If the mount point is busy at package startup and fs_mount_retry_count is set to zero, package startup will fail. If the mount point is busy and fs_mount_retry_count is greater than zero, the startup script will attempt to kill the user process responsible for the busy mount point (fuser -ku) and then try to mount the file system again.
NOTE: A volume group must be defined in this file, using vg (page 304), for each logical volume specified by an fs_name entry. See “File system parameters” (page 306), and the comments in the FILESYSTEMS section of the configuration file, for more information and examples. See also “Logical Volume and File System Planning ” (page 168), and the mount (1m) and mount_nfs (1m) manpages. fs_server The name or IP address (IPv4 or IPv6) of the NFS server for an NFS-imported filesystem.
fs_umount_opt The umount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. Using the-s option of umount will improve startup performance if the package uses a large number of file systems. See the mount (1m) manpage for more information. fs_fsck_opt The fsck options for the file system specified by fs_name.
The script is executed on package startup after volume groups and file systems are activated and IP addresses are assigned, but before services are started; and during package shutdown after services are halted but before IP addresses are removed and volume groups and file systems deactivated. If more than one external_script is specified, the scripts will be executed on package startup in the order they are entered into this file, and in the reverse order during package shutdown.
Additional Parameters Used Only by Legacy Packages IMPORTANT: The following parameters are used only by legacy packages. Do not try to use them in modular packages. See “Configuring a Legacy Package” (page 375) for more information. PATH Specifies the path to be used by the script. SUBNET Specifies the IP subnets that are to be monitored for the package. RUN_SCRIPT and HALT_SCRIPT Use the full pathname of each script.
cmmakepkg Examples The cmmakepkg command generates a package configuration file. Some examples follow; see the cmmakepkg (1m) manpage for complete information. All the examples create an editable configuration file pkg1.conf in the $SGCONF/pkg1 directory. NOTE: If you do not include a base module (or default or all) on the cmmakepkg command line, cmmakepkg will ignore the modules you specify and generate a default configuration file containing all the parameters.
NOTE: You can add more than one module at a time. Next Step The next step is to edit the configuration file you have generated; see “Editing the Configuration File” (page 313). Editing the Configuration File When you have generated the configuration file that contains the modules your package needs (see “Generating the Package Configuration File” (page 311)), you need to edit the file to set the package parameters to the values that will make the package function as you intend.
NOTE: Optional parameters are commented out in the configuration file (with a # at the beginning of the line). In some cases these parameters have default values that will take effect unless you uncomment the parameter (remove the #) and enter a valid value different from the default. Read the surrounding comments in the file, and the explanations in this chapter, to make sure you understand the implications both of accepting and of changing a given default.
• If this package will depend on another package or packages, enter values for dependency_name, (page 294)dependency_condition, dependency_location, and optionally priority. See “About Package Dependencies” (page 179) for more information. NOTE: The package(s) this package depends on must already be part of the cluster configuration by the time you validate this package (via cmcheckconf(1m); see “Verifying and Applying the Package Configuration” (page 317)); otherwise validation will fail.
• To configure the package to monitor a registered EMS resource, enter values for the following parameters (page 301): — resource_name — resource_polling_interval — resource_up_value — resource_start See “Parameters for Configuring EMS Resources” (page 178) for more information and an example.
• If your package uses a large number of volume groups or disk groups, or mounts a large number of file systems, consider increasing the values of the following parameters: — concurrent_vgchange_operations (page 302) — concurrent_fsck_operations (page 306) — concurrent_mount_and_umount_operations (page 306) You can also use the fsck_opt and fs_umount_opt parameters (page 309) to specify the -s option of the fsck and mount/umount commands.
cmcheckconf -v -P $SGCONF/pkg1/pkg1.conf Errors are displayed on the standard output. If necessary, re-edit the file to correct any errors, then run cmcheckconf again until it completes without errors. The following items are checked: • • • • • • • The package name is valid, and at least one node_name entry is included. There are no duplicate parameter entries (except as permitted for multiple volume groups, etc.) Values for all parameters are within permitted ranges.
import and deport these disk groups. (For details on importing and deporting disk groups, refer to the discussion of the import and deport options in the vxdg man page.) The control script imports disk groups using the vxdg command with the -tfC options. The -t option specifies that the disk is imported with the noautoimport flag, which means that the disk will not be automatically re-imported at boot time.
7 Cluster and Package Maintenance This chapter describes how to see cluster configuration and status information, how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
TIP: Some commands take longer to complete in large configurations. In particular, you can expect Serviceguard’s CPU usage to increase during cmviewcl -v as the number of packages and services increases. You can also specify that the output should be formatted as it was in a specific earlier release by using the -r option to specify the release format you want, for example: cmviewcl -r A.11.16 (See the cmviewcl(1m) manpage for the supported release formats.
Node Status and State The status of a node is either up (as an active member of the cluster) or down (inactive in the cluster), depending on whether its cluster daemon is running or not. Note that a node might be down from the cluster perspective, but still up and running HP-UX. A node may also be in one of the following states: • • • • • Failed. Active members of the cluster will see a node in this state if that node was active in a cluster, but is no longer, and is not Halted. Reforming.
• • • reconfiguring — The node where this package is running is adjusting the package configuration to reflect the latest changes that have been applied. reconfigure_wait — The node where this package is running is waiting to adjust the package configuration to reflect the latest changes that have been applied. unknown — Serviceguard could not determine the status at the time cmviewcl was run. A system multi-node package is up when it is running on all the active cluster nodes.
The following states are possible only for multi-node packages: • • blocked — The package has never run on this node, either because a dependency has not been met, or because auto_run is set to no. changing — The package is in a transient state, different from the status shown, on some nodes. For example, a status of starting with a state of changing would mean that the package was starting on at least one node, but in some other, transitory condition (for example, failing) on at least one other node.
Failover and Failback Policies Failover packages can be configured with one of two values for the failover_policy parameter (page 292), as displayed in the output of cmviewcl -v: • • configured_node. The package fails over to the next node in the node_name list in the package configuration file (page 289). min_package_node. The package fails over to the node in the cluster that has the fewest running packages.
NODE ftsys10 STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 28.1 32.1 NAME lan0 lan1 PACKAGE pkg2 STATE running AUTO_RUN enabled STATUS up NODE ftsys10 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled RESTARTS 0 0 NAME ftsys10 ftsys9 NAME service2 15.
Quorum Server Status: NAME STATUS lp-qs up STATE running CFS Package Status If the cluster is using the Veritas Cluster File System (CFS), the system multi-node package SG-CFS-pkg must be running on all active nodes, and the multi-node packages for disk group and mount point must also be running on at least one of their configured nodes. If the cluster is using the Veritas Cluster Volume Manager for raw access to disk groups (that is, without CFS), SG-CFS-pkg must be running on all active nodes.
Status After Halting a Package After we halt the failover package pkg2 with the cmhaltpkg command, the output of cmviewcl-v is as follows: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 56/36.
Failback manual Script_Parameters: ITEM STATUS Resource up Subnet up Resource down Subnet up NODE_NAME ftsys9 ftsys9 ftsys10 ftsys10 NAME /example/float 15.13.168.0 /example/float 15.13.168.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NAME ftsys10 ftsys9 pkg2 now has the status down, and it is shown as unowned, with package switching disabled. Resource /example/float, which is configured as a dependency of pkg2, is down on one node.
Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled PACKAGE pkg2 STATUS up STATE running NAME ftsys9 ftsys10 AUTO_RUN disabled (current) NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service2.1 Subnet up 15.13.168.
Both packages are now running on ftsys9 and pkg2 is enabled for switching. ftsys10 is running the cmcld daemon but no packages.
Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled Alternate up enabled Alternate up enabled NAME manx burmese tabby persian Viewing Information about System Multi-Node Packages The following example shows a cluster that includes system multi-node packages as well as failover packages. The system multi-node packages are running on all nodes in the cluster, whereas the standard packages run on only one node at a time.
tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/1vol1 regular lvol1 vg_for_cvm_dd5 MOUNTED /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/lvol4 regular lvol4 vg_for_cvm_dd5 MOUNTED Node : ftsys8 Cluster Manager : up CVM state : up MOUNT POINT TYPE /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/lvol1 /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/lvol4 SHARED VOLUME DISK GROUP STATUS regular lvol1 vg_for_cvm_veggie_dd5 MOUNTED regular lvol4 vg_for_cvm_dd5 MOUNTED Status of the Packa
Status of CFS Disk Group Packages To see the status of the disk group, use the cfsdgadm display command. For example, for the disk group logdata, enter: cfsdgadm display -v logdata NODE NAME ACTIVATION MODE ftsys9 MOUNT POINT sw (sw) SHARED VOLUME TYPE ftsys10 sw (sw) MOUNT POINT SHARED VOLUME TYPE ... To see which package is monitoring a disk group, use the cfsdgadm show_package command.
configuration of the cluster and all of its packages at any time by running cmcheckconf (1m) without arguments (or with -v; see below). These checks help you to ensure that packages will start up and fail over successfully. Specifically, the following capabilities are new as of Serviceguard A.11.20.
NOTE: • All of the checks below are performed when you run cmcheckconf without any arguments (or with only -v, with or without -k or -K). cmcheckconf validates the current cluster and package configuration, including external scripts and pre-scripts for modular packages, and runs cmcompare to check file consistency across nodes. (This new version of the command also performs all of the checks that were done in previous releases.) See “Checking Cluster Components” (page 337) for details.
to use, and where to find further information. NOTE: The table includes all the checks available as of A.11.20, not just the new ones. Table 7-1 Verifying Cluster Components Component (Context) Tool or Command; More Information Comments Volume groups (cluster) cmcheckconf (1m), cmapplyconf Checked for: (1m) • existence • availability on each node See also “Verifying the Cluster Configuration ” (page 258).
Table 7-1 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Lock disk(s) (cluster) cmcheckconf (1m), cmapplyconf Commands check that the disk(s) are accessible from all (1m) nodes, and that the lock volume group(s) are activated on at least one node.
Table 7-1 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Comments IP addresses (cluster) cmcheckconf (1m), cmapplyconf Commands check that all IP addresses configured into the (1m) cluster are in each node's /etc/hosts. Package IP addresses (package) cmcheckconf (1m), cmapplyconf (1m) See also “Verifying and Applying the Package Configuration” (page 317).
To run this script from cron, you would create the following entry in /var/spool/ cron/crontabs/root: 0 8,20 * * * verification.sh See the cron (1m) manpage for more information.
Note the distinction that is made in this chapter between adding an already configured node to the cluster and adding a new node to the cluster configuration. An already configured node is one that is already entered in the cluster configuration file; a new node is added to the cluster by modifying the cluster configuration file. NOTE: Manually starting or halting the cluster or individual nodes does not require access to the Quorum Server, if one is configured.
Using Serviceguard Commands to Add Previously Configured Nodes to a Running Cluster Use the cmrunnode command to join one or more nodes to an already running cluster. Any node you add must already be a part of the cluster configuration. The following example adds node ftsys8 to the cluster that was just started with only nodes ftsys9 and ftsys10.
This halts any packages running on the node ftsys9 by executing the halt instructions in each package's master control script. ftsys9 is halted and the packages start on their adoptive node. Halting the Entire Cluster You can use Serviceguard Manager, or Serviceguard commands as shown below, to halt a running cluster. Use cmhaltcl to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons.
NOTE: Keep in mind that the purpose of the LAD capabilities is to allow you do maintenance on one or more nodes, or the entire cluster. If you want to do maintenance on individual packages, or on elements of the cluster configuration that affect only one package, or a few packages, you should probably use package maintenance mode; see “Maintaining a Package: Maintenance Mode” (page 353). What You Can Do You can do the following.
• You cannot detach a package that is in maintenance mode, and you cannot place a package into maintenance mode if any of its dependent packages are detached. For more information about maintenance mode, see“Maintaining a Package: Maintenance Mode” (page 353). For more information about dependencies, see “About Package Dependencies” (page 179). • You cannot make configuration changes to a cluster in which any packages are detached. cmapplyconf (1m) will fail.
• You cannot detach a package that has relocatable IP addresses on a subnet that is not configured into the cluster. Such addresses will not be handled properly when the package is attached or re-attached. IMPORTANT: (page 298). • • HP does not recommend such configurations. See “ip_subnet” You cannot detach packages when the HP-UX Clustered I/O Congestion Control feature is enabled. As of the date of this manual, you cannot detach ECMT-based packages.
in fact failed while it was detached, Serviceguard will halt it and restart it on another eligible node. CAUTION: Serviceguard does not check LVM volume groups and mount points when re-attaching packages. • The detached state and status could appear to persist across a reboot. If a node reboots while packages are detached (or detaching, or re-attaching), and package components are left in an inconsistent state, the appropriate package halt scripts will run to clean things up when the node comes back up.
NOTE: 3. If you do not do this, the cmhaltnode in the next step will fail. Halt the node with the -d (detach) option: cmhaltnode -d node1 NOTE: -d and -f are mutually exclusive. See cmhaltnode (1m) for more information. To re-attach the packages, restart the node: cmrunnode -n node1 Halting a Detached Package To halt a package that is detached on node1, proceed as follows. 1. 2. Log in as superuser on another node that is still running in the cluster.
In this example we'll assume that packages pkg1 through pkg5 are unsupported for Live Application Detach, and pkg6 through pkgn are supported. Proceed as follows: 1. Halt all the unsupported packages: cmhaltpkg pkg1 pkg2 pkg3 pkg4 pkg5 2. Halt the cluster, detaching the remaining packages: cmhaltcl -d 3. 4. Upgrade the heartbeat networks as needed.
The cluster must be running, and if the package is dependent on other packages, those packages must be either already running, or started by the same command that starts this package (see the section that follows, and “About Package Dependencies” (page 179).) Starting a Package that Has Dependencies Before starting a package, it is a good idea to use the cmviewcl command to check for package dependencies. You cannot start a package unless all the packages that it depends on are running.
System multi-node packages run on all cluster nodes simultaneously; halting these packages stops them running on all nodes. A multi-node package can run on several nodes simultaneously; you can halt it on all the nodes it is running on, or you can specify individual nodes. Halting a Package that Has Dependencies Before halting a package, it is a good idea to use the cmviewcl command to check for package dependencies. You cannot halt a package unless all the packages that depend on it are down.
After it halts, run the package on the new node using the cmrunpkg command, then re-enable switching as described under “Using Serviceguard Commands to Start a Package” (page 351). Changing Package Switching Behavior There are two options to consider: • Whether the package can switch (fail over) or not. • Whether the package can switch to a particular node or not.
information about package types and modules.) These two methods are called maintenance mode and partial-startup maintenance mode. NOTE: If you need to do maintenance that requires halting a node, or the entire cluster, you should consider Live Application Detach; see “Halting a Node or the Cluster while Keeping Packages Running” (page 344). • Maintenance mode is chiefly useful for modifying networks and EMS resources used by a package while the package is running.
• • • A package in maintenance mode still has its configured (or default) weight, meaning that its weight, if any, is counted against the node's capacity; this applies whether the package is up or down. (See “About Package Weights” (page 187) for a discussion of weights and capacities.) Node-wide and cluster-wide events affect the package as follows: — If the node the package is running on is halted or crashes, the package will no longer be in maintenance mode but will not be automatically started.
• • • You cannot do online configuration as described under “Reconfiguring a Package” (page 385). You cannot configure new dependencies involving this package; that is, you cannot make it dependent on another package, or make another package depend on it. See also “Dependency Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode ” (page 356).
additional rules and instructions under “Performing Maintenance Using Partial-Startup Maintenance Mode” (page 357). If you want to reconfigure the package (using cmapplyconf (1m)) see “Reconfiguring a Package” (page 385) and “Allowable Package States During Reconfiguration ” (page 390). Procedure Follow these steps to perform maintenance on a package's networking or EMS components. In this example, we'll call the package pkg1 and assume it is running on node1. 1.
(page 283) for a list of package modules. The modules used by a package are started in the order shown near the top of its package configuration file.) cmrunpkg -m sg/package_ip pkg1 4. Perform maintenance on the services and test manually that they are working correctly. NOTE: If you now run cmviewcl, you'll see that the STATUS of pkg1 is up and its STATE is maintenance. 5.
NOTE: The full execution sequence for starting a package is: 1. The master control script itself 2. External pre-scripts 3. Volume groups 4. File systems 5. Package IPs 6. External scripts 7. Services Reconfiguring a Cluster You can reconfigure a cluster either when it is halted or while it is still running. Some operations can only be done when the cluster is halted. Table 7-2 shows the required cluster state for many kinds of changes.
Table 7-2 Types of Changes to the Cluster Configuration (continued) Change to the Cluster Configuration Required Cluster State Delete NICs and their IP addresses, if any, from Cluster can be running. the cluster configuration “Changing the Cluster Networking Configuration while the Cluster Is Running” (page 367). If removing the NIC from the system, see “Removing a LAN or VLAN Interface from a Node” (page 372). Change the designation of an existing interface Cluster can be running.
NOTE: If you are using CVM or CFS, you cannot change MEMBER_TIMEOUT or AUTO_START_TIMEOUT while the cluster is running. This is because they affect the aggregate failover time, which is only reported to the CVM stack on cluster startup. You also cannot change the quorum configuration while SG-CFS-pkg is running.
• • • • cmhaltpkg [–t] cmrunpkg [–t] [-n node_name] cmmodpkg { -e [-t] | -d } [-n node_name] cmruncl –v [–t] NOTE: You cannot use the -t option with any command operating on a package in maintenance mode; see “Maintaining a Package: Maintenance Mode” (page 353). For more information about these commands, see their respective manpages. You can also perform these preview functions in Serviceguard Manager: check the Preview [...] box for the action in question.
cmeval does not require you to be logged in to the cluster being evaluated, and in fact that cluster does not have to be running, though it must use the same Serviceguard release and patch version as the system on which you run cmeval.
IMPORTANT: For detailed information and examples, see the cmeval (1m) manpage. Updating the Cluster Lock Configuration Use the procedures that follow whenever you need to change the device file names of the cluster lock physical volumes.
Reconfiguring a Halted Cluster You can make a permanent change in the cluster configuration when the cluster is halted. This procedure must be used for changes marked “Cluster must not be running” in Table 7-2 (page 359), but it can be used for any other cluster configuration changes as well. Use the following steps: 1. 2. 3. 4. Halt the cluster on all nodes, using Serviceguard Manager’s Halt Cluster command, or cmhaltcl on the command line.
1. Use the following command to store a current copy of the existing cluster configuration in a temporary file: cmgetconf -c cluster1 temp.ascii 2. Specify a new set of nodes to be configured and generate a template of the new configuration. Specify the node name (39 bytes or less) without its full domain name; for example, ftsys8 rather than ftsys8.cup.hp.com. Enter a command such as the following (all on one line): cmquerycl -C clconfig.ascii -c cluster1 -n ftsys8 -n ftsys9 -n ftsys10 3. 4.
NOTE: If you want to remove a node from the cluster, run the cmapplyconf command from another node in the same cluster. If you try to issue the command on the node you want removed, you will get an error message. 1. Use the following command to store a current copy of the existing cluster configuration in a temporary file: cmgetconf -c cluster1 temp.ascii 2. Specify the new set of nodes to be configured (omitting ftsys10) and generate a template of the new configuration: cmquerycl -C clconfig.
• • • • • • Change the designation of an existing interface from HEARTBEAT_IP to STATIONARY_IP, or vice versa. Change the NETWORK_POLLING_INTERVAL. Change the NETWORK_FAILURE_DETECTION parameter. Change the NETWORK_AUTO_FAILBACK parameter. Change IP Monitor parameters: SUBNET, IP_MONITOR, POLLING TARGET; see the entries for these parameters under “Cluster Configuration Parameters ” (page 143) for more information.
• You cannot delete a subnet or IP address from a node while a package that uses it (as a monitored_subnet, ip_subnet, or ip_address) is configured to run on that node. See the package networking parameter descriptions (page 296) for more information. • You cannot change the IP configuration of an interface (NIC) used by the cluster in a single transaction (cmapplyconf).
1. Run cmquerycl to get a cluster configuration template file that includes networking information for interfaces that are available to be added to the cluster configuration: cmquerycl -c cluster1 -C clconfig.ascii NOTE: As of Serviceguard A.11.18, cmquerycl -c produces output that includes commented-out entries for interfaces that are not currently part of the cluster configuration, but are available. The networking portion of the resulting clconfig.
4. Apply the changes to the configuration and distribute the new binary configuration file to all cluster nodes: cmapplyconf -C clconfig.ascii If you were configuring the subnet for data instead, and wanted to add it to a package configuration, you would now need to: 1. 2. 3. 4.
4. Verify the new configuration: cmcheckconf -C clconfig.ascii 5. Apply the changes to the configuration and distribute the new binary configuration file to all cluster nodes: cmapplyconf -C clconfig.ascii Removing a LAN or VLAN Interface from a Node You must remove a LAN or VLAN interface from the cluster configuration before removing the interface from the system. On an HP-UX 11i v3 system, you can then remove the interface without shutting down the node.
NOTE: If you are removing a volume group from the cluster configuration, make sure that you also modify any package that activates and deactivates this volume group. In addition, you should use the LVM vgexport command on the removed volume group; do this on each node that will no longer be using the volume group. From the LVM’s cluster, follow these steps: 1. 2. 3. 4. Use the cmgetconf command to store a copy of the cluster's existing cluster configuration in a temporary file.
Similarly, you can delete VxVM or CVM disk groups provided they are not being used by a cluster node at the time. CAUTION: Serviceguard manages the Veritas processes, specifically gab and LLT. This means that you should never use administration commands such as gabconfig, llthosts, and lltconfig to administer a cluster. It is safe to use the read-only variants of these commands, such as gabconfig -a. But a Veritas administrative command could potentially crash nodes or the entire cluster.
Configuring a Legacy Package IMPORTANT: You can still create a new legacy package. If you are using a Serviceguard Toolkit such as Serviceguard NFS Toolkit, consult the documentation for that product. Otherwise, use this section to maintain and rework existing legacy packages rather than to create new ones.
3. Edit each configuration file to specify package name, prioritized list of nodes (with 39 bytes or less in the name), the location of the control script, and failover parameters for each package. Include the data you recorded on the Package Configuration Worksheet. Configuring a Package in Stages It is a good idea to configure failover packages in stages, as follows: 1. 2. 3. 4. 5. 6. 7. 8. Configure volume groups and mount points only. Distribute the control script to all nodes.
• • • • • • FAILBACK_POLICY. For failover packages, enter the failback_policy (page 293). NODE_NAME. Enter the node or nodes on which the package can run; see node_name (page 289). AUTO_RUN. Configure the package to start up automatically or manually; see auto_run (page 289). LOCAL_LAN_FAILOVER_ALLOWED. Enter the policy for local_lan_failover_allowed (page 296). NODE_FAIL_FAST_ENABLED. Enter the policy for node_fail_fast_enabled (page 290). RUN_SCRIPT and HALT_SCRIPT.
— RESOURCE_UP_VALUE — RESOURCE_START For more information, see “Parameters for Configuring EMS Resources” (page 178), and the resource_ parameter descriptions (page 301). NOTE: For legacy packages, DEFERRED resources must be specified in the package control script. • ACCESS_CONTROL_POLICY. You can grant a non-root user PACKAGE_ADMIN privileges for this package. See the entries for user_name, user_host, and user_role (page 310), and “Controlling Access to the Cluster” (page 251), for more information.
IMPORTANT: Serviceguard automatically creates the necessary control scripts when you create the multi-node or system multi-node packages for CFS/CVM (version 4.1 and later). HP strongly recommends that you never edit the configuration or control script files for these packages, although Serviceguard does not prevent it. For CFS, create and modify the information using cfs administration commands only. See “Creating a Storage Infrastructure with Veritas Cluster File System (CFS)” (page 261).
• • • • Select the appropriate options for the storage activation command (not applicable for basic VxVM disk groups), and also include options for mounting filesystems, if desired. Specify the filesystem mount and unmount retry options. If your package uses a large number of volume groups or disk groups or mounts a large number of file systems, consider increasing the number of concurrent vgchange, mount, umount, and fsck operations.
function customer_defined_run_cmds { # ADD customer defined run commands. : # do nothing instruction, because a function must contain some command. date >> /tmp/pkg1.datelog echo 'Starting pkg1' >> /tmp/pkg1.datelog test_return 51 } # This function is a place holder for customer defined functions. # You should define all actions you want to happen here, before the service is # halted. function customer_defined_halt_cmds { # ADD customer defined halt commands.
Verifying the Package Configuration Serviceguard checks the configuration you create and reports any errors. For legacy packages, you can do this in Serviceguard Manager: click Check to verify the package configuration you have done under any package configuration tab, or to check changes you have made to the control script. Click Apply to verify the package as a whole. See the local Help for more details.
Copying Package Control Scripts with HP-UX commands IMPORTANT: In a cross-subnet configuration, you cannot use the same package control script on all nodes if the package uses relocatable IP addresses. See “Configuring Cross-Subnet Failover” (page 383). Use HP-UX commands to copy legacy package control scripts from the node where you created the files, to the same pathname on all nodes which can possibly run the package. Use your favorite method of file transfer (e. g., rcp or ftp).
NOTE: You cannot use Serviceguard Manager to configure cross-subnet packages. Suppose that you want to configure a package, pkg1, so that it can fail over among all the nodes in a cluster comprising NodeA, NodeB, NodeC, and NodeD. NodeA and NodeB use subnet 15.244.65.0, which is not used by NodeC and NodeD; and NodeC and NodeD use subnet 15.244.56.0, which is not used by NodeA and NodeB. (See “Obtaining Cross-Subnet Information” (page 248) for sample cmquerycl output).
NOTE: Configuring monitored_subnet_access as FULL (or not configuring monitored_subnet_access) for either of these subnets will cause the package configuration to fail, because neither subnet is available on all the nodes. Creating Subnet-Specific Package Control Scripts Now you need to create control scripts to run the package on the four nodes.
whether the package is running or not; see “Allowable Package States During Reconfiguration ” (page 390). CAUTION: Be extremely cautious about changing a package's configuration while the package is running. If you reconfigure a package online (by executing cmapplyconf on a package while the package itself is running) it is possible that the package will fail, even if the cmapplyconf succeeds, validating the changes with no errors.
1. Halt the package if necessary: cmhaltpkg pkg1 CAUTION: Make sure you read and understand the information and caveats under“Allowable Package States During Reconfiguration ” (page 390) before you decide to reconfigure a running package. 2. If it is not already available, obtain a copy of the package's configuration file by using the cmgetconf command, specifying the package name. cmgetconf -p pkg1 pkg1.conf 3. Edit the package configuration file.
3. Distribute the new script to all nodes that are configured for that package. Make sure you place the new script in the correct directory with the proper file modes and ownership. 4. Run cmcheckconf to validate the package configuration with the new external script. CAUTION: If cmcheckconf fails, do not proceed to the next step until you have corrected all the errors. 5. Run cmapplyconf on the running package.
You can use Serviceguard Manager to delete the package. On the Serviceguard command line, you can (in most cases) delete a package from all cluster nodes by using the cmdeleteconf command. To delete one of the Veritas Cluster File System packages, usecfscluster, cfsdgadm, or cfsmntadm. This removes the package information from the binary configuration file on all the nodes in the cluster. The command can only be executed when the package is down; the cluster can be up.
Resetting the Service Restart Counter The service restart counter is the number of times a package service has been automatically restarted. This value is used to determine when the package service has exceeded its maximum number of allowable automatic restarts. When a package service successfully restarts after several attempts, the package manager does not automatically reset the restart count. You can reset the counter online using the cmmodpkg -R -s command.
to match that of modular packages as far as possible; these cases are shown in the table. For more information about legacy and modular packages, see Chapter 6 (page 279). NOTE: If neither legacy nor modular is called out under “Change to the Package”, the “Required Package State” applies to both types of package. Changes that are allowed, but which HP does not recommend, are labeled “should not be running”.
Table 7-3 Types of Changes to Packages (continued) Change to the Package Required Package State Change service_restart: modular Package can be running. package Serviceguard will not allow the change if the new value is less than the current restart count. (You can use cmmodpkg -R to reset the restart count if you need to.) Change SERVICE_RESTART: legacy package Package must not be running. Add or remove a SUBNET (in control script): legacy package Package must not be running.
Table 7-3 Types of Changes to Packages (continued) Change to the Package Required Package State Add or remove a resource: modular package Package can be running. Serviceguard will not allow the change if it would cause the package to fail. In addition, Serviceguard will reject the change if the resource is not UP within about 30 seconds. Multiple changes that take longer than this should be done incrementally; resources that are very slow to come up may need to be configured offline.
Table 7-3 Types of Changes to Packages (continued) Change to the Package Required Package State Change a file system: modular package Package should not be running (unless you are only changing fs_umount_opt). Changing file-system options other than fs_umount_opt may cause problems because the file system must be unmounted (using the existing fs_umount_opt) and remounted with the new options; the CAUTION under “Remove a file system: modular package” applies in this case as well.
Table 7-3 Types of Changes to Packages (continued) Change to the Package Required Package State Remove a vxvm_dg or cvm_dg: modular package Package should not be running. See CAUTION under “Remove a volume group: modular package”. Change vxvm_dg_retry: modular Package can be running. package Add or remove a VXVM_DG or CVM_DG: legacy package Package must not be running. Change VXVM_DG_RETRY: legacy package Package must not be running.
Table 7-3 Types of Changes to Packages (continued) Change to the Package Required Package State Add or delete a configured dependency Both packages can be either running or halted. Special rules apply to packages in maintenance mode; see “Dependency Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode ” (page 356). For dependency purposes, a package being reconfigured is considered to be UP.
Responding to Cluster Events Serviceguard does not require much ongoing system administration intervention. As long as there are no failures, your cluster will be monitored and protected. In the event of a failure, those packages that you have designated to be transferred to another node will be transferred automatically. Your ongoing responsibility as the system administrator will be to monitor the cluster and determine if a transfer of package has occurred.
Disabling Serviceguard If for some reason you want to disable Serviceguard on a system, you can do so by commenting out the following entries in /etc/inetd.conf: hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c Then force inetd to re-read inetd.conf: /usr/sbin/inetd -c You can check that this did in fact disable Serviceguard by trying the following command: cmquerycl -n nodename where nodename is the name of the local system.
8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
kill PID 3. To view the package status, enter cmviewcl -v The package should be running on the specified adoptive node. 4. Move the package back to the primary node (see “Moving a Failover Package ” (page 352)). Testing the Cluster Manager To test that the cluster manager is operating correctly, perform the following steps for each node on the cluster: 1. 2. Turn off the power to the node SPU.
3. 4. Verify that a local switch has taken place so that the Standby card is now the Primary card. In Serviceguard Manager, check the cluster properties. On the command line, use cmviewcl -v. Reconnect the LAN to the original Primary card, and verify its status. In Serviceguard Manager, check the cluster properties. On the command line, use cmviewcl -v.
Using EMS (Event Monitoring Service) Hardware Monitors A set of hardware monitors is available for monitoring and reporting on memory, CPU, and many other system values. Some of these monitors are supplied with specific hardware products. Hardware Monitors and Persistence Requests When hardware monitors are disabled using the monconfig tool, associated hardware monitor persistent requests are removed from the persistence files.
Replacing a Faulty Mechanism in an HA Enclosure If you are using software mirroring with Mirrordisk/UX and the mirrored disks are mounted in a high availability disk enclosure, you can use the following steps to hot plug a disk mechanism: 1. Identify the physical volume name of the failed disk and the name of the volume group in which it was configured. In the following example, the volume group name is shown as /dev/vg_sg01 and the physical volume name is shown as /dev/dsk/c2t3d0.
7. Finally, use the lvsync command for each logical volume that has extents on the failed physical volume. This synchronizes the extents of the new disk with the extents of the other mirror. lvsync /dev/vg_sg01/lvolname Replacing a Lock Disk You can replace an unusable lock disk while the cluster is running.
NOTE: If you restore or recreate the volume group for the lock disk and you need to re-create the cluster lock (for example if no vgcfgbackup is available), you can run cmdisklock to re-create the lock. See the cmdisklock (1m) manpage for more information. Replacing a Lock LUN You can replace an unusable lock disk while the cluster is running.
cmdisklock checks that the specified device is not in use by LVM, VxVM, ASM, or the file system, and will fail if the device has a label marking it as in use by any of those subsystems. cmdisklock -f overrides this check. CAUTION: You are responsible for determining that the device is not being used by any subsystem on any node connected to the device before using cmdisklock -f. If you use cmdisklock -f without taking this precaution, you could lose data.
1. 2. 3. 4. 5. 6. Halt the node. You can use Serviceguard Manager to do this, or use the cmhaltnode command. Packages should fail over normally to other nodes. Remove the SCSI cable from the card. Remove the defective SCSI card. Install the new SCSI card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed. You must set the SCSI ID for the new card to be the same as the card it is replacing. Attach the new SCSI card.
NOTE: After replacing a Fibre Channel I/O card, it may necessary to reconfigure the SAN to use the World Wide Name (WWN) of the new Fibre Channel card if Fabric Zoning or other SAN security requiring WWN is used.
5. 6. All nodes in all clusters that were using the old Quorum Server will connect to the new quorum server. Use the cmviewcl -v command from any cluster that is using the quorum server to verify that the nodes in that cluster have connected to the QS. The Quorum Server log file on the new quorum server will show a log message like the following for each cluster that uses the Quorum Server: Request for lock /sg/ succeeded. New lock owners: N1, N2 7.
NOTE: HP recommends you use Serviceguard Manager as a convenient way to observe the status of a cluster and the properties of cluster objects: from the System Management Homepage (SMH), select the cluster you need to troubleshoot. Reviewing Package IP Addresses The netstat -in command can be used to examine the LAN configuration. The command, if executed on ftsys9 after ftsys10 has been halted, shows that the package IP addresses are assigned to lan0 on ftsys9 along with the primary LANIP address.
NOTE: Many other products running on HP-UX in addition to Serviceguard use the syslog.log file to save messages. The HP-UX System Administrator’s Guide provides additional information on using the system log. Sample System Log Entries The following entries from the file /var/adm/syslog/syslog.log show a package that failed to run due to a problem in the pkg5_run script. You would look at the pkg5_run.log for details.
Reviewing Serviceguard Manager Log Files From the System Management Homepage (SMH), click Tools, then select Serviceguard Manager, select the cluster you are interested and then choose View -> Operation Log. Reviewing the System Multi-node Package Files If you are running Veritas Cluster Volume Manager and you have problems starting the cluster, check the log file for the system multi-node package. For CVM 4.1 and later, the file is SG-CFS-pkg.log.
The cmcheckconf command checks: • • • The network addresses and connections. The cluster lock disk connectivity. The validity of configuration parameters of the cluster and packages for: — The uniqueness of names. — The existence and permission of scripts. It doesn’t check: • • The correct setup of the power circuits. The correctness of the package configuration script.
• • • • • • • Cluster re-formations System administration errors Package control script hangs Problems with VxVM disk groups Package movement errors Node and network failures Quorum Server problems The first two categories of problems occur with the incorrect configuration of Serviceguard. The last category contains “normal” failures to which Serviceguard is designed to react and ensure the availability of your applications.
This may indicate a serious problem, such as a node failure, whose underlying cause is probably a too-aggressive setting for the MEMBER_TIMEOUT parameter; see the next section, “Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low”. Or it may be a transitory problem, such as excessive network traffic or system load.
confident that it won't recur, you need take no further action; otherwise you should increase MEMBER_TIMEOUT as instructed in item 1. 3. Member node_name seems unhealthy, not receiving heartbeats from it. This is the message that indicates that the node has been found “unhealthy” as described in the previous bullet. What to do: See item 2. For more information, including requirements and recommendations, see the MEMBER_TIMEOUT discussion under “Cluster Configuration Parameters ” (page 143).
Serviceguard kills the script and marks the package “Halted.” In both cases, the following also take place: • • • • Control of a failover package will not be transferred. The run or halt instructions may not run to completion. auto_run (automatic package switching) will be disabled. The current node will be disabled from running the package. Following such a failure, since the control script is terminated, some of the package's resources may be left activated.
logical volumes, as specified by the LV[] array variables in the package control script, appear under the “Filesystem” column, use umount to unmount them: fuser -ku umount Next, deactivate the package volume groups. These are specified by the VG[] array entries in the package control script. vgchange -a n 4. Finally, re-enable the package for switching.
The ports that must be up are: 1. a — llt, gab 2. b — vxfen 3. v w — cvm 4. f — cfs You can also check the syslog file for information. CAUTION: Do not use the HP-UX mount and umount commands in a CFS cluster; use cfsmount or cfsumount. Commands such asmount -o cluster, dbed_chkptmount, or sfrac_chkptmount could cause conflicts with subsequent command operations on the file system or Serviceguard packages.
CAUTION: This force import procedure should only be used when you are certain the disk is not currently being accessed by another node. If you force import a disk that is already being accessed on another node, data corruption can result. Package Movement Errors These errors are similar to the system administration errors, but they are caused specifically by errors in the control script for legacy packages.
Troubleshooting the Quorum Server NOTE: See the HP Serviceguard Quorum Server Version A.04.00 Release Notes for information about configuring the Quorum Server. Do not proceed without reading the Release Notes for your version. Authorization File Problems The following kind of message in a Serviceguard node’s syslog file or in the output of cmviewcl -v may indicate an authorization problem: Access denied to quorum server 192.6.7.4 The reason may be that you have not updated the authorization file.
This condition can be ignored. The request will be retried a few seconds later and will succeed. The following message is logged: Oct 008 16:10:06:0: Request for lock /sg/lockTest1 succeeded. New lock owners: 1,2.
A Enterprise Cluster Master Toolkit The Enterprise Cluster Master Toolkit (ECMT) provides a group of example scripts and package configuration files for creating Serviceguard packages for several major database and internet software products. Each toolkit contains a README file that explains how to customize the package for your needs. The ECMT can be installed on HP-UX 11i v2, or 11i v3.
B Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 427) • Designing Applications to Run on Multiple Systems (page 430) • Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY (page 435) • Restoring Client Connections (page 437) • Handling Application Failures
There are two principles to keep in mind for automating application relocation: • Insulate users from outages. • Applications must have defined startup and shutdown procedures. You need to be aware of what happens currently when the system your application is running on is rebooted, and whether changes need to be made in the application's response for high availability. Insulate Users from Outages Wherever possible, insulate your end users from outages.
To reduce the impact on users, the application should not simply abort in case of error, since aborting would cause an unneeded failover to a backup system. Applications should determine the exact error and take specific action to recover from the error rather than, for example, aborting upon receipt of any error.
is advisable to take certain actions to minimize the amount of data that will be lost, as explained in the following discussion. Minimize the Use and Amount of Memory-Based Data Any in-memory data (the in-memory context) will be lost when a failure occurs. The application should be designed to minimize the amount of in-memory data that exists unless this data can be easily recalculated.
A common example is a print job. Printer applications typically schedule jobs. When that job completes, the scheduler goes on to the next job.
the second system simply takes over the load of the first system. This eliminates the start up time of the application. There are many ways to design this sort of architecture, and there are also many issues with this sort of design. This discussion will not go into details other than to give a few examples. The simplest method is to have two applications running in a master/slave relationship where the slave is simply a hot standby application for the master.
Avoid Node-Specific Information Typically, when a new system is installed, an IP address must be assigned to each active network interface. This IP address is always associated with the node and is called a stationary IP address. The use of packages containing highly available applications adds the requirement for an additional set of IP addresses, which are assigned to the applications themselves. These are known as relocatable application IP addresses.
Avoid Using SPU IDs or MAC Addresses Design the application so that it does not rely on the SPU ID or MAC (link-level) addresses. The SPU ID is a unique hardware ID contained in non-volatile memory, which cannot be changed. A MAC address (also known as a LANIC id) is a link-specific address associated with the LAN hardware.
be avoided for the same reason. Also, the gethostbyaddr() call may return different answers over time if called with a stationary IP address. Instead, the application should always refer to the application name and relocatable IP address rather than the hostname and stationary IP address. It is appropriate for the application to call gethostbyname(2), specifying the application name rather than the hostname. gethostbyname(2) will pass in the IP address of the application.
Network applications can bind to a stationary IP address, a relocatable IP address, or INADDR_ANY. If the stationary IP address is specified, then the application may fail when restarted on another node, because the stationary IP address is not moved to the new system. If an application binds to the relocatable IP address, then the application will behave correctly when moved to another system. Many server-style applications will bind to INADDR_ANY, meaning that they will receive requests on any interface.
separate mount points. If possible, the application should not assume a specific mount point. To prevent one node from inadvertently accessing disks being used by the application on another node, HA software uses an exclusive access mechanism to enforce access by only one node at a time. This exclusive access applies to a volume group as a whole.
ip_strong_es_model is set to 1 and the sending socket (or communication endpoint) is bound to INADDR_ANY, IP will send the packet using the interface on which the inbound packet was received. For more information about this parameter, see: • The help menu for ndd –h ip_strong_es_model • The HP-UX IPSec Version A.03.00 Administrator's Guide which you can find at www.docs.hp.com/en/internet.
Put a command such as the following in the customer_defined_halt_commands function of a legacy package, or the stop_command function in theexternal_script (page 309) for a modular package: /usr/sbin/route delete net default 128.17.17.1 1 source 128.17.17.17 Once the per-interface default route(s) have been added, netstat –rn would show something like the following, where 128.17.17.17 is the package relocatable address and 128.17.17.19 is the physical address on the same subnet: Destination 127.0.0.1 128.17.
coffee and don't come back for 28 minutes, the perceived downtime is actually 30 minutes, not 5. Factors to consider are the number of reconnection attempts to make, the frequency of reconnection attempts, and whether or not to notify the user of connection loss. There are a number of strategies to use for client reconnection: • Design clients which continue to try to reconnect to their failed server. Put the work into the client application rather than relying on the user to reconnect.
Handling Application Failures What happens if part or all of an application fails? All of the preceding sections have assumed the failure in question was not a failure of the application, but of another component of the cluster. This section deals specifically with application problems. For instance, software bugs may cause an application to fail or system resource issues (such as low swap/memory space) may cause an application to die.
Minimizing Planned Downtime Planned downtime (as opposed to unplanned downtime) is scheduled; examples include backups, systems upgrades to new operating system revisions, or hardware replacements. For planned downtime, application designers should consider: • Reducing the time needed for application upgrades/patches.
the some of the application servers are at revision 4.0. The application must be designed to handle this type of situation. For more information about the rolling upgrades, see “Software Upgrades ” (page 447), and the Release Notes for your version of Serviceguard at http://www.hp.com/go/ hpux-serviceguard-docs. Do Not Change the Data Layout Between Releases Migration of the data to a new format can be very time intensive. It also almost guarantees that rolling upgrade will not be possible.
C Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the Appendix “Designing Highly Available Cluster Applications.” 2.
NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information about support for CFS (http:// www.hp.com/go/hpux-serviceguard-docs). Checklist for Integrating HA Applications This section contains a checklist for integrating HA applications in both single and multiple systems.
a. b. c. d. 3. 4. Create the cluster configuration. Create a package. Create the package script. Use the simple scripts you created in earlier steps as the customer defined functions in the package control script. Start the cluster and verify that applications run as planned. If you will be building an application that depends on a Veritas Cluster File System (CFS) and Cluster Volume Manager (CVM), then consider the following: a. Build storage on all nodes of the cluster. b.
15 seconds to run fsck on the filesystems, 30 seconds to start the application and 3 minutes to recover the database.
D Software Upgrades There are five types of upgrade you can do: • rolling upgrade • rolling upgrade using DRD • non-rolling upgrade • non-rolling upgrade using DRD • migration with cold install Each of these is discussed below. Special Considerations for Upgrade to Serviceguard A.11.20 • • • Before you proceed, read the sections Upgrading from an Earlier Serviceguard Release and Rolling Upgrade in the latest version of the release notes for A.11.20 Serviceguard A.11.20 is supported on HP–UX 11i v3 only.
CAUTION: From the time when the old cluster manager is shut down until the new cluster manager forms its first cluster, a node failure will cause the entire cluster to fail. HP strongly recommends that you use no Serviceguard commands other than cmviewcl (1m) until the new cluster manager successfully completes its first cluster re-formation.
upgrade can also be done any time one system needs to be taken offline for hardware maintenance or patch installations. This method is among the least disruptive, but your cluster must meet both general and release-specific requirements. See “Guidelines for Rolling Upgrade” (page 450). Rolling Upgrade Using DRD DRD stands for Dynamic Root Disk. Using a Dynamic Root Disk allows you to perform the update on a clone of the root disk, then halt the node and reboot it from the updated clone root disk.
Non-Rolling Upgrade Using DRD In a non-rolling upgrade with DRD, you clone each node's root disk and apply the upgrade to the clone, then halt the cluster and reboot each node from its updated clone root disk. This method involves much less cluster down time than a conventional non-rolling upgrade, and is particularly safe because the nodes can be quickly rolled back to their original (pre-upgrade) root disks. But you must make sure your cluster is eligible; see “Restrictions for DRD Upgrades” (page 449).
Performing a Rolling Upgrade Limitations of Rolling Upgrades The following limitations apply to rolling upgrades: CAUTION: Stricter limitations apply to an upgrade to A.11.19; do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 447). • • • During a rolling upgrade to a release other than A.11.
Before You Start Make sure you plan sufficient system capacity to allow moving the packages from node to node during the process without an unacceptable loss of performance. CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 447). Running the Rolling Upgrade 1. 2. Halt the node you want to upgrade. You can do this in Serviceguard Manager, or use the cmhaltnode command.
7. Repeat this process for each node in the cluster. If the cluster fails before the rolling upgrade is complete (because of a catastrophic power failure, for example), you can restart the cluster by entering the cmruncl command from a node which has been upgraded to the latest version of the software. Keeping Kernels Consistent If you change kernel parameters as a part of doing an upgrade, be sure to change the parameters to the same values on all nodes that can run the same packages in case of failover.
Performing a Rolling Upgrade Using DRD IMPORTANT: All the limitations listed under “Guidelines for Rolling Upgrade” (page 450) and “Limitations of Rolling Upgrades ” (page 451) also apply to a rolling upgrade with DRD. You should read the entire section on “Performing a Rolling Upgrade” (page 451) before you proceed. Before You Start CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 447).
AUTOSTART_CMCLD = 1 7. Restart the cluster on the upgraded node, using Serviceguard Manager or cmrunnode (1m). 8. Move the packages back to the upgraded node. 9. Verify that the applications are functioning properly. • If the applications do not function properly and this is not the last node to be upgraded, you can revert to the previous release on this node.
Figure D-1 Running Cluster Before Rolling Upgrade Step 1. Halt the first node, as follows # cmhaltnode -f node1 This will cause pkg1 to be halted cleanly and moved to node 2. The Serviceguard daemon on node 1 is halted, and the result is shown in Figure D-2. Figure D-2 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 to the next operating system release (“HP-UX (new)”), and install the next version of Serviceguard (“SG (new)”).
Figure D-3 Node 1 Upgraded to new HP-UX version Step 3. When upgrading is finished, enter the following command on node 1 to restart the cluster on node 1. # cmrunnode -n node1 At this point, different versions of the Serviceguard daemon (cmcld) are running on the two nodes, as shown in Figure D-4. Figure D-4 Node 1 Rejoining the Cluster Step 4. Repeat the process on node 2. Halt the node, as follows: # cmhaltnode -f node2 This causes both packages to move to node 1.
Figure D-5 Running Cluster with Packages Moved to Node 1 Step 5. Move pkg2 back to its original node. Use the following commands: cmhaltpkg pkg2 cmrunpkg -n node2 pkg2 cmmodpkg -e pkg2 The cmmodpkg command re-enables switching of the package, which was disabled by the cmhaltpkg command. The final running cluster is shown in Figure D-6.
Guidelines for Non-Rolling Upgrade Do a non-rolling upgrade if: • Your cluster does not meet the requirements for rolling upgrade as specified in the Release Notes for the target version of Serviceguard; or • The limitations imposed by rolling upgrades make it impractical for you to do a rolling upgrade (see “Limitations of Rolling Upgrades ” (page 451)); or • For some other reason you need or prefer to bring the cluster down before performing the upgrade.
4. Restart the cluster: cmruncl Performing a Non-Rolling Upgrade Using DRD Limitations of Non-Rolling Upgrades using DRD CAUTION: Stricter limitations apply to an upgrade to A.11.19; do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 447).
cluster by halting the cluster (cmhaltcl), rebooting each node from its original (pre-upgrade) root disk, and restarting the cluster (cmruncl). CAUTION: You must reboot all the nodes from their original disks before restarting the cluster; do not try to restart the cluster with some nodes booted from the upgraded disks and some booted from the pre-upgrade disks.
Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix, at www.hp.com/go/hpux-serviceguard-docs. 6. Recreate any user accounts needed for the cluster applications. 7. Recreate the cluster as described in Chapter 5: “Building an HA Cluster Configuration” (page 205). 8. Restart the cluster. 9. Reinstall the applications. 10. Restore or re-import the data. 11. Recreate and run the cluster packages as described in Chapter 6: “Configuring Packages and Their Services ” (page 279).
E Blank Planning Worksheets This appendix contains blank versions of the planning worksheets mentioned inChapter 4 “Planning and Documenting an HA Cluster ”. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Worksheet for Hardware Planning HARDWARE WORKSHEET Page ___ of ____ =============================================================================== Node Information: Host Name _____________________ Series No _____________________ Memory Capacity ____________________ Number of I/O Slots ________________ =============================================================================== LAN Information: Name of Subnet _________ Name of IP Interface __________ Addr_____________ Traffic Type ___________ Name o
Power Supply Worksheet POWER SUPPLY WORKSHEET Page ___ of ____ =============================================================================== SPU Power: Host Name _____________________ Power Supply _______________________ Host Name _____________________ Power Supply _______________________ =============================================================================== Disk Power: Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supp
Quorum Server Worksheet Quorum Server Data: ============================================================================== QS Hostname: _____________IP Address: _______________IP Address_______________ ============================================================================== Quorum Services are Provided for: Cluster Name: ___________________________________________________________ Host Names ____________________________________________ Host Names ____________________________________________ Cluster
LVM Volume Group and Physical Volume Worksheet PHYSICAL VOLUME WORKSHEET Page ___ of ____ =============================================================================== Volume Group Name: ______________________________________________________ Physical Volume Name:_____________________________________________________ Physical Volume Name:_____________________________________________________ Physical Volume Name:_____________________________________________________ Physical Volume Name: _____________________
VxVM Disk Group and Disk Worksheet DISK GROUP WORKSHEET Page ___ of ____ =========================================================================== Disk Group Name: __________________________________________________________ Physical Volume Name:______________________________________________________ Physical Volume Name:______________________________________________________ Physical Volume Name:______________________________________________________ Physical Volume Name: _____________________________________
Cluster Configuration Worksheet =============================================================================== Name and Nodes: =============================================================================== Cluster Name: __________________________ RAC Version: _______________ Node Names: _________________________________________________________ Volume Groups (for packages):________________________________________ =========================================================================== Subnets: =========
Package Configuration Worksheet Package Configuration File Data:===============================================================Package Name: __________________Package Type:______________ Primary Node: ____________________ First Failover Node:__________________ Additional Failover Nodes:__________________________________ Run Script Timeout: _____ Halt Script Timeout: _____________ Package AutoRun Enabled? ______ Local LAN Failover Allowed? _____ Node Failfast Enabled? ________ Failover Policy:_______________
F Migrating from LVM to VxVM Data Storage This appendix describes how to migrate LVM volume groups to VxVM disk groups for use with the Veritas Volume Manager (VxVM), or with the Cluster Volume Manager (CVM) on systems that support it.
3. 4. 5. Back up the volume group’s data, using whatever means are most appropriate for the data contained on this volume group. For example, you might use a backup/restore utility such as Omniback, or you might use an HP-UX utility such as dd. Back up the volume group configuration: vgcfgbackup Define the new VxVM disk groups and logical volumes. You will need to have enough additional disks available to create a VxVM version of all LVM volume groups.
3. Edit the new script to include the names of the new VxVM disk groups and logical volumes. The new portions of the package control script that are needed for VxVM use are as follows: • The VXVM_DG[] array. This defines the VxVM disk groups that are used for this package. The first VxVM_DG[] entry should be in index 0, the second in 1, etc. For example: VXVM_DG[0]="dg01" VXVM_DG[1]="dg02" • The LV[], FS[] and FS_MOUNT_OPT[] arrays are used the same as they are for LVM.
Customizing Packages for CVM NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information about support for CVM and CFS: www.hp.com/go/ hpux-serviceguard-docs. For instructions on configuring packages to use CVM disk groups for raw storage (that is, without CFS) see “Creating the Storage Infrastructure with Veritas Cluster Volume Manager (CVM)” (page 268).
G IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses. Topics: • IPv6 Address Types • Network Configuration Restrictions • Local Primary/Standby LAN Patterns • IPv6 Relocatable Address and Duplicate Address Detection Feature (page 480) IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces.
can appear only once in an address and it can be used to compress the leading, trailing, or contiguous sixteen-bit zeroes in an address. Example: fec0:1:0:0:0:0:0:1234 can be represented as fec0:1::1234. • When dealing with a mixed environment of IPv4 and IPv6 nodes there is an alternative form of IPv6 address that will be used. It is x:x:x:x:x:x:d.d.d.d, where 'x's are the hexadecimal values of higher order 96 bits of IPv6 address and the 'd's are the decimal values of the 32-bit lower order bits.
IPv4 and IPv6 Compatibility There are a number of techniques for using IPv4 addresses within the framework of IPv6 addressing. IPv4 Compatible IPv6 Addresses The IPv6 transition mechanisms use a technique for tunneling IPv6 packets over the existing IPv4 infrastructure. IPv6 nodes that support such mechanisms use a special kind of IPv6 addresses that carry IPv4 addresses in their lower order 32-bits. These addresses are called IPv4 Compatible IPv6 addresses.
TLA ID = Top-level Aggregation Identifier. RES = Reserved for future use. NLA ID = Next-Level Aggregation Identifier. SLA ID = Site-Level Aggregation Identifier. Interface ID = Interface Identifier. Link-Local Addresses Link-local addresses have the following format: Table G-6 10 bits 54 bits 64 bits 1111111010 0 interface ID Link-local address are supposed to be used for addressing nodes on a single link. Packets originating from or destined to a link-local address will not be forwarded by a router.
‘2’ indicates that the scope is link-local. A value of “5” indicates that the scope is site-local. The “group ID” field identifies the multicast group. Some frequently used multicast groups are the following: All Node Addresses = FF02:0:0:0:0:0:0:1 (link-local) All Router Addresses = FF02:0:0:0:0:0:0:2 (link-local) All Router Addresses = FF05:0:0:0:0:0:0:2 (site-local) Network Configuration Restrictions Serviceguard supports IPv6 for data and heartbeat IP.
NOTE: Even though link-local IP addresses are not supported in the Serviceguard cluster configuration, the primary link-local address on the Serviceguard primary interface will be switched over the standby during a local switch. This is because of two requirements: First, the dual stack (IPv4/IPv6) kernel requires that the primary IP address associated with an interface must always be a link-local address.
Local Primary/Standby LAN Patterns The use of IPv6 allows a number of different patterns of failover among LAN cards configured in the cluster. This is true because each LAN card can support several IP addresses when a dual IPv4/IPv6 configuration is used. This section describes several ways in that local failover to a standby LAN can be configured.
Following the loss of lan0 or lan2, lan1 can adopt either address, as shown below. The same LAN card can be configured with both IPv4 and IPv6 addresses, as shown in below.
This type of configuration allows failover of both addresses to the standby. This is shown in below.
H Using Serviceguard Manager HP Serviceguard Manager is a web-based, HP System Management Homepage (HP SMH) tool, that replaces the functionality of the earlier Serviceguard management tools. HP Serviceguard Manager allows you to monitor, administer and configure a Serviceguard cluster from any system with a supported web browser. Serviceguard Manager does not require additional software installation.
— A user with HP SMH Administrator access has full cluster management capabilities. — A user with HP SMH Operator access can monitor the cluster and has restricted cluster management capabilities as defined by the user’s Serviceguard role-based access configuration. — A user with HP SMH User access does not have any cluster management capabilities. See the online help topic About Security for more information. • • Have created the security “bootstrap” file cmclnodelist.
NOTE: • A user with HP SMH Administrator access has full cluster management capabilities. • A user with HP SMH Operator access can monitor the cluster and has restricted cluster management capabilities as defined by the user’s Serviceguard role-based access configuration. • A user with HP SMH User access does not have any cluster management capabilities. 1. Enter the standard URL “http://:2301/” For example http://clusternode1.cup.hp.com:2301/ 2.
Figure H-1 System Management Homepage with Serviceguard Manager Number What is it? Description 1 Cluster and Overall status and alerts Displays information about the Cluster status, alerts and general information. 2 Menu tool bar The menu tool bar is available from the HP Serviceguard Manager Homepage, and from any cluster, node or package view-only property page. Menu option availability depends on which type of property page (cluster, node or package) you are currently viewing.
• • One or more clusters running Serviceguard A.11.15.01 or later Serviceguard Manager version A.05.01 with HP SIM 5.10 or later installed on a server NOTE: Serviceguard Manager can be launched by HP Systems Insight Manager version 5.10 or later if Serviceguard Manager is installed on an HP Systems Insight Manager Central Management Server. For a Serviceguard A.11.17.01 cluster or later, Systems Insight Manager will attempt to launch Serviceguard Manager from one of the nodes in the cluster.
4. Expand HP Serviceguard, and click on a Serviceguard cluster. NOTE: If you click on a cluster running an earlier Serviceguard release, the page will display a link that will launch Serviceguard Manager A.05.01 (if installed) via Java Webstart.
I Maximum and Minimum Values for Parameters Table I-1 shows the range of possible values for cluster configuration parameters. Table I-1 Minimum and Maximum Values of Cluster Configuration Parameters Cluster Parameter Minimum Value Maximum Value Member Timeout See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4. See 14,000,000 MEMBER_TIMEOUT microseconds under “Cluster Configuration Parameters” in Chapter 4.
Index A Access Control Policies, 251 Access Control Policy, 167 Access roles, 167 active node, 31 adding a package to a running cluster, 388 adding cluster nodes advance planning, 204 adding nodes to a running cluster, 342 adding packages on a running cluster , 318 additional package resources monitoring, 79 addressing, SCSI, 125 administration adding nodes to a ruuning cluster, 342 cluster and package states, 322 halting a package, 351 halting the entire cluster, 344 moving a package, 352 of packages and s
configuring with commands, 243 redundancy of components, 37 Serviceguard, 30 typical configuration, 29 understanding components, 37 cluster administration, 341 solving problems, 413 cluster and package maintenance , 321 cluster configuration creating with SAM or Commands, 242 file on all nodes, 59 identifying cluster lock volume group, 246 identifying cluster-aware volume groups, 251 planning, 135 planning worksheet, 167 sample diagram, 123 verifying the cluster configuration, 258 cluster configuration file
defined, 162 Configuring clusters with Serviceguard command line, 243 configuring packages and their services, 279 control script adding customer defined functions, 380 in package configuration, 378 pathname parameter in package configuration, 311 support for additional productss, 381 troubleshooting, 412 controlling the speed of application failover, 427 creating the package configuration, 375 Critical Resource Analysis (CRA) LAN or VLAN, 372 customer defined functions adding to the control script, 380 CVM
using, 79 exclusive access relinquishing via TOC, 118 expanding the cluster planning ahead, 122 expansion planning for, 176 F failback policy used by package manager, 75 FAILBACK_POLICY parameter used by package manager, 75 failover controlling the speed in applications, 427 defined, 31 failover behavior in packages, 177 failover package, 68 failover policy used by package manager, 72 FAILOVER_POLICY parameter used by package manager, 72 failure kinds of responses, 116 network communication, 120 package, s
HA cluster defined, 37 objectives in planning, 121 host IP address hardware planning, 124, 131 host name hardware planning, 124 HOSTNAME_ADDRESS_FAMILY defined, 144 discussion and restrictions, 139 how the cluster manager works, 59 how the network manager works, 89 HP Predictive monitoring in troubleshooting, 402 I I/O bus addresses hardware planning, 127 I/O slots hardware planning, 124, 126 I/O subsystem changes as of HP-UX 11i v3, 46, 106 identifying cluster-aware volume groups, 251 in-line terminator p
M MAC addresses, 432 managing the cluster and nodes, 341 manual cluster startup, 61 MAX_CONFIGURED_PACKAGES defined, 166 maximum number of nodes, 37 MEMBER_TIMEOUT and cluster re-formation, 117 and safety timer, 55 configuring, 160 defined, 159 maximum and minimum values , 159 modifying, 251 membership change reasons for, 62 memory capacity hardware planning, 124 memory requirements lockable memory for Serviceguard, 122 minimizing planned down time, 440 mirror copies of data protection against disk failure,
primary, 31 NTP time protocol for clusters, 222 O olrad command removing a LAN or VLAN interface, 372 online hardware maintenance by means of in-line SCSI terminators, 406 OTS/9000 support, 491 outages insulating users from, 426 P package adding and deleting package IP addresses, 91 base modules, 283 basic concepts, 37 changes while the cluster is running, 391 configuring legacy, 375 failure, 116 halting, 351 legacy, 375 local interface switching, 93 modular, 283 modular and legacy, 279 modules, 283 movin
blanks, 463 point of failure in networking, 40 point to point connections to storage devices, 51 POLLING_TARGET defined, 165 ports dual and single aggregated, 104 power planning power sources, 128 worksheet, 129 power supplies blank planning worksheet, 464 power supply and cluster lock, 49 blank planning worksheet, 465 UPS for OPS on HP-UX, 49 Predictive monitoring, 402 primary LAN interfaces defined, 38 primary network interface, 38 primary node, 31 pvcreate creating a root mirror with, 224 PVG-strict mirr
adding or removing packages , 318 S safety timer and node TOC, 55 and syslog.
for clusters, 222 timeout node, 117 TOC and MEMBER_TIMEOUT, 117 and package availability, 118 and safety timer, 161 and the safety timer, 55 defined, 55 when a node fails, 116 toolkits for databases, 423 traffic type LAN hardware planning, 125 troubleshooting approaches, 409 monitoring hardware, 401 replacing disks, 402 reviewing control scripts, 412 reviewing package IP addresses, 410 reviewing system log file, 410 using cmquerycl and cmcheckconf, 412 troubleshooting your cluster, 399 typical cluster after
