Managing Serviceguard Nineteenth Edition HP Part Number: 5900-1492 Published: June 2011
Legal Notices © Copyright 1995-2011 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Publishing History .......................................................................................17 Preface......................................................................................................18 1 Serviceguard at a Glance.........................................................................20 What is Serviceguard? ...........................................................................................................20 Failover........................................
Cluster SNMP Agent Daemon: cmsnmpd..........................................................................40 Service Assistant Daemon: cmserviced.............................................................................40 Quorum Server Daemon: qs...........................................................................................41 Network Manager Daemon: cmnetd................................................................................41 Lock LUN Daemon: cmdisklockd..........................
Local Switching ............................................................................................................66 Switching Back to Primary LAN Interfaces after Local Switching...........................................69 Remote Switching .........................................................................................................69 Address Resolution Messages after Switching on the Same Subnet.......................................
Cluster Lock Planning..............................................................................................................94 Cluster Lock Disk and Re-formation Time ...............................................................................95 Planning for Expansion.......................................................................................................95 Using a Quorum Server......................................................................................................
Simple Method...........................................................................................................136 Example 1.............................................................................................................136 Points to Keep in Mind............................................................................................137 Comprehensive Method...............................................................................................137 Defining Capacities..........
Creating the Storage Infrastructure and file systems with LVM, VxVM and CVM........................168 Creating a Storage Infrastructure with LVM..........................................................................168 Using the EMS Disk Monitor.........................................................................................169 Using Mirrored Individual Data Disks.............................................................................169 Creating Volume Groups...............................
Managing Disk Groups and Mount Points Using Modular Packages...................................195 Creating Modular Disk Group and Mount Point Packages............................................196 Parallel Activation of Disk Groups and Parallel Mounting of Mount Points.......................198 Creating Modular Checkpoint and Snapshot Packages for CFS....................................199 Guidelines for Migrating from Legacy CFS Package to Modular CFS Package.....................
dependency_condition............................................................................................228 dependency_location..............................................................................................228 weight_name, weight_value.....................................................................................229 local_lan_failover_allowed......................................................................................229 monitored_subnet.................................
Adding the Package to the Cluster..........................................................................................246 How Control Scripts Manage VxVM Disk Groups.....................................................................246 7 Cluster and Package Maintenance............................................................248 Reviewing Cluster and Package Status.....................................................................................
Starting a Package that Has Dependencies.....................................................................271 Using Serviceguard Commands to Start a Package..........................................................272 Starting the Special-Purpose CVM and CFS Packages..................................................272 Halting a Package ..........................................................................................................272 Halting a Package that Has Dependencies.....................
Distributing the Binary Cluster Configuration File with HP-UX Commands ...........................295 Configuring Cross-Subnet Failover......................................................................................295 Configuring node_name..............................................................................................296 Configuring monitored_subnet_access............................................................................296 Creating Subnet-Specific Package Control Scripts.....
Using the cmcheckconf Command......................................................................................318 Using the cmviewconf Command.......................................................................................319 Reviewing the LAN Configuration ......................................................................................319 Solving Problems .................................................................................................................
Handling Application Failures ...............................................................................................337 Create Applications to be Failure Tolerant ..........................................................................337 Be Able to Monitor Applications .......................................................................................337 Minimizing Planned Downtime ..............................................................................................
E Blank Planning Worksheets......................................................................355 Worksheet for Hardware Planning..........................................................................................355 Power Supply Worksheet.......................................................................................................355 Quorum Server Worksheet.....................................................................................................
Publishing History Table 1 Publishing History Publishing Date Part Number Edition January 1995 B3936-90001 First June 1995 B3936-90003 Second December 1995 B3936-90005 Third August 1997 B3936-90019 Fourth January 1998 B3936-90024 Fifth October 1998 B3936-90026 Sixth December 2000 B3936-90045 Seventh September 2001 B3936-90053 Eighth March 2002 B3936-90065 Ninth June 2003 B3936-90070 Tenth June 2004 B3936-90076 Eleventh June 2005 B3936-90076 Eleventh, First reprint Octo
Preface This nineteenth edition of the manual applies to Serviceguard Version A.11.20 for the April 2011 release. Earlier versions are available at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard. The information in this manual has been updated for new features and functionality introduced in the Serviceguard A.11.20 April 2011 patches, or their superseding patches (PHSS_41628 or later). If the Serviceguard A.11.
Related Publications For information about the current version of Serviceguard, and about older versions, see the Serviceguard documents posted at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard. The following documents, which can all be found at http://www.hp.com/go/ hpux-serviceguard-docs, are particularly useful. • The latest edition of the HP Serviceguard Version A.11.20 Release Notes • HP Serviceguard Quorum Server Version A.04.
1 Serviceguard at a Glance This chapter introduces Serviceguard on HP-UX, and shows where to find information in this book. It covers the following: • What is Serviceguard? • Using Serviceguard Manager (page 23) • A Roadmap for Configuring Clusters and Packages (page 24) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4: “Planning and Documenting an HA Cluster ” (page 88).
control of the package to another cluster node, allowing services to remain available with minimal interruption. • There are also packages that run on several cluster nodes at once, and do not fail over. These are called system multi-node packages and multi-node packages. Examples are the packages HP supplies for use with the Veritas Cluster Volume Manager and Veritas Cluster File System from Symantec (on HP-UX releases that support them; see “About Veritas CFS and CVM from Symantec” (page 22)).
Figure 2 Typical Cluster After Failover After this transfer, the failover package typically remains on the adoptive node as long the adoptive node continues running. If you wish, however, you can configure the package to return to its primary node as soon as the primary node comes back online. Alternatively, you may manually transfer control of the package back to the primary node at the appropriate time. Figure 2 does not show the power connections to the cluster, but these are important as well.
Using Serviceguard Manager NOTE: For more-detailed information, see Appendix I (page 371) and the section on Serviceguard Manager in the latest version of the Serviceguard Release Notes. Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for up-to-date information about Serviceguard Manager compatibility. You can find both documents at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard.
on the command line. As of HP-UX 11i v3, SAM offers a Terminal User Interface (TUI) which also acts as a gateway to the web-based System Management Homepage (SMH). • To get to the SMH for any task area, highlight the task area in the SAM TUI and press w. • To go directly to the SMH from the command line, enter /usr/sbin/sam -w For more information, see the HP-UX Systems Administrator’s Guide, at the address given in the preface to this manual.
Figure 3 Tasks in Configuring a Serviceguard Cluster The tasks in Figure 3 are covered in step-by-step detail in chapters 4 through 7. HP recommends you gather all the data that is needed for configuration before you start. See “Planning and Documenting an HA Cluster ” (page 88) for tips on gathering data.
2 Understanding Serviceguard Hardware Configurations This chapter gives a broad overview of how the Serviceguard hardware components work. The following topics are presented: • Redundancy of Cluster Components • Redundant Network Components (page 27) • Redundant Disk Storage (page 31) • Redundant Power Supplies (page 35) • Larger Clusters (page 35) Refer to the next chapter for information about Serviceguard software components.
Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable, and the cables themselves are connected by a component such as a hub or a bridge. This arrangement of physical cables connected to each other via a bridge or concentrator or switch is known as a bridged net .
configured into the Serviceguard cluster, unless those IP addresses themselves will be immediately configured into the cluster as stationary IP addresses. CAUTION: If you configure any address other than a stationary IP address on a Serviceguard network interface, it could collide with a relocatable package IP address assigned by Serviceguard. See “Stationary and Relocatable IP Addresses ” (page 64).
NOTE: You should verify that network traffic is not too heavy on the heartbeat/data LAN. If traffic is too heavy, this LAN might not perform adequately in transmitting heartbeats if the dedicated heartbeat LAN fails. Cross-Subnet Configurations As of Serviceguard A.11.18 it is possible to configure multiple subnets, joined by a router, both for the cluster heartbeat and for data, with some nodes using one subnet and some another.
• Because Veritas Cluster File System from Symantec (CFS) requires link-level traffic communication (LLT) among the nodes, Serviceguard cannot be configured in cross-subnet configurations with CFS alone. But CFS is supported in specific cross-subnet configurations with Serviceguard and HP add-on products; see the documentation listed below. • Each package subnet must be configured with a standby interface on the local bridged net. The standby interface can be shared between subnets.
IMPORTANT: Although cross-subnet topology can be implemented on a single site, it is most commonly used by extended-distance clusters, and specifically site-aware disaster-tolerant clusters, which require Metrocluster (HP add-on software). Design and configuration of such clusters are covered in the disaster-tolerant documentation delivered with Serviceguard. For more information, see the following documents under http:// www.hp.
Data Protection It is required that you provide data protection for your highly available system, using one of two methods: • Disk Mirroring • Disk Arrays using RAID Levels and Multiple Data Paths Disk Mirroring Serviceguard itself does not provide protection for data on your disks, but protection is provided by HP’s Mirrordisk/UX product for LVM storage, and by the Veritas Volume Manager for VxVM and CVM.
Monitoring LVM Disks Through Event Monitoring Service If you are using LVM, you can configure disk monitoring to detect a failed mechanism by using the disk monitor capabilities of the EMS HA Monitors, available as a separate product . Monitoring can be set up to trigger a package failover or to report disk failure events to a Serviceguard, to another application, or by email. For more information, see “Using EMS to Monitor Volume Groups” (page 96).
Figure 5 Mirrored Disks Connected for High Availability Figure 6 below shows a similar cluster with a disk array connected to each node on two I/O channels. See “About Multipathing” (page 32). Figure 6 Cluster with High Availability Disk Array Details on logical volume configuration for Serviceguard are in the chapter “Building an HA Cluster Configuration.
node is attached to both switches, and both switches are attached to the disk array with redundant links. Figure 7 Cluster with Fibre Channel Switched Disk Array This type of configuration uses native HP-UX or other multipathing software; see “About Multipathing” (page 32). Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks.
shared SCSI buses, the practical limit on the number of nodes that can be attached to the same shared bus is four, because of bus loading and limits on cable length. Even in this case, 16 nodes could be set up as an administrative unit, and sub-groupings of four could be set up on different SCSI buses which are attached to different mass storage devices. In the case of non-shared SCSI connections to an XP series or EMC disk array, the four-node limit does not apply.
Figure 9 Eight-Node Cluster with XP or EMC Disk Array Fibre Channel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP Unix Servers Configuration Guide, available through your HP representative.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
• /usr/lbin/cmfileassistd—Serviceguard File Management daemon • /usr/lbin/cmlogd—Serviceguard Syslog Log Daemon • /usr/lbin/cmlvmd—Cluster Logical Volume Manager Daemon • /opt/cmom/lbin/cmomd—Cluster Object Manager Daemon • /usr/lbin/cmsnmpd—Cluster SNMP subagent (optionally running) • /usr/lbin/cmserviced—Serviceguard Service Assistant Daemon • /usr/lbin/qs—Serviceguard Quorum Server Daemon • /usr/lbin/cmnetd—Serviceguard Network Manager daemon.
NOTE: Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 20 on all cluster nodes. It is important that user processes should have a priority lower than 20, otherwise they may prevent Serviceguard from updating the kernel safety timer, causing a system reset. File Management Daemon: cmfileassistd The cmfileassistd daemon is used by cmcld to manage the files that it needs to read from, and write to, disk.
For services, cmcld monitors the service process and, depending on the number of service retries, cmcld either restarts the service through cmserviced or it causes the package to halt and moves the package to an available alternate node. Quorum Server Daemon: qs Using a Quorum Server is one way to break a tie and establish a quorum when the cluster is re-forming; the other way is to use a cluster lock. See “Cluster Quorum to Prevent Split-Brain Syndrome” and “Cluster Lock ” (page 44).
Veritas CFS components operate directly over Ethernet networks that connect the nodes within a cluster. Redundant networks are required to avoid single points of failure. The Veritas CFS components are: • GAB (Group Membership Services/Atomic Broadcast) — When Veritas Cluster Volume Manager (CVM) 4.1 or later, or Veritas Cluster File System (CFS), is deployed as part of the Serviceguard Storage Management Suite bundles, the file /etc/gabtab is automatically configured and maintained by Serviceguard.
(page 85) . At the end of the re-formation, information about the new cluster membership is passed to the package coordinator (described further in this chapter, in “How the Package Manager Works” (page 48)). Failover packages that were running on nodes that are no longer in the new cluster are transferred to their adoptive nodes.
modification of the configuration files. Re-formation of the cluster occurs under the following conditions (not a complete list): • An SPU or network failure was detected on an active node. • An inactive node wants to join the cluster. The cluster manager daemon has been started on that node. • A node has been added to or deleted from the cluster configuration. • The system administrator halted a node. • A node halts because of a package failure. • A node halts because of a service failure.
Use of a Lock LUN or LVM Lock Disk as the Cluster Lock A lock disk or lock LUN can be used for clusters up to and including four nodes in size. A cluster lock disk is a special area on an LVM disk located in a volume group that is shareable by all nodes in the cluster. Similarly, a cluster lock LUN is a small dedicated LUN, connected to all nodes in the cluster, that contains the lock information.
Dual Lock Disk If you are using disks that are internally mounted in the same cabinet as the cluster nodes, then a single lock disk would be a single point of failure, since the loss of power to the node that has the lock disk in its cabinet would also render the cluster lock unavailable.
Figure 12 Quorum Server Operation The Quorum Server runs on a separate system, and can provide quorum services for multiple clusters. IMPORTANT: For more information about the Quorum Server, see the latest version of the HP Serviceguard Quorum Server release notes at http://www.hp.com/go/hpux-serviceguard-docs under HP Serviceguard Quorum Server Software No Cluster Lock Normally, you should not configure a cluster of three or fewer nodes without a cluster lock.
IMPORTANT: During Step 1, while the nodes are using a strict majority quorum, node failures can cause the cluster to go down unexpectedly if the cluster has been using a quorum device before the configuration change. For example, suppose you change the Quorum Server polling interval while a two-node cluster is running. If a node fails during Step 1, the cluster will lose quorum and go down, because a strict majority of prior cluster members (two out of two in this case) is required.
Figure 13 Package Moving During Failover Configuring Failover Packages You configure each package separately. You create a failover package by generating and editing a package configuration file template, then adding the package to the cluster configuration database; see Chapter 6: “Configuring Packages and Their Services ” (page 216). For legacy packages (packages created by the method used on versions of Serviceguard earlier than A.11.
A package switch normally involves moving a failover package and its associated IP addresses to a new system on the same subnet. In this case, the new system must have the same subnet configured and working properly; otherwise the package will not be started. NOTE: It is possible to configure a cluster that spans subnets joined by a router, with some nodes using one subnet and some another. This is known as a cross-subnet configuration.
NOTE: For design and configuration information about clusters which span subnets, including site-aware disaster-tolerant clusters, see the documents listed under “Cross-Subnet Configurations” (page 29). Figure 15 After Package Switching Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file.
When the cluster starts, each package starts as shown in Figure 16.
NOTE: Using the min_package_node policy, when node 2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node.
Figure 19 Automatic Failback Configuration before Failover Table 3 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 CONFIGURED_NODE AUTOMATIC pkgB node2, node4 CONFIGURED_NODE AUTOMATIC pkgC node3, node4 CONFIGURED_NODE AUTOMATIC node1 panics, and after the cluster reforms, pkgA starts running on node4: Figure 20 Automatic Failback Configuration After Failover 54 Understanding Serviceguard Software Components
After rebooting, node 1 rejoins the cluster. At that point, pkgA will be automatically stopped on node 4 and restarted on node 1. Figure 21 Automatic Failback Configuration After Restart of Node 1 CAUTION: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period.
If a monitored resource is configured in a package, the package manager calls the resource registrar to launch an external monitor for the resource. Resources can be configured to start up either at the time the node enters the cluster or at the end of package startup. The monitor then sends messages back to Serviceguard, which checks to see whether the resource is available before starting the package.
A failover package can be configured to have a dependency on a multi-node or system multi-node package. The package manager cannot start a package on a node unless the package it depends on is already up and running on that node. The package manager will always try to keep a failover package running unless there is something preventing it from running on any node.
NOTE: This diagram applies specifically to legacy packages. Differences for modular scripts are called out below. Figure 22 Legacy Package Time Line Showing Important Events The following are the most important moments in a package’s life: 1. Before the control script starts. (For modular packages, this is the master control script.) 2. During run script execution. (For modular packages, during control script execution to start the package.) 3. While services are running 4.
During Run Script Execution Once the package manager has determined that the package can start on a particular node, it launches the script that starts the package (that is, a package’s control script or master control script is executed with the start parameter). This script carries out the following steps: 1. 2. 3. 4. 5. 6. 7. 8. Executes any external_pre_scripts (modular packages only (page 239)) Activates volume groups or disk groups. Mounts file systems.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled is set to yes, it will halt the node on which the package is running.
Some failures can result in a local switch. For example, if there is a failure on a specific LAN card and there is a standby LAN configured for that subnet, then the Network Manager will switch to the healthy LAN card. If a service fails but the restart parameter for that service is set to a value greater than 0, the service will restart, up to the configured number of restarts, without halting the package.
1. 2. 3. 4. 5. 6. 7. 8. Halts any deferred resources that had been started earlier. Halts all package services. Executes any customer-defined halt commands (legacy packages only) or external_scripts (modular packages only(page 239)). Removes package IP addresses from the LAN card on the node. Unmounts file systems. Deactivates volume groups. Exits with an exit code of zero (0). Executes any external_pre_scripts (modular packages only (page 239)).
NOTE: This diagram applies specifically to legacy packages. Differences for modular scripts are called out above. Normal and Abnormal Exits from the Halt Script The package’s ability to move to other nodes is affected by the exit conditions on leaving the halt script. The following are the possible exit codes: • 0—normal exit. The package halted normally, so all services are down on this node. • 1—abnormal exit, also known as no_restart exit. The package did not halt normally.
Table 4 Error Conditions and Package Movement for Failover Packages (continued) Package Error Condition Results Error or Exit Code Node Failfast Service Enabled Failfast Enabled HP-UX Status on Primary after Error Halt script Package Allowed to Package runs after Run on Primary Allowed to Run Error or Exit Node after Error on Alternate Node Loss of Network YES Either Setting system reset No N/A (system reset) Yes Loss of Network NO Either Setting Running Yes Yes Yes Loss of Monitored YES
A relocatable IP address is like a virtual host IP address that is assigned to a package. HP recommends that you configure names for each package through DNS (Domain Name Service). A program can then use the package’s name like a host name as the input to gethostbyname, which will return the package's relocatable IP address. Both stationary IP addresses, and relocatable IP addresses on subnets that are configured into the cluster, will switch to a standby LAN interface in the event of a LAN card failure.
Monitoring LAN Interfaces and Detecting Failure: Link Level At regular intervals, determined by the NETWORK_POLLING_INTERVAL (see “Cluster Configuration Parameters ” (page 105)) Serviceguard polls all the network interface cards specified in the cluster configuration file. Network failures are detected within each single node in the following manner. One interface on the node is assigned to be the poller.
IPv6 uses the Neighbor Discovery Protocol (NDP) instead of ARP. The NDP protocol is used by hosts and routers to do the following: • determine the link-layer addresses of neighbors on the same link, and quickly purge cached values that become invalid. • find neighboring routers willing to forward packets on their behalf. • actively keep track of which neighbors are reachable, and which are not, and detect changed link-layer addresses.
Figure 26 Cluster After Local Network Switching As the standby interface takes over, IP addresses will be switched to the hardware path associated with the standby interface. The switch is transparent at the TCP/IP level. All applications continue to run on their original nodes. During this time, IP traffic on Node 1 will be delayed as the transfer occurs. However, the TCP/IP connections will continue to be maintained and applications will continue to run. Control of the packages on Node 1 is not affected.
Local network switching will work with a cluster containing one or more nodes. You may wish to design a single-node cluster in order to take advantage of this local network switching feature in situations where you need only one node and do not wish to set up a more complex cluster. Switching Back to Primary LAN Interfaces after Local Switching If a primary interface fails, the IP address will be switched to a standby.
configuration, all subnets configured on that node, and identified as monitored subnets in the package configuration file, must be available.) Note that remote switching is supported only between LANs of the same type. For example, a remote switchover between an Ethernet interface on one machine and an IPoIB interface on the failover machine is not supported. The remote switching of relocatable IP addresses is shown in Figure 14 and Figure 15.
IMPORTANT: You should configure the IP Monitor in a cross-subnet configuration, because IP monitoring will detect some errors that link-level monitoring will not. See also “Cross-Subnet Configurations” (page 29). How the IP Monitor Works Using Internet Control Message Protocol (ICMP) and ICMPv6, the IP Monitor sends polling messages to target IP addresses and verifies that responses are received.
NOTE: This is the default if cmquerycl detects a gateway for the subnet in question; see SUBNET under “Cluster Configuration Parameters ” (page 105) for more information. IMPORTANT: By default, cmquerycl does not verify that the gateways it detects will work correctly for monitoring. But if you use the -w full option, cmquerycl will validate them as polling targets. SUBNET 192.168.1.0 IP_MONITOR ON POLLING_TARGET 192.168.1.
The following constraints apply to peer polling when there are only two interfaces on a subnet: • If one interface fails, both interfaces and the entire subnet will be marked down on each node, unless Local Switching (page 66) is configured and there is a working standby. • If the node that has one of the interfaces goes down, the subnet on the other node will be marked down.
PRIMARY PRIMARY down (Link and IP) up 0/3/1/0 0/5/1/0 lan2 lan3 cmviewcl -v -f line will report the same failure like this: node:gary|interface:lan2|status=down node:gary|interface:lan2|disabled=false node:gary|interface:lan2|failure_type=link+ip If local switching is not configured and a failure is detected by IP monitoring, output from cmviewcl -v will look like something like this: Network_Parameters: INTERFACE STATUS PRIMARY down (IP only) PRIMARY up PATH 0/3/1/0 0/5/1/0 NAME lan2 lan3 cmviewcl
Figure 28 Aggregated Networking Ports Both the Single and Dual ported LANs in the non-aggregated configuration have four LAN cards, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (lan0, lan1, lan2, lan3). When these ports are aggregated all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as lan900, the name by which the aggregate is known on HP-UX 11i.
remote failover of VLAN interfaces when failure is detected. Failure of a VLAN interface is typically the result of the failure of the underlying physical NIC port or aggregated (APA) ports. Configuration Restrictions HP-UX allows up to 1024 VLANs to be created from a physical NIC port. A large pool of system resources is required to accommodate such a configuration; Serviceguard could suffer performance degradation if many network interfaces are configured in each cluster node.
redundant storage in hardware. Two types of mirroring are RAID1 and RAID5. Here are some differences between the two storage methods: • If you are using JBODs, the basic element of storage is an individual disk. This disk must be paired with another disk to create a mirror (RAID1). (Serviceguard configurations usually have separate mirrors on different storage devices).
NOTE: Under agile addressing (see “About Device File Names (Device Special Files)” (page 77)), the storage units in this example would have names such as disk1, disk2, disk3, etc. As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 99). Figure 29 Physical Disks Within Shared Storage Units Figure 30 shows the individual disks combined in a multiple disk mirrored configuration.
Figure 31 Multiple Devices Configured in Volume Groups Examples of Storage on Disk Arrays Figure 32 shows an illustration of storage configured on a disk array. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system. Figure 32 Physical Disks Combined into LUNs NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer.
NOTE: Under agile addressing, the storage units in these examples would have names such as disk1, disk2, disk3, etc. See “About Device File Names (Device Special Files)” (page 77) As of A.11.20, Serviceguard supports cluster-wide DSFs, and HP recommends that you use them. See “About Cluster-wide Device Special Files (cDSFs)” (page 99). . Figure 33 Multiple Paths to LUNs Finally, the multiple paths are configured into volume groups as shown in Figure 34.
Types of Volume Manager Serviceguard allows a choice of volume managers for data storage: • HP-UX Logical Volume Manager (LVM) and (optionally) Mirrordisk/UX • Veritas Volume Manager for HP-UX (VxVM)—Base and add-on Products • Veritas Cluster Volume Manager for HP-UX Separate sections in Chapters 5 and 6 explain how to configure cluster storage using all of these volume managers.
Veritas Cluster Volume Manager (CVM) NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information on CVM support: http://www.hp.com/go/hpux-serviceguard-docs. You may choose to configure cluster storage with the Veritas Cluster Volume Manager (CVM) instead of the Volume Manager (VxVM).
Redundant Heartbeat Subnets HP recommends that you configure all subnets that connect cluster nodes as heartbeat networks; this increases protection against multiple faults at no additional cost. For CVM 4.1 and later, the minimum recommended configurations are: 1) dual (multiple) heartbeat networks 2) single heartbeat network with standby LAN card(s) Comparison of Volume Managers The following table summarizes the advantages and disadvantages of the volume managers.
Table 5 Pros and Cons of Volume Managers with Serviceguard Product Advantages Tradeoffs Logical Volume Manager (LVM) • Software is provided with all versions of HP-UX. • Lacks flexibility and extended features of some other volume managers • Provides up to 3-way mirroring using optional Mirrordisk/UX software. • Dynamic multipathing (DMP) is active by default as of HP-UX 11i v3.
Table 5 Pros and Cons of Volume Managers with Serviceguard (continued) Product Advantages Tradeoffs • Supports shared activation. • Requires purchase of additional license • Supports exclusive activation. • No support for RAID 5 • Supports activation in different modes on different nodes at the same time • CVM requires all nodes to have connectivity to the shared disk groups • RAID 1+0 mirrored stripes • Not currently supported on all versions of HP-UX • RAID 0+1 striped mirrors • CVM versions 4.
Failure. Only one LAN has been configured for both heartbeat and data traffic. During the course of operations, heavy application traffic monopolizes the bandwidth of the network, preventing heartbeat packets from getting through. Since SystemA does not receive heartbeat messages from SystemB, SystemA attempts to reform as a one-node cluster. Likewise, since SystemB does not receive heartbeat messages from SystemA, SystemB also attempts to reform as a one-node cluster.
Responses to Package and Service Failures In the default case, the failure of a failover package, or of a service within the package, causes the package to shut down by running the control script with the ‘stop’ parameter, and then restarting the package on an alternate node. A package will also fail if it is configured to have a dependency on another package, and that package fails.
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather information about all the hardware and software components of the configuration.
your cluster without the need to bring it down, careful planning of the initial configuration is required. Use the following guidelines: • Remember the rules for cluster locks when considering expansion. A one-node cluster does not require a cluster lock. A two-node cluster must have a cluster lock. In clusters larger than 3 nodes, a cluster lock is strongly recommended. If you have a cluster with more than 4 nodes, you can use a Quorum Server but a cluster lock disk is not allowed.
slots, and the bus address for each adapter; you can update the details as you do the cluster configuration (described in Chapter 5). SPU Information SPU information includes the basic characteristics of the systems you are using in the cluster. Different models of computers can be mixed in the same cluster. This configuration model also applies to HP Integrity servers. HP-UX workstations are not supported for Serviceguard.
For more details of IPv6 address format, see the Appendix H (page 364). NETWORK_FAILURE_DETECTION When there is a primary and a standby network card, Serviceguard needs to determine when a card has failed, so it knows whether to fail traffic over to the other card. The configuration file specifies one of two ways to decide when the network interface card has failed: • INOUT • INONLY_OR_INOUT The default is INOUT.
Table 6 SCSI Addressing in Cluster Configuration (continued) System or Disk Host Interface SCSI Address Disk #6 14 Others 13 - 8 NOTE: When a boot/root disk is configured with a low-priority address on a shared SCSI bus, a system panic can occur if there is a timeout on accessing the boot/root device. This can happen in a cluster when many nodes and many disks are configured on the same bus.
Hardware Configuration Worksheet You may find a worksheet such as the following useful for organizing and recording your cluster hardware configuration. This worksheet is an example; blank worksheets are in Appendix E. Make as many copies as you need.
units you are using and the power supply to which they will be connected; record the following information: Host Name The host name for each SPU. Disk Unit The disk drive unit number for each disk. Tape Unit The tape unit number for each backup device. Other Unit Tthe number of any other unit. Power Supply The power supply unit number of the UPS to which the host or other device is connected. Be sure to follow UPS and cabinet power limits as well as SPU power limits.
Quorum Server as the cluster lock. In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes. For more information on lock disks, lock LUNs, and the Quorum Server, see “Choosing Cluster Lock Disks” (page 164), “Setting Up a Lock LUN” (page 165), and “Setting Up and Running the Quorum Server” (page 168).
Supported Node Names The name (39 characters or fewer) of each cluster node that will be supported by this quorum server. These entries will be entered into qs_authfile on the system that is running the quorum server process.
NOTE: EMS cannot be used to monitor the status of VxVM disk groups. For this you should use the volume monitor cmvolmond which is supplied with Serviceguard. cmvolmond can also monitor LVM volumes. See “About the Volume Monitor” (page 123). resource_name /vg/vgpkg/pv_summary resource_polling_interval 60 resource_start AUTOMATIC resource_up_value = UP The example above will monitor all PV links for the volume group vgpkg.
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ CVM and VxVM Planning You can create storage groups using the HP-UX Logical Volume Manager (LVM, described in the previous section), or using Veritas VxVM and CVM software.
Cluster Configuration Planning A cluster should be designed to provide the quickest possible recovery from failures. The actual time required to recover from a failure depends on several factors: • The value of the cluster MEMBER_TIMEOUT. See MEMBER_TIMEOUT under “Cluster Configuration Parameters ” (page 105) for recommendations. • The availability of raw disk access. Applications that use raw disk access should be designed with crash recovery services. • The application and database recovery time.
Where cDSFs Reside cDSFs reside in two new HP-UX directories, /dev/cdisk for cluster-wide block devicefiles and /dev/rcdisk for cluster-wide character devicefiles. Persistent DSFs that are not cDSFs continue to reside in /dev/disk and /dev/rdisk, and legacy DSFs (DSFs using the naming convention that was standard before HP–UX 11i v3) in /dev/dsk and /dev/rdsk.
The Easy Deployment tool consists of three commands: cmpreparecl (1m), cmdeploycl (1m), and cmpreparestg (1m). These commands allow you to get a cluster up and running in the minimum amount of time.
NOTE: For heartbeat configuration requirements, see the discussion of the HEARTBEAT_IP parameter later in this chapter. For more information about managing the speed of cluster re-formation, see the discussion of the MEMBER_TIMEOUT parameter, and further discussion under “What Happens when a Node Times Out” (page 85) ,“Modifying the MEMBER_TIMEOUT Parameter” (page 183), and, for troubleshooting, “Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low” (page 320).
For more information about IPv6, see Appendix H (page 364). Rules and Restrictions for IPv6-Only Mode IMPORTANT: See the latest version of the Serviceguard release notes for the most current information on these and other restrictions. • All addresses used by the cluster must be in each node's /etc/hosts file.
NOTE: CVM/CFS can be installed on any system in an IPv6–only cluster using HP Serviceguard A.11.20 April 2011 patches with Veritas Storage Foundation™ 5.1 SP1 and later, even if they are not configured. This also means that the Serviceguard component of bundles that include Serviceguard cannot be configured in IPV6 mode prior to Veritas Storage Foundation™ 5.1 SP1. • HPVM is not supported IPv6-only mode. You cannot configure a virtual machine either as a node or a package in an IPv6-only cluster.
NOTE: • This also applies if HOSTNAME_ADDRESS_FAMILY is set to IPV6. HPVM is not supported. You cannot have a virtual machine that is either a node or a package if HOSTNAME_ADDRESS_FAMILY is set to ANY or IPV6. Cluster Configuration Parameters You need to define a set of cluster parameters. These are stored in the binary cluster configuration file, which is distributed to each node in the cluster.
IMPORTANT: See “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 102) for important information. See also the latest Serviceguard release notes at http://www.hp.com/go/ hpux-serviceguard-docs. FIRST_CLUSTER_LOCK_VG, SECOND_CLUSTER_LOCK_VG The volume group containing the physical disk volume on which a cluster lock is written. This parameter is used only when you employ a lock disk for tie-breaking services in the cluster.
only if you use a Quorum Server and want to specify an address on an alternate subnet by which it can be reached. The alternate subnet need not use the same address family as QS_HOST if HOSTNAME_ADDRESS_FAMILY is set to ANY. For more information, see “Using a Quorum Server” (page 95) and “Specifying a Quorum Server” (page 180).
the documents listed under “Cross-Subnet Configurations” (page 29) for more information. You can define multiple SITE_NAMEs. SITE_NAME entries must precede any NODE_NAME entries. See also SITE. NODE_NAME The hostname of each system that will be a node in the cluster. CAUTION: Make sure that the node name is unique within the subnets configured on the cluster nodes; under some circumstances Serviceguard may not be able to detect a duplicate name and unexpected problems may result.
NETWORK_INTERFACE The name of each LAN that will be used for heartbeats or for user data on the node identified by the preceding NODE_NAME. An example is lan0. See also HEARTBEAT_IP, STATIONARY_IP, and “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 102).
on a given subnet must all be of the same type: IPv4 or IPv6 site-local or IPv6 global. For information about changing the configuration online, see “Changing the Cluster Networking Configuration while the Cluster Is Running” (page 284).
subnet on another node (that is, each heartbeat path must be physically separate). See “Cross-Subnet Configurations” (page 29) for more information. NOTE: Limitations: • Because Veritas Cluster File System from Symantec (CFS) requires link-level traffic communication (LLT) among the nodes, Serviceguard cannot be configured in cross-subnet configurations with CFS alone.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 120)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
this list). Used only if a lock disk is used for tie-breaking services. Use FIRST_CLUSTER_LOCK_PV for the first physical lock volume and SECOND_CLUSTER_LOCK_PV for the second physical lock volume, if any. If there is a second physical lock volume, SECOND_CLUSTER_LOCK_PV must be on a separate line. These parameters are used only when you employ a lock disk for tie-breaking services in the cluster.
of four capacities per cluster, unless you use the reserved CAPACITY_NAME package_limit; in that case, you can use only that capacity throughout the cluster. For all capacities other than package_limit, the default weight for all packages is zero, though you can specify a different default weight for any capacity other than package_limit; see the entry for WEIGHT_NAME and WEIGHT_DEFAULT later in this list. See “About Package Weights” (page 135) for more information.
NOTE: • For most clusters that use an LVM cluster lock or lock LUN, a minimum MEMBER_TIMEOUT of 14 seconds is appropriate. • For most clusters that use a MEMBER_TIMEOUT value lower than 14 seconds, a quorum server is more appropriate than a lock disk or lock LUN. The cluster will fail if the time it takes to acquire the disk lock exceeds 0.2 times the MEMBER_TIMEOUT.
See also “What Happens when a Node Times Out” (page 85), “Cluster Daemon: cmcld” (page 39), and the white paper Optimizing Failover Time in a Serviceguard Environment (version A.11.19 and later) on http:// www.hp.com/go/hpux-serviceguard-docs. Can be changed while the cluster is running. AUTO_START_TIMEOUT The amount of time a node waits before it stops trying to join a cluster during automatic cluster startup.
first determine the Maximum Bridge Transit Delay (MBTD) for each switch and router. The value should be in the vendors' documentation. Set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the values for the switches and routers. If there is more than one possible path between the NFS server and the cluster nodes, sum the values for each path and use the largest number. CAUTION: Serviceguard supports NFS-mounted file systems only over switches and routers that support MBTD.
By default, each of the cluster subnets is listed under SUBNET, and, if at least one gateway is detected for that subnet, IP_MONITOR is set to ON and POLLING_TARGET entries are populated with the gateway addresses, enabling target polling; otherwise the subnet is listed with IP_MONITOR set to OFF. See “Monitoring LAN Interfaces and Detecting Failure: IP Level” (page 70) for more information.
NOTE: cmquerycl (1m) detects first-level routers in the cluster (by looking for gateways in each node's routing table) and lists them here as polling targets. If you run cmquerycl with the -w full option (for full network probing) it will also verify that the gateways will work correctly for monitoring purposes. Can be changed while the cluster is running; must be removed if the preceding SUBNET entry is removed.
Access Control Policies (also known as Role Based Access) For each policy, specify USER_NAME, USER_HOST, and USER_ROLE. Policies set in the configuration file of a cluster and its packages must not be conflicting or redundant. For more information, see “Setting up Access-Control Policies” (page 185). VOLUME_GROUP The name of an LVM volume group whose disks are attached to at least two nodes in the cluster. Such disks are considered cluster-aware.
As part of planning, you need to decide the following: • What volume groups are needed? • How much disk space is required, and how should this be allocated in logical volumes? • What file systems need to be mounted for each package? • Which nodes need to import which logical volume configurations? • If a package moves to an adoptive node, what effect will its presence have on performance? Create a list by package of volume groups, logical volumes, and file systems.
The system multi-node package SG-CFS-pkg manages the cluster’s volumes. When you use CFS, Serviceguard enables you to create and manage multi-node packages for disk groups and mount points using modular style packages or legacy style packages. To create modular CVM disk group and CFS mount point packages, use the cmmakepkg command and edit the parameters in the package configuration file. You can choose to name the packages. You can also use the Serviceguard Manager to create the modular CFS packages.
3. The mount point packages should not run unless the disk group packages are running. If the disk groups and mount points are in separate packages, specify the dependencies on the disk group packages in the configuration file. CAUTION: Once you create the modular CVM disk group and CFS mount point packages, you must administer the cluster with cmcheckconf, cmapplyconf, cmrunpkg, cmmodpkg, and cmrunpkg commands.
When a monitored volume fails or becomes inaccessible, the monitor service will exit, causing the package to fail on the current node. The package’s failure behavior depends on its configured settings. For prompt recovery, HP recommends setting the value of service_restart (page 233) for the monitoring service to none.
The full path to at least one VxVM volume or LVM logical volume device file for monitoring (required). The pathname must identify a block device file. Examples /usr/sbin/cmvolmond -O /log/monlog.log -D 3 /dev/vx/dsk/cvm_dg0/lvol2 This command monitors a single VxVM volume, /dev/vx/dsk/cvm_dg0/lvol2, at log level 3, with a polling interval of 60 seconds, and prints all log messages to /log/monlog.log.
IMPORTANT: Find out the MBTD value for each affected router and switch from the vendors' documentation; determine all of the possible paths; find the worst case sum of the MBTD values on these paths; and use the resulting value to set the Serviceguard CONFIGURED_IO_TIMEOUT_EXTENSION parameter. For instructions, see the discussion of this parameter under “Cluster Configuration Parameters ” (page 105). Switches and routers that do not support MBTD value must not be used in a Serviceguard NFS configuration.
This paper includes instructions for setting up a sample package that uses an NFS-imported file system. See also the description of fs_name (page 238), fs_type (page 238), and the other file system-related package parameters. Planning for Expansion You can add packages to a running cluster. This process is described in “Cluster and Package Maintenance” (page 248).
Parameters for Configuring EMS Resources NOTE: The default form for parameter names and literal values in the modular package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameters are concerned. This manual uses lower case, unless the parameter in question is used only in legacy packages, or the context refers exclusively to such a package.
are discussed later in this section under “Extended Dependencies” (page 132). You should read the next section, “Simple Dependencies” (page 129), first. Simple Dependencies A simple dependency occurs when one package requires another to be running on the same node. You define these conditions by means of the parameters dependency_condition and dependency_location, using the literal values UP and same_node, respectively.
• If pkg1 is a failover package and pkg2 is a multi-node or system multi-node package, and pkg2 fails, pkg1 will halt and fail over to the next node on its node_name list on which pkg2 is running (and any other dependencies, such as resource dependencies or a dependency on a third package, are met). • In the case of failover packages with a configured_node failover_policy, a set of rules governs under what circumstances pkg1 can force pkg2 to start on a given node.
NOTE: Keep the following in mind when reading the examples that follow, and when actually configuring priorities: 1. auto_run (page 224) should be set to yes for all the packages involved; the examples assume that it is. 2. Priorities express a ranking order, so a lower number means a higher priority (10 is a higher priority than 30, for example).
In a simple dependency, if pkg1 depends on pkg2, and pkg1’s priority is higher than pkg2’s, pkg1’s node order dominates. Assuming pkg1’s node order is node1, node2, node3, then: • On startup: ◦ pkg1 will select node1 to start on, provided pkg2 can run there. ◦ pkg2 will start on node1, provided it can run there (no matter where node1 appears on pkg2’s node_name list). – ◦ If pkg2 is already running on another node, it will be dragged to node1, provided it can run there.
another package be down as an exclusionary dependency; see “Rules for Exclusionary Dependencies” (page 133). • You can specify where the dependency_condition must be satisfied: on the same node, a different node, all nodes, or any node in the cluster. You define this by means of the dependency_location parameter (page 228), using one of the literals same_node, different_node, all_nodes, or any_node. different_node and any_node are allowed only if dependency_condition is UP.
Rules for different_node and any_node Dependencies These rules apply to packages whose dependency_condition is UP and whose dependency_location is different_node or any_node. For same-node dependencies, see Simple Dependencies (page 129); for exclusionary dependencies, see “Rules for Exclusionary Dependencies” (page 133). • Both packages must be failover packages whose failover_policy (page 226) is configured_node.
3. Halts packages the failing package depends on, starting with the package this package immediately depends on. The packages are halted only if: • these are failover packages, and • the failing package can “drag” these packages to a node on which they can all run. Otherwise the failing package halts and the packages it depends on continue to run 4. Starts the packages the failed package depends on (those halted in step 3, if any).
Simple Method Use this method if you simply want to control the number of packages that can run on a given node at any given time. This method works best if all the packages consume about the same amount of computing resources. If you need to make finer distinctions between packages in terms of their resource consumption, use the Comprehensive Method (page 137) instead. To implement the simple method, use the reserved keyword package_limit to define each node's capacity.
you wanted to ensure that the larger packages, pkg2 and pkg3, did not run on node1 at the same time, you could raise the weight_value of one or both so that the combination exceeded 10 (or reduce node1's capacity to 8). Points to Keep in Mind The following points apply specifically to the Simple Method (page 136). Read them in conjunction with the Rules and Guidelines (page 141), which apply to all weights and capacities.
meanings of the names processor and memory; there is no mapping to actual processor and memory usage and you would get exactly the same results if you used the names apples and oranges, for example. Suppose you have the following configuration: • A two node cluster running four packages. These packages contend for resource we'll simply call A and B. • node1 has a capacity of 80 for A and capacity of 50 for B. • node2 has a capacity of 60 for A and capacity of 70 for B.
NOTE: You do not have to define capacities for every node in the cluster. If any capacity is not defined for any node, Serviceguard assumes that node has an infinite amount of that capacity.
NOTE: Option 4 means that the package is “weightless” as far as this particular capacity is concerned, and can run even on a node on which this capacity is completely consumed by other packages. (You can make a package “weightless” for a given capacity even if you have defined a cluster-wide default weight; simply set the corresponding weight to zero in the package's cluster configuration file.
(page 141)). This is true whenever a package has a weight that exceeds the available amount of the corresponding capacity on the node. Rules and Guidelines The following rules and guidelines apply to both the Simple Method (page 136) and the Comprehensive Method (page 137) of configuring capacities and weights. • You can define a maximum of four capacities, and corresponding weights, throughout the cluster.
decide which package to start if it cannot start them both because there is not enough node capacity to support their weight. Example 1 • pkg1 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 10. It is down and has switching disabled. • pkg2 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 20. It is running on node turkey and has switching enabled. • turkey and griffon can run one package each (package_limit is set to 1).
Each external script must have three entry points: start, stop, and validate, and should exit with one of the following values: • 0 - indicating success. • 1 - indicating the package will be halted, and should not be restarted, as a result of failure in this script. • 2 - indicating the package will be restarted on another node, or halted if no other node is available.
elif (( PEV_MONITORING_INTERVAL < 1 )) then sg_log 0 "ERROR: PEV_MONITORING_INTERVAL value ($PEV_MONITORING_INTERVAL) not within legal limits!" ret=1 fi # check monitoring service we are expecting for this package is configured while (( i < ${#SG_SERVICE_NAME[*]} )) do case ${SG_SERVICE_CMD[i]} in *monitor.
commands in their external scripts. If a timeout is not specified and your configuration has a command loop as described above, inconsistent results can occur, including a hung cluster. Determining Why a Package Has Shut Down You can use an external script (or CUSTOMER DEFINED FUNCTIONS area of a legacy package control script) to find out why a package has shut down.
monitored_subnet_access unconfigured for a monitored subnet is equivalent to FULL.) (For legacy packages, see “Configuring Cross-Subnet Failover” (page 295).) • You should not use the wildcard (*) for node_name (page 223) in the package configuration file, as this could allow the package to fail over across subnets when a node on the same subnet is eligible. Failing over across subnets can take longer than failing over on the same subnet. List the nodes in order of preference instead of using the wildcard.
Configuring node_name First you need to make sure that pkg1 will fail over to a node on another subnet only if it has to. For example, if it is running on NodeA and needs to fail over, you want it to try NodeB, on the same subnet, before incurring the cross-subnet overhead of failing over to NodeC or NodeD.
Planning for Changes in Cluster Size If you intend to add additional nodes to the cluster online (while it is running) ensure that they are connected to the same heartbeat subnets and to the same lock disks as the other cluster nodes. For cross-subnet considerations, see “Cross-Subnet Configurations” (page 29). In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes.
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. These procedures are carried out on one node, called the configuration node, and the resulting binary file is distributed by Serviceguard to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
################## cmcluster.conf ############### # Highly Available Cluster file locations # This file must not be edited ################################################# SGCONF=/etc/cmcluster SGSBIN=/usr/sbin SGLBIN=/usr/lbin SGLIB=/usr/lib SGRUN=/var/adm/cmcluster SGAUTOSTART=/etc/rc.config.d/cmcluster SGFFLOC=/opt/cmcluster/cmff CMSNMPD_LOG_FILE=/var/adm/SGsnmpsuba.log NOTE: If these variables are not defined on your system, then source the file /etc/ cmcluster.
1. Identify the nodes for which you want to create cluster-wide device files; this is known as the cDSF group. This should be all the nodes in the cluster (or prospective cluster). IMPORTANT: 2. Remember that: • You cannot create a cDSF group that crosses cluster boundaries; that is, the group must consist of the nodes of a single cluster. • cDSFs use agile addressing; see “About Device File Names (Device Special Files)” (page 77) for information about agile addressing.
“Specifying a Lock LUN” (page 180), and “Creating the Storage Infrastructure and file systems with LVM, VxVM and CVM” (page 168). Adding a Node to a cDSF Group When you add a new node to a cluster that uses cDSFs, you also need to add the node to the cDSF group.
• Make sure all the subnets used by the prospective nodes are accessible to all the nodes. Easy Deployment commands will fail if they detect an asymmetric network configuration, in which only a subset of nodes has access to a given subnet. • Make sure that the LAN or LANs that will be used for the cluster heartbeat meet high-availability requirements. See the requirements for HEARTBEAT_IP listed under “Cluster Configuration Parameters ” (page 105), and the networking “Rules and Restrictions” (page 27).
2. Edit the output file ($SGCONF/mynetwork in this example) to add entries for additional heartbeats, if necessary. NOTE: For information about heartbeat and networking requirements, see the sections listed under “Before You Start” (page 152). If you omit steps 1 and 2, and all the prospective nodes are connected to at least one subnet, cmdeploycl behaves as follows (the configuration is actually done by cmquerycl (1m) which is called by cmdeploycl).
NOTE: You can also accomplish this by running cmpreparecl; cmdeploycl calls cmpreparecl. For more information, see “Configuring Root-Level Access” (page 157) and “Configuring Name Resolution” (page 159). Those sections describe the “manual” (HP-UX command-line) method of accomplishing what cmpreparecl does. • Configures a two-node cluster (cluster1). • Configures the LVM volume group and physical volume for the lock disk. • Starts the cluster on ftsys9 and ftsys10.
5. Create shared storage for a LVM volume group with mirroring for use by cluster packages. cmpreparestg –l –p ... –n –n is the path name of the volume group that is to be created and imported to each specified node, and each is the name of a physical volume on the local node from which the volume group specified by is to be created.
6. Verify the storage using the following commands: • For LVM volume groups: vgdisplay • For VxVM/CVM disk groups: vxdg list The volume group or the disk group can now be used by packages; see Chapter 6 (page 216). NOTE: cmpreparestg requires either -l or -g in all cases. The command must be run on one of the specified nodes (ftsys9 or ftsys10 in this example).
For example: gryf sly bit root root root #cluster1, node1 #cluster1, node2 #cluster1, node3 This example grants root access to the node on which this cmclnodelist file resides to root users on the nodes gryf, sly, and bit. Serviceguard also accepts the use of a “+” in the cmclnodelist file; this indicates that the root user on any Serviceguard node can configure Serviceguard on this node. IMPORTANT: If $SGCONF/cmclnodelist does not exist, Serviceguard will look at ~/.rhosts.
Configuring Name Resolution Serviceguard uses the name resolution services built into HP-UX. NOTE: If you plan to use cmpreparecl (1m) (or cmpdeploycl (1m), which calls cmpreparecl), the /etc/hosts and /etc/nsswitch.conf configuration described in this subsection will be done automatically, but you should still read the entire subsection and make sure you understand the issues. In particular, you still need to make sure that aliases are properly represented in /etc/hosts, as described below.
15.145.162.131 10.8.0.131 10.8.1.131 gryf.uksr.hp.com gryf2.uksr.hp.com gryf3.uksr.hp.com gryf gryf gryf 15.145.162.132 10.8.0.132 10.8.1.132 sly.uksr.hp.com sly2.uksr.hp.com sly3.uksr.hp.com sly sly sly If applications require the use of hostname aliases, the Serviceguard hostname must be one of the aliases in all the entries for that host.
NOTE: If a NIC fails, the affected node will be able to fail over to a standby LAN so long as the node is running in the cluster. But if a NIC that is used by Serviceguard fails when the affected node is not running in the cluster, Serviceguard will not be able to restart the node. (For instructions on replacing a failed NIC, see “Replacing LAN or Fibre Channel Cards” (page 314).) NOTE: If you plan to use cmpreparecl (1m) (orcmpdeploycl (1m), which calls cmpreparecl), the /etc/hosts and /etc/nsswitch.
cluster node, they may also need to be changed on other cluster nodes that can run the same packages. Enabling the Network Time Protocol HP strongly recommends that you enable network time protocol (NTP) services on each node in the cluster. The use of NTP, which runs as a daemon process on each system, ensures that the system time on all nodes is consistent, resulting in consistent timestamps in log files and consistent behavior of message services.
model, see the HP-UX IPSec Version A.03.00 Administrator's Guide which you can find at http://www.hp.com/go/hpux-security-docs —> HP-UX IPSec Software. ip_strong_es_model is disabled (set to 0) by default. Setting it to 1 enables it. CAUTION: HP supports enabling this parameter only if you are not using a cross-subnet configuration (page 29). Otherwise, leave the parameter at its default setting (zero, meaning disabled).
The following is an example of mirroring the root logical volume: lvextend -m 1 /dev/vg00/lvol3 /dev/dsk/c4t6d0 5. Update the boot information contained in the BDRA for the mirror copies of boot, root and primary swap. /usr/sbin/lvlnboot -b /dev/vg00/lvol1 /usr/sbin/lvlnboot -s /dev/vg00/lvol2 /usr/sbin/lvlnboot -r /dev/vg00/lvol3 6. Verify that the mirrors were properly created.
NOTE: You must use the vgcfgbackup and vgcfgrestore commands to back up and restore the lock volume group configuration data regardless of how you create the lock volume group. Setting Up a Lock LUN LUN stands for Logical Unit Number. The term can refer to a single physical disk, but these days is more often used in a SAN (Storage Area Network) or NAS (Network-Attached Storage) context to denote a virtual entity derived from one or more physical disks.
IMPORTANT: On HP 9000 systems, there is no means of partitioning a disk or LUN, so you will need to dedicate an entire small disk or LUN for the lock LUN. This means that in a mixed cluster containing both Integrity and HP-PA systems, you must also use an entire disk or LUN; if you partition the device as described below, the HP-PA nodes will not be able to see the partitions.
Defining the Lock LUN Use cmquerycl -L to create a cluster configuration file that defines the lock LUN. • If the pathname for the lock LUN is the same on all nodes, use a command such as: cmquerycl -C $SGCONF/config.ascii -L /dev/dsk/c0t1d1 -n -n • If the pathname for the lock LUN is different on some nodes, you must specify the path on each node; for example (all on one line): cmquerycl -C $SGCONF/config.
IMPORTANT: The purpose of cmnotdisk.conf is to exclude specific devices, usually CD and DVD drives, that Serviceguard does not recognize and which should not be probed. Make sure you do not add a DEVICE_FILE entry in cmnotdisk.conf for any device that should be probed; that is, disk devices being managed by LVM or LVM2. Excluding any such device will cause cmquerycl to fail.
NOTE: The procedures that follow describe the command-line method of configuring LVM storage. There are two other, more automated methods you can use. • System Management Homepage You can use the System Management Homepage to create or extend volume groups and create logical volumes. From the System Management Homepage, choose Disks and File Systems. Make sure you create mirrored logical volumes with PVG-strict allocation; see“Using Mirrored Individual Data Disks” (page 169).
lssf /dev/d*/* In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which happen to be the device names for the same disks on both ftsys9 and ftsys10. In the event that the device file names are different on the different nodes, make a careful note of the correspondences. NOTE: Under agile addressing, the physical devices in these examples would have names such as /dev/rdisk/disk1 and /dev/rdisk/disk2. See “About Device File Names (Device Special Files)” (page 77).
Creating Logical Volumes NOTE: You can create a single logical volume or multiple logical volumes by means of the cmpreparestg (1m) command. See “Using Easy Deployment Commands to Configure the Cluster” (page 153) and the manpage for more information. If you use cmpreparestg, you can skip this step and proceed to “Making Physical Volume Group Files Consistent” (page 173). Use a command such as the following to create a logical volume (the example is for /dev/ vgdatabase).
Distributing Volume Groups to Other Nodes After creating volume groups for cluster packages, you must make them available to any cluster node that will need to activate the volume group. The cluster lock volume group must be made available to all nodes. NOTE: You can distribute the volume groups by means of the cmpreparestg (1m) command. See “Using Easy Deployment Commands to Configure the Cluster” (page 153) for more information.
Note that the disk device names onftsys10 may be different from their names on ftsys9. Make sure the physical volume names are correct throughout the cluster. When the volume group can be activated on this node, perform a vgcfgbackup. (This backup will be available in the unlikely event that a vgcfgrestore must be performed on this node because of a disaster on the primary node and an LVM problem with the volume group.
3. 4. 5. If /etc/lvmpvg on ftsys10 contains entries for volume groups that do not appear in /etc/ lvmpvg.new, copy all physical volume group entries for that volume group to/etc/ lvmpvg.new. Adjust any physical volume names in /etc/lvmvpg.new to reflect their correct names on ftsys10. On ftsys10, copy /etc/lvmpvg to /etc/lvmpvg.old, to create a backup. Copy /etc/ lvmpvg.new to /etc/lvmpvg on ftsys10.
NOTE: These commands make the disk and its data unusable by LVM, and allow it to be initialized by VxVM. (The commands should only be used if you have previously used the disk with LVM and do not want to save the data on it.
NOTE: You can create file systems by means of the cmpreparestg (1m) command. See “Using Easy Deployment Commands to Configure the Cluster” (page 153) for more information. If you use cmpreparestg, you can skip the following steps, but it is a good idea to read them so that you understand what cmpreparestg does for you. Use the following commands to create a file system for mounting on the logical volume just created: 1.
Note that the clearimport is done for disks previously imported with noautoimport set on any system that has Serviceguard installed, whether it is configured in a cluster or not. Configuring the Cluster This section describes how to define the basic cluster configuration using traditional Serviceguard commands. This must be done on a system that is not part of a Serviceguard cluster (that is, on which Serviceguard is installed but not configured).
Specifying the Address Family for the Cluster Hostnames You can use the -a option to tell Serviceguard to resolve cluster node names (as well as Quorum Server hostnames, if any) to IPv4 addresses only (-a ipv4) IPv6 addresses only (-a ipv6), or both (-a any). You can also configure the address family by means of the HOSTNAME_ADDRESS_FAMILY in the cluster configuration file.
NOTE: You can specify only one lock disk on the command line; if you need to specify a second cluster lock disk, you must do so in the cluster configuration file. For more information, see “Specifying a Lock Disk” (page 179), “Specifying a Lock LUN” (page 180), and “Specifying a Quorum Server” (page 180). Generating a Network Template File As of Serviceguard A.11.
Do not include the node’s entire domain name; for example, specify ftsys9, not ftsys9.cup.hp.com: cmquerycl -v -n ftsys9 -n ftsys10 cmquerycl will not print out the re-formation time for a volume group that currently belongs to a cluster. If you want cmquerycl to print the re-formation time for a volume group, run vgchange -c n to clear the cluster ID from the volume group.
cmquerycl -q -n ftsys9 -n ftsys10 -C .conf To specify an alternate hostname or IP address by which the Quorum Server can be reached, use a command such as (all on one line): cmquerycl -q -n ftsys9 -n ftsys10 -C .conf Enter the QS_HOST (IPv4 or IPv6), optional QS_ADDR (IPv4 or IPv6), QS_POLLING_INTERVAL, and optionally a QS_TIMEOUT_EXTENSION; and also check the HOSTNAME_ADDRESS_FAMILY setting, which defaults to IPv4.
15.13.164.0 lan1 lan1 (nodeA) (nodeB) 15.13.172.0 lan1 lan1 (nodeC) (nodeD) 15.13.165.0 lan2 lan2 (nodeA) (nodeB) 15.13.182.0 lan2 lan2 (nodeC) (nodeD) 15.244.65.0 lan3 lan3 (nodeA) (nodeB) 15.244.56.0 lan4 lan4 (nodeC) (nodeD) 3ffe:1111::/64 lan3 lan3 (nodeA) (nodeB) 3ffe:2222::/64 lan3 lan3 (nodeC) (nodeD) IPv6: Possible Heartbeat IPs: 15.13.164.0 15.13.164.1 15.13.164.2 (nodeA) (nodeB) 15.13.172.0 15.13.172.158 15.13.172.159 (nodeC) (nodeD) 15.13.165.0 15.13.165.1 15.13.
IMPORTANT: Note that in this example subnet 15.244.65.0, used by NodeA and NodeB, is not routed to 15.244.56.0, used by NodeC and NodeD. But subnets 15.13.164.0 and 15.13.165.0, used by NodeA and NodeB, are routed respectively to subnets 15.13.172.0 and15.13.182.0, used by NodeC and NodeD. At least one such routing among all the nodes must exist for cmquerycl to succeed.
A Note about Terminology Although you will also sometimes see the term role-based access (RBA) in the output of Serviceguard commands, the preferred set of terms, always used in this manual, is as follows: • Access-control policies- the set of rules defining user access to the cluster. ◦ • Access-control policy - one of these rules, comprising the three parameters USER_NAME, USER_HOST, USER_ROLE. See “Setting up Access-Control Policies” (page 185).
Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 36 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
NOTE: For more information and advice, see the white paper Securing Serviceguard at http:// www.hp.com/go/hpux-serviceguard-docs. Define access-control policies for a cluster in the cluster configuration file; see “Cluster Configuration Parameters ” (page 105). You can define up to 200 access policies for each cluster. A root user can create or modify access control policies while the cluster is running.
only. These roles are not exclusive; for example, you can configure more than one PACKAGE_ADMIN for the same package. NOTE: You do not have to halt the cluster or package to configure or modify access control policies. Here is an example of an access control policy: USER_NAME john USER_HOST bit USER_ROLE PACKAGE_ADMIN If this policy is defined in the cluster configuration file, it grants user john the PACKAGE_ADMIN role for any package on node bit.
NOTE: Check spelling especially carefully when typing wildcards, such as ANY_USER and ANY_SERVICEGUARD_NODE. If they are misspelled, Serviceguard will assume they are specific users or nodes.
• That each node is connected to each heartbeat network. • That all heartbeat networks are of the same type of LAN. • That network interface device files specified are valid LAN device files. • That VOLUME_GROUP entries are marked as cluster-aware. If the cluster is online, the check also verifies that all the conditions for the specific change in configuration have been met.
NOTE: The apply will not complete unless the cluster lock volume group is activated on exactly one node before applying. There is one exception to this rule: a cluster lock had been previously configured on the same physical volume and volume group. After the configuration is applied, the cluster lock volume group must be deactivated.
Table 8 Differences between Legacy CFS and Modular CFS Legacy Modular The CFS legacy disk groups and mount points can be created or managed only by using the cfsdgadm, cfsmntadm, cfsmount, or cfsumount. Unlike the legacy CFS packages, the modular CVM disk groups and CFS mount points cannot be created or managed by using the cfs commands. These are created and managed using cmmakepkg, cmcheckconf, cmapplyconf, cmdeleteconf, cmrunpkg, cmhaltpkg, or cmmodpkg commands.
Table 9 Operational commands for Legacy CFS and Modular CFS (continued) Operation Modifying attributes for a disk group in a package Commands used by legacy style cfsdgadm modify Equivalent commands in modular style For modifying the attributes in an offline mode: • cmhaltpkg • edit parameters in the configuration file • cmapplyconf with modified parameters • cmrunpkg For modifying the attributes in an online mode: • edit parameters in the configuration file • cmapplyconf with modified parameters Create
NOTE: The “Creating the Storage Infrastructure with Veritas Cluster Volume Manager (CVM)” (page 208) section explains how to configure Veritas Cluster Volume Manager (CVM) disk groups without CFS; that is, for raw access only. Both solutions use many of the same commands, but in a slightly different order. Refer to the Serviceguard man pages for more information about the commands cfscluster, cfsdgadm, cfsmntadm, cfsmount, cfsumount, and cmgetpkgenv.
NOTE: Because the CVM system multi-node package automatically starts up the Veritas processes, do not edit /etc/llthosts, /etc/llttab, or /etc/gabtab. The cfscluster status command displays the status of the disk groups and mount point packages created only for legacy CFS packages. Creating the Disk Groups Initialize the disk group from the master node. 1. Find the master node using vxdctl or cfscluster status 2.
SG-CFS-DG-1 Creating Volumes 1. Make log_files volume on the logdata disk group: vxassist -g logdata make log_files 1024m 2.
you to create a package configuration file. The command uses the sg/cfs module that contains attributes for CVM disk groups, mount points, storage snapshot, and checkpoint parameters. A modular CFS package template can be generated using the sg/cfs_all template. The modular CFS packages run as a multi-node package and have a dependency on CVM system multi-node package, SG-CFS-pkg.
node_name node_name node3 node4 cvm_disk_group cvm_activation_mode cvm_dg1_node1 "node1=sw node2=sw node3=sw node4=sw" cfs_mount_point cfs_volume cfs_mount_options cfs_primary_policy /mnt2 cvm_dg1_node1/lvol1 "node1=cluster node2=cluster node3=cluster node4=cluster" "" NOTE: By default, the package parameters are commented. You must uncomment them and specify the values.
5. Run the package: cmrunpkg cfspkg1 6. Verify that the modular package of disk group(s) and mount point(s) is running.
IMPORTANT: It is recommended to increment the parameter values in smaller numbers and monitor the effect on performance at each level. As soon as the performance starts to decline, stop increasing and reset it to a lower level. However, you must consider the factors like the number of CPUs, the amount of available memory, the HP-UX kernel settings, and the number and characteristics of other packages that will be running on the node.
4. Check the package configuration file: cmcheckconf -P /etc/cmcluster/ckpt1.ascii cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration 5. Apply the package configuration file: cmapplyconf -P /etc/cmcluster/ckpt1.ascii Modify the package configuration ([y]/n)? y Completed the cluster update 6.
NOTE: By default, the package parameters are commented out. You must uncomment them and specify the values. 4. snapshot_mount_point name of the mount point for the snapshot snapshot_name Diskgroups/Logical volume pair used by the snapshot snapshot_cfs_mount_point unique name for CFS mount point; supports up to two levels of nested mount points snapshot_mount_options the CFS mount options for the snapshot. For detailed information about the mount options, see the mount_vxfs (1m) manpage.
/dev/vg00/lvol4 2048000 18290 1902964 /dev/vg00/lvol6 10076160 5666019 4134547 /dev/vg00/lvol8 3063808 17747 2855686 /dev/odm 0 0 0 /dev/vx/dsk/cvm_dg3_node1/lvol3 167776 3260 154234 1% /tmp 58% /opt 1% /home 0% /dev/odm 2% /snap1 Guidelines for Migrating from Legacy CFS Package to Modular CFS Package • You must migrate the legacy CFS packages to modular CFS package only when the package is offline.
Figure 37 Legacy Style of Packaging Figure 38 illustrates an example in which all the disk group and mount point packages used by Application 1 are consolidated into a single modular package as compared to four separate packages in the legacy style of packaging. Figure 38 Modular Style of Packaging Use Case 2: Migration of two different applications using checkpoint and snapshot packages Figure 39 illustrates a cluster with two applications independent of each other.
Figure 39 Legacy Style of Packaging Figure 40 illustrates the consolidated modular CFS packages; all the disk group and mount point packages used by Application 1 are consolidated into a single modular package. Similarly, the disk group and mount point packages used by Application 2 are consolidated into a single modular package. The disk group, cvm_dg3, is merged with the snapshot package to enable consolidation of storage within the same package.
Managing Disk Groups and Mount Points Using Legacy Packages To manage disk groups and mount points in a Serviceguard cluster using CVM/CFS, create disk group and mount point packages. These packages are configured for activating disk groups and mounting the volumes. The applications that use CVM/CFS require the disk group and mount point packages to be up and running. Creating a File System and Mount Point Package 1.
SG-CFS-MP-1 7. After creating the mount point packages for the cluster file system, configure your application package to depend on the mount points. In the configuration file, configure a dependency, setting dependency_condition to SG-CFS-MP-pkg-# =UP and dependency_location to SAME_NODE. For more information about these parameters, see “Package Parameter Explanations” (page 222).
SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-CK-1 up up up running running running enabled enabled disabled no no no /tmp/check_logfiles now contains a point-in-time view of /tmp/logdata/log_files, and it is persistent.
SG-CFS-MP-1 SG-CFS-SN-1 up up running running enabled disabled no no The snapshot file system /local/snap1 is now mounted and provides a point in time view of /tmp/logdata/log_files.
Initializing the Veritas Volume Manager If you are about to create disk groups for the first time, you need to initialize the Volume Manager. Use the following command after installing VxVM/CVM on each node: vxinstall This displays a menu-driven program that steps you through the VxVM/CVM initialization sequence.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: /usr/lib/vxvm/bin/vxdisksetup -i c4t3d4 Creating Disk Groups Use the following steps to create disk groups. 1. Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: vxdg -s init logdata c0t3d2 2.
packages. Use one cvm_dg for each disk group the package will use, and select the appropriate cvm_activation_cmd. Package configuration is described in detail in Chapter 6 (Chapter 7 for legacy packages).
You can use these commands to test cluster operation, as in the following: 1. If the cluster is not already running, start it. From the Serviceguard Manager menu, choose Run Cluster. From the command line, use cmruncl -v. By default, cmruncl will check the networks. Serviceguard will probe the actual network configuration with the network information in the cluster configuration. If you do not need this validation, use cmruncl -v -w none instead, to turn off validation and save time 2.
There are three cases: • The cluster is not running on any node, all cluster nodes must be reachable, and all must be attempting to start up. In this case, the node attempts to form a cluster consisting of all configured nodes. • The cluster is already running on at least one node. In this case, the node attempts to join that cluster. • Neither is true: the cluster is not running on any node, and not all the nodes are reachable and trying to start.
Single-Node Operation Single-node operation occurs in a single-node cluster or in a multi-node cluster, following a situation where all but one node has failed, or where you have shut down all but one node, which will probably have applications running. As long as the Serviceguard daemon cmcld is active, other nodes can rejoin the cluster at a later time. If the Serviceguard daemon fails when in single-node operation, it will leave the single node up and your applications running.
It is recommended that you do not proceed with the configuration operation unless you are sure these nodes are permanently unavailable.Do you want to continue? Reply Yes to remove the configuration. Later, if the inaccessible node becomes available, you should run the cmdeleteconf command on that node to remove the configuration file.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. See “What is Serviceguard? ” (page 20), “How the Package Manager Works” (page 48), and “Package Configuration Planning ” (page 120) for more information.
NOTE: This is a new process for configuring packages, as of Serviceguard A.11.18. This manual refers to packages created by this method as modular packages, and assumes that you will use it to create new packages; it is simpler and more efficient than the older method, allowing you to build packages from smaller modules, and eliminating the separate package control script and the need to distribute it manually. Packages created using Serviceguard A.11.17 or earlier are referred to as legacy packages.
IMPORTANT: If the package uses volume groups, they must be activated in shared mode: vgchange -a s. NOTE: On systems that support CFS, you configure the legacy CFS multi-node packages by means of the cfsdgadm and cfsmntadm commands, not by editing a package configuration file. For modular CFS packages, you cannot edit using the cfs commands, but must edit the parameters in the package configuration files; see “Creating a Storage Infrastructure with Veritas Cluster File System (CFS)” (page 190).
and start the package for the first time. But if you then halt the multi-node package via cmhaltpkg, it can be re-started only by means of cmrunpkg, not cmmodpkg. • If a multi-node package is halted via cmhaltpkg, package switching is not disabled. This means that the halted package will start to run on a rebooted node, if it is configured to run on that node and its dependencies are met.
Table 10 Base Modules (continued) Module Name multi_node system_multi_node Parameters (page) Comments package_type (page 223) package_description (page 223) * node_name (page 223) auto_run (page 224) node_fail_fast_enabled (page 224) run_script_timeout (page 224) halt_script_timeout (page 225) successor_halt_timeout (page 225) script_log_file (page 225) operation_sequence (page 225) * log_level (page 226) * failover_policy (page 226) failback_policy (page 227) priority (page 227) * Cannot be used if p
Table 11 Optional Modules (continued) Module Name package_ip Parameters (page) Comments * ip_subnet (page 231) (S) ip_subnet_node (page 231) ip_address (page 232) * (S) Add to failover module to assign relocatable IP addresses to a failover package. * service service_name (page 232) * (S) service_cmd (page 232) (S) service_restart (page 233) * (S) service_fail_fast_enabled (page 233) service_halt_timeout (page 233) Add to a base module to create a package that runs an application or service.
Table 11 Optional Modules (continued) Module Name Parameters (page) Comments all all parameters Use if you are creating a complex package that requires most or all of the optional parameters; or if you want to see the specifications and comments for all available parameters. multi_node_all all parameters that can be used by a multi-node package; includes multi_node, dependency, monitor_subnet, service, resource, volume_group, filesystem, pev, external_pre, external, and acp modules.
IMPORTANT: Restrictions on package names in previous Serviceguard releases were less stringent. Packages whose names do not conform to the above rules will continue to run, but if you reconfigure them, you will need to change the name; cmcheckconf and cmapplyconf will enforce the new rules. module_name The module name (for example, failover, service, etc.) Do not change it.
IMPORTANT: node names. See “Cluster Configuration Parameters ” (page 105) for important information about See “About Cross-Subnet Failover” (page 145) for considerations affecting cross-subnet packages, which are further explained under “Cross-Subnet Configurations” (page 29). See “Rules for Simple Dependencies” (page 129) for considerations affecting a package that depends on another package, or that is depended on. auto_run Can be set to yes or no. The default is yes.
If no timeout is specified (no_timeout), Serviceguard will wait indefinitely for the package to start. If a timeout occurs: • Switching will be disabled. • The current node will be disabled from running the package. NOTE: VxVM disk groups are imported at package run time and exported at package halt time. If a package uses a large number of VxVM disk, the timeout value must be large enough to allow all of them to finish the import or export.
This parameter is not configurable; do not change the entries in the configuration file. New for modular packages. log_level Determines the amount of information printed to stdout when the package is validated, and to the script_log_file (page 225) when the package is started and halted. Valid values are 0 through 5, but you should normally use only the first two (0 or 1); the remainder (2 through 5) are intended for use by HP Support.
failback_policy Specifies what action the package manager should take when a failover package is not running on its primary node (the first node on its node_name list) and the primary node is once again available. Can be set to automatic or manual. The default is manual. • manual means the package will continue to run on the current (adoptive) node.
dependency_condition The condition that must be met for this dependency to be satisfied. The syntax is: = . is the name of the package depended on. Valid values for are UP, or DOWN. • UP means that this package requires the package identified by to be up (that is, the status reported by cmviewcl is UP).
weight_name, weight_value These parameters specify a weight for a package; this weight is compared to a node's available capacity (defined by the CAPACITY_NAME and CAPACITY_VALUE parameters in the cluster configuration file) to determine whether the package can run there. Both parameters are optional, but if weight_value is specified, weight_name must also be specified, and must come first. You can define up to four weights, corresponding to four different capacities, per cluster.
If any monitored_subnet fails, Serviceguard will switch the package to any other node specified by node_name which can communicate on the monitored_subnets defined for this package. See the comments in the configuration file for more information and examples. monitored_subnet_access In cross-subnet configurations, specifies whether each monitored_subnet is accessible on all nodes in the package’s node list (see node_name (page 223)), or only some.
ip_subnet Specifies an IP subnet used by the package for relocatable addresses; see also ip_address (page 232) and “Stationary and Relocatable IP Addresses ” (page 64). Replaces SUBNET, which is still supported in the package control script for legacy packages. CAUTION: HP recommends that this subnet be configured into the cluster.
Can be added or deleted while the package is running, with these restrictions: • The package must not be running on the node that is being added or deleted. • The node must not be the first to be added to, or the last deleted from, the list of ip_subnet_nodes for this ip_subnet. See also monitored_subnet_access (page 230) and “About Cross-Subnet Failover” (page 145). New for modular packages. For legacy packages, see “Configuring Cross-Subnet Failover” (page 295).
NOTE: Be careful when defining service run commands. Each run command is executed in the following way: • The cmrunserv command executes the run command. • Serviceguard monitors the process ID (PID) of the process the run command creates. • When the command exits, Serviceguard determines that a failure has occurred and takes appropriate action, which may include transferring the package to an adoptive node.
See “Parameters for Configuring EMS Resources” (page 128) for more information and example. resource_polling_interval How often, in seconds, the resource identified by resource_name (page 233) is to be monitored. Default is 60 seconds. The minimum value is 1. (There is no practical maximum.) resource_start Specifies when Serviceguard will begin monitoring the resource identified by resource_name. Valid values are automatic and deferred.
in parallel, rather than serially. That can improve a package’s startup performance if its volume groups contain a large number of physical volumes. Note that, in the context of a Serviceguard package, this affects the way physical volumes are activated within a volume group; concurrent_vgchange_operations (page 234) controls how many volume groups the package can activate simultaneously.
vg Specifies an LVM volume group (one per vg, each on a new line) on which a file system needs to be mounted. A corresponding vgchange_cmd (page 235) specifies how the volume group is to be activated. The package script generates the necessary file system commands on the basis of the fs_ parameters (page 238). NOTE: A volume group must be configured into the cluster (via the VOLUME_GROUP parameter) before you can configure it into a modular package.
fs_name /dev/vg01/lvol1 fs_directory /pkg01aa fs_mount_opt "-o rw" fs_umount_opt "-s" fs_fsck_opt "-s" fs_type "vxfs" A logical volume can be created in an LVM volume group, a Veritas VxVM disk group, or a Veritas CVM disk group. Logical volumes can be entered in any order, regardless of the type of storage group. The parameter explanations that follow provide more detail. For an NFS-imported file system, see the discussion under fs_name (page 238) and fs_server (page 238).
fs_name • For a local file system, this parameter, in conjunction with fs_directory, fs_type, fs_mount_opt, fs_umount_opt, and fs_fsck_opt, specifies the file system that is to be mounted by the package. fs_name must specify the block devicefile for a logical volume. • For an NFS-imported file system, the additional parameters required are fs_server, fs_directory, fs_type, and fs_mount_opt; see fs_server (page 238) for an example.
fs_mount_opt The mount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. See the mount (1m) manpage for more information. fs_umount_opt The umount options for the file system specified by fs_name. This parameter is in the package control script for legacy packages. Using the-s option of umount will improve startup performance if the package uses a large number of file systems. See the mount (1m) manpage for more information.
user_name Specifies the name of a user who has permission to administer this package. See also user_host and user_role; these three parameters together define the access-control policy for this package (see “Controlling Access to the Cluster” (page 183)). These parameters must be defined in this order: user_name, user_host, user_role. Legal values for user_name are any_user or a maximum of eight login names from /etc/ passwd on user_host.
Generating the Package Configuration File When you have chosen the configuration modules your package needs (see “Choosing Package Modules” (page 217)), you are ready to generate a package configuration file that contains those modules. This file will consist of a base module (usually failover, multi-node or system multi-node) plus the modules that contain the additional parameters you have decided to include.
NOTE: You can add more than one module at a time. Next Step The next step is to edit the configuration file you have generated; see “Editing the Configuration File” (page 242). Editing the Configuration File When you have generated the configuration file that contains the modules your package needs (see “Generating the Package Configuration File” (page 241)), you need to edit the file to set the package parameters to the values that will make the package function as you intend.
• auto_run. For failover packages, enter yes to allow Serviceguard to start the package on the first available node specified by node_name, and to automatically restart it later if it fails. Enter no to keep Serviceguard from automatically starting the package. • node_fail_fast_enabled. Enter yes to cause the node to be halted (system reset) if the package fails; otherwise enter no. For system multi-node packages, you must enter yes. • run_script_timeout and halt_script_timeout.
• • For each service the package will run, enter values for the following parameters (page 232): ◦ service_name (for example, a daemon or long-running process) ◦ service_cmd (for example, the command that starts the process) ◦ service_fail_fast_enabled and service_halt_timeout if you need to change them from their defaults. ◦ service_restart if you want the package to restart the service if it exits.
• If your package uses a large number of volume groups or disk groups, or mounts a large number of file systems, consider increasing the values of the following parameters: ◦ concurrent_vgchange_operations (page 234) ◦ concurrent_fsck_operations (page 237) ◦ concurrent_mount_and_umount_operations (page 237) You can also use the fsck_opt and fs_umount_opt parameters (page 239) to specify the -s option of the fsck and mount/umount commands.
• File systems and volume groups are valid (for a modular package, cmcheckconf checks that volume groups are already configured into the cluster). • Services are executable. • Any package that this package depends on is already part of the cluster configuration. For more information, see the manpage for cmcheckconf (1m) and “Checking Cluster Components” (page 261). When cmcheckconf has completed without errors, apply the package configuration, for example: cmapplyconf -P $SGCONF/pkg1/pkg1.
disks in disk group dg_01 and sets the noautoimport flag for the disks. This flag prevents a disk group from being automatically re-imported by a node following a reboot. If a node in the cluster fails, the host ID is still written on each disk in the disk group. However, if the node is part of a Serviceguard cluster then on reboot the host ID will be cleared by the owning node from all disks which have the noautoimport flag set, even if the disk group is not under Serviceguard control.
7 Cluster and Package Maintenance This chapter describes how to see cluster configuration and status information, how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Viewing CFS Multi-Node Information If you are running Veritas Cluster File System (CFS) cluster, you can use cfs commands to see multi-node package configuration information, status, and dependencies; for example: cfsdgadm show_package cfsmntadm show_package getconf -p | grep dependency Types of Cluster and Package States A cluster or its component nodes may be in several different states at different points in time.
• halt_wait — A cmhaltpkg command is in progress for this package. The package is waiting to be halted, but the halt script cannot start because the package is waiting for packages that depend on it (successors) to halt. The parameter description for successor_halt_timeout (page 225) provides more information. • failing — The package is halting because it, or a package it depends on, has failed.
The following states are possible only for multi-node packages: • blocked — The package has never run on this node, either because a dependency has not been met, or because auto_run is set to no. • changing — The package is in a transient state, different from the status shown, on some nodes. For example, a status of starting with a state of changing would mean that the package was starting on at least one node, but in some other, transitory condition (for example, failing) on at least one other node.
Failover packages can also be configured with one of two values for the failback_policy parameter (page 227), and these are also displayed in the output of cmviewcl -v: • automatic: Following a failover, a package returns to its primary node when the primary node becomes available again. • manual: Following a failover, a package will run on the adoptive node until moved back to its original node by a system administrator.
Service Subnet up up 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled 0 0 NAME ftsys10 ftsys9 service2 15.13.168.0 (current) NOTE: The Script_Parameters section of the PACKAGE output of cmviewcl shows the Subnet status only for the node that the package is running on. In a cross-subnet configuration, in which the package may be able to fail over to a node on another subnet, that other subnet is not shown; see “Cross-Subnet Configurations” (page 29).
Service Service Service Service NODE_NAME ftsys10 up up up up 5 5 0 0 STATUS up 0 0 0 0 SG-CFS-sgcvmd SG-CFS-vxfsckd SG-CFS-cmvxd SG-CFS-cmvxpingd SWITCHING enabled Script_Parameters: ITEM STATUS MAX_RESTARTS Service up 0 Service up 5 Service up 5 Service up 0 Service up 0 RESTARTS 0 0 0 0 0 NAME SG-CFS-vxconfigd SG-CFS-sgcvmd SG-CFS-vxfsckd SG-CFS-cmvxd SG-CFS-cmvxpingd Status After Halting a Package After we halt the failover package pkg2 with the cmhaltpkg command, the output of cmviewcl-v is as
Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Resource up Subnet up Resource down Subnet up NODE_NAME ftsys9 ftsys9 ftsys10 ftsys10 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NAME /example/float 15.13.168.0 /example/float 15.13.168.0 NAME ftsys10 ftsys9 pkg2 now has the status down, and it is shown as unowned, with package switching disabled.
pkg2 up running disabled ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service2.1 Subnet up 15.13.168.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys10 Alternate up enabled ftsys9 (current) NODE ftsys10 STATUS up Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up STATE running PATH 28.1 32.
NODE ftsys10 STATUS down STATE halted This output can be seen on both ftsys9 and ftsys10. Viewing Information about Unowned Packages The following example shows packages that are currently unowned, that is, not running on any configured node. cmviewcl provides information on monitored resources for each node on which the package can run; this allows you to identify the cause of a failure and decide where to start the package up again.
Checking Status of the Cluster File System (CFS) If the cluster is using the cluster file system, you can check status with thecfscluster command, as in the example that follows. The cfscluster status command displays the status of the disk groups and mount points created only using legacy style of packaging, and not modular style of packaging.
Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd Status of CFS Modular Disk Group and Mount Point Packages To see the status of CFS modular disk group and mount point packages, use the cmviewcl command.
Mount Point : /tmp/logdata/log_files Shared Volume : lvol1 Disk Group : logdata To see which package is monitoring a mount point, use the cfsmntadm show_package command. For example, for the disk group logdata: cfsmntadm show_package /tmp/logdata/log_files SG-CFS_MP-1 Checking the Cluster Configuration and Components Serviceguard provides tools that allow you to check the soundness of the cluster configuration, and the health of its components.
NOTE: • All of the checks below are performed when you run cmcheckconf without any arguments (or with only -v, with or without -k or -K). cmcheckconf validates the current cluster and package configuration, including external scripts and pre-scripts for modular packages, and runs cmcompare to check file consistency across nodes. (This new version of the command also performs all of the checks that were done in previous releases.) See “Checking Cluster Components” (page 261) for details.
NOTE: The table includes all the checks available as of A.11.20, not just the new ones. Table 12 Verifying Cluster Components Component (Context) Tool or Command; More Information Comments Volume groups (cluster) cmcheckconf (1m), cmapplyconf (1m) Checked for: See also “Verifying the Cluster Configuration ” (page 188).
Table 12 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Comments File consistency (cluster) cmcheckconf (1m), cmcompare (1m). To check file consistency across all nodes in the cluster, do the IMPORTANT: See the manpage for following: differences in return codes from 1. Customize /etc/cmcluster/ cmcheckconf without options versus cmfiles2check cmcheckconf -C 2. Distribute it to all nodes using cmsysnc (1m) 3.
NOTE: The job must run on one of the nodes in the cluster. Because only the root user can run cluster verification, and cron (1m) sets the job’s user and group ID’s to those of the user who submitted the job, you must edit the file /var/spool/cron/crontabs/root as the root user. Example The short script that follows runs cluster verification and sends an email to admin@hp.com when verification fails. #!/bin/sh cmcheckconf -v >/tmp/cmcheckconf.
In Serviceguard A.11.16 and later, these tasks can be performed by non-root users with the appropriate privileges, except where specifically noted. See “Controlling Access to the Cluster” (page 183) for more information about configuring access. You can use Serviceguard Manager or the Serviceguard command line to start or stop the cluster, or to add or halt nodes. Starting the cluster means running the cluster daemon on one or more of the nodes in a cluster.
ftsys8 to the cluster that was just started with only nodes ftsys9 and ftsys10. The-v (verbose) option prints out all the messages: cmrunnode -v ftsys8 By default, cmrunnode will do network validation, making sure the actual network setup matches the configured network setup. This is the recommended method. If you have recently checked the network and find the check takes a very long time, you can use the -w none option to bypass the validation.
Halting a Node or the Cluster while Keeping Packages Running There may be circumstances in which you want to do maintenance that involves halting a node, or the entire cluster, without halting or failing over the affected packages. Such maintenance might consist of anything short of rebooting the node or nodes, but a likely case is networking changes that will disrupt the heartbeat. New command options in Serviceguard A.11.
See Chapter 6 (page 216) for more information about package types. • You cannot detach a package that is in maintenance mode, and you cannot place a package into maintenance mode if any of its dependent packages are detached. For more information about maintenance mode, see“Maintaining a Package: Maintenance Mode” (page 273). For more information about dependencies, see “About Package Dependencies” (page 128). • You cannot make configuration changes to a cluster in which any packages are detached.
Additional Points To Note Keep the following points in mind. • When packages are detached, they continue to run, but without high availability protection. Serviceguard does not detect failures of components of detached packages, and packages are not failed over. Serviceguard will not provide local LAN failover of IP addresses if the node has been halted in detach mode.
if the IP address is switched to the standby LAN and NETWORK_AUTO_FAILBACK cluster parameter is set to FALSE. If the primary LAN recovers while the node is halted and you want the IP address to fail back to the primary LAN, run cmmodnet –e to re-enable the primary LAN interface and trigger the failback. Halting a Node and Detaching its Packages To halt a node and detach its packages, proceed as follows. 1. Make sure that the conditions spelled out under “Rules and Restrictions” (page 267) are met. 2.
assume that packages pkg1 through pkg5 are unsupported for Live Application Detach, and pkg6 through pkgn are supported. Proceed as follows: 1. Halt all the unsupported packages: cmhaltpkg pkg1 pkg2 pkg3 pkg4 pkg5 2. Halt the cluster, detaching the remaining packages: cmhaltcl -d 3. 4. Upgrade the heartbeat networks as needed.
Using Serviceguard Commands to Start a Package Use the cmrunpkg command to run the package on a particular node, then use the cmmodpkg command to enable switching for the package. For example, to start a failover package: cmrunpkg -n ftsys9 pkg1 cmmodpkg -e pkg1 This starts up the package on ftsys9, then enables package switching. This sequence is necessary when a package has previously been halted on some node, since halting the package disables switching.
Moving a Failover Package You can use Serviceguard Manager, or Serviceguard commands as shown below, to move a failover package from one node to another. Using Serviceguard Commands to Move a Running Failover Package Before you move a failover package to a new node, it is a good idea to run cmviewcl -v -l package and look at dependencies. If the package has dependencies, be sure they can be met on the new node. To move the package, first halt it where it is running using the cmhaltpkg command.
NOTE: If you need to do maintenance that requires halting a node, or the entire cluster, you should consider Live Application Detach; see “Halting a Node or the Cluster while Keeping Packages Running” (page 267). • Maintenance mode is chiefly useful for modifying networks and EMS resources used by a package while the package is running. See “Performing Maintenance Using Maintenance Mode” (page 276).
◦ A script times out ◦ The limit of a restart count is exceeded Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode IMPORTANT: See the latest Serviceguard release notes for important information about version requirements for package maintenance. • The package must have package switching disabled before you can put it in maintenance mode. • You can put a package in maintenance mode only on one node.
Dependency Rules for a Package in Maintenance Mode or Partial-Startup Maintenance Mode You cannot configure new dependencies involving a package running in maintenance mode, and in addition the following rules apply (we'll call the package in maintenance mode pkgA). • The packages that depend on pkgA must be down with package switching disabled when you place pkgA in maintenance mode.
1. Halt the package: cmhaltpkg pkg1 2. Place the package in maintenance mode: cmmodpkg -m on -n node1 pkg1 NOTE: 3. The order of the first two steps can be reversed. Run the package in maintenance mode. In this example, we'll start pkg1 such that only the modules up to and including the package_ip module are started. (See “Package Modules and Parameters” (page 219) for a list of package modules.
NOTE: The full execution sequence for starting a package is: 1. The master control script itself 2. External pre-scripts 3. Volume groups 4. File systems 5. Package IPs 6. External scripts 7. Services Reconfiguring a Cluster You can reconfigure a cluster either when it is halted or while it is still running. Some operations can only be done when the cluster is halted. Table 13 shows the required cluster state for many kinds of changes.
Table 13 Types of Changes to the Cluster Configuration (continued) Change to the Cluster Configuration Required Cluster State See “What You Must Keep in Mind” (page 284). Cluster can be running throughout. Change NETWORK_FAILURE_DETECTION parameter Cluster can be running. (see “Monitoring LAN Interfaces and Detecting Failure: Link Level” (page 66)) Change NETWORK_AUTO_FAILBACK parameter Cluster can be running. Change NETWORK_POLLING_INTERVAL Cluster can be running.
• Availability of package subnets, EMS resources, and storage • Changes in package priority, node order, dependency, failover and failback policy, node capacity and package weight Using Preview mode for Commands and in Serviceguard Manager The following commands support the -t option, which allows you to run the command in preview mode: • cmhaltnode [–t] [–f] • cmrunnode [–t] • cmhaltpkg [–t] • cmrunpkg [–t] [-n node_name] • cmmodpkg { -e [-
running, though it must use the same Serviceguard release and patch version as the system on which you run cmeval. Use cmeval rather than command preview mode when you want to see more than the effect of a single command, and especially when you want to see the results of large-scale changes, or changes that may interact in complex ways, such as changes to package priorities, node order, dependencies and so on. Using cmeval involves three major steps: 1.
IMPORTANT: See “What Happens when You Change the Quorum Configuration Online” (page 47) for important information. 1. 2. 3. In the cluster configuration file, modify the values of FIRST_CLUSTER_LOCK_PV and SECOND_CLUSTER_LOCK_PV for each node. Run cmcheckconf to check the configuration. Run cmapplyconf to apply the configuration. For information about replacing the physical disk, see “Replacing a Lock Disk” (page 311).
Adding Nodes to the Cluster While the Cluster is Running You can use Serviceguard Manager to add nodes to a running cluster, or use Serviceguard commands as in the example below. In this example, nodes ftsys8 and ftsys9 are already configured in a running cluster named cluster1, and you are adding node ftsys10. 1. Use the following command to store a current copy of the existing cluster configuration in a temporary file: cmgetconf -c cluster1 temp.ascii 2.
2. Specify the new set of nodes to be configured (omitting ftsys10) and generate a template of the new configuration: cmquerycl -C clconfig.ascii -c cluster1 -n ftsys8 -n ftsys9 3. 4. Edit the file clconfig.ascii to check the information about the nodes that remain in the cluster. Halt the node you are going to remove (ftsys10in this example): cmhaltnode -f -v ftsys10 5. Verify the new configuration: cmcheckconf -C clconfig.ascii 6.
• You cannot add interfaces or modify their characteristics unless those interfaces, and all other interfaces in the cluster configuration, are healthy. There must be no bad NICs or non-functional or locally switched subnets in the configuration, unless you are deleting those components in the same operation.
1. Run cmquerycl to get a cluster configuration template file that includes networking information for interfaces that are available to be added to the cluster configuration: cmquerycl -c cluster1 -C clconfig.ascii NOTE: As of Serviceguard A.11.18, cmquerycl -c produces output that includes commented-out entries for interfaces that are not currently part of the cluster configuration, but are available. The networking portion of the resulting clconfig.
Example: Deleting a Subnet Used by a Package In this example, we are deleting subnet 15.13.170.0 (lan0). This will also mean deleting lan3, which is a standby for lan0 and not shared by any other primary LAN. Proceed as follows. 1. Halt any package that uses this subnet and delete the corresponding networking information: monitored_subnet, ip_subnet, ip_address (page 229). See “Reconfiguring a Package on a Running Cluster ” (page 297) for more information. 2.
6. Runolrad -d to remove the NIC. See also “Replacing LAN or Fibre Channel Cards” (page 314). Changing the LVM Configuration while the Cluster is Running You can do this in Serviceguard Manager, or use HP-UX commands as in the example that follows. NOTE: If you are removing a volume group from the cluster configuration, make sure that you also modify any package that activates and deactivates this volume group.
cmgetconf -c clconfig.ascii Edit the clconfig.ascii file to include the new value for MAX_CONFIGURED_PACKAGES. Then use the cmcheckconf command to verify the new configuration. Using the -k or -K option can significantly reduce the response time. Use cmapplyconf to apply the changes to the configuration and send the new configuration file to all cluster nodes. Using -k or -K can significantly reduce the response time.
3. 4. 5. 6. 7. 8. Apply the configuration. Run the package and ensure that it can be moved from node to node. Halt the package. Configure package IP addresses and application services in the control script. Distribute the control script to all nodes. Run the package and ensure that applications run as expected and that the package fails over correctly when services are disrupted. Editing the Package Configuration File Edit the file you generated with cmmakepkg.
• Enter the SUBNET or MONITORED_SUBNETs that are to be monitored for this package. They can be IPv4 or an IPv6 subnets, but must not be link-local subnets (link-local package IPs are not allowed). IMPORTANT: Each subnet specified here must already be specified in the cluster configuration file via the NETWORK_INTERFACE parameter and either the HEARTBEAT_IP or STATIONARY_IP parameter. See “Cluster Configuration Parameters ” (page 105) for more information.
You can use a single script for both run and halt operations, or, if you wish, you can create separate scripts. IMPORTANT: Serviceguard automatically creates the necessary control scripts when you create the multi-node or system multi-node packages for CVM/CFS (version 4.1 and later). HP strongly recommends that you never edit the configuration or control script files for these packages, although Serviceguard does not prevent it.
• If your package will use relocatable IP addresses, define IP subnet and IP address pairs. IPv4 or IPv6 addresses are allowed. CAUTION: HP recommends that the subnet(s) be specified in the cluster configuration file via the NETWORK_INTERFACE parameter and either the HEARTBEAT_IP or STATIONARY_IP parameter; see “Cluster Configuration Parameters ” (page 105).
Adding Serviceguard Commands in Customer Defined Functions You can add Serviceguard commands (such as cmmodpkg) in the Customer Defined Functions section of a package control script. These commands must not interact with the package itself. If a Serviceguard command interacts with another package, be careful to avoid command loops. For instance, a command loop might occur under the following circumstances. Suppose pkg1 does a cmmodpkg -d of pkg2, and pkg2 does a cmmodpkg -d of pkg1.
Distributing the Configuration And Control Script with Serviceguard Manager When you have finished creating a legacy package in Serviceguard Manager, click Apply Configuration. If the package control script has no errors, it is converted to a binary file and distributed to the cluster nodes. Copying Package Control Scripts with HP-UX commands IMPORTANT: In a cross-subnet configuration, you cannot use the same package control script on all nodes if the package uses relocatable IP addresses.
Configuring node_name First you need to make sure that pkg1 will fail over to a node on another subnet only if it has to. For example, if it is running on NodeA and needs to fail over, you want it to try NodeB, on the same subnet, before incurring the cross-subnet overhead of failing over to NodeC or NodeD. NOTE: If you are using a site-aware disaster-tolerant cluster, which requires Metrocluster (additional HP software), you can use the SITE to accomplish this.
SUBNET[1] = 15.244.56.0 Reconfiguring a Package You reconfigure a package in much the same way as you originally configured it; for modular packages, see Chapter 6: “Configuring Packages and Their Services ” (page 216); for older packages, see “Configuring a Legacy Package” (page 289). The cluster, and the package itself, can be either halted or running during package reconfiguration; see “Reconfiguring a Package on a Running Cluster ” (page 297).
1. Halt the package if necessary: cmhaltpkg pkg1 CAUTION: Make sure you read and understand the information and caveats under“Allowable Package States During Reconfiguration ” (page 300) before you decide to reconfigure a running package. 2. If it is not already available, obtain a copy of the package's configuration file by using the cmgetconf command, specifying the package name. cmgetconf -p pkg1 pkg1.conf 3. Edit the package configuration file.
Adding a Package to a Running Cluster You can create a new package and add it to the cluster configuration while the cluster is up and while other packages are running. The number of packages you can add is subject to the value of MAX_CONFIGURED_PACKAGES in the cluster configuration file. To create the package, follow the steps in the chapter “Configuring Packages and Their Services ” (page 216).
To remove the legacy CVM disk groups and CFS mount points, follow these steps: CAUTION: You must not use the HP-UX mount and umount commands in a CFS environment; use cfsmount or cfsumount for legacy CFS packages. For modular packages, you must use cmcheckconf, cmapplyconf, cmrunpkg, cmmodpkg, and cmrunpkg.
which the results might not be what you expect — as well as differences between modular and legacy packages. CAUTION: is running. Be extremely cautious about changing a package's configuration while the package If you reconfigure a package online (by executing cmapplyconf on a package while the package itself is running) it is possible that the package will fail, even if the cmapplyconf succeeds, validating the changes with no errors.
Table 14 Types of Changes to Packages (continued) Change to the Package Required Package State Change run script contents: legacy Package can be running, but should not be starting. package Timing problems may occur if the script is changed while the package is starting. Change halt script contents: legacy Package can be running, but should not be halting. package Timing problems may occur if the script is changed while the package is halting.
Table 14 Types of Changes to Packages (continued) Change to the Package Required Package State incrementally; resources that are very slow to come up may need to be configured offline. Serviceguard treats a change to resource_name as deleting the existing resource and adding a new one, meaning that the existing resource is stopped. Change Package can be running. resource_polling_interval, Serviceguard will not allow a change to resource_up_value if it would cause resource_start, the package to fail.
Table 14 Types of Changes to Packages (continued) Change to the Package Required Package State CAUTION: Removing a file system may cause problems if the file system cannot be unmounted because it's in use by a running process. In this case Serviceguard kills the process; this could cause the package to fail. Remove a file system: legacy package Package must not be running. Change Package can be running.
Table 14 Types of Changes to Packages (continued) Change to the Package Required Package State Add, change, or delete pev_: modular package Package can be running. Change package auto_run Package can be running. See “Choosing Switching and Failover Behavior” (page 127). Add or delete a configured dependency Both packages can be either running or halted.
• Package priority • auto_run • failback_policy Responding to Cluster Events Serviceguard does not require much ongoing system administration intervention. As long as there are no failures, your cluster will be monitored and protected. In the event of a failure, those packages that you have designated to be transferred to another node will be transferred automatically.
NOTE: You should not disable Serviceguard on a system on which it is actually running. If you are not sure, you can get an indication by means of the command: ps -e | grep cmclconfd If there are cmclconfd processes running, it does not mean for certain that Serviceguard is running on this system (cmclconfd could simply be handling UDP queries from a Serviceguard cluster on the same subnet) but it does mean you should investigate further before disabling Serviceguard.
8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
1. 2. Turn off the power to the node SPU. To observe the cluster reforming, enter the following command on some other configured node: cmviewcl -v You should be able to observe that the powered down node is halted, and that its packages have been correctly switched to other nodes. 3. 4. Turn on the power to the node SPU.
since the applications are still running normally. But at this point, there is no redundant path if another failover occurs, so the mass storage configuration is vulnerable. Using Event Monitoring Service Event Monitoring Service (EMS) allows you to configure monitors of specific devices and system resources. You can direct alerts to an administrative workstation where operators can be notified of further action in case of a problem.
Replacing a Faulty Mechanism in an HA Enclosure If you are using software mirroring with Mirrordisk/UX and the mirrored disks are mounted in a high availability disk enclosure, you can use the following steps to hot plug a disk mechanism: 1. Identify the physical volume name of the failed disk and the name of the volume group in which it was configured. In the following example, the volume group name is shown as /dev/ vg_sg01 and the physical volume name is shown as /dev/dsk/c2t3d0.
IMPORTANT: If you need to replace a disk under the HP-UX 11i v3 agile addressing scheme, also used by cDSFs (see “About Device File Names (Device Special Files)” (page 77) and “About Cluster-wide Device Special Files (cDSFs)” (page 99)), and you use the same DSF, you may need to use the io_redirect_dsf(1M) command to reassign the existing DSF to the new device, depending on whether the operation changes the WWID of the device.
IMPORTANT: If you need to replace a LUN under the HP-UX 11i v3 agile addressing scheme, also used by cDSFs (see “About Device File Names (Device Special Files)” (page 77) and “About Cluster-wide Device Special Files (cDSFs)” (page 99), and you use the same DSF, you may need to use the io_redirect_dsf(1M) command to reassign the existing DSF to the new device, depending on whether the operation changes the WWID of the LUN; see the section on io_redirect_dsf in the white paper The Next Generation Mass Storage
Normally disconnecting any portion of the SCSI bus will leave the SCSI bus in an unterminated state, which will cause I/O errors for other nodes connected to that SCSI bus, so the cluster would need to be halted before disconnecting any portion of the SCSI bus. However, it is not necessary to bring the cluster down to do this if you are using a SCSI configuration that allows disconnection of a portion of the SCSI bus without losing termination.
NOTE: After replacing a Fibre Channel I/O card, it may necessary to reconfigure the SAN to use the World Wide Name (WWN) of the new Fibre Channel card if Fabric Zoning or other SAN security requiring WWN is used.
NOTE: While the old Quorum Server is down and the new one is being set up, these things can happen: • These three commands will not work: cmquerycl -q cmapplyconf -C cmcheckconf -C • If there is a node or network failure that creates a 50-50 membership split, the quorum server will not be available as a tie-breaker, and the cluster will fail. NOTE: Make sure that the old Quorum Server system does not rejoin the network with the old IP address.
directory, by default. You can use a text editor, such as vi, or the more command to view the log file for historical information on your cluster. It is always a good idea to review the syslog.log file on each of the nodes in the cluster when troubleshooting cluster problems. This log provides information on the following: • Commands executed and their outcome. • Major cluster events which may, or may not, be errors. • Cluster status information.
Reviewing Serviceguard Manager Log Files From the System Management Homepage (SMH), click Tools, then select Serviceguard Manager, select the cluster you are interested and then choose View -> Operation Log. Reviewing the System Multi-node Package Files If you are running Veritas Cluster Volume Manager and you have problems starting the cluster, check the log file for the system multi-node package. For CVM 4.1 and later, the file is SG-CFS-pkg.log.
Using the cmviewconf Command cmviewconf allows you to examine the binary cluster configuration file, even when the cluster is not running. The command displays the content of this file on the node where you run the command. Reviewing the LAN Configuration The following networking commands can be used to diagnose problems: • netstat -in can be used to examine the LAN configuration. This command lists all IP addresses assigned to each LAN interface card.
Networking and Security Configuration Errors Many Serviceguard commands, including cmviewcl, depend on name resolution services to look up the addresses of cluster nodes. When name services are not available (for example, if a name server is down), Serviceguard commands may hang, or may return a network-related error message. If this happens, use the nslookup command on each cluster node to see whether name resolution is correct. For example: nslookup ftsys9 Name Server: server1.cup.hp.com Address: 15.13.
largest number says that cmcld was unable to run for the last 1.6 seconds, increase MEMBER_TIMEOUT to more than 16 seconds. 2. This node is at risk of being evicted from the running cluster. Increase MEMBER_TIMEOUT. This means that the hang was long enough for other nodes to have noticed the delay in receiving heartbeats and marked the node “unhealthy”.
“Halted.” Similarly, when a package control script fails, Serviceguard kills the script and marks the package “Halted.” In both cases, the following also take place: • Control of a failover package will not be transferred. • The run or halt instructions may not run to completion. • auto_run (automatic package switching) will be disabled. • The current node will be disabled from running the package.
If after cleaning up the node on which the timeout occurred it is desirable to have that node as an alternate for running the package, remember to re-enable the package to run on the node: cmmodpkg -e -n The default Serviceguard control scripts are designed to take the straightforward steps needed to get an application running or stopped.
Force Import and Deport After Node Failure After certain failures, packages configured with VxVM disk groups will fail to start, logging an error such as the following in the package log file: vxdg: Error dg_01 may still be imported on ftsys9 ERROR: Function check_dg failed This can happen if a package is running on a node which then fails before the package control script can deport the disk group. In these cases, the host name of the node that had failed is still written on the disk group header.
• arp -a - to check the arp tables. • lanadmin - to display, test, and reset the LAN cards. Since your cluster is unique, there are no cookbook solutions to all possible problems. But if you apply these checks and commands and work your way through the log files, you will be successful in identifying and solving problems. Troubleshooting the Quorum Server NOTE: See the HP Serviceguard Quorum Server Version A.04.00 Release Notes for information about configuring the Quorum Server.
A Enterprise Cluster Master Toolkit The Enterprise Cluster Master Toolkit (ECMT) provides a group of example scripts and package configuration files for creating Serviceguard packages for several major database and internet software products. Each toolkit contains a README file that explains how to customize the package for your needs. The ECMT can be installed on HP-UX 11i v2, or 11i v3.
B Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 328) • Designing Applications to Run on Multiple Systems (page 331) • Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY (page 335) • Restoring Client Connections (page 336) • Handling Applicatio
Insulate Users from Outages Wherever possible, insulate your end users from outages. Issues include the following: • Do not require user intervention to reconnect when a connection is lost due to a failed server. • Where possible, warn users of slight delays due to a failover in progress. • Minimize the reentry of data. • Engineer the system for reserve capacity to minimize the performance degradation experienced by users.
Replicate Non-Data File Systems Non-data file systems should be replicated rather than shared. There can only be one copy of the application data itself. It will be located on a set of disks that is accessed by the system that is running the application. After failover, if these data disks are file systems, they must go through file systems recovery (fsck) before the data can be accessed. To help reduce this recovery time, the smaller these file systems are, the faster the recovery will be.
Use Restartable Transactions Transactions need to be restartable so that the client does not need to re-enter or back out of the transaction when a server fails, and the application is restarted on another system. In other words, if a failure occurs in the middle of a transaction, there should be no need to start over again from the beginning. This capability makes the application more robust and reduces the visibility of a failover to the user. A common example is a print job.
Another possibility is having two applications that are both active. An example might be two application servers which feed a database. Half of the clients connect to one application server and half of the clients connect to the second application server. If one server fails, then all the clients connect to the remaining application server. Design for Replicated Data Sites Replicated data sites are a benefit for both fast failover and disaster recovery.
network interface must have a stationary IP address associated with it. This IP address does not move to a remote system in the event of a network failure. Obtain Enough IP Addresses Each application receives a relocatable IP address that is separate from the stationary IP address assigned to the system itself. Therefore, a single system might have many IP addresses, one for itself and one for each of the applications that it normally runs.
One way to look for problems in this area is to look for calls to gethostname(2) in the application. HA services should use gethostname() with caution, since the response may change over time if the application migrates. Applications that use gethostname() to determine the name for a call to gethostbyname(2) should also be avoided for the same reason. Also, the gethostbyaddr() call may return different answers over time if called with a stationary IP address.
With UDP datagram sockets, however, there is a problem. The client may connect to multiple servers utilizing the relocatable IP address and sort out the replies based on the source IP address in the server’s response message. However, the source IP address given in this response will be the stationary IP address rather than the relocatable application IP address.
Using a Relocatable Address as the Source Address for an Application that is Bound to INADDR_ANY CAUTION: The procedure in this section depends on setting the HP-UX kernel parameter ip_strong_es_model. HP supports setting this parameter for use with Serviceguard only if you are not using a cross-subnet configuration (page 29). Otherwise, leave the parameter at its default setting (zero, meaning disabled) and do not attempt to use the procedure that follows.
/usr/sbin/route delete net default 128.17.17.1 1 source 128.17.17.17 Once the per-interface default route(s) have been added, netstat –rn would show something like the following, where 128.17.17.17 is the package relocatable address and 128.17.17.19 is the physical address on the same subnet: Destination 127.0.0.1 128.17.17.19 128.17.17.17 192.168.69.82 128.17.17.0 128.17.17.0 192.168.69.0 127.0.0.0 default default Gateway 127.0.0.1 128.17.17.19 128.17.17.17 192.168.69.82 128.17.17.19 128.17.17.17 192.168.
application. If the application can be restarted on the same node after a failure (see “Handling Application Failures ” following), the retry to the current server should continue for the amount of time it takes to restart the server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness.
Minimizing Planned Downtime Planned downtime (as opposed to unplanned downtime) is scheduled; examples include backups, systems upgrades to new operating system revisions, or hardware replacements. For planned downtime, application designers should consider: • Reducing the time needed for application upgrades/patches.
Providing Online Application Reconfiguration Most applications have some sort of configuration information that is read when the application is started. If to make a change to the configuration, the application must be halted and a new configuration file read, downtime is incurred. To avoid this downtime use configuration tools that interact with an application and make dynamic changes online. The ideal solution is to have a configuration tool which interacts with the application.
C Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the Appendix “Designing Highly Available Cluster Applications.” 2.
1. 2. 3. 4. 5. Install the application, database, and other required resources on one of the systems. Be sure to follow Serviceguard rules in doing this: • Install all shared data on separate external volume groups. • Use JFS/VxFS file systems as appropriate. Perform some sort of standard test to ensure the application is running correctly. This test can be used later in testing with Serviceguard. If possible, try to connect to the application through a client.
NOTE: Check the Serviceguard/SGeRAC/SMS/Serviceguard Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information about support for CVM and CFS: http://www.hp.com/go/hpux-serviceguard-docs. Testing the Cluster 1. Test the cluster: • Have clients connect. • Provide a normal system load. • Halt the package on the first node and move it to the second node: cmhaltpkg pkg1 cmrunpkg -n node2 pkg1 cmmodpkg -e pkg1 • Move it back.
D Software Upgrades There are five types of upgrade you can do: • rolling upgrade • rolling upgrade using DRD • non-rolling upgrade • non-rolling upgrade using DRD • migration with cold install Each of these is discussed below. Special Considerations for Upgrade to Serviceguard A.11.20 • Before you proceed, read the sections Upgrading from an Earlier Serviceguard Release and Rolling Upgrade in the latest version of the release notes for A.11.20 • Serviceguard A.11.
CAUTION: From the time when the old cluster manager is shut down until the new cluster manager forms its first cluster, a node failure will cause the entire cluster to fail. HP strongly recommends that you use no Serviceguard commands other than cmviewcl (1m) until the new cluster manager successfully completes its first cluster re-formation.
Restrictions for DRD Upgrades • Upgrade using DRD is supported only on HP-UX 11i v3. • You must use the DRD software released with the September 2009 release of HP-UX 11i v3 or later. You can obtain the DRD software free from HP (see “Rolling Upgrade Using DRD” ); you do not have to upgrade the operating system itself, so long as you are running 11i v3. • The cluster must meet both general and release-specific requirements for a rolling upgrade; see “Guidelines for Rolling Upgrade” (page 345).
IMPORTANT: Do not proceed without reading and understanding the following points: • The above are general guidelines. The requirements for any particular Serviceguard release may be more restrictive. Rolling upgrade is supported only for the HP-UX/Serviceguard combinations listed in the Release Notes for the target version of Serviceguard.
Before You Start Make sure you plan sufficient system capacity to allow moving the packages from node to node during the process without an unacceptable loss of performance. CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 343). Running the Rolling Upgrade 1. 2. Halt the node you want to upgrade. You can do this in Serviceguard Manager, or use the cmhaltnode command.
Performing a Rolling Upgrade Using DRD IMPORTANT: All the limitations listed under “Guidelines for Rolling Upgrade” (page 345) and “Limitations of Rolling Upgrades ” (page 346) also apply to a rolling upgrade with DRD. You should read the entire section on “Performing a Rolling Upgrade” (page 346) before you proceed. Before You Start CAUTION: Do not proceed with an upgrade to A.11.19 until you have read and understood the Special Considerations for Upgrade to Serviceguard A.11.19 (page 343).
4. Restart the cluster on the node, using Serviceguard Manager or cmrunnode (1m). • If the applications do not function properly and this is the last node to be upgraded, you cannot revert to the previous release on just this node. You must either solve the problems with this release on this node, or revert the entire cluster to the previous release by halting the cluster (cmhaltcl), rebooting each node from its original (pre-upgrade) root disk, and restarting the cluster (cmruncl).
Figure 42 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 to the next operating system release (“HP-UX (new)”), and install the next version of Serviceguard (“SG (new)”). Figure 43 Node 1 Upgraded to new HP-UX version Step 3. When upgrading is finished, enter the following command on node 1 to restart the cluster on node 1. # cmrunnode -n node1 At this point, different versions of the Serviceguard daemon (cmcld) are running on the two nodes, as shown in Figure 44.
Figure 44 Node 1 Rejoining the Cluster Step 4. Repeat the process on node 2. Halt the node, as follows: # cmhaltnode -f node2 This causes both packages to move to node 1. Then upgrade node 2 to the new versions of HP-UX and Serviceguard. Figure 45 Running Cluster with Packages Moved to Node 1 Step 5. Move pkg2 back to its original node.
Figure 46 Running Cluster After Upgrades Guidelines for Non-Rolling Upgrade Do a non-rolling upgrade if: • Your cluster does not meet the requirements for rolling upgrade as specified in the Release Notes for the target version of Serviceguard; or • The limitations imposed by rolling upgrades make it impractical for you to do a rolling upgrade (see “Limitations of Rolling Upgrades ” (page 346)); or • For some other reason you need or prefer to bring the cluster down before performing the upgrade.
1. Halt all nodes in the cluster: cmhaltcl -f 2. If necessary, upgrade all the nodes in the cluster to the new HP-UX release. (See Step 3 under “Running the Rolling Upgrade” (page 347) for more information.) Upgrade all the nodes in the cluster to the new Serviceguard release. (See Step 3 under “Running the Rolling Upgrade” (page 347) for more information.) Restart the cluster: cmruncl 3. 4.
CAUTION: You must reboot all the nodes from their original disks before restarting the cluster; do not try to restart the cluster with some nodes booted from the upgraded disks and some booted from the pre-upgrade disks. Guidelines for Migrating a Cluster with Cold Install There may be circumstances when you prefer to do a cold install of the HP-UX operating system rather than an upgrade.
E Blank Planning Worksheets This appendix contains blank versions of the planning worksheets mentioned in“Planning and Documenting an HA Cluster ” (page 88). You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
=============================================================================== Tape Backup Power: Tape Unit __________________________ Power Supply _______________________ Tape Unit __________________________ Power Supply _______________________ =============================================================================== Other Power: Unit Name __________________________ Power Supply _______________________ Unit Name __________________________ Power Supply _______________________ .
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ VxVM Disk Group and Disk Worksheet DISK GROUP WORKSHEET Page ___ of ____ ==========================================================
Heartbeat Subnet: _________________________________ Monitored Non-heartbeat Subnet: ___________________ Monitored Non-heartbeat Subnet: ___________________ =====================================----====================================== Cluster Lock: Volume Groups, LUN, or Quorum Server =============================================================================== Quorum Server: QS_HOST___________________________QS_ADDR_________________ QS_POLLING_INTERVAL________________________________ QS_TIMEOUT_EXTENSIO
Concurrent fsck operations: ______________________________________________ Kill processes accessing raw devices?_____ =============================================================================== Network Information: IP ________ IP___________IP___________IP___________subnet ___________ IP__________IP___________IP___________IP___________subnet____________ IP_subnet_node______________IP_subnet_node______________IP_subnet_node__________ Monitored subnet:___________________________monitored_subnet_access_____
F Migrating from LVM to VxVM Data Storage This appendix describes how to migrate LVM volume groups to VxVM disk groups for use with the Veritas Volume Manager (VxVM), or with the Cluster Volume Manager (CVM) on systems that support it.
separation.) The same mirroring pattern should be followed in creating the VxVM plexes, with different plexes configured on disks that are attached to different buses. As an alternative to defining the VxVM disk groups on a new set of disks, it is possible to convert existing LVM volume groups into VxVM disk groups in line using the vxvmconvert(1M) utility.
4. 5. 6. 7. Be sure to copy from the old script any user-specific code that may have been added, including environment variables and customer defined functions. Distribute the new package control scripts to all nodes in the cluster. Test to make sure the disk group and data are intact. Deport the disk group: vxdg deport DiskGroupName 8. Make the disk group visible to the other nodes in the cluster by issuing the following command on all other nodes: vxdctl enable 9. Restart the package.
G Migrating from Legacy CFS Packages to Modular CFS Packages To migrate from Legacy CFS packages to Modular CFS packages for use with the CVM or CFS on systems that support it, follow these steps: 1. Halt the application package: cmhaltpkg 2. Unconfigure the CFS legacy disk group and the mount point package: cfsumount cfsdgadm deactivate cfsmntadm delete cfsdgadm delete 3.
H IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses. Topics: • IPv6 Address Types • Network Configuration Restrictions • Local Primary/Standby LAN Patterns • IPv6 Relocatable Address and Duplicate Address Detection Feature (page 367) IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces.
IPv6 Address Prefix IPv6 Address Prefix is similar to CIDR in IPv4 and is written in CIDR notation. An IPv6 address prefix is represented by the notation: IPv6-address/prefix-length where ipv6-address is an IPv6 address in any notation listed above and prefix-length is a decimal value representing how many of the leftmost contiguous bits of the address comprise the prefix. Example: fec0:0:0:1::1234/64 The first 64-bits of the address fec0:0:0:1 forms the address prefix.
::ffff:192.168.0.1 Aggregatable Global Unicast Addresses The global unicast addresses are globally unique IPv6 addresses. This address format is very well defined in the RFC 2374 (An IPv6 Aggregatable Global Unicast Address Format). The format is: 3 13 8 24 16 64 bits FP TLA ID RES NLA ID SLA ID Interface ID where: FP = Format prefix. Value of this is “001” for Aggregatable Global unicast addresses. TLA ID = Top-level Aggregation Identifier. RES = Reserved for future use.
All Node Addresses = FF02:0:0:0:0:0:0:1 (link-local) All Router Addresses = FF02:0:0:0:0:0:0:2 (link-local) All Router Addresses = FF05:0:0:0:0:0:0:2 (site-local) Network Configuration Restrictions Serviceguard supports IPv6 for data and heartbeat IP. To configure IPv6, the system should be set up in what is called a dual-stack configuration, which requires the IPv6 product bundle. The restrictions for supporting IPv6 in Serviceguard are listed below.
ndd -get /dev/ip6 ip6_nd_dad_solicit_count If the result is 1, the feature is turned on. If the result is 0, the feature is turned off. To temporarily change the state of DAD on your computer, use the ndd -set command to change the kernel parameter. ndd -set /dev/ip6 ip6_nd_dad_solicit_countn where n is a number: either 1 to turn the feature on, or 0 to turn it off.
Following the loss of lan0 or lan2, lan1 can adopt either address, as shown below. The same LAN card can be configured with both IPv4 and IPv6 addresses, as shown in below. This type of configuration allows failover of both addresses to the standby. This is shown in below.
IPv6 Network Support
I Using Serviceguard Manager HP Serviceguard Manager is a web-based, HP System Management Homepage (HP SMH) tool, that replaces the functionality of the earlier Serviceguard management tools. HP Serviceguard Manager allows you to monitor, administer and configure a Serviceguard cluster from any system with a supported web browser. Serviceguard Manager does not require additional software installation.
Table 16 Accessing Serviceguard Manager Scenario Scenario Use Case Number of Clusters Serviceguard Version Serviceguard Manager Version 1 Single cluster management 1 A.11.20 B.03.10 2 Multi-cluster management more than 1 cluster B.03.00 A.11.19 B.02.00 A.11.18 B.01.01 A.11.17.01 B.01.00 A.11.17.01 and later HP SIM or multiple browser sessions of B.02.00 or later Launching Serviceguard Manager This section provides information about three common scenarios.
NOTE: If a cluster is not yet configured, then you will not see the Serviceguard Cluster section on this screen. To create a cluster, from the SMH Tools menu, you must click Serviceguard Manager link in the Serviceguard box first, then click Create Cluster. The figure below shows a browser session at the HP Serviceguard Manager Main Page.
NOTE: Serviceguard Manager can be launched by HP Systems Insight Manager version 5.10 or later if Serviceguard Manager is installed on an HP Systems Insight Manager Central Management Server. For a Serviceguard A.11.17 cluster or earlier, Serviceguard Manager A.05.01 will be launched via Java Web Start. You must ensure that the hostname for each Serviceguard node can be resolved by DNS. For more information about this older version of Serviceguard Manager, see the Serviceguard Manager Version A.05.
J Maximum and Minimum Values for Parameters Table 17 shows the range of possible values for cluster configuration parameters. Table 17 Minimum and Maximum Values of Cluster Configuration Parameters Cluster Parameter Minimum Value Maximum Value Default Value Member Timeout See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4. See MEMBER_TIMEOUT under “Cluster Configuration Parameters” in Chapter 4.
Index A Access Control Policies, 183 Access Control Policy, 120 Access roles, 120 active node, 21 adding a package to a running cluster, 299 adding cluster nodes advance planning, 148 adding nodes to a running cluster, 265 adding packages on a running cluster , 246 additional package resources monitoring, 55 addressing, SCSI, 91 administration adding nodes to a ruuning cluster, 265 cluster and package states, 249 halting a package, 272 halting the entire cluster, 266 moving a package, 273 of packages and se
understanding components, 26 cluster administration, 264 solving problems, 319 cluster and package maintenance , 248 cluster configuration creating with SAM or Commands, 177 file on all nodes, 42 identifying cluster lock volume group, 179 identifying cluster-aware volume groups, 183 planning, 99 planning worksheet, 120 sample diagram, 89 verifying the cluster configuration, 188 cluster configuration file Autostart Delay parameter (AUTO_START_TIMEOUT), 116 cluster coordinator defined, 42 cluster lock, 44 4 o
support for additional productss, 294 troubleshooting, 318 controlling the speed of application failover, 328 creating the package configuration, 289 Critical Resource Analysis (CRA) LAN or VLAN, 287 customer defined functions adding to the control script, 293 CVM, 82 creating a storage infrastructure, 208 not supported on all HP-UX versions, 22 planning, 98 CVM planning Version 4.1 with CFS, 121 Version 4.
used by package manager, 53 FAILBACK_POLICY parameter used by package manager, 53 failover controlling the speed in applications, 328 defined, 21 failover behavior in packages, 127 failover package, 48 failover policy used by package manager, 51 FAILOVER_POLICY parameter used by package manager, 51 failure kinds of responses, 85 network communication, 87 package, service, node, 85 response to hardware failures, 86 responses to package and service failures, 87 restarting a service after failure, 87 failures
I I/O bus addresses hardware planning, 92 I/O slots hardware planning, 90, 92 I/O subsystem changes as of HP-UX 11i v3, 32, 77 identifying cluster-aware volume groups, 183 in-line terminator permitting online hardware maintenance, 313 Installing Serviceguard, 149 installing software quorum server, 168 integrating HA applications with Serviceguard, 340 internet toolkits, 326 introduction Serviceguard at a glance, 20 IP in sample package control script, 292 IP address adding and deleting in packages, 65 for n
memory requirements lockable memory for Serviceguard, 88 minimizing planned down time, 338 mirror copies of data protection against disk failure, 21 MirrorDisk/UX, 32 mirrored disks connected for high availability figure, 34 mirroring disks, 32 mirroring disks, 32 mkboot creating a root mirror with, 163 modular package, 216 monitor cluster with Serviceguard commands, 211 monitored non-heartbeat subnet parameter in cluster manager configuration, 111 monitored resource failure Serviceguard behavior, 26 monito
local interface switching, 66 modular, 219 modular and legacy, 216 modules, 219 moving, 273 optional modules, 220 parameters, 222 reconfiguring while the cluster is running, 297 reconfiguring with the cluster offline, 298 remote switching, 69 starting, 271 toolkits for databases, 326 types, 217 package administration, 271 solving problems, 319 package and cluster maintenance, 248 package configuration distributing the configuration file, 245, 294 planning, 120 run and halt script timeout parameters, 240 ste
Q qs daemon, 39 QS_ADDR parameter in cluster manager configuration, 107 QS_HOST parameter in cluster manager configuration, 106 QS_POLLING_INTERVAL parameter in cluster manager configuration, 107 QS_TIMEOUT_EXTENSION parameter in cluster manager configuration, 107 quorum and cluster reformation, 85 quorum server and safety timer, 39 blank planning worksheet, 356 installing, 168 parameters in cluster manager configuration, 106, 107 planning, 95 status and state, 253 use in re-forming a cluster, 46 worksheet,
introduction, 20 Serviceguard at a glance, 20 Serviceguard behavior after monitored resource failure, 26 Serviceguard behavior in LAN failure, 26 Serviceguard behavior in software failure, 26 Serviceguard commands to configure a package, 289 Serviceguard Manager, 23 overview, 23 SG-CFS-DG-id# multi-node package, 122 SG-CFS-MP-id# multi-node package, 122 SG-CFS-pkg system multi-node package, 121 SGCONF, 150 shared disks planning, 92 single cluster lock choosing, 45 single point of failure avoiding, 20 single
USER_NAME, 120 USER_ROLE, 120 V verifying cluster configuration, 188 verifying the cluster and package configuration, 245, 294 VERITAS CFS and CVM not supported on all HP-UX versions, 22 Dynamic Multipathing (DMP), 32 VERITAS CFS components, 41 VERITAS disk group packages creating, 194 VERITAS mount point packages creating, 205 VERITAS system multi-node packages, 193 VG in sample package control script, 292 vgcfgbackup and cluster lock data, 190 VGCHANGE in package control script, 292 vgextend creating a r
