Red Hat Enterprise Linux 6 Cluster Administration Configuring and Managing the High Availability Add-On
Red Hat Enterprise Linux 6 Cluster Administration Configuring and Managing the High Availability Add-On
Legal Notice Co pyright © 20 14 Red Hat, Inc. and o thers. This do cument is licensed by Red Hat under the Creative Co mmo ns Attributio n-ShareAlike 3.0 Unpo rted License. If yo u distribute this do cument, o r a mo dified versio n o f it, yo u must pro vide attributio n to Red Hat, Inc. and pro vide a link to the o riginal. If the do cument is mo dified, all Red Hat trademarks must be remo ved.
T able of Cont ent s T able of Contents .Int . .roduct . . . . . .ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6. . . . . . . . . . 1. Do c ument Co nventio ns 6 1.1. Typ o g rap hic Co nventio ns 6 1.2. Pull-q uo te Co nventio ns 8 1.3. No tes and Warning s 8 2 . Feed b ac k 9 . .hapt C . . . .er . .1. .. Red . . . . Hat . . . .High . . . . Availabilit . . . . . . . . . y. .Add. . . .O. n .
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 3 .5.3. Netwo rk Co nfig uratio n 44 3 .5.4. Co nfig uring Red und ant Ring Pro to c o l 45 3 .5.5. Q uo rum Dis k Co nfig uratio n 45 3 .5.6 . Lo g g ing Co nfig uratio n 46 3 .6 . Co nfig uring Fenc e Devic es 3 .6 .1. Creating a Fenc e Devic e 47 48 3 .6 .2. Mo d ifying a Fenc e Devic e 3 .6 .3. Deleting a Fenc e Devic e 3 .7. Co nfig uring Fenc ing fo r Clus ter Memb ers 3 .7.1.
T able of Cont ent s 5 .10 . Ad d ing a Clus ter Servic e to the Clus ter 90 5 .11. Lis ting Availab le Clus ter Servic es and Res o urc es 5 .12. Virtual Mac hine Res o urc es 93 94 5 .13. Co nfig uring a Q uo rum Dis k 5 .14. Mis c ellaneo us Clus ter Co nfig uratio n 95 97 5 .14.1. Clus ter Co nfig uratio n Vers io n 5 .14.2. Multic as t Co nfig uratio n 5 .14.3. Co nfig uring a Two -No d e Clus ter 97 97 98 5 .14.4. Lo g g ing 5 .14.5.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion . .hapt C . . . .er . .9. .. Diagnosing . . . . . . . . . . .and . . . .Correct . . . . . . ing . . . Problems . . . . . . . . . in . . a. .Clust . . . . er . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 50 ........... 9 .1. Co nfig uratio n Chang es Do No t Take Effec t 150 9 .2. Clus ter Do es No t Fo rm 151 9 .3. No d es Unab le to Rejo in Clus ter after Fenc e o r Reb o o t 151 9 .4. Clus ter Daemo n c ras hes 152 9 .4.1.
T able of Cont ent s . . . . . . . . . .Line Command . . . .T. ools . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.1. 8. . . . . . . . . . .High . . . . Availabilit . . . . . . . . .y. LVM . . . . (HA. . . . LVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.1. 9. . . . . . . . . . F.1.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Introduction This document provides information about installing, configuring and managing Red Hat High Availability Add-On components. Red Hat High Availability Add-On components allow you to connect a group of computers (called nodes or members) to work together as a cluster. In this document, the use of the word cluster or clusters is used to refer to a group of computers running the Red Hat High Availability Add-On.
Int roduct ion Used to highlight system input, including shell commands, file names and paths. Also used to highlight keys and key combinations. For example: To see the contents of the file my_next_bestsel l i ng _no vel in your current working directory, enter the cat my_next_bestsel l i ng _no vel command at the shell prompt and press Enter to execute the command. The above includes a file name, a shell command and a key, all presented in mono-spaced bold and all distinguishable thanks to context.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion The mo unt -o remo unt file-system command remounts the named file system. For example, to remount the /ho me file system, the command is mo unt -o remo unt /ho me. To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release. Note the words in bold italics above: username, domain.name, file-system, package, version and release.
Int roduct ion Finally, we use three visual styles to draw attention to information that might otherwise be overlooked. Note Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier. Important Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Chapter 1. Red Hat High Availability Add-On Configuration and Management Overview Red Hat High Availability Add-On allows you to connect a group of computers (called nodes or members) to work together as a cluster. You can use Red Hat High Availability Add-On to suit your clustering needs (for example, setting up a cluster for sharing files on a GFS2 file system or setting up service failover).
Chapt er 1 . Red Hat High Availabilit y Add- O n Configurat ion and Management O verview You can now configure an independent subtree as non-critical, indicating that if the resource fails then only that resource is disabled. For information on this feature see Section 3.10, “ Adding a Cluster Service to the Cluster” and Section C.4, “ Failure Recovery and Independent Subtrees” . This document now includes the new chapter Chapter 9, Diagnosing and Correcting Problems in a Cluster.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion This document includes a new appendix, Appendix D , Cluster Service Resource Check and Failover Timeout. This appendix describes how rg manag er monitors the status of cluster resources, and how to modify the status check interval. The appendix also describes the __enfo rce_ti meo uts service parameter, which indicates that a timeout for an operation should cause a service to fail. This document includes a new section, Section 2.3.
Chapt er 1 . Red Hat High Availabilit y Add- O n Configurat ion and Management O verview 1.1.4 . New and Changed Feat ures for Red Hat Ent erprise Linux 6.4 Red Hat Enterprise Linux 6.4 includes the following documentation and feature updates and changes. The Red Hat Enterprise Linux 6.4 release provides support for the Eaton Network Power Controller (SNMP Interface) fence agent, the HP BladeSystem fence agent, and the IBM iPD U fence agent.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 1.1.6. New and Changed Feat ures for Red Hat Ent erprise Linux 6.6 Red Hat Enterprise Linux 6.6 includes the following documentation and feature updates and changes. The tables of fence device parameters in Appendix A, Fence Device Parameters have been updated to reflect small updates to the lu ci interface. The tables of resource agent parameters in Appendix B, HA Resource Parameters have been updated to reflect small updates to the lu ci interface.
Chapt er 1 . Red Hat High Availabilit y Add- O n Configurat ion and Management O verview Fencing device — A fencing device is required. A network power switch is recommended to perform fencing in an enterprise-level cluster. For information about supported fencing devices, see Appendix A, Fence Device Parameters. Storage — Some type of storage is required for a cluster. Figure 1.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Note that installing only the rg manag er will pull in all necessary dependencies to create an HA cluster from the HighAvailability channel. The l vm2-cl uster and g fs2-uti l s packages are part of ResilientStorage channel and may not be needed by your site. Warning After you install the Red Hat High Availability Add-On packages, you should ensure that your software update preferences are set so that nothing is installed automatically.
Upgrading Red Hat High Availabilit y Add- O n Soft ware Note system-co nfi g -cl uster is not available in Red Hat Enterprise Linux 6.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Chapter 2. Before Configuring the Red Hat High Availability AddOn This chapter describes tasks to perform and considerations to make before installing and configuring the Red Hat High Availability Add-On, and consists of the following sections. Important Make sure that your deployment of Red Hat High Availability Add-On meets your needs and can be supported.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n Only single site clusters are fully supported at this time. Clusters spread across multiple physical locations are not formally supported. For more details and to discuss multi-site clusters, please speak to your Red Hat sales or support representative.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion The High Availability Add-On supports both IPv4 and IPv6 Internet Protocols. Support of IPv6 in the High Availability Add-On is new for Red Hat Enterprise Linux 6. 2.2. Compat ible Hardware Before configuring Red Hat High Availability Add-On software, make sure that your cluster uses appropriate hardware (for example, supported fence devices, storage devices, and Fibre Channel switches). Refer to the Red Hat Hardware Catalog at https://hardware.redhat.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n T ab le 2.2. En ab led IP Po rt o n a C o mp u t er T h at R u n s lu ci IP Po rt N u mb er Pro t o co l C o mp o n en t 8084 TCP lu ci (C o n g a user interface server) As of the Red Hat Enterprise Linux 6.1 release, which enabled configuration by means of the /etc/sysco nfi g /l uci file, you can specifically configure the only IP address lu ci is being served at.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion After executing these commands, run the following command to save the current configuration for the changes to be persistent during reboot. $ servi ce i ptabl es save ; servi ce i ptabl es restart 2.4 . Configuring luci wit h /etc/sysco nfi g /l uci As of the Red Hat Enterprise Linux 6.1 release, you can configure some aspects of lu ci's behavior by means of the /etc/sysco nfi g /l uci file.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n For more complete information on the parameters you can configure with the /etc/sysco nfi g /l uci file, refer to the documentation within the file itself. 2.5. Configuring ACPI For Use wit h Int egrat ed Fence Devices If your cluster uses integrated fence devices, you must configure ACPI (Advanced Configuration and Power Interface) to ensure immediate and complete fencing.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Important This method completely disables ACPI; some computers do not boot correctly if ACPI is completely disabled. Use this method only if the other methods are not effective for your cluster. The following sections provide procedures for the preferred method and alternate methods of disabling ACPI Soft-Off: Section 2.5.1, “ D isabling ACPI Soft-Off with chkco nfi g Management” — Preferred method Section 2.5.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n Note D isabling ACPI Soft-Off with the BIOS may not be possible with some computers. You can disable ACPI Soft-Off by configuring the BIOS of each cluster node as follows: 1. Reboot the node and start the BIO S C MO S Setup Uti l i ty program. 2. Navigate to the Po wer menu (or equivalent power management menu). 3.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion | x KB Power ON Password Enter | | | x Hot Key Power ON Ctrl-F1 | | | | | | | | +---------------------------------------------|-------------------+ This example shows AC PI Fu n ct io n set to En ab led , and So f t - O f f b y PWR - B T T N set to In st an t - O f f . 2.5.3. Disabling ACPI Complet ely in t he g rub. co nf File The preferred method of disabling ACPI Soft-Off is with chkco nfi g management (Section 2.5.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n # initrd /initrd-[generic-]version.img #boot=/dev/hda default=0 timeout=5 serial --unit=0 --speed=115200 terminal --timeout=5 serial console title Red Hat Enterprise Linux Server (2.6.32-193.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-193.el6.x86_64 ro root=/dev/mapper/vg_doc01-lv_root console=ttyS0,115200n8 acpi=off initrd /initramrs-2.6.32-131.0.15.el6.x86_64.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Fig u re 2.1. Web Server C lu st er Service Examp le Clients access the HA service through the IP address 10.10.10.201, enabling interaction with the web server application, httpd-content. The httpd-content application uses the gfs2-content-webserver file system. If node B were to fail, the content-webserver HA service would fail over to node D . If node D were not available or also failed, the service would fail over to node A.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n specialized services to clients. An HA service is represented as a resource tree in the cluster configuration file, /etc/cl uster/cl uster. co nf (in each cluster node). In the cluster configuration file, each resource tree is an XML representation that specifies each resource, its attributes, and its relationship among other resources in the resource tree (parent, child, and sibling relationships).
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Invalid XML — Example 2.4, “ cl uster. co nf Sample Configuration: Invalid XML” Invalid option — Example 2.5, “ cl uster. co nf Sample Configuration: Invalid Option” Invalid option value — Example 2.6, “ cl uster. co nf Sample Configuration: Invalid Option Value” Examp le 2.3. cl uster. co nf Samp le C o n f ig u rat io n : Valid File PAGE 35Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n <----------------INVALID In this example, the last line of the configuration (annotated as " INVALID " here) is missing a slash — it is instead of . Examp le 2.5. cl uster. co nf Samp le C o n f ig u rat io n : In valid O p t io n <----------------INVALID PAGE 36Red Hat Ent erprise Linux 6 Clust er Administ rat ion
In this example, the fourth line of the configuration (annotated as " INVALID " here) contains an invalid value for the XML attribute, no d ei d in the cl usterno d e line for no d e0 1. exampl e. co m. The value is a negative value (" -1" ) instead of a positive value (" 1" ). For the no d ei d attribute, the value must be a positive value. 2.8.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n Important Overall, heuristics and other q d i skd parameters for your deployment depend on the site environment and special requirements needed. To understand the use of heuristics and other q d i skd parameters, refer to the qdisk(5) man page. If you require assistance understanding and using q d i skd for your site, contact an authorized Red Hat support representative.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Note Using JBOD as a quorum disk is not recommended. A JBOD cannot provide dependable performance and therefore may not allow a node to write to it quickly enough. If a node is unable to write to a quorum disk device quickly enough, the node is falsely evicted from a cluster. 2.10.
Chapt er 2 . Before Configuring t he Red Hat High Availabilit y Add- O n You can configure the Red Hat High-Availability Add-On to use UD P unicast by setting the cman transpo rt= "ud pu" parameter in the cl uster. co nf configuration file. You can also specify Unicast from the N et wo rk C o n f ig u rat io n page of the C o n g a user interface, as described in Section 3.5.3, “ Network Configuration” . 2.13. Considerat ions for ri cci For Red Hat Enterprise Linux 6, ri cci replaces ccsd .
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Chapter 3. Configuring Red Hat High Availability Add-On With Conga This chapter describes how to configure Red Hat High Availability Add-On software using C o n g a. For information on using C o n g a to manage a running cluster, see Chapter 4, Managing Red Hat High Availability Add-On With Conga. Note Conga is a graphical user interface that you can use to administer the Red Hat High Availability Add-On.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga 7. Creating resources. Refer to Section 3.9, “ Configuring Global Cluster Resources” . 8. Creating cluster services. Refer to Section 3.10, “ Adding a Cluster Service to the Cluster” . 3.2. St art ing luci Installing ricci Using l uci to configure a cluster requires that ri cci be installed and running on the cluster nodes, as described in Section 2.13, “ Considerations for ri cci ” .
Red Hat Ent erprise Linux 6 Clust er Administ rat ion https: //luci_server_hostname: luci_server_port. The default value of luci_server_port is 80 84 . The first time you access lu ci, a web browser specific prompt regarding the self-signed SSL certificate (of the lu ci server) is displayed. Upon acknowledging the dialog box or boxes, your Web browser displays the lu ci login page. 4. Any user able to authenticate on the system that is hosting lu ci can log in to lu ci. As of Red Hat Enterprise Linux 6.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga lu ci for the first time. As of Red Hat Enterprise Linux 6.4, the root user or a user who has been granted lu ci administrator permissions can also use the lu ci interface to delete users from the lu ci interface, which resets any permissions you have configured for that user. Note You can modify the way in which lu ci performs authentication by editing the /etc/pam. d /l uci file on the system.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion C an Sto p, Start, and R ebo o t C l uster No d es Allows the user to manage the individual nodes of a cluster, as described in Section 4.3, “ Managing Cluster Nodes” . C an Ad d and D el ete No d es Allows the user to add and delete nodes from a cluster, as described in Section 3.4, “ Creating a Cluster” . C an R emo ve T hi s C l uster fro m Luci Allows the user to remove a cluster from the lu ci interface, as described in Section 4.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga Fig u re 3.3. lu ci clu st er creat io n d ialo g b o x 3. Enter the following parameters on the C reat e N ew C lu st er dialog box, as necessary: At the C lu st er N ame text box, enter a cluster name. The cluster name cannot exceed 15 characters. If each node in the cluster has the same ricci password, you can check U se t h e same p asswo rd f o r all n o d es to autofill the p asswo rd field as you add nodes.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Note Whether you select the U se lo cally in st alled p ackag es or the D o wn lo ad Packag es option, if any of the base cluster components are missing (cman, rg manag er, mo d cl uster and all their dependencies), they will be installed. If they cannot be installed, the node creation will fail. Check R eb o o t n o d es b ef o re jo in in g clu st er if desired.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga information on deleting a node from an existing cluster that is currently in operation, see Section 4.3.4, “ D eleting a Member from a Cluster” . Warning Removing a cluster node from the cluster is a destructive operation that cannot be undone. 3.5. Global Clust er Propert ies When you select a cluster to configure, a cluster-specific page is displayed. The page provides an interface for configuring cluster-wide properties.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Note For more information about Po st Jo in D elay and Po st Fail D elay, refer to the fenced(8) man page. 3.5.3. Net work Configurat ion Clicking on the N et wo rk tab displays the N et wo rk C o n f ig u rat io n page, which provides an interface for configuring the network transport type.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga As of the Red Hat Enterprise Linux 6.2 release, the nodes in a cluster can communicate with each other using the UD P Unicast transport mechanism. It is recommended, however, that you use IP multicasting for the cluster network. UD P Unicast is an alternative that can be used when IP multicasting is not available. For GFS2 deployments using UD P Unicast is not recommended. Click Appl y.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Paramet er D escrip t io n Sp ecif y Ph ysical D evice: B y D evice Lab el Specifies the quorum disk label created by the mkq d i sk utility. If this field is used, the quorum daemon reads the /pro c/parti ti o ns file and checks for qdisk signatures on every block device found, comparing the label against the specified label. This is useful in configurations where the quorum device name differs among nodes.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga C o n f ig u rat io n page. After selecting the daemon, you can check whether to log the debugging messages for that particular daemon. You can also specify the sysl o g and log file settings for that daemon. Click Appl y for the logging configuration changes you have specified to take effect. 3.6. Configuring Fence Devices Configuring fence devices consists of creating, updating, and deleting fence devices for the cluster.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Fig u re 3.5. lu ci f en ce d evices co n f ig u rat io n p ag e 3.6.1. Creat ing a Fence Device To create a fence device, follow these steps: 1. From the Fen ce D evices configuration page, click Ad d . Clicking Ad d displays the Ad d Fence D evi ce (Instance) dialog box. From this dialog box, select the type of fence device to configure. 2. Specify the information in the Ad d Fence D evi ce (Instance) dialog box according to the type of fence device.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga Note Fence devices that are in use cannot be deleted. To delete a fence device that a node is currently using, first update the node fence configuration for any node using the device and then delete the device. To delete a fence device, follow these steps: 1. From the Fen ce D evices configuration page, check the box to the left of the fence device or devices to select the devices to delete. 2.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 4. Enter a Met h o d N ame for the fencing method that you are configuring for this node. This is an arbitrary name that will be used by Red Hat High Availability Add-On; it is not the same as the D NS name for the device. 5. Click Submi t. This displays the node-specific screen that now displays the method you have just added under Fen ce D evices. 6.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga You can continue to add fencing methods as needed. You can rearrange the order of fencing methods that will be used for this node by clicking on Mo ve U p and Mo ve D o wn . 3.7.3. Configuring a Node wit h Redundant Power If your cluster is configured with redundant power supplies for your nodes, you must be sure to configure fencing so that your nodes fully shut down when they need to be fenced.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Fig u re 3.6 . D u al- Po wer Fen cin g C o n f ig u rat io n 3.7.4 . T est ing t he Fence Configurat ion As of Red Hat Enterprise Linux Release 6.4, you can test the fence configuration for each node in a cluster with the fence_check utility. The following example shows the output of a successful execution of this command.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga Restricted — Allows you to restrict the members that can run a particular cluster service. If none of the members in a restricted failover domain are available, the cluster service cannot be started (either manually or by the cluster software).
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Section 3.8.1, “ Adding a Failover D omain” Section 3.8.2, “ Modifying a Failover D omain” Section 3.8.3, “ D eleting a Failover D omain” 3.8.1. Adding a Failover Domain To add a failover domain, follow the steps in this section. 1. From the cluster-specific page, you can configure failover domains for that cluster by clicking on Failo ver D o main s along the top of the cluster display.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga 4. To enable setting failover priority of the members in the failover domain, click the Prio rit iz ed checkbox. With Prio rit iz ed checked, you can set the priority value, Prio rit y, for each node selected as members of the failover domain. 5. To restrict failover to members in this failover domain, click the R est rict ed checkbox.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion To add a global cluster resource, follow the steps in this section. You can add a resource that is local to a particular service when you configure the service, as described in Section 3.10, “ Adding a Cluster Service to the Cluster” . 1. From the cluster-specific page, you can add resources to that cluster by clicking on R eso u rces along the top of the cluster display. This displays the resources that have been configured for that cluster. 2.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga 3. On the Ad d Servi ce G ro up to C l uster dialog box, at the Service N ame text box, type the name of the service. Note Use a descriptive name that clearly distinguishes the service from other services in the cluster. 4. Check the Au t o mat ically St art T h is Service checkbox if you want the service to start automatically when a cluster is started and running.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion When adding a resource to a service, whether it is an existing global resource or a resource available only to this service, you can specify whether the resource is an In d ep en d en t Su b t ree or a N o n - C rit ical R eso u rce. If you specify that a resource is an independent subtree, then if that resource fails only that resource is restarted (rather than the entire service) before the system attempting normal recovery.
Chapt er 3. Configuring Red Hat High Availabilit y Add- O n Wit h Conga Note To verify the existence of the IP service resource used in a cluster service, you can use the /sbi n/i p ad d r sho w command on a cluster node (rather than the obsoleted i fco nfi g command). The following output shows the /sbi n/i p ad d r sho w command executed on a node running a cluster service: 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Chapter 4. Managing Red Hat High Availability Add-On With Conga This chapter describes various administrative tasks for managing Red Hat High Availability Add-On and consists of the following sections: Section 4.1, “ Adding an Existing Cluster to the luci Interface” Section 4.2, “ Removing a Cluster from the luci Interface” Section 4.3, “ Managing Cluster Nodes” Section 4.4, “ Starting, Stopping, Restarting, and D eleting Clusters” Section 4.
Chapt er 4 . Managing Red Hat High Availabilit y Add- O n Wit h Conga 3. Click R emo ve. The system will ask you to confirm whether to remove the cluster from the lu ci management GUI. For information on deleting a cluster entirely, stopping all cluster services removing the cluster configuration information from the nodes themselves, refer to Section 4.4, “ Starting, Stopping, Restarting, and D eleting Clusters” . 4 .3.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 1. From the cluster-specific page, click on N o d es along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath Man ag e C lu st ers from the menu on the left side of the lu ci H o meb ase page. 2. Select the node you want to leave the cluster by clicking the checkbox for that node. 3.
Chapt er 4 . Managing Red Hat High Availabilit y Add- O n Wit h Conga 7. When the process of adding a node is complete, click on the node name for the newly-added node to configure fencing for this node, as described in Section 3.6, “ Configuring Fence D evices” . 4 .3.4 . Delet ing a Member from a Clust er To delete a member from an existing cluster that is currently in operation, follow the steps in this section.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion To start a cluster, perform the following steps: 1. Select all of the nodes in the cluster by clicking on the checkbox next to each node. 2. Select the Jo in C lu st er function from the menu at the top of the page. 3. Refresh the page to see the updated status of the nodes. To restart a running cluster, first stop all of the nodes in cluster, then start all of the nodes in the cluster, as described above.
Chapt er 4 . Managing Red Hat High Availabilit y Add- O n Wit h Conga D elet in g a service — To delete any services that are not currently running, select any services you want to disable by clicking the checkbox for that service and clicking D elet e. R elo cat in g a service — To relocate a running service, click on the name of the service in the services display.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 3. Execute servi ce l uci resto re-d b /var/l i b/l uci /d ata/lucibackupfile where lucibackupfile is the backup file to restore. For example, the following command restores the lu ci configuration information that was stored in the backup file l uci -backup20 110 9 230 6 2526 . d b: service luci restore-db /var/lib/luci/data/lucibackup20110923062526.db 4. Execute servi ce l uci start. If you need to restore a lu ci database but you have lost the ho st.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Chapter 5. Configuring Red Hat High Availability Add-On With the ccs Command As of the Red Hat Enterprise Linux 6.1 release and later, the Red Hat High Availability Add-On provides support for the ccs cluster configuration command. The ccs command allows an administrator to create, modify and view the cl uster. co nf cluster configuration file.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Important This chapter references commonly used cl uster. co nf elements and attributes. For a comprehensive list and description of cl uster. co nf elements and attributes, refer to the cluster schema at /usr/share/cl uster/cl uster. rng , and the annotated schema at /usr/share/d o c/cman-X. Y . ZZ/cl uster_co nf. html (for example /usr/share/d o c/cman-3. 0 . 12/cl uster_co nf. html ). 5.1.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command 5.1.2. Viewing t he Current Clust er Configurat ion If at any time during the creation of a cluster configuration file you want to print the current file, use the following command, specifying a node in the cluster as the host: ccs -h host --getconf If you are creating your cluster configuration file on a local system you can specify the -f option instead of the -h option, as described in Section 5.1.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion --setrm --setcman --setmul ti cast --setal tmul ti cast --setfenced aemo n --setl o g g i ng --setq uo rumd For example, to reset all of the fence deamon properties, you can run the following command. # ccs -h ho stname --setfenced aemo n Note, however, that if you use one of these commands to reset a property, then the other properties of the command will be reset to their default values.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command 2. Creating a cluster. Refer to Section 5.4, “ Creating and Modifying a Cluster” . 3. Configuring fence devices. Refer to Section 5.5, “ Configuring Fence D evices” . 4. Configuring fencing for cluster members. Refer to Section 5.7, “ Configuring Fencing for Cluster Members” . 5. Creating failover domains. Refer to Section 5.8, “ Configuring a Failover D omain” . 6. Creating resources. Refer to Section 5.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion ccs -h node-01.example.com --createcluster mycluster The cluster name cannot exceed 15 characters. If a cl uster. co nf file already exists on the host that you specify, executing this command will replace that existing file. If you want to create a cluster configuration file on your local system you can specify the -f option instead of the -h option. For information on creating the file locally, refer to Section 5.1.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command When you add a node to the cluster, you can specify the number of votes the node contributes to determine whether there is a quorum. To set the number of votes for a cluster node, use the following command: ccs -h host --addnode host --votes votes When you add a node, the ccs assigns the node a unique integer that is used as the node identifier.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion For example, to configure a value for the po st_fai l _d el ay attribute, execute the following command. This command will overwrite the values of all other existing fence daemon properties that you have set with this command and restore them to their default values. ccs -h host --setfencedaemon post_fail_delay=value To configure a value for the po st_jo i n_d el ay attribute, execute the following command.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command ccs -h host --rmfencedev fence_device_name For example, to remove a fence device that you have named myfence from the cluster configuration file on cluster node no d e1, execute the following command: ccs -h node1 --rmfencedev myfence If you need to modify the attributes of a fence device you have already configured, you must first remove that fence device then add it again with the modified attributes.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion fence_vmware - Fence agent for VMware fence_vmware_soap - Fence agent for VMware over SOAP API fence_wti - Fence agent for WTI fence_xvm - Fence agent for virtual machines To print a list of the options you can specify for a particular fence type, execute the following command: ccs -h host --lsfenceopts fence_type For example, the following command lists the fence options for the fence_wti fence agent.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Note It is recommended that you configure multiple fencing mechanisms for each node. A fencing device can fail due to network split, a power outage, or a problem in the fencing device itself. Configuring multiple fencing mechanisms can reduce the likelihood that the failure of a fencing device will have fatal results. This section documents the following procedures: Section 5.7.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion option, as described in Section 5.5, “ Configuring Fence D evices” . Each node is configured with a unique APC switch power port number: The port number for no d e-0 1. exampl e. co m is 1, the port number for no d e-0 2. exampl e. co m is 2, and the port number for no d e-0 3. exampl e. co m is 3. ccs -h ccs -h ccs -h ccs -h port=1 ccs -h port=2 ccs -h port=3 node01.example.com node01.example.com node01.example.com node01.example.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command sync the cluster configuration file to all of the nodes, as described in Section 5.15, “ Propagating the Configuration File to the Cluster Nodes” . 5.7.2. Configuring a Single St orage-Based Fence Device for a Node When using non-power fencing methods (that is, SAN/storage fencing) to fence a node, you must configure unfencing for the fence device.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion unique SAN physical port number: The port number for no d e-0 1. exampl e. co m is 11, the port number for no d e-0 2. exampl e. co m is 12, and the port number for no d e-0 3. exampl e. co m is 13. ccs -h node01.example.com ccs -h node01.example.com ccs -h node01.example.com ccs -h node01.example.com SAN port=11 ccs -h node01.example.com SAN port=12 ccs -h node01.example.com SAN port=13 ccs -h node01.example.com port=11 action=on ccs -h node01.example.
Chapt er 5.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion ccs -h node01.example.com --addmethod APC node01.example.com 2. Add a fence instance for the primary method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node: ccs -h host --addfenceinst fencedevicename node method [options] For example, to configure a fence instance in the configuration file on the cluster node no d e0 1.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Example 5.4, “ cl uster. co nf After Adding Backup Fence Methods ” shows a cl uster. co nf configuration file after you have added a power-based primary fencing method and a storage-based backup fencing method to each node in the cluster. Examp le 5.4 . cl uster. co nf Af t er Ad d in g B acku p Fen ce Met h o d s PAGE 88Red Hat Ent erprise Linux 6 Clust er Administ rat ion
Note that when you have finished configuring all of the components of your cluster, you will need to sync the cluster configuration file to all of the nodes, as described in Section 5.15, “ Propagating the Configuration File to the Cluster Nodes” . Note The order in which the system will use the fencing methods you have configured follows their order in the cluster configuration file.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command 3. Add a fence instance for the first power supply to the fence method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node. At this point you configure the acti o n attribute as o ff.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion ccs -h host --addfenceinst fencedevicename node method [options] action=on For example, to configure a second fence instance in the configuration file on the cluster node no d e-0 1. exampl e. co m that uses the APC switch power port 1 on the fence device named apc2 to fence cluster node no d e-0 1. exampl e.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Note that when you have finished configuring all of the components of your cluster, you will need to sync the cluster configuration file to all of the nodes, as described in Section 5.15, “ Propagating the Configuration File to the Cluster Nodes” . 5.7.5.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics: Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member. Restricted — Allows you to restrict the members that can run a particular cluster service.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Note To configure a preferred member, you can create an unrestricted failover domain comprising only one cluster member. D oing that causes a cluster service to run on that cluster member primarily (the preferred member), but allows the cluster service to fail over to any of the other members. To configure a failover domain, perform the following procedure: 1.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion ccs -h host --rmfailoverdomainnode failoverdomain node Note that when you have finished configuring all of the components of your cluster, you will need to sync the cluster configuration file to all of the nodes, as described in Section 5.15, “ Propagating the Configuration File to the Cluster Nodes” . 5.9.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Note Use a descriptive name that clearly distinguishes the service from other services in the cluster. When you add a service to the cluster configuration, you configure the following attributes: auto start — Specifies whether to autostart the service when the cluster starts. Use " 1" to enable and " 0" to disable; the default is enabled. d o mai n — Specifies a failover domain (if required).
Red Hat Ent erprise Linux 6 Clust er Administ rat ion options. For example, if you had not previously defined web_fs as a global service, you could add it as a service-specific resource with the following command: ccs -h node01.example.com --addsubservice example_apache fs name=web_fs device=/dev/sdd2 mountpoint=/var/www fstype=ext3 3. To add a child service to the service, you also use the --ad d subservi ce option of the ccs command, specifying the service options.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command To remove a service and all of its subservices, execute the following command: ccs -h host --rmservice servicename To remove a subservice, execute the following command: ccs -h host --rmsubservice servicename subservice [service options] Note that when you have finished configuring all of the components of your cluster, you will need to sync the cluster configuration file to all of the nodes, as described in Section 5.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion To print a list of the options you can specify for a particular service type, execute the following command: ccs -h host --lsserviceopts service_type For example, the following command lists the service options for the vm service.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command A virtual machine resource requires at least a name and a path attribute. The name attribute should match the name of the l i bvi rt domain and the path attribute should specify the directory where the shared virtual machine definitions are stored. Note The path attribute in the cluster configuration file is a path specification or a directory name, not a path to an individual file.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Paramet er D escrip t io n min _sco re The minimum score for a node to be considered " alive" . If omitted or set to 0, the default function, fl o o r((n+ 1)/2), is used, where n is the sum of the heuristics scores. The Min imu m Sco re value must never exceed the sum of the heuristic scores; otherwise, the quorum disk cannot be available. The storage device the quorum daemon uses. The device must be the same on all nodes.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Note Syncing and activating propagates and activates the updated cluster configuration file. However, for the quorum disk to operate, you must restart the cluster (refer to Section 6.2, “ Starting and Stopping a Cluster” ), ensuring that you have restarted the q d i skd daemon on each node. 5.14 . Miscellaneous Clust er Configurat ion This section describes using the ccs command to configure the following: Section 5.14.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion If you do not specify a multicast address in the cluster configuration file, the Red Hat High Availability Add-On software creates one based on the cluster ID . It generates the lower 16 bits of the address and appends them to the upper portion of the address according to whether the IP protocol is IPv4 or IPv6: For IPv4 — The address formed is 239.192. plus the lower 16 bits generated by Red Hat High Availability Add-On software.
Chapt er 5. Configuring Red Hat High Availabilit y Add- O n Wit h t he ccs Command Note that this command resets all other properties that you can set with the --setcman option to their default values, as described in Section 5.1.5, “ Commands that Overwrite Previous Settings” . When you use the ccs --setcman command to add, remove, or modify the two _no d e option, you must restart the cluster for this change to take effect.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion As of Red Hat Enterprise Linux 6.4, the Red Hat High Availability Add-On supports the configuration of redundant ring protocol. When using redundant ring protocol, there are a variety of considerations you must take into account, as described in Section 7.6, “ Configuring Redundant Ring Protocol” .
Chapt er 5.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Chapter 6. Managing Red Hat High Availability Add-On With ccs This chapter describes various administrative tasks for managing the Red Hat High Availability AddOn by means of the ccs command, which is supported as of the Red Hat Enterprise Linux 6.1 release and later. This chapter consists of the following sections: Section 6.1, “ Managing Cluster Nodes” Section 6.2, “ Starting and Stopping a Cluster” Section 6.
Chapt er 6 .
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Chapter 7. Configuring Red Hat High Availability Manually This chapter describes how to configure Red Hat High Availability Add-On software by directly editing the cluster configuration file (/etc/cl uster/cl uster. co nf) and using command-line tools. The chapter provides procedures about building a configuration file one section at a time, starting with a sample file provided in the chapter.
Chapt er 7 . Configuring Red Hat High Availabilit y Manually Important Certain procedure in this chapter call for using the cman_to o l versi o n -r command to propagate a cluster configuration throughout a cluster. Using that command requires that ri cci is running. Using ricci requires a password the first time you interact with ricci from any specific machine. For information on the ri cci service, refer to Section 2.13, “ Considerations for ri cci ” .
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 2. (O pti o nal ) If you are configuring a two-node cluster, you can add the following line to the configuration file to allow a single node to maintain quorum (for example, if one node fails): When you add or remove the two _no d e option from the cl uster. co nf file, you must restart the cluster for this change to take effect when you update the configuration.
Basic Configurat ion Examples [root@ example-01 ~]# servi ce cman start Starting cluster: Checking Network Manager... Global setup... Loading kernel modules... Mounting configfs... Starting cman... Waiting for quorum... Starting fenced... Starting dlm_controld... Starting gfs_controld... Unfencing self... Joining fence domain... [ OK [ OK [ OK [ OK [ OK [ OK [ OK [ OK [ OK [ OK [ OK ] ] ] ] ] ] ] ] ] ] ] 9.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion Examp le 7.2. cl uster. co nf Samp le: B asic T wo - N o d e C o n f ig u rat io n PAGE 113T he consensus Value for t ot em in a T wo- Node Clust er The advantage of using the optimized consensus timeout for two-node clusters is that overall failover time is reduced for the two-node case, since consensus is not a function of the token timeout. Note that for two-node autodetection in cman, the number of physical nodes is what matters and not the presence of the two _no d e= 1 directive in the cl uster. co nf file. 7.3.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 5. Save /etc/cl uster/cl uster. co nf. 6. (O pti o nal ) Validate the updated file against the cluster schema (cl uster. rng ) by running the ccs_co nfi g _val i d ate command. For example: [root@ example-01 ~]# ccs_co nfi g _val i d ate Configuration validates 7. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes. This will also run additional validation.
Fencing Configurat ion Examples Example 7.7, “ cl uster. co nf: Fencing Nodes with D ual Power Supplies” Note The examples in this section are not exhaustive; that is, there may be other ways to configure fencing depending on your requirements. Examp le 7.3. APC Fen ce D evice Ad d ed t o cl uster. co nf PAGE 116Red Hat Ent erprise Linux 6 Clust er Administ rat ion
PAGE 117Fencing Configurat ion Examples
PAGE 118Red Hat Ent erprise Linux 6 Clust er Administ rat ion
PAGE 119Fencing Configurat ion Examples PAGE 120Red Hat Ent erprise Linux 6 Clust er Administ rat ion Note The failback characteristic is applicable only if ordered failover is configured. Note Changing a failover domain configuration has no effect on currently running services. Note Failover domains are not required for operation. By default, failover domains are unrestricted and unordered.
Fencing Configurat ion Examples Note The number of fai l o verd o mai nno d e attributes depends on the number of nodes in the failover domain. The skeleton fai l o verd o mai n section in preceding text shows three fai l o verd o mai nno d e elements (with no node names specified), signifying that there are three nodes in the failover domain. 3. In the fai l o verd o mai n section, provide the values for the elements and attributes.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion PAGE 123Fencing Configurat ion Examples 7.5.1. Adding Clust er Resources You can configure two types of resources: Global — Resources that are available to any service in the cluster. These are configured in the reso urces section of the configuration file (within the rm element). Service-specific — Resources that are available to only one service. These are configured in each servi ce section of the configuration file (within the rm element). This section describes how to add a global resource.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion 7. Run the cman_to o l versi o n -r command to propagate the configuration to the rest of the cluster nodes. 8. Verify that the updated configuration file has been propagated. 9. Proceed to Section 7.5.2, “ Adding a Cluster Service to the Cluster” . Examp le 7.9 . cl uster. co nf File wit h R eso u rces Ad d ed PAGE 125Fencing Configurat ion Examples
7.5.2. Adding a Clust er Service t o t he Clust er To add a cluster service to the cluster, follow the steps in this section. Note The examples provided in this section show a cluster service in which all of the resources are at the same level.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion PAGE 127Fencing Configurat ion Examples Examp le 7.10. cl uster. co nf wit h Services Ad d ed : O n e U sin g G lo b al R eso u rces an d O n e U sin g Service- Sp ecif ic R eso u rces PAGE 128Red Hat Ent erprise Linux 6 Clust er Administ rat ion
7.6.
Fencing Configurat ion Examples The following example specifies cl usternet-no d e1-eth2 as the alternate name for cluster node cl usternet-no d e1-eth1.
Red Hat Ent erprise Linux 6 Clust er Administ rat ion
