Red Hat Cluster Suite for RHEL 4 Overview
Red Hat Cluster Suitefor RHEL 4: Overview Copyright © 2000-2006 Red Hat, Inc. Red Hat, Inc. 1801 Varsity Drive Raleigh NC 27606-2072 USA Phone: +1 919 754 3700 Phone: 888 733 4281 Fax: +1 919 754 3701 PO Box 13588 Research Triangle Park NC 27709 USA rh-cs [RHCS overview](EN)-4-Print-RHI (2006-1129T16:36) This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, V1.0 or later (the latest version is presently available at http://www.opencontent.
Table of Contents About This Document........................................................................................................ i 1. Document Conventions ........................................................................................ i 2. Feedback .............................................................................................................. v 1. Red Hat Cluster Suite Overview ..................................................................................1 1.1.
About This Document This document provides a high-level overview of Red Hat Cluster Suite for Red Hat Enterprise Linux 4. Although the information in this document is an overview, you should have advanced working knowledge of Red Hat Enterprise Linux and understand the concepts of server computing to gain a good comprehension of the information.
ii About This Document contains words that would be displayed in a different style on their own (such as file names). In these cases, they are considered to be part of the command, so the entire phrase is displayed as a command. For example: Use the cat testfile command to view the contents of a file, named testfile, in the current working directory. file name File names, directory names, paths, and RPM package names are represented this way.
About This Document iii top level of a menu on a GUI screen or window A word in this style indicates that the word is the top level of a pulldown menu. If you click on the word on the GUI screen, the rest of the menu should appear. For example: Under File on a GNOME terminal, the New Tab option allows you to open multiple shell prompts in the same window.
iv About This Document Text used in examples that is meant to be replaced with data provided by the user is displayed in this style. In the following example, is displayed in this style: The directory for the kernel source is /usr/src/kernels//, where is the version and type of kernel installed on this system. Additionally, we use several different strategies to draw your attention to certain pieces of information.
About This Document v Warning Be careful to remove only the necessary partitions. Removing other partitions could result in data loss or a corrupted system environment. 2. Feedback If you spot a typo, or if you have thought of a way to make this document better, we would love to hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against the component rh-cs.
vi About This Document
Chapter 1. Red Hat Cluster Suite Overview Clustered systems provide reliability, scalability, and availability to critical production services. Using Red Hat Cluster Suite, you can create a cluster to suit your needs for performance, high availability, load balancing, scalability, file sharing, and economy. This chapter provides an overview of Red Hat Cluster Suite components and functions, and consists of the following sections: • Section 1.1 Cluster Basics • Section 1.
2 Chapter 1. Red Hat Cluster Suite Overview High-availability clusters provide continuous availability of services by eliminating single points of failure and by failing over services from one cluster node to another in case a node becomes inoperative. Typically, services in a high-availability cluster read and write data (via read-write mounted file systems). Therefore, a high-availability cluster must maintain data integrity as one cluster node takes over control of a service from another cluster node.
Chapter 1. Red Hat Cluster Suite Overview 3 • Red Hat GFS (Global File System) — Provides a cluster file system for use with Red Hat Cluster Suite. GFS allows multiple nodes to share storage at a block level as if the storage were connected locally to each cluster node. • Cluster Logical Volume Manager (CLVM) — Provides volume management of cluster storage. • Global Network Block Device (GNBD) — An ancillary component of GFS that exports block-level storage to Ethernet.
4 Chapter 1. Red Hat Cluster Suite Overview 1.3. Cluster Infrastructure The Red Hat Cluster Suite cluster infrastructure provides the basic functions for a group of computers (called nodes or members) to work together as a cluster. Once a cluster is formed using the cluster infrastructure, you can use other Red Hat Cluster Suite components to suit your clustering needs (for example, setting up a cluster for sharing files on a GFS file system or setting up service failover).
Chapter 1. Red Hat Cluster Suite Overview 5 Note In a CMAN cluster, by default each node has one quorum vote for establishing quorum. Optionally, you can configure each node to have more than one vote. In a GULM cluster, the quorum consists of a majority of nodes designated as GULM servers according to the number of GULM servers configured: • Configured with one GULM server — Quorum equals one GULM server. • Configured with three GULM servers — Quorum equals two GULM servers.
6 Chapter 1. Red Hat Cluster Suite Overview Figure 1-3. GULM Overview 1.3.2. Lock Management Lock management is a common cluster-infrastructure service that provides a mechanism for other cluster infrastructure components to synchronize their access to shared resources. In a Red Hat cluster, one of the following Red Hat Cluster Suite components operates as the lock manager: DLM (Distributed Lock Manager) or GULM (Grand Unified Lock Manager).
Chapter 1. Red Hat Cluster Suite Overview • Configured with CMAN/DLM — fenced, the fence daemon, performs fencing. • Configured with GULM servers — GULM performs fencing. 7 When the cluster manager determines that a node has failed, it communicates to other cluster-infrastructure components that the node has failed. The fencing program (either fenced or GULM), when notified of the failure, fences the failed node.
8 Chapter 1. Red Hat Cluster Suite Overview Figure 1-4. Power Fencing Example Figure 1-5.
Chapter 1. Red Hat Cluster Suite Overview 9 Specifying a fencing method consists of editing a cluster configuration file to assign a fencing-method name, the fencing agent, and the fencing device for each node in the cluster. Note Other fencing parameters may be necessary depending on the type of cluster manager (either CMAN or GULM) selected in a cluster. The way in which a fencing method is specified depends on if a node has either dual power supplies or multiple paths to storage.
10 Chapter 1. Red Hat Cluster Suite Overview Figure 1-7. Fencing a Node with Dual Fibre Channel Connections You can configure a node with one fencing method or multiple fencing methods. When you configure a node for one fencing method, that is the only fencing method available for fencing that node.
Chapter 1. Red Hat Cluster Suite Overview 11 Figure 1-8. CCS Overview Other cluster components (for example, CMAN) access configuration information from the configuration file through CCS (refer to Figure 1-8). Figure 1-9.
12 Chapter 1. Red Hat Cluster Suite Overview The cluster configuration file (/etc/cluster/cluster.conf) is an XML file that describes the following cluster characteristics: • Cluster name — Displays the cluster name, cluster configuration file revision level, locking type (either DLM or GULM), and basic fence timing properties used when a node joins a cluster or is fenced from the cluster.
Chapter 1. Red Hat Cluster Suite Overview 13 cluster service can start on any cluster node in the event no member of the failover domain is available.) In Figure 1-10, Failover Domain 1 is configured to restrict failover within that domain; therefore, Cluster Service X can only fail over between Node A and Node B. Failover Domain 2 is also configured to restrict failover with its domain; additionally, it is configured for failover priority.
14 Chapter 1. Red Hat Cluster Suite Overview Figure 1-11 shows an example of a high-availability cluster service that is a web server named "content-webserver". It is running in cluster node B and is in a failover domain that consists of nodes A, B, and D. In addition, the failover domain is configured with a failover priority to fail over to node D before node A and to restrict failover to nodes only in that failover domain.
Chapter 1. Red Hat Cluster Suite Overview 15 1.5. Red Hat GFS Red Hat GFS is a cluster file system that allows a cluster of nodes to simultaneously access a block device that is shared among the nodes. GFS is a native file system that interfaces directly with the VFS layer of the Linux kernel file-system interface. GFS employs distributed metadata and multiple journals for optimal operation in a cluster. To maintain file system integrity, GFS uses a lock manager to coordinate I/O.
16 Chapter 1. Red Hat Cluster Suite Overview You can deploy GFS in a variety of configurations to suit your needs for performance, scalability, and economy. For superior performance and scalability, you can deploy GFS in a cluster that is connected directly to a SAN. For more economical needs, you can deploy GFS in a cluster that is connected to a LAN with servers that use GNBD (Global Network Block Device) or to iSCSI (Internet Small Computer System Interface) devices.
Chapter 1. Red Hat Cluster Suite Overview 17 Figure 1-12. GFS with a SAN 1.5.2. Performance, Scalability, Moderate Price Multiple Linux client applications on a LAN can share the same SAN-based data as shown in Figure 1-13. SAN block storage is presented to network clients as block storage devices by GNBD servers. From the perspective of a client application, storage is accessed as if it were directly attached to the server in which the application is running. Stored data is actually on the SAN.
18 Chapter 1. Red Hat Cluster Suite Overview Figure 1-13. GFS and GNBD with a SAN 1.5.3. Economy and Performance Figure 1-14 shows how Linux client applications can take advantage of an existing Ethernet topology to gain shared access to all block storage devices. Client data files and file systems can be shared with GFS on each client. Application failover can be fully automated with Red Hat Cluster Suite.
Chapter 1. Red Hat Cluster Suite Overview 19 Figure 1-14. GFS and GNBD with Directly Connected Storage 1.6. Cluster Logical Volume Manager The Cluster Logical Volume Manager (CLVM) provides a cluster-wide version of LVM2. CLVM provides the same capabilities as LVM2 on a single node, but makes the volumes available to all nodes in a Red Hat cluster. The logical volumes created with CLVM make logical volumes available to all nodes in a cluster. The key component in CLVM is clvmd.
20 Chapter 1. Red Hat Cluster Suite Overview Note Using CLVM requires minor changes to /etc/lvm/lvm.conf for cluster-wide locking. Figure 1-15. CLVM Overview You can configure CLVM using the same commands as LVM2 or by using the LVM graphical user interface (refer to Figure 1-16). Figure 1-17 shows the basic concept of creating logical volumes from Linux partitions and shows the commands used to create logical volumes.
Chapter 1. Red Hat Cluster Suite Overview Figure 1-16. LVM Graphical User Interface Figure 1-17.
22 Chapter 1. Red Hat Cluster Suite Overview 1.7. Global Network Block Device Global Network Block Device (GNBD) provides block-device access to Red Hat GFS over TCP/IP. GNBD is similar in concept to NBD; however, GNBD is GFS-specific and tuned solely for use with GFS. GNBD is useful when the need for more robust technologies — Fibre Channel or single-initiator SCSI — are not necessary or are cost-prohibitive. GNBD consists of two major components: a GNBD client and a GNBD server.
Chapter 1. Red Hat Cluster Suite Overview • To balance the load across the real servers. • To check the integrity of the services on each real server. 23 The backup LVS router monitors the active LVS router and takes over from it in case the active LVS router fails. Figure 1-19 provides an overview of the LVS components and their interrelationship. Figure 1-19. Components of a Running LVS Cluster The pulse daemon runs on both the active and passive LVS routers.
24 Chapter 1. Red Hat Cluster Suite Overview router via both the public and private network interfaces to shut down the lvs daemon on the active LVS router, and starts the lvs daemon on the backup LVS router to accept requests for the configured virtual servers. To an outside user accessing a hosted service (such as a website or database application), LVS appears as one server. However, the user is actually accessing real servers behind the LVS routers.
Chapter 1. Red Hat Cluster Suite Overview 25 Figure 1-20. Two-Tier LVS Topology Service requests arriving at an LVS router are addressed to a virtual IP address or VIP. This is a publicly-routable address that the administrator of the site associates with a fullyqualified domain name, such as www.example.com, and which is assigned to one or more virtual servers1.
26 Chapter 1. Red Hat Cluster Suite Overview • Weighted Round-Robin Scheduling — Distributes each request sequentially around a pool of real servers but gives more jobs to servers with greater capacity. Capacity is indicated by a user-assigned weight factor, which is then adjusted up or down by dynamic load information. This is a preferred choice if there are significant differences in the capacity of real servers in a server pool.
Chapter 1. Red Hat Cluster Suite Overview 27 of the active LVS router. During failover, the backup LVS router takes over the VIP addresses serviced by the failed router using a technique known as ARP spoofing — where the backup LVS router announces itself as the destination for IP packets addressed to the failed node. When the failed node returns to active service, the backup LVS router assumes its backup role again.
28 Chapter 1. Red Hat Cluster Suite Overview Figure 1-21. Three-Tier LVS Topology This topology is suited well for busy FTP servers, where accessible data is stored on a central, highly available server and accessed by each real server via an exported NFS directory or Samba share. This topology is also recommended for websites that access a central, high-availability database for transactions.
Chapter 1. Red Hat Cluster Suite Overview 29 1.8.3. Routing Methods You can use Network Address Translation (NAT) routing or direct routing with LVS. The following sections briefly describe NAT routing and direct routing with LVS. 1.8.3.1. NAT Routing Figure 1-22, illustrates LVS using NAT routing to move requests between the Internet and a private network. Figure 1-22. LVS Implemented with NAT Routing In the example, there are two NICs in the active LVS router.
30 Chapter 1. Red Hat Cluster Suite Overview IP address to its physical device on the LVS router nodes, having more than two NICs is not a requirement. Using this topology, the active LVS router receives the request and routes it to the appropriate server. The real server then processes the request and returns the packets to the LVS router. The LVS router uses network address translation to replace the address of the real server in the packets with the LVS routers public VIP address.
Chapter 1. Red Hat Cluster Suite Overview 31 Figure 1-23. LVS Implemented with Direct Routing In a typical direct-routing LVS configuration, an LVS router receives incoming server requests through a virtual IP (VIP) and uses a scheduling algorithm to route the request to real servers. Each real server processes requests and sends responses directly to clients, bypassing the LVS routers.
32 Chapter 1. Red Hat Cluster Suite Overview The issue with ARP requests in a direct-routing LVS configuration is that because a client request to an IP address must be associated with a MAC address for the request to be handled, the virtual IP address of the LVS router must also be associated to a MAC. However, because both the LVS router and the real servers have the same VIP, the ARP request is broadcast to all the nodes associated with the VIP.
Chapter 1. Red Hat Cluster Suite Overview 33 1.8.4.2. Firewall Marks Firewall marks are an easy and efficient way to a group ports used for a protocol or group of related protocols. For example, if LVS is deployed to run an e-commerce site, firewall marks can be used to bundle HTTP connections on port 80 and secure, HTTPS connections on port 443.
34 Chapter 1. Red Hat Cluster Suite Overview Command Line Tool Used With Purpose ccs_tool — Cluster Infrastructure ccs_tool is a program for making online updates to the cluster configuration file. It provides the capability to create and modify cluster infrastructure components (for example, creating a cluster, adding and removing a node). For more information about this tool, refer to the ccs_tool(8) man page.
Chapter 1. Red Hat Cluster Suite Overview 35 1.9.1. Cluster Configuration Tool You can access the Cluster Configuration Tool (Figure 1-24) through the Cluster Configuration tab in the Cluster Administration GUI. Figure 1-24. Cluster Configuration Tool The Cluster Configuration Tool represents cluster configuration components in the configuration file (/etc/cluster/cluster.conf) with a hierarchical graphical display in the left panel.
36 Chapter 1. Red Hat Cluster Suite Overview • Fence Devices — Displays fence devices. Fence devices are represented as subordinate elements under Fence Devices. Using configuration buttons at the bottom of the right frame (below Properties), you can add fence devices, delete fence devices, and edit fence-device properties. Fence devices must be defined before you can configure fencing (with the Manage Fencing For This Node button) for each node.
Chapter 1. Red Hat Cluster Suite Overview Figure 1-25.
38 Chapter 1. Red Hat Cluster Suite Overview 1.9.2. Cluster Status Tool You can access the Cluster Status Tool (Figure 1-26) through the Cluster Management tab in Cluster Administration GUI. Figure 1-26. Cluster Status Tool The nodes and services displayed in the Cluster Status Tool are determined by the cluster configuration file (/etc/cluster/cluster.conf). You can use the Cluster Status Tool to enable, disable, restart, or relocate a high-availability service.
Chapter 1. Red Hat Cluster Suite Overview 39 (Relocating a service to its current node — that is, dragging a service to its current node and dropping the service onto that node — restarts the service.) The following tables describe the members and services status information displayed by the Cluster Status Tool. Members Status Description Member The node is part of the cluster. Note: A node can be a member of a cluster; however, the node may be inactive and incapable of running services.
40 Chapter 1. Red Hat Cluster Suite Overview 1.10. Linux Virtual Server Administration GUI This section provides an overview of the LVS configuration tool available with Red Hat Cluster Suite — the Piranha Configuration Tool. The Piranha Configuration Tool is a Web-browser graphical user interface (GUI) that provides a structured approach to creating the configuration file for LVS — /etc/sysconfig/ha/lvs.cf.
Chapter 1. Red Hat Cluster Suite Overview 41 Figure 1-28. The CONTROL/MONITORING Panel Auto update Enables the status display to be updated automatically at a user-configurable interval set in the Update frequency in seconds text box (the default value is 10 seconds). It is not recommended that you set the automatic update to an interval less than 10 seconds. Doing so may make it difficult to reconfigure the Auto update interval because the page will update too frequently.
42 Chapter 1. Red Hat Cluster Suite Overview Figure 1-29. The GLOBAL SETTINGS Panel The top half of this panel sets up the primary LVS router’s public and private network interfaces. Primary server public IP The publicly routable real IP address for the primary LVS node. Primary server private IP The real IP address for an alternative network interface on the primary LVS node. This address is used solely as an alternative heartbeat channel for the backup router.
Chapter 1. Red Hat Cluster Suite Overview 43 NAT Router netmask If the NAT router’s floating IP needs a particular netmask, select it from drop-down list. NAT Router device Defines the device name of the network interface for the floating IP address, such as eth1:1. 1.10.3. REDUNDANCY The REDUNDANCY panel allows you to configure of the backup LVS router node and set various heartbeat monitoring options. Figure 1-30.
44 Chapter 1. Red Hat Cluster Suite Overview Redundant server private IP The backup router’s private real IP address. The rest of the panel is for configuring the heartbeat channel, which is used by the backup node to monitor the primary node for failure. Heartbeat Interval (seconds) Sets the number of seconds between heartbeats — the interval that the backup node will check the functional status of the primary LVS node.
Chapter 1. Red Hat Cluster Suite Overview 45 Figure 1-31. The VIRTUAL SERVERS Panel Each server displayed in the VIRTUAL SERVERS panel can be configured on subsequent screens or subsections. To add a service, click the ADD button. To remove a service, select it by clicking the radio button next to the virtual server and click the DELETE button. To enable or disable a virtual server in the table click its radio button and click the (DE)ACTIVATE button.
46 Chapter 1. Red Hat Cluster Suite Overview Figure 1-32. The VIRTUAL SERVERS Subsection Name A descriptive name to identify the virtual server. This name is not the hostname for the machine, so make it descriptive and easily identifiable. You can even reference the protocol used by the virtual server, such as HTTP. Application port The port number through which the service application will listen. Protocol Provides a choice of UDP or TCP, in a drop-down menu.
Chapter 1. Red Hat Cluster Suite Overview 47 Firewall Mark For entering a firewall mark integer value when bundling multi-port protocols or creating a multi-port virtual server for separate, but related protocols. Device The name of the network device to which you want the floating IP address defined in the Virtual IP Address field to bind. You should alias the public floating IP address to the Ethernet interface connected to the public network.
48 Chapter 1. Red Hat Cluster Suite Overview Persistence Network Mask To limit persistence to particular subnet, select the appropriate network mask from the drop-down menu. 1.10.4.2. REAL SERVER Subsection Clicking on the REAL SERVER subsection link at the top of the panel displays the EDIT REAL SERVER subsection. It displays the status of the physical server hosts for a particular virtual service. Figure 1-33. The REAL SERVER Subsection Click the ADD button to add a new server.
Chapter 1. Red Hat Cluster Suite Overview 49 Figure 1-34. The REAL SERVER Configuration Panel This panel consists of three entry fields: Name A descriptive name for the real server. Tip This name is not the hostname for the machine, so make it descriptive and easily identifiable. Address The real server’s IP address. Since the listening port is already specified for the associated virtual server, do not add a port number.
50 Chapter 1. Red Hat Cluster Suite Overview 1.10.4.3. EDIT MONITORING SCRIPTS Subsection Click on the MONITORING SCRIPTS link at the top of the page. The EDIT MONITORING SCRIPTS subsection allows the administrator to specify a send/expect string sequence to verify that the service for the virtual server is functional on each real server. It is also the place where the administrator can specify customized scripts to check services requiring dynamically changing data. Figure 1-35.
Chapter 1. Red Hat Cluster Suite Overview 51 Send A string for the nanny daemon to send to each real server in this field. By default the send field is completed for HTTP. You can alter this value depending on your needs. If you leave this field blank, the nanny daemon attempts to open the port and assume the service is running if it succeeds. Only one send sequence is allowed in this field, and it can only contain printable, ASCII characters as well as the following escape characters: • \n for new line.
52 Chapter 1.
Chapter 2. Red Hat Cluster Suite Component Summary This chapter provides a summary of Red Hat Cluster Suite components and consists of the following sections: • Section 2.1 Cluster Components • Section 2.2 Man Pages 2.1. Cluster Components Table 2-1 summarizes cluster components. Function Components Description Cluster system-config-cluster Command used to manage cluster Configuration Tool configuration in a graphical setting.
54 Chapter 2. Red Hat Cluster Suite Component Summary Function Components Description Cluster Configuration System (CCS) ccs_tool ccs_tool is part of the Cluster ccs_test Diagnostic and testing command that is used to retrieve information from configuration files through ccsd. ccsd CCS daemon that runs on all cluster nodes and provides configuration file data to cluster software. cluster.conf This is the cluster configuration file. The full path is cman.ko The kernel module for CMAN.
Chapter 2. Red Hat Cluster Suite Component Summary Function Fence 55 Components Description clurgmgrd Daemon used to handle user service requests including service start, service disable, service relocate, and service restart clurmtabd Daemon used to handle Clustered NFS mount tables fence_node Command used by lock_gulmd when a fence operation is required. This command takes the name of a node and fences it based on the node’s fencing configuration. fence_apc Fence agent for APC power switch.
56 Function DLM GFS Chapter 2. Red Hat Cluster Suite Component Summary Components Description fence_egenera Fence agent used with Egenera BladeFrame system. fence_xcat Fence agent used with xCAT-managed cluster. fence_manual Fence agent for manual interaction. NOTE This component is not supported for production environments. fence_ack_manual User interface for fence_manual agent. libdlm.so.1.0.0 Library for Distributed Lock Manager (DLM) support. dlm.
Chapter 2. Red Hat Cluster Suite Component Summary Function GNBD 57 Components Description gfs_tool Command that configures or tunes a GFS file system. This command can also gather a variety of information about the file system. lock_harness.ko Implements a pluggable lock module interface for GFS that allows for a variety of locking mechanisms to be used (for example, the DLM lock module, lock_dlm.ko). lock_dlm.ko A lock module that implements DLM locking for GFS.
58 Chapter 2. Red Hat Cluster Suite Component Summary Function Components Description LVS pulse This is the controlling process which starts all other daemons related to LVS routers. At boot time, the daemon is started by the /etc/rc.d/init.d/pulse script. It then reads the configuration file /etc/sysconfig/ha/lvs.cf. On the active LVS router, pulse starts the LVS daemon.
Chapter 2. Red Hat Cluster Suite Component Summary Function 59 Components Description nanny The nanny monitoring daemon runs on the active LVS router. Through this daemon, the active LVS router determines the health of each real server and, optionally, monitors its workload. A separate process runs for each service defined on each real server. lvs.cf This is the LVS configuration file. The full path for the file is /etc/sysconfig/ha/lvs.cf.
60 Chapter 2. Red Hat Cluster Suite Component Summary • • • cluster.conf [cluster] (5) - The configuration file for cluster products • QDisk 1.
Chapter 2. Red Hat Cluster Suite Component Summary • • • • gfs_quota (8) - Manipulate GFS disk quotas • gfs_tool (8) - interface to gfs ioctl calls Cluster Logical Volume Manager • clvmd (8) - cluster LVM daemon • lvm (8) - LVM2 tools • lvm.
62 Chapter 2.
Index M members status table, 39 A about this document, i other Red Hat Enterprise Linux documents, i C cluster displaying status, 39 cluster administration displaying cluster and service status, 39 cluster components table, 53 Cluster Configuration Tool accessing, 36 cluster service displaying status, 39 command line tools table, 33 conventions document, i F feedback, v L LVS direct routing requirements, hardware, 30 requirements, network, 30 requirements, software, 30 routing methods NAT, 29 three ti
64 S services status table, 39 T table command line tools, 33 tables cluster components, 53 members status, 39 services status, 39
