Linux Virtual Server Administration 5.1 Linux Virtual Server (LVS) for Red Hat Enterprise Linux 5.
Linux Virtual Server Administration Building a Linux Virtual Server (LVS) system offers highly-available and scalable solution for production services using specialized routing and load-balancing techniques configured through the PIRANHA. This book discusses the configuration of high-performance systems and services with Red Hat Enterprise Linux and LVS.
Linux Virtual Server Administration: Linux Virtual Server (LVS) for Red Hat Enterprise Linux 5.1 Copyright © You need to override this in your local ent file Red Hat, Inc. Copyright © You need to override this in your local ent file Red Hat Inc.. This material may only be distributed subject to the terms and conditions set forth in the Open Publication License, V1.0 or later with the restrictions noted below (the latest version of the OPL is presently available at http://www.opencontent.org/openpub/).
Linux Virtual Server Administration
Introduction .............................................................................................................. vii 1. Document Conventions ................................................................................ viii 2. Feedback ...................................................................................................... ix 1. Linux Virtual Server Overview ................................................................................. 1 1. A Basic LVS Configuration ..........
Linux Virtual Server Administration 3. CONTROL/MONITORING .............................................................................38 4. GLOBAL SETTINGS ....................................................................................40 5. REDUNDANCY ............................................................................................42 6. VIRTUAL SERVERS .....................................................................................44 6.1. The VIRTUAL SERVER Subsection ................
Introduction This document provides information about installing, configuring, and managing Red Hat Virtual Linux Server (LVS) components. LVS provides load balancing through specialized routing techniques that dispatch traffic to a pool of servers. This document does not include information about installing, configuring, and managing Red Hat Cluster software. Information about that is in a separate document.
Introduction • Using Device-Mapper Multipath — Provides information about using the Device-Mapper Multipath feature of Red Hat Enterprise Linux 5. • Using GNBD with Global File System — Provides an overview on using Global Network Block Device (GNBD) with Red Hat GFS. • Red Hat Cluster Suite Release Notes — Provides information about the current release of Red Hat Cluster Suite.
Feedback Additionally, the manual uses different strategies to draw your attention to pieces of information. In order of how critical the information is to you, these items are marked as follows: Note A note is typically information that you need to understand the behavior of the system. Tip A tip is typically an alternative way of performing a task. Important Important information is necessary, but possibly unexpected, such as a configuration change that will not persist after a reboot.
Introduction rh-lvs(EN)-5.1 (2007-10-30T17:36) By mentioning this manual's identifier, we know exactly which version of the guide you have. If you have a suggestion for improving the documentation, try to be as specific as possible. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.
Chapter 1. Linux Virtual Server Overview Linux Virtual Server (LVS) is a set of integrated software components for balancing the IP load across a set of real servers. LVS runs on a pair of equally configured computers: one that is an active LVS router and one that is a backup LVS router. The active LVS router serves two roles: • To balance the load across the real servers. • To check the integrity of the services on each real server.
Chapter 1. Linux Virtual Server Overview Figure 1.1. A Basic LVS Configuration Service requests arriving at the LVS routers are addressed to a virtual IP address, or VIP. This is a publicly-routable address the administrator of the site associates with a fully-qualified domain name, such as www.example.com, and is assigned to one or more virtual servers. A virtual server is a service configured to listen on a specific virtual IP.
Data Replication and Data Sharing Between The active router also dynamically monitors the overall health of the specific services on the real servers through simple send/expect scripts. To aid in detecting the health of services that require dynamic data, such as HTTPS or SSL, the administrator can also call external executables. If a service on a real server malfunctions, the active router stops sending jobs to that server until it returns to normal operation.
Chapter 1. Linux Virtual Server Overview Each of the real servers then accesses a shared data source over the network. Figure 1.2. A Three-Tier LVS Configuration This configuration is ideal for busy FTP servers, where accessible data is stored on a central, highly available server and accessed by each real server via an exported NFS directory or Samba share.
Real Servers available database for transactions. Additionally, using an active-active configuration with Red Hat Cluster Manager, administrators can configure one high-availability cluster to serve both of these roles simultaneously. The third tier in the above example does not have to use Red Hat Cluster Manager, but failing to use a highly available solution would introduce a critical single point of failure. 3.
Chapter 1. Linux Virtual Server Overview Weighted Round-Robin Scheduling Distributes each request sequentially around the pool of real servers but gives more jobs to servers with greater capacity. Capacity is indicated by a user-assigned weight factor, which is then adjusted upward or downward by dynamic load information. Refer to Section 3.2, “Server Weight and Scheduling” for more on weighting real servers.
Server Weight and Scheduling Distributes requests to the pool of real servers by looking up the source IP in a static hash table. This algorithm is designed for LVS routers with multiple firewalls. 3.2. Server Weight and Scheduling The administrator of LVS can assign a weight to each node in the real server pool.
Chapter 1. Linux Virtual Server Overview Figure 1.3. LVS Implemented with NAT Routing In the example, there are two NICs in the active LVS router. The NIC for the Internet has a real IP address on eth0 and has a floating IP address aliased to eth0:1. The NIC for the private network interface has a real IP address on eth1 and has a floating IP address aliased to eth1:1.
Direct Routing which uses network address translation to replace the address of the real server in the packets with the LVS routers public VIP address. This process is called IP masquerading because the actual IP addresses of the real servers is hidden from the requesting clients. Using this NAT routing, the real servers may be any kind of machine running various operating systems.
Chapter 1. Linux Virtual Server Overview Figure 1.4. LVS Implemented with Direct Routing In the typical direct routing LVS setup, the LVS router receives incoming server requests through the virtual IP (VIP) and uses a scheduling algorithm to route the request to the real servers. The real server processes the request and sends the response directly to the client, bypassing the LVS routers.
Persistence and Firewall Marks While there are many advantages to using direct routing in LVS, there are limitations as well. The most common issue with LVS via direct routing is with Address Resolution Protocol (ARP). In typical situations, a client on the Internet sends a request to an IP address. Network routers typically send requests to their destination by relating IP addresses to a machine's MAC address with ARP.
Chapter 1. Linux Virtual Server Overview it is handled according to the scheduling rules in place. Persistence also allows the administrator to specify a subnet mask to apply to the client IP address test as a tool for controlling what addresses have a higher level of persistence, thereby grouping connections to that subnet. Grouping connections destined for different ports can be important for protocols which use more than one port to communicate, such as FTP.
LVS Components Figure 1.5. LVS Components The pulse daemon runs on both the active and passive LVS routers. On the backup router, pulse sends a heartbeat to the public interface of the active router to make sure the active router is still properly functioning. On the active router, pulse starts the lvs daemon and responds to heartbeat queries from the backup LVS router.
Chapter 1. Linux Virtual Server Overview 6.1. LVS Components Section 6.1.1, “pulse” shows a detailed list of each software component in an LVS router. 6.1.1. pulse This is the controlling process which starts all other daemons related to LVS routers. At boot time, the daemon is started by the /etc/rc.d/init.d/pulse script. It then reads the configuration file /etc/sysconfig/ha/lvs.cf. On the active router, pulse starts the LVS daemon.
LVS Components Chapter 2, Initial LVS Configuration reviews important post-installation configuration steps you should take before configuring Red Hat Enterprise Linux to be an LVS router.
16
Chapter 2. Initial LVS Configuration After installing Red Hat Enterprise Linux, you must take some basic steps to set up both the LVS routers and the real servers. This chapter covers these initial steps in detail. Note The LVS router node that becomes the active node once LVS is started is also referred to as the primary node. When configuring LVS, use the Piranha Configuration Tool on the primary node. 1.
Chapter 2. Initial LVS Configuration /sbin/chkconfig --level 35 daemon on In the above command, replace daemon with the name of the service you are activating. To get a list of services on the system as well as what runlevel they are set to activate on, issue the following command: /sbin/chkconfig --list Warning Turning any of the above services on using chkconfig does not actually start the daemon. To do this use the /sbin/service command.
Configuring the Piranha Configuration Tool following command as root: /sbin/service piranha-gui start or /sbin/service piranha-gui restart Issuing this command starts a private session of the Apache HTTP Server by calling the symbolic link /usr/sbin/piranha_gui -> /usr/sbin/httpd. For security reasons, the piranha-gui version of httpd runs as the piranha user in a separate process. The fact that piranha-gui leverages the httpd service means that: 1.
Chapter 2. Initial LVS Configuration Now that the Piranha Configuration Tool is running, you may wish to consider limiting who has access to the tool over the network. The next section reviews ways to accomplish this task. 4. Limiting Access To the Piranha Configuration Tool The Piranha Configuration Tool prompts for a valid username and password combination.
Web Server Port configuration pages in the /etc/sysconfig/ha/web/secure/ directory but not to the login and the help pages in /etc/sysconfig/ha/web/. To limit access to this directory, create a .htaccess file in the /etc/sysconfig/ha/web/ directory with order, allow, and deny lines identical to /etc/sysconfig/ha/web/secure/.htaccess. 5.
22
Chapter 3. Setting Up LVS LVS consists of two basic groups: the LVS routers and the real servers. To prevent a single point of failure, each groups should contain at least two member systems. The LVS router group should consist of two identical or very similar systems running Red Hat Enterprise Linux. One will act as the active LVS router while the other stays in hot standby mode, so they need to have as close to the same capabilities as possible.
Chapter 3. Setting Up LVS which link to the real servers (eth1) will be on the 10.11.12/24 network. So on the active or primary LVS router node, the public interface's network script, /etc/sysconfig/network-scripts/ifcfg-eth0, could look something like this: DEVICE=eth0 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.26.9 NETMASK=255.255.255.0 GATEWAY=192.168.26.
Enabling NAT Routing on the LVS Routers 1.2. Routing on the Real Servers The most important thing to remember when configuring the real servers network interfaces in a NAT topology is to set the gateway for the NAT floating IP address of the LVS router. In this example, that address is 10.11.12.10. Note Once the network interfaces are up on the real servers, the machines will be unable to ping or connect in other ways to the public network. This is normal.
Chapter 3. Setting Up LVS firewall marks, see Section 4, “Multi-port Services and LVS”. Once forwarding is enabled on the LVS routers and the real servers are set up and have the clustered services running, use the Piranha Configuration Tool to configure LVS as shown in Chapter 4, Configuring the LVS Routers with Piranha Configuration Tool. Warning Do not configure the floating IP for eth0:1 or eth1:1 by manually editing network scripts or using a network configuration tool.
Direct Routing and arptables_jf topologies. While the LVS router needs to be running Red Hat Enterprise Linux to process the incoming requests and perform load-balancing for the real servers, the real servers do not need to be Linux machines to function correctly. The LVS routers need one or two NICs each (depending on if there is a back-up router).
Chapter 3. Setting Up LVS they contain the real IP of the server instead. The only node that should respond to ARP requests for any of the VIPs is the current active LVS node. 2. Once this has been completed on each real server, save the ARP table entries by typing the following commands on each real server: service arptables_jf save chkconfig --level 2345 arptables_jf on The chkconfig command will cause the system to reload the arptables configuration on bootup — before the network is started. 3.
Putting the Configuration Together two separate Apache HTTP Server services bound to port 80, because both must bind to INADDR_ANY instead of the virtual IP addresses. To configure direct routing using the iptables method, perform the following steps: 1.
Chapter 3. Setting Up LVS Configuration Tool. 3.1. General LVS Networking Tips Configure the real IP addresses for both the public and private networks on the LVS routers before attempting to configure LVS using the Piranha Configuration Tool. The sections on each topology give example network addresses, but the actual network addresses are needed. Below are some useful commands for bringing up network interfaces or checking their status.
Assigning Firewall Marks occur within the length of time specified by the persistence parameter. For more on assigning persistence to a virtual server, see Section 6.1, “The VIRTUAL SERVER Subsection”. Unfortunately, the mechanism used to balance the loads on the real servers — IPVS — can recognize the firewall marks assigned to a packet, but cannot itself assign firewall marks.
Chapter 3. Setting Up LVS HTTP and HTTPS virtual servers. These commands have the net effect of assigning any traffic addressed to the VIP on the appropriate ports a firewall mark of 80, which in turn is recognized by IPVS and forwarded appropriately. Warning The commands above will take effect immediately, but do not persist through a reboot of the system. To ensure network packet filter settings are restored upon reboot, refer to Section 6, “Saving Network Packet Filter Settings” 5.
How This Affects LVS Routing effectively cluster FTP, you must configure the LVS routers to handle both active and passive connections. The FTP client/server relationship can potentially open a large number of ports that the Piranha Configuration Tool and IPVS do not know about. 5.2. How This Affects LVS Routing IPVS packet forwarding only allows connections in and out of the cluster based on it recognizing its port number or its firewall mark.
Chapter 3. Setting Up LVS Warning If you are limiting the port range for passive connections, you must also configure the VSFTP server to use a matching port range. This can be accomplished by adding the following lines to /etc/vsftpd.conf: pasv_min_port=10000 pasv_max_port=20000 You must also control the address that the server displays to the client for passive FTP connections. In a NAT routed LVS system, add the following line to /etc/vsftpd.
Saving Network Packet Filter Settings runlevels. For more on this, refer to Section 1, “Configuring Services on the LVS Routers”. 6. Saving Network Packet Filter Settings After configuring the appropriate network packet filters for your situation, save the settings so they get restored after a reboot. For iptables, type the following command: /sbin/service iptables save This saves the settings in /etc/sysconfig/iptables so they can be recalled at boot time.
36
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool The Piranha Configuration Tool provides a structured approach to creating the necessary configuration file for LVS — /etc/sysconfig/ha/lvs.cf. This chapter describes the basic operation of the Piranha Configuration Tool and how to activate LVS once configuration is complete. Important The configuration file for LVS follows strict formatting rules. Using the Piranha Configuration Tool is the best way to prevent syntax errors in the lvs.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Figure 4.1. The Welcome Panel Click on the Login button and enter piranha for the Username and the administrative password you created in the Password field. The Piranha Configuration Tool is made of four main screens or panels. In addition, the Virtual Servers panel contains four subsections. The CONTROL/MONITORING panel is the first panel after the login screen. 3.
CONTROL/MONITORING Figure 4.2. The CONTROL/MONITORING Panel Auto update The status display on this page can be updated automatically at a user configurable interval. To enable this feature, click on the Auto update checkbox and set the desired update frequency in the Update frequency in seconds text box (the default value is 10 seconds). It is not recommended that you set the automatic update to an interval less than 10 seconds.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool CHANGE PASSWORD Clicking this button takes you to a help screen with information on how to change the administrative password for the Piranha Configuration Tool. 4. GLOBAL SETTINGS The GLOBAL SETTINGS panel is where the you define the networking details for the primary LVS router's public and private network interfaces. Figure 4.3.
GLOBAL SETTINGS Primary server private IP Enter the real IP address for an alternative network interface on the primary LVS node. This address is used solely as an alternative heartbeat channel for the backup router and does not have to correlate to the real private IP address assigned in Section 1.1, “Configuring Network Interfaces for LVS with NAT”.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Tip You should alias the NAT floating IP address to the Ethernet interface connected to the private network. In this example, the private network is on the eth1 interface, so eth1:1 is the floating IP address. Warning After completing this page, click the ACCEPT button to make sure you do not lose any changes when selecting a new panel. 5.
REDUNDANCY Figure 4.4. The REDUNDANCY Panel Redundant server public IP Enter the public real IP address for the backup LVS router node. Redundant server private IP Enter the backup node's private real IP address in this text field. If you do not see the field called Redundant server private IP, go back to the GLOBAL SETTINGS panel and enter a Primary server private IP address and click ACCEPT.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Assume dead after (seconds) If the primary LVS node does not respond after this number of seconds, then the backup LVS router node will initiate failover. Heartbeat runs on port This field sets the port at which the heartbeat communicates with the primary LVS node. The default is set to 539 if this field is left blank.
The VIRTUAL SERVER Subsection Figure 4.5. The VIRTUAL SERVERS Panel Each server displayed in the VIRTUAL SERVERS panel can be configured on subsequent screens or subsections. To add a service, click the ADD button. To remove a service, select it by clicking the radio button next to the virtual server and click the DELETE button. To enable or disable a virtual server in the table click its radio button and click the (DE)ACTIVATE button.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool any of the subsections related to this virtual server, complete this page and click on the ACCEPT button. Figure 4.6. The VIRTUAL SERVERS Subsection Name Enter a descriptive name to identify the virtual server. This name is not the hostname for the machine, so make it descriptive and easily identifiable. You can even reference the protocol used by the virtual server, such as HTTP.
The VIRTUAL SERVER Subsection Enter the virtual server's floating IP address in this text field. Virtual IP Network Mask Set the netmask for this virtual server with the drop-down menu. Firewall Mark Do not enter a firewall mark integer value in this field unless you are bundling multi-port protocols or creating a multi-port virtual server for separate, but related protocols.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Load monitoring tool The LVS router can monitor the load on the various real servers by using either rup or ruptime. If you select rup from the drop-down menu, each real server must run the rstatd service. If you select ruptime, each real server must run the rwhod service. Caution Load monitoring is not the same as load balancing and can result in hard to predict scheduling behavior when combined with weighted scheduling algorithms.
REAL SERVER Subsection Warning Remember to click the ACCEPT button after making any changes in this panel. To make sure you do not lose changes when selecting a new panel. 6.2. REAL SERVER Subsection Clicking on the REAL SERVER subsection link at the top of the panel displays the EDIT REAL SERVER subsection. It displays the status of the physical server hosts for a particular virtual service. Figure 4.7. The REAL SERVER Subsection Click the ADD button to add a new server.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Figure 4.8. The REAL SERVER Configuration Panel This panel consists of three entry fields: Name A descriptive name for the real server. Tip This name is not the hostname for the machine, so make it descriptive and easily identifiable. Address The real server's IP address. Since the listening port is already specified for the associated virtual server, do not add a port number.
EDIT MONITORING SCRIPTS Subsection Weight An integer value indicating this host's capacity relative to that of other hosts in the pool. The value can be arbitrary, but treat it as a ratio in relation to other real servers in the pool. For more on server weight, see Section 3.2, “Server Weight and Scheduling”. Warning Remember to click the ACCEPT button after making any changes in this panel. To make sure you do not lose any changes when selecting a new panel. 6.3.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Figure 4.9. The EDIT MONITORING SCRIPTS Subsection Sending Program For more advanced service verification, you can use this field to specify the path to a service-checking script. This functionality is especially helpful for services that require dynamically changing data, such as HTTPS or SSL.
Synchronizing Configuration Files Only one send sequence is allowed in this field, and it can only contain printable, ASCII characters as well as the following escape characters: • \n for new line. • \r for carriage return. • \t for tab. • \ to escape the next character which follows it. Expect Enter a the textual response the server should return if it is functioning properly. If you wrote your own sending program, enter the response you told it to send if it was successful.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool • /etc/sysconfig/iptables — If you are using firewall marks, you should synchronize one of these files based on which network packet filter you are using. Important The /etc/sysctl.conf and /etc/sysconfig/iptables files do not change when you configure LVS using the Piranha Configuration Tool. 7.1. Synchronizing lvs.cf Anytime the LVS configuration file, /etc/sysconfig/ha/lvs.
Synchronizing Network Packet Filtering Important If you are not sure whether or not packet forwarding is enabled in the kernel, see Section 5, “Turning on Packet Forwarding” for instructions on how to check and, if necessary, enable this key functionality. 7.3. Synchronizing Network Packet Filtering Rules If you are using iptables, you will need to synchronize the appropriate configuration file on the backup LVS router.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool requests to LVS at this point, you should start the backup LVS router before putting LVS into service. To do this, simply repeat the process described above on the backup LVS router node. After completing this final step, LVS will be up and running.
Appendix A. Using LVS with Red Hat Cluster You can use LVS routers with a Red Hat Cluster to deploy a high-availability e-commerce site that provides load balancing, data integrity, and application availability. The configuration in Figure A.1, “LVS with a Red Hat Cluster” represents an e-commerce site used for online merchandise ordering through a URL. Client requests to the URL pass through the firewall to the active LVS load-balancing router, which then forwards the requests to one of the Web servers.
Appendix A. Using LVS with Red Hat Cluster Figure A.1. LVS with a Red Hat Cluster Serving dynamic Web content with LVS requires a three-tier configuration (as shown in Figure A.1, “LVS with a Red Hat Cluster”). This combination of LVS and Red Hat Cluster allows for the configuration of a high-integrity, no-single-point-of-failure e-commerce site. The Red Hat Cluster can run a high-availability instance of a database or a set of databases that are network-accessible to the Web servers.
configuration is suitable if the Web servers serve only static Web content (consisting of small amounts of infrequently changing data), a two-tier configuration is not suitable if the Web servers serve dynamic content. Dynamic content could include product inventory, purchase orders, or customer databases, which must be consistent on all the Web servers to ensure that customers have access to up-to-date and accurate information.
60
job scheduling, LVS, 5 requirements, hardware, 9, 26 requirements, network, 9, 26 requirements, software, 9, 26 initial configuration, 17 ipvsadm program, 14 job scheduling, 5 lvs daemon, 14 LVS routers configuring services, 17 necessary services, 17 primary node, 17 multi-port services, 30 FTP, 32 nanny daemon, 14 NAT routing enabling, 25 requirements, hardware, 23 requirements, network, 23 requirements, software, 23 overview of, 1 packet forwarding, 21 Piranha Configuration Tool, 14 pulse daemon, 14 real
Index network address translation (see NAT) W P packet forwarding, 21 (see also LVS) Piranha Configuration Tool, 14 CONTROL/MONITORING, 38 EDIT MONITORING SCRIPTS Subsection, 51 GLOBAL SETTINGS, 40 limiting access to, 20 login panel, 37 necessary software, 37 overview of, 37 REAL SERVER subsection, 49 REDUNDANCY, 42 setting a password, 18 VIRTUAL SERVER subsection, 45 Firewall Mark, 47 Persistence, 48 Scheduling, 48 Virtual IP Address, 46 VIRTUAL SERVERS, 44 piranha-gui service, 17 piranha-passwd, 18 puls
