Red Hat Enterprise Linux 4 Virtual Server Administration Linux Virtual Server (LVS) for Red Hat Enterprise Linux Edition 1.
Red Hat Enterprise Linux 4 Virtual Server Administration Linux Virtual Server (LVS) for Red Hat Enterprise Linux Edition 1.0 Landmann rlandmann@redhat.
Legal Notice Copyright © 2009 Red Hat, Inc. T his document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Table of Contents Table of Contents .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . 1. Document Conventions 5 1.1. T ypographic Conventions 5 1.2. Pull-quote Conventions 6 1.3. Notes and Warnings 7 2. Feedback 7 .Chapter . . . . . . . . 1. . . .Linux . . . . . .Virtual . . . . . . . Server . . . . . . . .Overview . . . . . . . . . . . . . .
Red Hat Enterprise Linux 4 Virtual Server Administration 3.4.1. Assigning Firewall Marks 3.5. Configuring FT P 3.5.1. How FT P Works 3.5.2. How T his Affects LVS Routing 3.5.3. Creating Network Packet Filter Rules 3.5.3.1. Rules for Active Connections 3.5.3.2. Rules for Passive Connections 3.6. Saving Network Packet Filter Settings 30 31 31 32 32 32 33 34 .Chapter ........4 . ...Configuring . . . . . . . . . . . . .the . . . .LVS . . . . Routers . . . . . . . . .with . . . . .Piranha . . . . . . . .
Table of Contents 3
Red Hat Enterprise Linux 4 Virtual Server Administration Introduction T his document provides information about installing, configuring, and managing Red Hat Virtual Linux Server (LVS) components. LVS provides load balancing through specialized routing techniques that dispatch traffic to a pool of servers. T his document does not include information about installing, configuring, and managing Red Hat Cluster software. Information about that is in a separate document.
Introduction 1. Document Conventions T his manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information. In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts set. T he Liberation Fonts set is also used in HT ML editions if the set is installed on your system. If not, alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later include the Liberation Fonts set by default. 1.1.
Red Hat Enterprise Linux 4 Virtual Server Administration Next. T he character you sought will be highlighted in the Character T able. Double-click this highlighted character to place it in the T ext to copy field and then click the Copy button. Now switch back to your document and choose Edit → Paste from the gedit menu bar.
Introduction static int kvm_vm_ioctl_deassign_device(struct kvm *kvm, struct kvm_assigned_pci_dev *assigned_dev) { int r = 0; struct kvm_assigned_dev_kernel *match; mutex_lock(&kvm->lock); match = kvm_find_assigned_dev(&kvm->arch.
Red Hat Enterprise Linux 4 Virtual Server Administration Be sure to mention the manual's identifier: Virtual_Server_Administration(EN)-4.8 (2009-04-23T15:41) By mentioning this manual's identifier, we know exactly which version of the guide you have. If you have a suggestion for improving the documentation, try to be as specific as possible. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.
Chapter 1. Linux Virtual Server Overview Chapter 1. Linux Virtual Server Overview Linux Virtual Server (LVS) is a set of integrated software components for balancing the IP load across a set of real servers. LVS runs on a pair of equally configured computers: one that is an active LVS router and one that is a backup LVS router. T he active LVS router serves two roles: T o balance the load across the real servers. T o check the integrity of the services on each real server.
Red Hat Enterprise Linux 4 Virtual Server Administration Figure 1.1. A Basic LVS Configuration Service requests arriving at the LVS routers are addressed to a virtual IP address, or VIP. T his is a publicly-routable address the administrator of the site associates with a fully-qualified domain name, such as www.example.com, and is assigned to one or more virtual servers. A virtual server is a service configured to listen on a specific virtual IP. Refer to Section 4.
Chapter 1. Linux Virtual Server Overview However, this type of data synchronization does not optimally function if the configuration is overloaded with users constantly uploading files or issuing database transactions. For a configuration with a high load, a three-tier topology is the ideal solution. 1.2. A Three-Tier LVS Configuration Figure 1.2, “A T hree-T ier LVS Configuration” shows a typical three-tier LVS topology.
Red Hat Enterprise Linux 4 Virtual Server Administration 1.3. LVS Scheduling Overview One of the advantages of using LVS is its ability to perform flexible, IP-level load balancing on the real server pool. T his flexibility is due to the variety of scheduling algorithms an administrator can choose from when configuring LVS. LVS load balancing is superior to less flexible methods, such as RoundRobin DNS where the hierarchical nature of DNS and the caching by client machines can lead to load imbalances.
Chapter 1. Linux Virtual Server Overview dynamic scheduling algorithm, making it a better choice if there is a high degree of variation in the request load. It is best suited for a real server pool where each member node has roughly the same capacity. If a group of servers have different capabilities, weighted least-connection scheduling is a better choice. Weighted Least-Connections (default) Distributes more requests to servers with fewer active connections relative to their capacities.
Red Hat Enterprise Linux 4 Virtual Server Administration suppose there are three servers in the real server pool. Servers A and B are weighted at 1 and the third, server C, is weighted at 2. If server C goes down for any reason, servers A and B evenly distributes the abandoned load. However, once server C comes back online, the LVS router sees it has zero connections and floods the server with all incoming requests until it is on par with servers A and B.
Chapter 1. Linux Virtual Server Overview physical device on the LVS router nodes, having more than two NICs is not a requirement. Using this topology, the active LVS router receives the request and routes it to the appropriate server. T he real server then processes the request and returns the packets to the LVS router which uses network address translation to replace the address of the real server in the packets with the LVS routers public VIP address.
Red Hat Enterprise Linux 4 Virtual Server Administration bottleneck under heavy network load. 1.4 .2.1. Direct Routing and the ARP Limitation While there are many advantages to using direct routing in LVS, there are limitations as well. T he most common issue with LVS via direct routing is with Address Resolution Protocol (ARP). In typical situations, a client on the Internet sends a request to an IP address.
Chapter 1. Linux Virtual Server Overview the problem of grouping together connections destined for different ports. For these situations, it is best to use firewall marks. 1.5.2. Firewall Marks Firewall marks are an easy and efficient way to a group ports used for a protocol or group of related protocols. For instance, if LVS is deployed to run an e-commerce site, firewall marks can be used to bundle HT T P connections on port 80 and secure, HT T PS connections on port 443.
Red Hat Enterprise Linux 4 Virtual Server Administration nanny process checks the state of one configured service on one real server, and tells the lvs daemon if the service on that real server is malfunctioning. If a malfunction is detected, the lvs daemon instructs ipvsadm to remove that real server from the IPVS routing table.
Chapter 1. Linux Virtual Server Overview Chapter 2, Initial LVS Configuration reviews important post-installation configuration steps you should take before configuring Red Hat Enterprise Linux to be an LVS router.
Red Hat Enterprise Linux 4 Virtual Server Administration Chapter 2. Initial LVS Configuration After installing Red Hat Enterprise Linux, you must take some basic steps to set up both the LVS routers and the real servers. T his chapter covers these initial steps in detail. Note T he LVS router node that becomes the active node once LVS is started is also referred to as the primary node. When configuring LVS, use the Piranha Configuration T ool on the primary node. 2.1.
Chapter 2. Initial LVS Configuration Warning T urning any of the above services on using chkconfig does not actually start the daemon. T o do this use the /sbin/service command. See Section 2.3, “Starting the Piranha Configuration T ool Service” for an example of how to use the /sbin/service command.
Red Hat Enterprise Linux 4 Virtual Server Administration Warning If the command /sbin/service httpd stop or /sbin/service httpd restart is issued on an LVS router, you must start the piranha-gui service by issuing the following command: /sbin/service piranha-gui start T he piranha-gui service is all that is necessary to begin configuring LVS. However, if you are configuring LVS remotely, the sshd service is also required.
Chapter 2. Initial LVS Configuration Order deny,allow Deny from all Allow from 127.0.0.1 You can also allow specific hosts or subnets as seen in this example: Order deny,allow Deny from all Allow from 192.168.1.100 Allow from 172.16.57 In this example, only Web browsers from the machine with the IP address of 192.168.1.100 and machines on the 172.16.57/24 network can access the Piranha Configuration T ool. Warning Editing the Piranha Configuration T ool.
Red Hat Enterprise Linux 4 Virtual Server Administration Chapter 3. Setting Up LVS LVS consists of two basic groups: the LVS routers and the real servers. T o prevent a single point of failure, each groups should contain at least two member systems. T he LVS router group should consist of two identical or very similar systems running Red Hat Enterprise Linux. One will act as the active LVS router while the other stays in hot standby mode, so they need to have as close to the same capabilities as possible.
Chapter 3. Setting Up LVS DEVICE=eth0 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.26.9 NETMASK=255.255.255.0 GATEWAY=192.168.26.254 T he /etc/sysconfig/network-scripts/ifcfg-eth1 for the private NAT interface on the LVS router could look something like this: DEVICE=eth1 BOOTPROTO=static ONBOOT=yes IPADDR=10.11.12.9 NETMASK=255.255.255.0 In this example, the VIP for the LVS router's public interface will be 192.168.26.10 and the VIP for the NAT or private interface will be 10.11.12.10.
Red Hat Enterprise Linux 4 Virtual Server Administration So the real server's /etc/sysconfig/network-scripts/ifcfg-eth0 file could look similar to this: DEVICE=eth0 ONBOOT=yes BOOTPROTO=static IPADDR=10.11.12.1 NETMASK=255.255.255.0 GATEWAY=10.11.12.10 Warning If a real server has more than one network interface configured with a GAT EWAY= line, the first one to come up will get the gateway.
Chapter 3. Setting Up LVS response to the client. So, for example, if the client is on the Internet, and sends the packet through the LVS router to a real server, the real server must be able to go directly to the client via the Internet. T his can be done by configuring a gateway for the real server to pass packets to the Internet.
Red Hat Enterprise Linux 4 Virtual Server Administration T o configure each real server to ignore ARP requests for each virtual IP addresses, perform the following steps: 1.
Chapter 3. Setting Up LVS 1. On each real server, run the following command for every VIP, port, and protocol (T CP or UDP) combination intended to be serviced for the real server: iptables -t nat -A PREROUT ING -p -d --dport -j REDIRECT T his command will cause the real servers to process packets destined for the VIP and port that they are given. 2.
Red Hat Enterprise Linux 4 Virtual Server Administration Warning Do not use the ifup scripts to bring up any floating IP addresses you may configure using Piranha Configuration T ool (eth0:1 or eth1:1). Use the service command to start pulse instead (see Section 4.8, “Starting LVS” for details). Bringing Down Real Network Interfaces T o bring down a real network interface, use the following command as root, replacing N with the number corresponding to the interface (eth0 and eth1).
Chapter 3. Setting Up LVS network packets. Before creating network packet filter rules, make sure there are no rules already in place. T o do this, open a shell prompt, login as root, and type: /sbin/service iptables status If iptables is not running, the prompt will instantly reappear. If iptables is active, it displays a set of rules.
Red Hat Enterprise Linux 4 Virtual Server Administration determines how the server responds and on what ports transactions will occur. T he two types of data connections are: Active Connections When an active connection is established, the server opens a data connection to the client from port 20 to a high range port on the client machine. All data from the server is then passed over this connection.
Chapter 3. Setting Up LVS servers that IPVS does not know about: /sbin/iptables -t nat -A POST ROUT ING -p tcp -s n.n.n.0/24 --sport 20 -j MASQUERADE In the iptables command, n.n.n should be replaced with the first three values for the floating IP for the NAT interface's internal network interface defined in the GLOBAL SET T INGS panel of Piranha Configuration T ool. 3.5.3.2.
Red Hat Enterprise Linux 4 Virtual Server Administration Finally, you need to be sure that the appropriate service is set to activate on the proper runlevels. For more on this, refer to Section 2.1, “Configuring Services on the LVS Routers”. 3.6. Saving Network Packet Filter Settings After configuring the appropriate network packet filters for your situation, save the settings so they get restored after a reboot.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool T he Piranha Configuration T ool provides a structured approach to creating the necessary configuration file for LVS — /etc/sysconfig/ha/lvs.cf. T his chapter describes the basic operation of the Piranha Configuration T ool and how to activate LVS once configuration is complete. Important T he configuration file for LVS follows strict formatting rules.
Red Hat Enterprise Linux 4 Virtual Server Administration Figure 4 .1. T he Welcome Panel Click on the Login button and enter piranha for the Usernam e and the administrative password you created in the Password field. T he Piranha Configuration T ool is made of four main screens or panels. In addition, the Virtual Servers panel contains four subsections. T he CONT ROL/MONIT ORING panel is the first panel after the login screen. 4.3.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Figure 4 .2. T he CONT ROL/MONIT ORING Panel Auto update T he status display on this page can be updated automatically at a user configurable interval. T o enable this feature, click on the Auto update checkbox and set the desired update frequency in the Update frequency in seconds text box (the default value is 10 seconds). It is not recommended that you set the automatic update to an interval less than 10 seconds.
Red Hat Enterprise Linux 4 Virtual Server Administration Figure 4 .3. T he GLOBAL SET T INGS Panel T he top half of this panel sets up the primary LVS router's public and private network interfaces. T hese are the interfaces already configured in Section 3.1.1, “Configuring Network Interfaces for LVS with NAT ”. Prim ary server public IP In this field, enter the publicly routable real IP address for the primary LVS node.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Use network type Click the NAT button to select NAT routing. Click the Direct Routing button to select direct routing. T he next three fields deal specifically with the NAT router's virtual network interface connecting the private network with the real servers. T hese fields do not apply to the direct routing network type. NAT Router IP Enter the private floating IP in this text field.
Red Hat Enterprise Linux 4 Virtual Server Administration Figure 4 .4 . T he REDUNDANCY Panel Redundant server public IP Enter the public real IP address for the backup LVS router node. Redundant server private IP Enter the backup node's private real IP address in this text field. If you do not see the field called Redundant server private IP, go back to the GLOBAL SET T INGS panel and enter a Prim ary server private IP address and click ACCEPT .
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Warning Remember to click the ACCEPT button after making any changes in this panel to make sure you do not lose any changes when selecting a new panel. 4.6. VIRTUAL SERVERS T he VIRT UAL SERVERS panel displays information for each currently defined virtual server.
Red Hat Enterprise Linux 4 Virtual Server Administration allows you to configure an individual virtual server. Links to subsections related specifically to this virtual server are located along the top of the page. But before configuring any of the subsections related to this virtual server, complete this page and click on the ACCEPT button. Figure 4 .6. T he VIRT UAL SERVERS Subsection Nam e Enter a descriptive name to identify the virtual server.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool protocols or creating a multi-port virtual server for separate, but related protocols. In this example, the above virtual server has a Firewall Mark of 80 because we are bundling connections to HT T P on port 80 and to HT T PS on port 443 using the firewall mark value of 80. When combined with persistence, this technique will ensure users accessing both insecure and secure webpages are routed to the same real server, preserving state.
Red Hat Enterprise Linux 4 Virtual Server Administration Warning Load monitoring is not the same as load balancing and can result in hard to predict scheduling behavior when combined with weighted scheduling algorithms. Also, if you use load monitoring, the real servers must be Linux machines. Scheduling Select your preferred scheduling algorithm from the drop-down menu. T he default is Weighted least-connection. For more information on scheduling algorithms, see Section 1.3.1, “Scheduling Algorithms”.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool subsection. It displays the status of the physical server hosts for a particular virtual service. Figure 4 .7. T he REAL SERVER Subsection Click the ADD button to add a new server. T o delete an existing server, select the radio button beside it and click the DELET E button. Click the EDIT button to load the EDIT REAL SERVER panel, as seen in Figure 4.8, “T he REAL SERVER Configuration Panel”. Figure 4 .8.
Red Hat Enterprise Linux 4 Virtual Server Administration T his panel consists of three entry fields: Nam e A descriptive name for the real server. Note T his name is not the hostname for the machine, so make it descriptive and easily identifiable. Address T he real server's IP address. Since the listening port is already specified for the associated virtual server, do not add a port number. Weight An integer value indicating this host's capacity relative to that of other hosts in the pool.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool Figure 4 .9. T he EDIT MONIT ORING SCRIPT S Subsection Sending Program For more advanced service verification, you can use this field to specify the path to a servicechecking script. T his functionality is especially helpful for services that require dynamically changing data, such as HT T PS or SSL.
Red Hat Enterprise Linux 4 Virtual Server Administration Note If an external program is entered in the Sending Program field, then the Send field is ignored. Send Enter a string for the nanny daemon to send to each real server in this field. By default the send field is completed for HT T P. You can alter this value depending on your needs. If you leave this field blank, the nanny daemon attempts to open the port and assume the service is running if it succeeds.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool /etc/sysconfig/ha/lvs.cf — the configuration file for the LVS routers. /etc/sysctl — the configuration file that, among other things, turns on packet forwarding in the kernel. /etc/sysconfig/iptables — If you are using firewall marks, you should synchronize one of these files based on which network packet filter you are using. Important T he /etc/sysctl.
Red Hat Enterprise Linux 4 Virtual Server Administration 4.7.3. Synchronizing Network Packet Filtering Rules If you are using iptables, you will need to synchronize the appropriate configuration file on the backup LVS router. If you alter the any network packet filter rules, enter the following command as root from the primary LVS router: scp /etc/sysconfig/iptables n.n.n.n:/etc/sysconfig/ In the command, replace n.n.n.n with the real IP address of the backup LVS router.
Using LVS with Red Hat Cluster Using LVS with Red Hat Cluster You can use LVS routers with a Red Hat Cluster to deploy a high-availability e-commerce site that provides load balancing, data integrity, and application availability. T he configuration in Figure A.1, “LVS with a Red Hat Cluster” represents an e-commerce site used for online merchandise ordering through a URL.
Red Hat Enterprise Linux 4 Virtual Server Administration Each tier provides the following functions: First tier — LVS routers performing load-balancing to distribute Web requests. Second tier — A set of Web servers to serve the requests. T hird tier — A Red Hat Cluster to serve data to the Web servers. In an LVS configuration like the one in Figure A.1, “LVS with a Red Hat Cluster”, client systems issue requests on the World Wide Web.
Revision History Revision History Revision 1.0-5.4 00 Rebuild with publican 4.0.0 2013-10-31 Rüdiger Landmann Revision 1.0-5 Rebuild for Publican 3.0 2012-07-18 Anthony T owns Revision 1.0-0 Fri Apr 24 2009 Index Symbols /etc/sysconfig/ha/lvs.cf file, /etc/sysconfig/ha/lvs.
Red Hat Enterprise Linux 4 Virtual Server Administration ipvsadm program, ipvsadm J job scheduling, LVS, LVS Scheduling Overview L least connections (see job scheduling, LVS) LVS - /etc/sysconfig/ha/lvs.cf file, /etc/sysconfig/ha/lvs.
Revision History - starting LVS, Starting LVS - synchronizing configuration files, Synchronizing Configuration Files - three-tier - Red Hat Cluster Manager, A T hree-T ier LVS Configuration - using LVS with Red Hat Cluster, Using LVS with Red Hat Cluster lvs daemon, lvs M multi-port services, Multi-port Services and LVS - (see also LVS) N nanny daemon, nanny NAT - enabling, Enabling NAT Routing on the LVS Routers - routing methods, LVS, Routing Methods network address translation (see NAT ) P packet f
Red Hat Enterprise Linux 4 Virtual Server Administration piranha-passwd, Setting a Password for the Piranha Configuration T ool pulse daemon, pulse pulse service, Configuring Services on the LVS Routers R real servers - configuring services, Configuring Services on the Real Servers Red Hat Cluster - and LVS, Using LVS with Red Hat Cluster - using LVS with, Using LVS with Red Hat Cluster round robin (see job scheduling, LVS) routing - prerequisites for LVS, Configuring Network Interfaces for LVS with NAT
