Building highly efficient Red Hat Enterprise Virtualization 3.0 Infrastructure with Mellanox Interconnect Reference Design Eli Karpilovski Manager, Cloud Market Development Mellanox Technologies, Inc. Itzik Brown Sr. Engineer, Solution Integration Mellanox Technologies, Inc. Version 1.
1801 Varsity Drive™ Raleigh NC 27606-2072 USA Phone: +1 919 754 3700 Phone: 888 733 4281 Fax: +1 919 754 3701 PO Box 13588 Research Triangle Park NC 27709 USA Linux is a registered trademark of Linus Torvalds. Red Hat, Red Hat Enterprise Linux and the Red Hat "Shadowman" logo are registered trademarks of Red Hat, Inc. in the United States and other countries. UNIX is a registered trademark of The Open Group. Mellanox®, Mellanox logo are registered trademarks of Mellanox Technologies, Ltd.
Table of Contents 1 Introduction........................................................................................................1 2 Getting Started..................................................................................................2 2.1 Basic Test Bed Configuration........................................................................................................2 2.2 Required Hardware..................................................................................................
7 Conclusion.......................................................................................................41 Appendix A: Troubleshooting............................................................................42 A.1.1 Host is not Added to a Logical Server in UFM.......................................................................42 A.1.2 Migration of VM Fails............................................................................................................ 42 A.1.
List of Figures Figure 1: Basic Test Bed Scenario - Example........................................................................................2 Figure 2: Red Hat Enterprise Linux Installation......................................................................................4 Figure 3: Red Hat Enterprise Linux Installation......................................................................................4 Figure 4: Red Hat Network - Register.........................................................
Figure 25: Adding a New Virtual Machine – Configuration...................................................................27 Figure 26: Adding a New Virtual Machine – Adding a New Network Interface.....................................28 Figure 27: Adding a New Virtual Machine – Adding a New Virtual Disk...............................................28 Figure 28: Adding a New Virtual Machine – Finishing Configuration....................................................
Index of Tables Table 1: Required Hardware.................................................................................................................. 3 Table 2: List of Related Documents.....................................................................................................43 refarch-feedback@redhat.com vii www.redhat.
1 Introduction This reference design describes how to integrate and use Red Hat Enterprise Virtualization Manager (RHEV-M) to control a cloud based on: • Servers with Red Hat OS and KVM • Mellanox products for network connectivity and storage Through significant customer engagements, building data centers, and working closely with IaaS architects and administrators, Mellanox in collaboration with Red Hat formed a new Cloud-X architecture which enables an integrated computing, network and storage technol
2 Getting Started 2.1 Basic Test Bed Configuration Figure 1: Basic Test Bed Scenario - Example www.redhat.com 2 refarch-feedback@redhat.
2.2 Required Hardware Table 1: Required Hardware Equipment Notes Used for Data/Storage network. Mellanox SX6036/Grid Director 4036E InfiniBand/Gateway switch An InfiniBand switch can be used, The gateway is used to be connected to external Ethernet networks. 1GE - Used for Management network.
• Red Hat Enterprise Linux (RHEL) 6.2 (or higher) • Red Hat Enterprise Virtualization 3.0 (RHEV, RHEV-M) or higher 3 Software Installation and Basic Configuration The reference solution contains several software applications and HW components. The following chapter supplies basic software installation procedures. 3.
Step 3: Activate NTP services. [root@host2-1]# /etc/init.d/ntpd status ntpd is stopped [root@host2-1]# /etc/init.d/ntpd start Starting ntpd: [ OK ] [root@host2-1]# /sbin/chkconfig ntpd on [root@host2-1]# /etc/init.d/ntpd status ntpd (pid 5197) is running... [root@host2-1]# Step 4: Register to the Red Hat Network (RHN). [root@host2-1]#rhn_register … Figure 4: Red Hat Network - Register Step 5: Subscribe to the required channels.
-channel=jbappplatform-5-x86_64-server-6-rpm Username: meldcs Password: [root@host2-1]#rhn-channel --add --channel=rhel-x86_64-server-supplementary-6 Username: meldcs Password: [root@host2-1]# Step 6: Confirm the list of channels to which the server is subscribed. [root@host2-1]#rhn-channel –list jbappplatform-5-x86_64-server-6-rpm rhel-x86_64-server-6 rhel-x86_64-server-6-rhevm-3 [root@host2-1]# Step 7: If installed, the classpathx-jaf package must be removed.
rhel-x86_64-server-6-rhevm-3 121/121 rhel-x86_64-server-supplementary-6 | 1.8 kB 00:00 rhel-x86_64-server-supplementary-6/primary | 91 kB 00:00 rhel-x86_64-server-supplementary-6 249/249 Package(s) classpathx-jaf available, but not installed. No Packages marked for removal [root@host2-1]# Step 8: Use yum to ensure that the most up to date versions of all installed packages are in use. [root@host2-1]#yum upgrade … Step 9: Use yum to initiate installation of the RHEV-M package and all dependencies.
Database password (required for secure authentication with the locally created database) : Warning: Weak Password. Confirm password : Organization Name for the Certificate: Mellanox The default storage type you will be using ISCSI ['NFS'| 'FC'| 'ISCSI'] [NFS] : Should the installer configure NFS share on this server to be used as an ISO Domain? ['yes'| 'no'] [yes] : no Firewall ports need to be opened. You can let the installer configure iptables automatically overriding the current configuration.
Editing RHEV Manager Configuration... [ DONE ] Configuring Firewall (iptables)... [ DONE ] Starting JBoss Service... [ DONE ] **** Installation completed successfully ****** (Please allow RHEV Manager a few moments to start up.....
LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= Note: Access the administration portal by pointing Internet Explorer to http://your_server:8080 (assuming defaults are used during installation). Use the administrator username and password supplied in the configuration step. .NET Framework installation prompting may appear if accessing the RHEV-M Portal for the first time.
3.2 RHEV Host Installation (Host3) Follow these steps for RHEV installation: Step 1: Install RHEL 6.2. You may use the default installation of RHEL – “Basic Server”. Figure 6: Red Hat Enterprise Linux Installation Figure 7: Red Hat Enterprise Linux Installation Step 2: Ensure the VM has installed RHEL 6.2 successfully. Step 3: Activate NTP services. [root@host3]# /etc/init.d/ntpd status ntpd is stopped [root@host3]# /etc/init.
Step 4: Register to the Red Hat Network (RHN). [root@host3]#rhn_register … Figure 8: Red Hat Network - Register Step 5: Subscribe to the required channels. Run: [root@host3]# rhn-channel --add --channel=rhel-x86_64-server Username: meldcs Password: [root@host3]# rhn-channel --add --channel=rhel-x86_64-rhev-mgmt-agent-6 Username: meldcs Password: [root@host3]# Step 6: Confirm the list of channels to which the server is subscribed.
Step 7: Add a manual host entry to the /etc/hosts file (on the Red Hat Enterprise Linux host) for the RHEV-M server to enable vdsm and other services to connect properly to the host (if not using DNS services). 10.0.0.1 server1.example.com rhev-manager.server1.example.com For example: 172.30.40.147 host2-1.lab.mtl.com rhev-manager.host2-1.lab.mtl.com Step 8: Open firewall ports. The following commands will remove existing firewall rules and add the required ports by RHEV-M to the iptables rules.
PermitRootLogin yes Step 11: Restart the SSH server. [root@host3]# service sshd restart Note: For advanced configuration of the RHEV-M refer to “Red Hat Enterprise Virtualization 3.0 – Installation Guide”. 3.3 Mellanox OFED Driver Installation (All Hosts) Any host in the fabric shall have Mellanox OFED installed. Follow the steps below for basic Mellanox OFED installation on all hosts. Step 1: Download Mellanox OFED from www.mellanox.com and locate it in your file system.
sys_image_guid: 0002:c903:000d:1413 vendor_id: 0x02c9 vendor_part_id: 26428 hw_ver: 0xB0 board_id: MT_0DD0110009 phys_port_cnt: 2 port: port: Step 6: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 24 port_lid: 22 port_lmc: 0x00 link_layer: IB 2 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet Set up an IP address for the “ib0” interface by editing the ifcfg-ib0 file and
TYPE=Infiniband # ifup ib0 Step 7: Add eIPoIB Interfaces. Make sure the host is connected to an InfiniBand network and that the latest Mellanox OFED that supports eIPoIB is installed. Step 8: Locate the interface. Edit the following line in the file /etc/infiniband/openib.conf: E_IPOIB_LOAD=yes If the E_IPOIB_LOAD = no , please change it and reload openibd process, as follows: #/etc/init.
iSER- ConnectX’s RDMA capabilities can be used to accelerate hypervisor traffic such as storage access, VM migration, data and VM replication. The use of RDMA moves the data from node-to-node to the ConnectX hardware, producing much faster performance, lower latency/access-time, and lower CPU overhead and provides zero-copy message transfers for SCSI packets.
State Serial Idx Name Rev Size Cache Vendor Model paths running 1 3600605b0032a49601601f69931f3bb42 667GB 8i 0042bbf33199f6011660492a03b00506 3.14 1 0 LSI MR9265- running 2 3600605b0032882501643ddec0204767e 890GB 8i 007e760402ecdd431650822803b00506 3.14 1 0 LSI MR9265- running 3 3600605b0032867601643c9ecd0d3de2c 890GB 8i 002cded3d0ecc9431660672803b00506 3.14 1 0 LSI MR9265- VSA-/# add servers/ rhev-servers ips=192.168.20.101;192.168.20.103 VSA-/# add targets/iqn.iser.
# cd mellanox_nm_client # ./install.sh 4 RHEV Manager Configuration Before configuring Red Hat Enterprise Virtualization Manager (RHEV-M), please ensure it is installed and running. 4.1 Add Data-Center To add a new data center in the RHEV-M portal, click on the New Data Center button. Figure 10: New Data Center refarch-feedback@redhat.com 19 www.redhat.
4.2 Add Cluster To add new cluster for the data center in RHEV-M, click on the New Cluster button: Figure 11: New Cluster Figure 12: Data Center www.redhat.com 20 refarch-feedback@redhat.
4.3 Add Host Follow the steps below in order to add a host: Step 1: Go to Hosts tab and click on New. Figure 13: Adding a Host Step 2: Fill in the details as desired. Note: If you intend to use UFM or Mellanox Network Plugin, it is necessary that the host name you give is the same as the host name in UFM. After filling in the required details the installation starts. Figure 14: Installation in Progress refarch-feedback@redhat.com 21 www.redhat.
After finishing the installation, the installer restarts the host. The virtual desktop and server manager daemon should be up and running. Figure 15: Installation Complete Step 3: Add iSER support by applying the iscsi.py.patch. Note: Retrieve the iscsi.py.patch from Mellanox support, ready for RHEV 3.0. For RHEV 3.1 and later skip this section. Step 4: Copy iscsi.py.patch to /tmp, and run: [root@host3]# cd /usr/share/vdsm/storage [root@host3]# patch iscsi.py < /tmp/iscsi.py.
Step 5: Click on Discover. Figure 17: Discovering Targets Step 6: Click on Login located on the right of your chosen target. Figure 18: Login to Target refarch-feedback@redhat.com 23 www.redhat.
Step 7: Choose the LUN to add and Click OK. Figure 19: Choosing an LUN The discovered storage domain will change from a non-operational status to operational. Figure 20: Successfully Adding a Storage Domain www.redhat.com 24 refarch-feedback@redhat.
4.5 Adding Virtual Machines to a Cluster Step 1: Click on System à Default à Clusters à Default à VMs à New Server. Step 2: Fill the details in General tab. Figure 21: Adding New Virtual Machine - General refarch-feedback@redhat.com 25 www.redhat.
Step 3: Select VNC protocol in Console tab. Figure 22: Adding a New Virtual Machine - Console Step 4: In the Host tab, select the host you want the VM to run on. Figure 23: Adding a New Virtual Machine - Host www.redhat.com 26 refarch-feedback@redhat.
Step 5: Go to Boot Options tab and choose Hard Disk as the First Device and PXE as the Second Device then click the OK button at the bottom. Figure 24: Adding a New Virtual Machine – Boot Options Step 6: A wizard will pop up. Choose Configure Network Interface. Figure 25: Adding a New Virtual Machine – Configuration refarch-feedback@redhat.com 27 www.redhat.
Step 7: Complete the details for the new HCA. Figure 26: Adding a New Virtual Machine – Adding a New Network Interface Step 8: Click on Configure Virtual Disks and fill in the details. Figure 27: Adding a New Virtual Machine – Adding a New Virtual Disk www.redhat.com 28 refarch-feedback@redhat.
Step 9: Click Configure Later to finish. Figure 28: Adding a New Virtual Machine – Finishing Configuration Figure 29: Adding a New Virtual Machine – VMs Screen refarch-feedback@redhat.com 29 www.redhat.
Step 10: Right click on the line of the VM and choose Run. Step 11: In order to start the VM console, right-click and select Console. Figure 30: Adding a New Virtual Machine – VNC Screen 4.6 Add a Network to the Cluster Step 1: Go to System à Default. Step 2: Click on Logical Networks and then on New. Figure 31: Logical Networks www.redhat.com 30 refarch-feedback@redhat.
Step 3: Fill in the details for the new Logical Network. Figure 32: Adding a New Logical Network The new logical network is available for use. Figure 33: Added the New Logical Network refarch-feedback@redhat.com 31 www.redhat.
Step 4: Go to each host you want to connect to the new logical network and click Edit on the interface. Step 5: Find which interface is eIPoIB. Run: (config) # cat /sys/class/net/eth_ipoib_interfaces eth5 over IB port: ib0 Step 6: Provide an IP address and save the configuration. Figure 34: Adding a Network Interface to the Logical Network www.redhat.com 32 refarch-feedback@redhat.
The logical network name appears under the column Network Name for this interface. Figure 35: Added the Network Interface to the Logical Network 4.7 Add an Interface to VM Step 1: Go to the VMs pane. Step 2: Click on Network Interface tab. Step 3: Click on New button – a pop-up will open. Figure 36: Virtual Machine – Network Interfaces View refarch-feedback@redhat.com 33 www.redhat.
Step 4: Fill in the details for the HCA. Figure 37: Adding a New Network Interface The newly added network interface appears. Figure 38: Added the New Network Interface www.redhat.com 34 refarch-feedback@redhat.
Step 5: Start the VM. Step 6: –a. Verify that the host has a new network interface for the VM. Run the command ifconfig Figure 39: Verifying the New HCA is Up refarch-feedback@redhat.com 35 www.redhat.
5 Using UFM to Automate Network Management Mellanox's Unified Fabric Manager™ (UFM™) is a powerful platform for managing scale-out computing environments. UFM enables data center operators to efficiently monitor and operate the entire fabric, boost application performance and maximize fabric resource utilization.
5.1 Basic UFM Configuration Flow The following steps show how to create a logical server and UFM Network, and finally connecting between them. Step 1: Create an environment. Figure 40: UFM Environment refarch-feedback@redhat.com 37 www.redhat.
Step 2: Add a logical server. UFM logical server is equivalent to datacenter cluster in the RHEV-M architecture model. Figure 41: New Logical Server Step 3: Add all hosts in the RHEV-M cluster. Figure 42: Add Hosts www.redhat.com 38 refarch-feedback@redhat.
Step 4: Create a new network. Add partition key (PKey) Figure 43: Add Hosts refarch-feedback@redhat.com 39 www.redhat.
Step 5: Connect the logical server (cluster) to the network. By doing this, all hosts located under this logical server (cluster) will be connected. Figure 44: Connect the Logical Sever to the Network Figure 45: UFM Network Connected to the UFM Logical Server Refer to UFM User Manual6 for advanced configuration options. www.redhat.com 40 refarch-feedback@redhat.
6 Mellanox Network Manager Plugin Mellanox Network Manager Plugin performs seamless integration between Mellanox UFM and the RHEV Manager. After installing the plugin, (see section 3.6 Mellanox Network Manager Plugin), the interconnectivity between the hosts in the network over eIPoIB interface is performed seamlessly. For advanced configuration, please contact cloudsupport@mellanox.com.
Appendix A: Troubleshooting A.1.1 Host is not Added to a Logical Server in UFM Verify the server is visible in UFM. If it does not appear there, run: #cat /sys/class/infiniband/mlx4_0/node_desc The output should be something other than localhost HCA-1. You can change it by running: #echo “web1 HCA-1” > /sys/class/infiniband/mlx4_0/node_desc A.1.2 Migration of VM Fails Step 1: Check that libvirtd on the target is listening on TCP port. # netstat -nap |grep libvirtd tcp LISTEN Step 2: 0 0 0.0.0.
A.1.4 Low Latency Performance Tuning The below links provide a tactical tuning overview of Red Hat Enterprise Linux 6 for latency sensitive workloads and describes important tuning parameters and settings that can improve performance for Mellanox adapters. Each setting, along with its potential effect, is described to help in making an informed judgment concerning its relevance to the user's system, the system workload, and the performance goals.
