EMC® Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Fibre Channel over Ethernet Converged Network Adapters (CNAs) for the Linux Environments P/N 300-002-583 REV A20 EMC Corporation Corporate Headquarters: Hopkinton, MA 01748-9103 1-508-435-1000 www.EMC.
Copyright © 2011, 2012 EMC Corporation. All rights reserved. Published: April, 2012 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Contents Preface.............................................................................................................................. 9 Chapter 1 Introduction Purpose of this document................................................................ 14 Host connectivity .............................................................................. 14 Fibre Channel ..............................................................................14 Fibre Channel over Ethernet ...........................
Contents Method Two – Booting from a DOS bootable diskette and using the lp6dutil utility .................................................................. 37 Method Three – Using lputil at the command line (for the Emulex 7.x driver) ............................................................................ 38 Updating the HBA firmware with lputil ................................ 38 Updating the HBA boot BIOS with lputil...............................
Contents Description of the Emulex lpfc driver parameters.................87 Descriptions of the Emulex parameters...................................88 Editing the Emulex driver parameters on hosts with SLES 9, SLES 10, and SLES 11 ...................................................90 Editing the Emulex driver parameters on hosts with RHEL 4 and RHEL 5 ...................................................................91 Editing the Emulex driver parameters on hosts with Asianux 2.0 and Asianux 3.0 .......
Contents Configuring a Symmetrix boot device......................................... 128 Preparing the Symmetrix system ........................................... 128 Preparing the host .................................................................... 128 Configuring the Emulex boot BIOS for SAN boot............... 129 Configuring a VNX series or CLARiiON boot device............... 132 Preparing the VNX series or CLARiiON system ................. 132 Preparing the host ................................
Contents Linux device-naming convention ................................................. 157 Block device file names ............................................................157 Red Hat distributions ...............................................................158 SuSE distributions.....................................................................158 PowerPath examples ................................................................159 Character device file names...................................
Contents 8 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the LiInux Environment
Preface As part of an effort to improve and enhance the performance and capabilities of its product line, EMC from time to time releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all revisions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes.
Preface Related documentation Conventions used in this guide The following documents are available on Powerlink: ◆ EMC Host Connectivity Guide for Linux ◆ EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux Environment ◆ EMC Linux iSCSI Attach Release Notes ◆ The EMC Networked Storage Topology Guide has been divided into several TechBooks and reference manuals and are available through the E-Lab Interoperability Nav
Preface Where to get help Italic Used in all text (including procedures) for: • Full titles of publications referenced in text • Emphasis (for example a new term) • Variables Courier Used for: • System output, such as an error message or script • URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Courier bold Used for: • Specific user input (such as commands) Courier italic Used in procedures for: • Variables on command line • User input variables <> Angle bra
Preface 12 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the Linux Environment
1 Invisible Body Tag Introduction This document describes the procedures for installing an EMC-approved Emulex adapter into a Linux host environment and configuring the host for connection to an EMC storage array over Fibre Channel (FC) or Fibre Channel over Ethernet (FCoE). Review the EMC Support Matrix for the latest information on approved adapters and drivers. ◆ ◆ ◆ ◆ ◆ ◆ Purpose of this document ................................................................. Host connectivity ....................
Introduction Purpose of this document This document is meant to assist in the installation and setup of Emulex Fibre Channel Host Bus Adapters (HBAs) and Fibre Channel Over Ethernet (FCoE) Converged Network Adapters (CNAs) and with the v7.x-series driver on the Linux v2.4.x hosts and the v8.x-series driver on Linux v2.6.x hosts. The focus of this document is to: ◆ Enable the integrated Emulex drivers in the Linux distributions ◆ Set up Linux hosts using the Emulex v7.
Introduction The benefits of FCoE technology become apparent in large data centers: ◆ Where dense, rack-mounted and blade server chassis exist. ◆ Where physical cable topology simplification is a priority. ◆ In virtualization environments, where several physical storage and network links are commonly required.
Introduction Zoning When using Linux hosts in a fabric environment, the zoning must be set up as single-initiator and single-target zoning. A single initiator/target zone has one adapter port and one EMC storage array port. Storage arrays ports can be shared among adapters; however, each adapter port must be in its own zone. Note: Multi-initiator zones are not supported in a Linux fabric environment.
Introduction Useful Linux utilities and functions These utilities and functions can be helpful in performing configuration operations. Table 1 Useful Linux utilities and functions Utility/Function Description fdisk Command used to create and manipulate partition tables. grep Command used to search through a file or files to find specific text. fsck Command used to check and repair a Linux filesystem. mkfs Command used to create a Linux filesystem on a device partition.
Introduction EMC storage array-specific settings Refer to the EMC Host Connectivity Guide for Linux, available at http://Powerlink.EMC.com, for EMC storage array-specific settings.
2 Invisible Body Tag Installation Steps This chapter outlines the prerequisites for first-time installation, offers a summary of the installation steps with links to the appropriate sections, and provides information on installing the adapter. Review the EMC Support Matrix for the latest information on approved adapters and drivers. ◆ ◆ ◆ Prerequisites for first-time installation ........................................... 20 Summary of installation steps.................................................
Installation Steps Prerequisites for first-time installation In order to complete a first-time installation of the Emulex adapter in your server, you will need the following: Operating system ! ◆ “Operating system” on page 20 ◆ “HBAnyware and hbacmd utilities” on page 20 ◆ “BIOS and firmware” on page 20 ◆ “Linux driver” on page 21 Before the Emulex adapter is installed, the Linux operating system must be installed and properly configured.
Installation Steps Follow the links to your adapter for the appropriate version. Linux driver The Linux driver for your HBA or CNA per theEMC Support Matrix for your supported configuration. EMC supports both in-kernel and out-of-kernel drivers. Note: The installation of the in-kernel driver occurs when you install your Linux distribution of choice.
Installation Steps Summary of installation steps Table 2 describes the procedures for installing an EMC-approved Emulex adapters into a Linux host and configuring the host for connection to an EMC Storage Array over Fibre Channel (FC) or Fibre Channel over Ethernet (FCoE). Table 2 Step Instructions For Fibre Channel, refer to For Fibre Channel over Ethernet, refer to 1 Install the HBA .
Installation Steps Table 2 Installation steps (page 2 of 2) Step Instructions For Fibre Channel, refer to For Fibre Channel over Ethernet, refer to 5 Install the HBanywhere applications kit. “Installing the Emulex HBAnywhere applications kit” on page 84 “Installing the Emulex HBAnywhere applications kit” on page 84 6 Install the firmware.
Installation Steps Installing the adapter This section contains the following information needed for installing the adapter: ◆ “Prerequisites” on page 24 ◆ “Installation procedure” on page 24 ◆ “Matching the adapter with the correct PCI slot” on page 26 Prerequisites Before the Emulex HBA is installed, the host must be configured with Linux. Install the Linux kernel (including sources) from the distribution installation CD, following the procedure provided in the distribution installation guide.
Installation Steps 2. With host system power removed, install the adapter card and cables as instructed in the server documentation. The card installs into a single slot. 3. (Optical cable only.) Remove the protective covers on each fiber-optic cable. 4. Plug one end of the cable into the connector on the adapter as shown in the appropriate figure under this step. (The hardware might be rotated 90 degrees clockwise from the orientation shown.
Installation Steps – LC optical cable: 1 2 3 – Cisco SFP+ (Twinax cable) 5. Plug the other end of the cable into a connector on the storage system or a hub/switch port. For FCoE switch connections, do not connect cables to the switch port until the switch has been properly configured. 6. Label each cable to identify the adapter and the storage/switch/hub port to which it connects. 7. Reapply power and allow the system to boot normally.
Installation Steps ◆ PCI-X ◆ PCI-X 2.0 ◆ PCI-Express PCI slots can be 32-bit and 64-bit (denoted by their 124-pin or 188-pin connectors.) These slots have plastic "keys" that prevent certain adapters from fitting into them. These keys work with the cutout notches in the adapter edge connector so only compatible adapters will fit into them. This is done because of the voltage characteristics of the adapter. (For example, inserting a 3.
Installation Steps Figure 3 Adapter edge connectors Note adapter 5, which shows a universal adapter edge connector. Universal adapters are compatible with both 3.3 V and 5 V PCI slots. PCI-X (or PCI Extended) slots increase the speed with which data travels over the bus. PCI-X slots appear identical to a 64-bit PCI slot keyed for 3.3 V. (Refer to number 3 in Figure 2 on page 27 and Figure 3.) PCI-X slots are backwards compatible with 3.3 V PCI adapters and universal adapters.
Installation Steps negotiate with the slot to determine the highest mutually supported number of lanes. However, an adapter requiring x16 lanes will not fit into a smaller slot. Figure 4 PCI Express slots Figure 5 shows x1, x4, and x16 lane slots aligned on a mainboard. You can see how the slots are keyed so that low-lane adapters can fit into larger slots. Figure 5 PCI Express slots aligned Emulex offers adapters for each bus/slot type available.
Installation Steps Slot requirements of EMC-supported Emulex adapters Table 3 Adapter model Protocol PCI spec BUS length Power Slot key LP9002L FC PCI 2.2 64-bit 3.3V Universal LP9002DC FC PCI 2.2 64-bit 3.3V Universal LP982 FC PCI-X 1.0a & PCI 2.2 64-bit 3.3V Universal LP9802 FC PCI-X 1.0a & PCI 2.2 64-bit 3.3V Universal LP9802DC FC PCI-X 1.0a & PCI 2.2 64-bit 3.3V, 5V 3.3V LP1050/LP1050DC FC PCI-X 1.0a & PCI 2.3 64-bit 3.
3 Invisible Body Tag Installing and Configuring the BIOS Settings This chapter provides information on installing and configuring the BIOS settings. ◆ ◆ ◆ ◆ Verifying and updating the Emulex firmware and boot BIOS .......... 32 Method One — Updating the adapter boot BIOS with hbacmd....... 34 Method Two – Booting from a DOS bootable diskette and using the lp6dutil utility ...................................................................
Installing and Configuring the BIOS Settings Verifying and updating the Emulex firmware and boot BIOS Each Emulex Fibre Channel controller has a flash upgradable firmware and boot BIOS. In most instances, it is necessary to keep only the firmware up-to-date, because the BIOS is used only when using connected storage as a boot device. The latest supported firmware can be found on the Emulex website: http://www.emulex.com/ts/docoem/framemc.htm From this site, select the appropriate file for your adapter.
Installing and Configuring the BIOS Settings • If the banner does not display the required version, update the BIOS as described in “Updating the boot BIOS” on page 33. • If the banner displays the required version, continue to the “Installation instructions for the Emulex v8.x-series driver” on page 44.
Installing and Configuring the BIOS Settings Method One — Updating the adapter boot BIOS with hbacmd To update the boot BIOS on the adapters, follow these steps: 1. Ensure that the firmware is updated prior to updating the boot BIOS. 2. Download the appropriate boot BIOS file from the EMC-approved section of the Emulex website and copy it to the appropriate directory, /usr/sbin/lpfc.: Table 4 34 Adapter model and Boot BIOS file (page 1 of 2) Adapter model Boot BIOS file LP982-E lbXXX.
Installing and Configuring the BIOS Settings Table 4 Adapter model and Boot BIOS file (page 2 of 2) Adapter model Boot BIOS file LPe111-E ybXX.X..all (whereXXX. is the version) Example: yb170a3.prg LPe1150-E wbXXX.prg (where XXX is the version) Example: wb170a3.prg LPe11002-E zbXXX.prg (where XXX is the version) Example: zb170a3.prg LP11002-E bbXXX..prg (where XXX. is the version) Example: bb170a3.prg LP11250-E obXXX..prg (where XXX. is the version) Example: ob202a1.prg LPe12000-E ubXXX..
Installing and Configuring the BIOS Settings 3. Execute the command /usr/sbin/hbanyware/hbacmd ListHBA, and record the "Port WWN" information for all adapters within the host. 4. Execute the command /usr/sbin/hbanyware/hbacmd Download , where WWPN is the first Port WWN recorded from Step 2, and firmwarefile is the firmware file name determined in Step 1. The utility will report "Download Complete." when the firmware download has completed successfully. 5.
Installing and Configuring the BIOS Settings Method Two – Booting from a DOS bootable diskette and using the lp6dutil utility This section discusses how to update the firmware and boot BIOS, using the Ip6util utility. Follow these steps to upgrade the flash firmware from a MS-DOS bootable diskette: 1. Boot the computer using a DOS/WIN95/WIN98 bootable diskette. 2. Insert the diskette containing the flash utility (LP6DUTIL.EXE). 3. At the A:\> prompt, type lp6dutil and press ENTER to start the utility. 4.
Installing and Configuring the BIOS Settings Method Three – Using lputil at the command line (for the Emulex 7.x driver) This section discusses how to update the firmware and boot BIOS, using the lputil utility. Updating the HBA firmware with lputil The lputil is dependent upon the existence of the Emulex driver. Therefore, the operating system and the Emulex driver must be installed prior to using the lputil. The lputil is included in the Emulex Application Kit along with the HBA API and HBAnyware.
Installing and Configuring the BIOS Settings HBA model Firmware file LP1050EX-E mfXXX.all (where XXX is the version) Example: mf1.91a5.all LP10000-E tdXXX.all (where XXX is the version) Example: td1.91a5.all LP10000DC-E tdXXX.all (where XXX is the version) Example: td1.91a5.all LP10000EXDC-E tdXXX.all (where XXX is the version) Example: td1.91a5.all LP101-E embXXX.all (where XXX is the version) Example: emb1.91a5.all LPe1150-E wfXXX.all (where XXX is the version) Example: wf2.50a6.
Installing and Configuring the BIOS Settings 2. Start the firmware update utility: cd /usr/sbin/lpfc ./lputil 3. Select Option 3 – Firmware Maintenance. 4. Select the number of the adapter to update. 5. Select Option 1 – Load Firmware Update. 6. Enter the full name of the image file (as shown in the table above; for example, td190a4.all) and press ENTER. 7. Select Option 0 – Return to Main Menu. 8. Repeat steps 3 through 7 for each additional adapter in the system. 9.
Installing and Configuring the BIOS Settings 2. Download the appropriate boot BIOS file from the EMC-approved section of the Emulex website and copy it to the appropriate directory, /usr/sbin/lpfc: HBA model Boot BIOS file LP982-E lbXXX.prg (where XXX is the version) Example: lb170a3.prg LP9002-E cbXXX.prg (where XXX is the version) Example: cb170a3.prg LP9802-E hbXXX.prg (where XXX is the version) Example: hb170a3.prg LP9802DC-E hbXXX.prg (where XXX is the version) Example: hb170a3.
Installing and Configuring the BIOS Settings HBA model Boot BIOS file LP11002-E bbXXX.prg (where XXX is the version) Example: bb170a3.prg LP1005DC Mezzanine for HPQ Blade Servers mpXXX.prg (where XXX is the version) Example: mp501a5.prg LP1005DC Mezzanine for IBM BladeServers mcXXX.prg (where XXX is the version) Example: mc501a3.prg 3. Copy the file for Open Boot support from the CD-ROM or the EMC-approved section of the Emulex website to the appropriate directory, /usr/sbin/lpfc. 4.
4 Invisible Body Tag Installing and Configuring the Linux Host with the Emulex Driver This chapter describes the procedures for installing an EMC-approved Emulex host bus adapter (HBA) into a Linux host. Refer to the EMC Support Matrix for the most up-to-date information on approved adapters and drivers. ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ Installation instructions for the Emulex v8.x-series driver.......... Uninstalling the Emulex v8.x-series driver.................................... Exercising the Emulex v8.
Installing and Configuring the Linux Host with the Emulex Driver Installation instructions for the Emulex v8.x-series driver Using the Emulex adapter with the Linux operating system requires adapter driver software. The driver functions at a layer below the Linux SCSI driver to present Fibre Channel devices to the operating system as if they were standard SCSI devices. Refer to the latest EMC Support Matrix for specific qualified kernel versions and distributions.
Installing and Configuring the Linux Host with the Emulex Driver Downloading the Emulex v8.x-series adapter driver for the v2.6.x kernel Note: This procedure is for both Fibre Channel and FCoE adapters. Ensure that you have the latest qualified driver. Refer to the EMC Support Matrix for the latest driver revision. Follow these steps to download the Emulex driver from the Emulex website: 1. Change to the directory to which you will save the driver. For example: mkdir /home/emulex cd /home/emulex 2.
Installing and Configuring the Linux Host with the Emulex Driver Installing the Emulex CNA v8.x-series driver on RHEL 5.2 and SLES 10 SP2 hosts To install the Emulex FCoE v8.x-series driver on RHEL 5.2 and SLES 10 SP2 hosts using the installer script included in the package. 1. Boot into the qualified and supported kernel onto which the driver will be installed. 2. Obtain the lpfc_2.6_driver_kit-8.2.0.29-1.tar.gz from the Emulex website as instructed in “Downloading the Emulex v8.
Installing and Configuring the Linux Host with the Emulex Driver Removing Emulex DFC driver startup script ... Restoring old Emulex drivers... Configuring /etc/modprobe.conf ... Original saved as /etc/modprobe.elx Original ramdisk image /boot/initrd-2.6.9-5.0.3.ELsmp.img saved as /boot/initrd-2.6.9-5.0.3.ELsmp.img.elx Creating ramdisk ... The ramdisk for this kernel has been updated. Uninstallation complete c. Reboot the host. 6.
Installing and Configuring the Linux Host with the Emulex Driver 3. Uncompress and extract the source files from the tar archive: cd /home/emulex tar xzvf lpfc_2.6_driver_kit-8.0.16.6_x2-1.tar.gz The initial uncompression will create a new directory and will provide you with the following files within the new directory: lpfc_2.6_driver_kit-8.0.16.6_x2-1/ lpfc_2.6_driver_kit-8.0.16.6_x2-1/lpfc_2.6_ioctl_module-2.0.2-1.noarch.rpm lpfc_2.6_driver_kit-8.0.16.6_x2-1/lpfcdriver_2.6-8.0.16.6_x2-1.noarch.
Installing and Configuring the Linux Host with the Emulex Driver The output from running the installer script is as follows: ./lpfc-install Emulex LPFC Driver Installer Version 1.30.1.2 Determining distribution type and kernel version ... Checking for old Emulex driver ... Looking for kernel source package ... Looking for compilers ... Searching for LPFC Driver source RPM from /home/emulex/lpfc_2.6_driver_kit-8.0.16.6_x2-1/. ... Installing package ... Preparing...
Installing and Configuring the Linux Host with the Emulex Driver compiled as a module and configured to load automatically on boot up. Also, a new RAM disk will be created for the system that includes the new driver in it. This section describes the previously mentioned process using the installer script included in the package. 1. Boot into the qualified and supported kernel onto which the driver will be installed. 2. Obtain the lpfc_2.6_driver_kit-8.0.16.6_x2-1.tar.
Installing and Configuring the Linux Host with the Emulex Driver Restoring old Emulex drivers... Configuring /etc/modprobe.conf ... Uninstallation complete. 6. The installer script will install both the lpfc and the lpfcdfc drivers. No options are needed when executing this script. The installer script unpacks the source RPMs included in the package in the /usr/src/lpfc directory and builds the driver using the sources for the currently running kernel. The script then modifies /etc/modprobe.
Installing and Configuring the Linux Host with the Emulex Driver Original ramdisk image /boot/initrd-2.6.9-5.0.3.ELsmp.img saved as /boot/initrd-2.6.9-5.0.3.ELsmp.img.elx Creating ramdisk ... 1 message generated during installation: o The ramdisk for this kernel has been updated. Please reboot this system to complete the installation of the Emulex LPFC driver kit. Note: The system must be rebooted so that the lpfc driver is loaded through the newly created ramdisk.
Installing and Configuring the Linux Host with the Emulex Driver Table 5 OS Supported Emulex driver versions (page 2 of 4) Driver version Supported adapters 1/2 Gb 4 Gb 8 Gb CNA Red Hat RHEL 4 U2 SuSe SLES 9 SP3 8.0.16.17 √ √ Red Hat RHEL 4 U3 Asianux 2.0 SP1 8.0.16.18 √ √ Red Hat RHEL 4 U4, RHEL 4.5, Oracle OEL 4 U4, OEL 4.5 Asianux 2.0 SP2 8.0.16.27 √ √ Red Hat RHEL 4.6 SuSE SLES 9 SP4 Oracle OEL 4.6 8.0.16.34 √ √ Red Hat RHEL 4.7 Oracle OEL 4.7 Asianux 2.
Installing and Configuring the Linux Host with the Emulex Driver Table 5 OS 54 Supported Emulex driver versions (page 3 of 4) Driver version Supported adapters 1/2 Gb 4 Gb 8 Gb √ √ √ √ √ CNA SuSE SLES 10 SP1 (errata kernels equal to or greater than 2.6.16.54-0.2.3) 8.1.10.12-update Red Hat RHEL 5.2 SuSE SLES 10 SP2 Oracle OEL 5.2 Asianux 3.0 SP1 8.2.0.22 RedHat RHEL 5.4 SuSE SLES 10 SP3 8.2.0.48.2p √ √ √ √a RHEL 5.3 OEL 5.3 8.2.0.33.3p √ √ √ √a RHEL 5.4 OEL 5.4 8.2.0.48.2p 8.
Installing and Configuring the Linux Host with the Emulex Driver Table 5 OS Supported Emulex driver versions (page 4 of 4) Driver version Supported adapters 1/2 Gb 4 Gb 8 Gb CNA SuSE SLES 11 SP1 8.3.5.8.1p √ √ √ √b SuSE SLES 11 SP2 8.3.5.48.2p √ √ √ √b RHEL 6.1 8.3.5.30.1p √ √ √ √ RHEL 6.2 8.3.5.45.4p √ √ √ √ a. For models LP21000/LP21002 CNAs only. b. For models LP21000/LP21002 and models OCe10102-FM-E/OCe10102-FX CNAs.
Installing and Configuring the Linux Host with the Emulex Driver Uninstalling the Emulex v8.x-series driver This section guides you through the process of uninstalling the Emulex v8.x-series driver. The installer script contained within the tar archive may be used to remove the Emulex driver from both SLES 9 SP1 and RHEL 4 hosts. The installer script will restore the old Emulex driver. 1. Stop all applications using the Emulex 8.
Installing and Configuring the Linux Host with the Emulex Driver Exercising the Emulex v8.x-series device driver This section describes the following functionality available with the Emulex v8.x-Series driver: ◆ “Loading and unloading the Emulex v8.x-series driver”, next ◆ “Target information exported to the /proc filesystem” on page 58 ◆ “Dynamically modifying the driver and device information exported to the /sys filesystem” on page 58 Loading and unloading the Emulex v8.
Installing and Configuring the Linux Host with the Emulex Driver ◆ Start the Emulex Application daemons: cd /usr/sbin/hbanyware ./start_elxdiscovery ./start_myserver Note: While loading the Emulex driver using modprobe, it is possible to specify driver parameters. However, any parameters set in the /etc/modprobe.conf or /etc/modprobe.conf.local files will override the values specified on the command line.
Installing and Configuring the Linux Host with the Emulex Driver disc_adisc disc_map disc_npr disc_plogi disc_prli disc_reglgn disc_unmap disc_unused issue_lip lpfc_ack0 lpfc_drvr_version lpfc_fcp_bind_method lpfc_fcp_class lpfc_fdmi_on lpfc_link_speed lpfc_log_verbose management_version mbox modeldesc modelname node_name nport_evt_cnt num_discovered_ports option_rom_version programtype scan serialnum sg_tablesize speed state unchecked_isa_dma unique_id This directory contains information regarding the
Installing and Configuring the Linux Host with the Emulex Driver HW Version : Opt ROM Version: FW Version : Vender Spec ID : Number of Ports: Driver Name : Device ID : HBA Type : Operational FW : SLI1 FW : SLI2 FW : IEEE Address : Boot BIOS : Driver Version : 1001206d 1.90A4 (T2D1.90A4) 10DF 1 lpfc FA00 LP10000DC SLI-2 Overlay SLI-1 Overlay 1.90a4 SLI-2 Overlay 1.90a4 00 00 c9 3d e3 3a Disabled 8.0.16.6; HBAAPI(I) v2.1.a, 11-04-04 host:/usr/sbin/hbanyware # .
Installing and Configuring the Linux Host with the Emulex Driver host:/usr/sbin/hbanyware # ./hbacmd allnodeinfo 10:00:00:00:c9:3d:e3:3a All Node Info for 10:00:00:00:c9:3d:e3:3a Node Type : FCP ID : SCSI Bus Number: SCSI Target Num: Node WWN : Port WWN : OS Device Name : READY 744E13 0 0 50:06:04:8A:CB:37:C6:63 50:06:04:8A:CB:37:C6:63 /sys/class/scsi_host/host0/00,0 host:/usr/sbin/hbanyware # .
Installing and Configuring the Linux Host with the Emulex Driver 01: 02: 03: 04: 05: 06: 07: 08: 09: 0a: 0b: 0c: 0d: 0e: 0f: lun-queue-depth scan-down nodev-tmo topology link-speed fcp-class use-adisc ack0 fcp-bind-method cr-delay cr-count fdmi-on discovery-threads max-luns scsi-req-tmo 1 0 0 0 0 2 0 0 1 0 1 0 1 1 0 80 1 ff 6 2 3 1 1 4 3f ff 2 40 8000 ff host:/usr/sbin/hbanyware # .
Installing and Configuring the Linux Host with the Emulex Driver Installation instructions for the Emulex v7.x-Series driver Using the Emulex adapter with the Linux operating system requires HBA driver software. The driver functions at a layer below the Linux SCSI driver to present fibre Channel devices to the operating system as if they were standard SCSI devices. Refer to the latest EMC Support Matrix for specific qualified kernel versions and distributions.
Installing and Configuring the Linux Host with the Emulex Driver alias eth0 tg3 options scsi_mod max_scsi_luns=255 scsi_allow_ghost_devices=1 2. If any modifications are made to the /etc/modules.conf file to include references to the Emulex HBAs, create a new ramdisk to reflect the changes made: cd /boot mkinitrd –v initrd-$1.img $1 where $1 is the currently running v2.4.x kernel version. Example: cd /boot cp initrd-2.4.21-27.ELsmp.img initrd-2.4.21-27.ELsmp.img.orig mkinitrd –v initrd-2.4.21-27.ELsmp.
Installing and Configuring the Linux Host with the Emulex Driver Note: When attaching to a VNX series or CLARiiON storage array, the lpfc_inq_pqb_filter parameter must be enabled in the /etc/modules.conf file. Refer to “CLARiiON “Ghost” LUN reported with the Emulex v7.1.14 and v7.3.2 drivers (PQB filter)” on page 168. Installing the IOCTL Application Helper Module on RHEL 3.0 U4 The Emulex IOCTL Application Helper Module allows Emulex applications to communicate with the driver itself.
Installing and Configuring the Linux Host with the Emulex Driver mptscsih mptbase diskdumplib sd_mod scsi_mod 40944 41888 3808 13744 110312 4 3 0 8 4 [mptscsih] [mptscsih mptbase] [lpfc sg mptscsih sd_mod] 2. Review the files in /proc/scsi/lpfc/ to verify the Emulex driver version. Example: [root@l82bi250 /]# cat /proc/scsi/lpfc/* Emulex LightPulse FC SCSI 7.1.14 Emulex LightPulse LP10000 2 Gigabit PCI Fibre Channel Adapter on PCI bus 09 device 08 irq 25 SerialNum: MM40706045 Firmware Version: 1.
Installing and Configuring the Linux Host with the Emulex Driver 4. Download the IOCTL Application Helper Module from the Emulex website to a directory on the server. Refer to the instructions for downloading the driver as described in the “Downloading the Emulex v7.x-Series HBA driver for the v2.4.x kernel” on page 68. Select the link for the IOCTL Application Helper Module and download it to the system. Note: The name of the package to be downloaded is lpfc_2.4_ioctl_module_kit-1.0.13-1 5.
Installing and Configuring the Linux Host with the Emulex Driver Installing the Emulex v7.x-Series driver Installation methods for the Emulex v7.x-Series driver The Emulex v7.x-series lpfc and lpfcdfc drivers can be installed onto a Linux host using one of the two methods listed below. ◆ Method one – Run the installer script provided in the driver tar archive to compile and install the modular lpfc and lpfcdfc drivers on RHEL 2.1, RHEL 3.0, and SLES 8 systems.
Installing and Configuring the Linux Host with the Emulex Driver 7. Select the appropriate link to download the desired tar archive for your distribution. 8. In addition to the actual driver installation tar archive, the IOCTL Application Helper Module and Emulex Applications Kit tar archives may also be downloaded. The file names for the kits are as follows: lpfc_2.4_ioctl_module_kit-1.0.13-1.tar.gz elxlinuxapps-2.1a24-7.3.2-1-2.
Installing and Configuring the Linux Host with the Emulex Driver The initial uncompression will create a new directory and will provide you with the following files within the new directory: lpfc_2.4_driver_kit-7.1.14-4/ lpfc_2.4_driver_kit-7.1.14-4/elxlpfc lpfc_2.4_driver_kit-7.1.14-4/lpfcdfcdriver-1.0.13-1.noarch.rpm lpfc_2.4_driver_kit-7.1.14-4/lpfc-install lpfc_2.4_driver_kit-7.1.14-4/README Contained within the newly created directory is the lpfc-install script.
Installing and Configuring the Linux Host with the Emulex Driver Looking for kernel source ... Looking for compilers ... Searching for LPFC Driver source RPM from /home/emulex/lpfc_2.4_driver_kit-7.1.14-4/. ... Installing package ... Preparing... ########################################### [100%] 1:lpfcdriver ########################################### [100%] *************** * Source Files are now available under /usr/src/lpfc * RPM installer will now attempt to : * Step 1: Install lpfc.
Installing and Configuring the Linux Host with the Emulex Driver o The ramdisk for this kernel has been updated. Please reboot this system to complete the installation of the Emulex LPFC driver kit. An example of a newly modified /etc/modules.conf is shown below. The last three lines of the file have been newly added by the installer script. [root@l82bi250 /]# more /etc/modules.
Installing and Configuring the Linux Host with the Emulex Driver VNX series or CLARiiON storage system as well as allow the SCSI stack to scan up to 255 devices. The Unisphere/Navisphere Host Agent requires that the disconnected LUN 0 be reported. The DKMS RPM will install the Emulex driver as a module. Note: After the driver installation, the system should be rebooted so that the lpfc driver may be loaded through the newly created ramdisk. 1.
Installing and Configuring the Linux Host with the Emulex Driver The output from the installation of DKMS is as follows: warning: dkms-2.0.4-1.noarch.rpm: V3 DSA signature: NOKEY, key ID 23b66a9d Preparing... ########################################### [100%] 1:dkms ########################################### [100%] 3. Install the Emulex lpfc driver DKMS RPM: rpm -ivh lpfcdkms-7.1.14-2.noarch.rpm The output from the installation is as follows: Preparing...
Installing and Configuring the Linux Host with the Emulex Driver - Installation - Installing to /lib/modules/2.4.21-15.0.4.ELsmp/kernel/drivers/scsi/lpfc/ /etc/modules.conf: added alias reference for 'lpfc' /etc/modules.conf: added 'options scsi_mod scsi_allow_ghost_devices=1 max_scsi_luns=255' depmod.... Saving old initrd as /boot/initrd-2.4.21-15.0.4.ELsmp_old.img Making new initrd as /boot/initrd-2.4.21-15.0.4.ELsmp.img (If next boot fails, revert to the _old initrd image) mkinitrd....
Installing and Configuring the Linux Host with the Emulex Driver Creating symlink /var/lib/dkms/lpfcdfc/1.0.13/source -> /usr/src/lpfcdfc-1.0.13 DKMS: add Completed. Preparing kernel 2.4.21-27.0.1.ELsmp for module build: (This is not compiling a kernel, only just preparing kernel symbols) Storing current .config to be restored when complete Running Red Hat style preparation routine make clean.... using /lib/modules/2.4.21-27.0.1.ELsmp/build/configs/kernel-2.4.21-i686-smp.config make oldconfig.....
Installing and Configuring the Linux Host with the Emulex Driver Use the following to unload the lpfc driver: modprobe –r lpfcdfc modprobe –r lpfc Installing the Emulex applications kit The lpfc driver must be installed first because HBAnyware is dependent upon the existence of the lpfc driver. 1. Create a directory to which the application kit will be saved. mkdir /home/emulex_apps 2. Download the Applications Kit from the Emulex website and save it to the newly created directory.
Installing and Configuring the Linux Host with the Emulex Driver 5. To install the application kit, change to the directory and run the installation script. cd /home/emulex_apps/enterprise_kitfiles ./install The output from running the installer script is as follows: [root@l82bi250 emulex_apps]# ./install Untarring EmlxApps tgz file... Untarring lpfcutil tgz file... Untarring hbaapi libs tgz file... Untarring lputil tgz file...
Installing and Configuring the Linux Host with the Emulex Driver Uninstallation of the Emulex v7.x-Series driver This section describes how to successfully uninstall the Emulex v7.x-series driver from your Linux host. Uninstallation methods for the Emulex v7.x-Series driver The Emulex v7.x-series lpfc and lpfcdfc drivers can be uninstalled from a Linux host one of two methods: ◆ Method one – Using the installer script with the –-uninstall switch, remove the modular lpfc and lpfcdfc drivers.
Installing and Configuring the Linux Host with the Emulex Driver Removing LPFC Driver source RPM ... lpfc: Device or resource busy /usr/src/lpfc/lpfc.conf saved as /usr/src/lpfc/lpfc.conf.backup The Emulex LPFC Driver has been removed from your system. Removing Emulex DFC driver startup script ... Restoring old Emulex drivers... 2. Reboot the host. Note: When using SLES 8 SP3, the uninstallation procedure does not fully remove all of the files.
Installing and Configuring the Linux Host with the Emulex Driver Uninstalling the Emulex v7.x-Series driver, Method two If you have used the DKMS RPMs, then the RPMs should be removed from the system in order to uninstall the drivers. Note: Please note that the DKMS RPM method does not function properly with SLES 8 distributions and may not be used. The driver names may be verified by querying the RPMs. For example: [root@l82bi250 /]# rpm -qa | grep lpfc lpfcdkms-7.1.14-2 lpfcdfcdkms-1.0.13-2 1.
Installing and Configuring the Linux Host with the Emulex Driver -----------------------------Deleting module version: 1.0.13 completely from the DKMS tree. -----------------------------Done. The Emulex DFC module has been removed from your system. 3. With the successful uninstallation of the lpfcdfc driver RPM, the lpfc driver RPM may be removed from the system. To uninstall the lpfc driver, use the following command: rpm -ev lpfcdkms-7.1.
Installing and Configuring the Linux Host with the Emulex Driver ...Restoring lib/modules/2.4.21-27.0.1.ELsmp/kernel/drivers/addon/lpfc_703/lpfc_703.o The Emulex LPFC Driver has been removed from your system. Uninstallation of the Emulex v7.
Installing and Configuring the Linux Host with the Emulex Driver Installing the Emulex HBAnywhere applications kit ! IMPORTANT The lpfc driver must be installed first, because HBAnyware is dependent upon the existence of the lpfc driver. Note: There is a one-to-one relationship between versions of the lpfc driver and the versions of the applications kit. For example, for the 8.0.16.6_x2 lpfc driver the application kit is ElxLinuxApps-2.1a8-8.0.16.4-1.tar.
Installing and Configuring the Linux Host with the Emulex Driver Note: To install HBAnyware, you will need the rpm-build-xxxx package (e.g., rpm-build-4.2.1-4.2) and the gcc-xxx package (e.g., gcc-3.2.3-20). Note: There is a one-to-one relationship between versions of the lpfc driver and the versions of the applications kit. For example, for the 8.0.16.6_x2 lpfc driver the application kit is ElxLinuxApps-2.1a8-8.0.16.4-1.tar.
Installing and Configuring the Linux Host with the Emulex Driver Untarring HBAnyware local jre... HBAnyware installation complete. Cleaning up... 5. The installation utility will prompt for the "mode of operation" for HBAnyware to manage adapters. Select the desired mode of operation: • Option 1 (Local Mode) allows management of only the adapters within the host running HBAnyware.
Installing and Configuring the Linux Host with the Emulex Driver Editing the configuration files This section contains information on editing the configuration files.
Installing and Configuring the Linux Host with the Emulex Driver Table 7 provides a listing of the EMC-recommended settings for the driver. These values will be used as default when installing the driver.
Installing and Configuring the Linux Host with the Emulex Driver Note: For FCoE adapters, the FC link is virtualized within a 10 gigabit ethernet link. As such, modification of this setting will not affect the FC link speed. lpfc_topology Specifies the link topology.
Installing and Configuring the Linux Host with the Emulex Driver What next? Certain customization features are available in the form of driver parameter options. These options are read every time the driver is loaded and included in the newly created RAM disk that includes the Emulex v8.x driver.
Installing and Configuring the Linux Host with the Emulex Driver To create a new RAM disk, use the script lpfc-install with the --createramdisk option as follows: 1. Change to the directory where the installation files are located. cd /home/emulex/lpfc_2.6_driver_kit-8.0.16.6_x2-1/ 2. Run the install script with the --createramdisk option: ./lpfc-install --createramdisk Emulex LPFC Driver Installer Version 1.30.1.2 Determining distribution type and kernel version ... Creating ramdisk .....
Installing and Configuring the Linux Host with the Emulex Driver Note: All options must be contained on a single line. The above example contains all options on a single line; however, because of line wrap, they appear to be on separate lines. EMC recommends creating a new RAM disk after modifying the parameters in /etc/modprobe.conf and rebooting the host in order to use the new RAM disk. To create a new RAM disk, use the script lpfc-install with the --createramdisk option as follows: 1.
Installing and Configuring the Linux Host with the Emulex Driver Editing the Emulex driver parameters on hosts with OEL 4 and OEL 5 Refer to “Editing the Emulex driver parameters on hosts with RHEL 4 and RHEL 5” on page 91.
Installing and Configuring the Linux Host with the Emulex Driver Using the Emulex lun_scan script Contained within the Emulex driver applications kit is a script, called lun_scan, which automates the SCSI scan requests for Emulex adapters. This script is located in the following applications kit directory: enterprise_kitfiles/scripts/lun_scan When the applications kit is installed, the lun_scan script is installed into the /usr/sbin/lpfc directory with the dfc and lputil utilities.
Installing and Configuring the Linux Host with the Emulex Driver 2. If the system contained multiple Emulex HBAs, the command could be run with specific SCSI host numbers. For example, this system has four Emulex HBAs: [root@l82bi183 /]# ls /proc/scsi/lpfc/ 2 3 4 5 To rescan the SCSI bus for LUNs on some of the HBAs, the SCSI host numbers for those HBAs may be specified: [root@l82bi250 /]# lun_scan 2 5 Scanning lpfc2, scsi host number : 2 Scanning lpfc5, scsi host number : 5 3.
Installing and Configuring the Linux Host with the Emulex Driver 96 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the Linux Environment
5 Invisible Body Tag Updating the Adapter and CEE/Menlo Firmware This chapter provides information on updating the adapter and CEE/Menlo firmware for Fibre Channel over Ethernet adapters. ◆ ◆ Updating the adapter firmware with hbacmd..................................... 98 Updating Emulex CEE /Menlo firmware for LP21000-E/ LP21002-E FCoE adapters .....................................................................
Updating the Adapter and CEE/Menlo Firmware Updating the adapter firmware with hbacmd The hbacmd utility is dependent upon the existence of the Emulex driver. Therefore, the operating system and the Emulex driver must be installed prior to using hbacmd. The hbacmd utility is included in the Emulex Application Kit along with the adapter API and HBAnyware.
Updating the Adapter and CEE/Menlo Firmware Table 8 HBA and CNA Fibre Channel firmware (page 2 of 3) Adapter model Firmware file LP10000DC-E tdXXX.all (where XXX is the version) Example: td191a1.all LP10000EXDC-E tdXXX.all (where XXX is the version) Example: td191a1.all LP101-E embXXX.all (where XXX is the version) Example: emb191a1.all LPe111-E ymXXX.all (where XXXis the version) Example: ym250a8.all LPe1150-E wfXXX.all (where XXX is the version) Example: wf250a4.all LPe11002-E zfXXX.
Updating the Adapter and CEE/Menlo Firmware Table 8 HBA and CNA Fibre Channel firmware (page 3 of 3) Adapter model Firmware file LP1005-BC Mezzanine for IBM Blade Servers bfXXX.all (where XXX is the version) Example: bf210a10.all LPe1105-M Mezzanine for Dell Blade Servers zfXXX.all (where XXX is the version) Example: zf280a4.all LPe1105-M4 Mezzanine for zfXXX.all (where XXX is the version) Dell Blade Servers Example: zf280a4.all LPe1205-M Mezzanine for Dell Blade Servers udXXX.
Updating the Adapter and CEE/Menlo Firmware Note: If you are using the adapter to boot from the SAN, update the Boot BIOS immediately after updating the firmware as the firmware overwrites part of the Boot BIOS.
Updating the Adapter and CEE/Menlo Firmware Updating Emulex CEE /Menlo firmware for LP21000-E/LP21002-E FCoE adapters Each Emulex Fibre Channel controller has a flash upgradable firmware and boot BIOS. In most instances, it is necessary to keep only the firmware up-to-date, because the BIOS is applied only when using connected storage as a boot device. The latest supported firmware can be found on the Emulex website: http://www.emulex.com/ts/docoem/framemc.
Updating the Adapter and CEE/Menlo Firmware the firmware file name determined in Step 1. The utility will report "Download Complete" when the firmware download has completed successfully. 4. If the installed adapter is a dual-ported model, then the CEE/Menlo firmware does not need to be downloaded to each adapter port; downloading it to either port results in both ports being updated.
Updating the Adapter and CEE/Menlo Firmware Updating Emulex BIOS/firmware for the OCe10102-FM-E/OCe10102-FX CNA adapters OneCommand Manager is the GUI software that is designed for administration of Emulex storage networking products, including both HBAs and CNAs. It can be used for retrieving card information, tuning of card parameters, and updating of firmware and BIOS. It is the upgraded version of previous HBAnyware GUI software offered by Emulex.
Updating the Adapter and CEE/Menlo Firmware ◆ Figure 7 For HBA or FCoE CNA, the button to invoke a firmware update process is under the Maintenance tab on the right pane of each port interface, as shown in Figure 7 on page 105. Firmware update interface for Emulex FCoE CNA ◆ However, the button for the OCe10102-FM-E/OCe10102-FX CNA iSCSI CNA is under the Firmware tab of the card entity, as shown in Figure 8 on page 106.
Updating the Adapter and CEE/Menlo Firmware Figure 8 Firmware update interface for Emulex OCe10102-FM-E/OCe10102-FX iSCSI CNA Since firmware and BIOS are offered in separate packages for HBAs and CNAs released before the OCe10102-FM-E/OCe10102-FX CNA, users need to update twice by clicking the Update Firmware… button and select appropriate packages for firmware and BIOS (firmware packages have .all as extension, while BIOS packages' extension is .PRG).
Updating the Adapter and CEE/Menlo Firmware To update the OCe10102-FM-E/OCe10102-FX CNA firmware and BIOS, complete the following steps. 1. Invoke the OneCommand Manager GUI client by issuing /usr/sbin/hbanyware/ocmanager in the Linux command prompt. 2. Click on the card entity with model name (such as "OneConnect OCe10100"; refer to Figure 8 on page 106); 3. Select the Firmware tab on the right pane of the interface and click Update Firmware… to invoke the firmware download window. 4.
Updating the Adapter and CEE/Menlo Firmware The updating process is completed once the latest versions are verified, as shown in Figure 10.
6 Invisible Body Tag Connecting to the Storage This chapter provides information on connecting to the storage. ◆ ◆ ◆ Zoning and connection planning in a fabric environment ........ 110 Enabling persistent binding ............................................................ 111 Configuring Emulex OCe10102-FM-E/OCe10102-FX iSCSI CNA for iSCSI connectivity............................................................
Connecting to the Storage Zoning and connection planning in a fabric environment In a fabric environment, the user should plan for the switch topology, target-to-hosts mapping, and the zone. Planning procedure The recommended procedure is as follows: 1. Draw the connectivity among the hosts, switch, and storage array to verify the correct fabric configuration. 2. Configure the zone capability in the switch.
Connecting to the Storage Enabling persistent binding In a fabric environment, EMC recommends, but does not require, enabling persistent binding in the driver. When using persistent binding, EMC strongly recommends the use of binding via WWPN. Without a persistent binding mechanism, the host will be unable maintain persistent logical routing of the communication from a storage device object across the fabric to an EMC storage array volume.
Connecting to the Storage Editing /usr/src/lpfc/lpfc.conf The Emulex v7.1.14 driver uses the lpfc_fcp_bind_method parameter. When using the lpfc_fcp_bind_method parameter, the following values will force a particular type of binding: ◆ A value of 1 forces WWNN binding. ◆ A value of 2 forces WWPN binding. ◆ A value of 3 forces DID binding. ◆ A value of 4 forces the driver to derive the binding from ALPA.
Connecting to the Storage 4. Add the WWPN and SCSI ID information to the lpfc_fcp_bind_WWPN parameter. The SCSI ID consists of the lpfc interface to bind with and the target number for that lpfc interface. For example, lpfc0t1 would reference target 1 on interface lpfc0. Example: char *lpfc_fcp_bind_WWPN[MAX_FC_BINDINGS]={"50060482cadfe242:lpfc0t1","50 060482cadfe243:lpfc0t2", 0}; Note: The entry must be all on one line; the length of the line forces it to wrap to multiples lines in this document. 5.
Connecting to the Storage Original ramdisk image /boot/initrd-2.4.21-27.0.1.ELsmp.img saved as /boot/initrd-2.4.21-27.0.1.ELsmp.img.elx Creating ramdisk .. The ramdisk for this kernel has been updated. • Depending upon your distribution, use one of the following instruction sets when manually creating a new ramdisk. – For Red Hat distributions, use: cd /boot mkinitrd -v initrd-$1.img $1 where $1 is the currently running v2.4.x kernel version. Example: cd /boot cp initrd-2.4.21-15.0.4.ELsmp.img initrd-2.
Connecting to the Storage 3. Set the lpfc_scan_down parameter as follows: The default setting is: lpfc_scan_down=1 Change it to zero: lpfc_scan_down=0 4. Set the lpfc_automap parameter as follows: The default setting is: lpfc_automap=1 Change it to zero: lpfc_automap=0 5. Set the lpfc_fcp_bind_WWPN parameter to the required number of persistent binding entries.
Connecting to the Storage 7. Create a new ramdisk to reflect the changes made. This may be done by using the lpfc-install script or by manually creating a new ramdisk. • When using the lpfc-install script, use the -configramdisk option. For example: cd /home/emulex/lpfc_2.4_driver_kit-7.1.14-4/ ./lpfc-install -configramdisk The output will be similar to the following: Emulex LPFC Driver Installer Version 1.4.1.10 Determining distribution type and kernel version ... Configuring ramdisk...
Connecting to the Storage Configuring Emulex OCe10102-FM-E/OCe10102-FX iSCSI CNA for iSCSI connectivity If OCM is not available for a particular OS, iSCSI connectivity can still be configured for the OCe10102-FM-E/OCe10102-FX iSCSI CNA by using the iscsiadm utility. Iscsiadm creates an interface (iface) for each OCe10102-FM-E/OCe10102-FX iSCSI CNA port and uses such structure to bind sessions and support iSCSI offload. First, ensure the be2iscsi driver is already available as an in-kernel driver.
Connecting to the Storage An IP must be configured on the OCe10102-FM-E/OCe10102-FX port to ensure the host can ping each target port without any issues. If you cannot ping the portal, then you will not be able to bind a session to the portal. The open-iscsi package needs to be installed and started to use the iscsiadm utility. When iscsiadm is run for the first time, it creates a default iface config file under the /etc/iscsi/ifaces directory for each of your OCe10102-FM-E/OCe10102-FX CNA portals.
Connecting to the Storage Table 9 lists a description for each field in the iface config file. Table 9 iface config file fields Setting Description Ifacename Iface configuration name Iface.hwaddress MAC address of the port Ip_address IP address to use for this port Iface.transport_name Name of driver You can update the value of any of the fields above by using iscsiadm -m iface -o update -I to set up the interface you require.
Connecting to the Storage 120 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the Linux Environment
7 Invisible Body Tag Configuring a Boot Device on an EMC Storage Array With the introduction of the Emulex v7.x-series driver, EMC supports booting RHEL 2.1 and 3.0 hosts from an EMC storage array through an EMC-qualified Emulex Fibre Channel HBA. With the introduction of the Emulex v8.
Configuring a Boot Device on an EMC Storage Array Cautions and restrictions booting from an EMC storage array Note the following cautions and restructions when booting from an EMC storage array: ! CAUTION If the Linux host booting from the SAN loses connectivity long enough, the disks will disappear from the system. To prevent further data from being lost in a situation like this, EMC recommends that the error behavior be changed from continue to remount read-only.
Configuring a Boot Device on an EMC Storage Array VNX series- and CLARiiON-specific cautions and restrictions The following are VNX series and CLARiiON-specific restrictions: ◆ Lost connection to VNX series and CLARiiON array (pulled or damaged cable connection). ◆ VNX series service and upgrade procedures. ◆ CLARiiON service and upgrade procedures, such as online CLARiiON FLARE® upgrades and/or configuration changes. ◆ VNX series and CLARiiON SP failures, including failed lasers.
Configuring a Boot Device on an EMC Storage Array Limitations This section contains the following limitation information: ◆ “Common limitations” on page 124 ◆ “Symmetrix-specific limitations” on page 126 ◆ “VNX series- and CLARiiON-specific limitations” on page 126 Common limitations Boot configurations must not deviate from the following limitations established by EMC: ◆ LUN 000 must exist on the EMC storage array in order for the SCSI layer to scan all devices on RHEL 4 (all releases), SLES 9 SP1,
Configuring a Boot Device on an EMC Storage Array ◆ The Symmetrix, VNX series, or CLARiiON device that is to contain the Master Boot Record (MBR) for the host must have a lower logical unit number (LUN) than any other device visible to the host. This device must be mapped as /dev/sda by the Linux operating system for the boot to succeed from the device.
Configuring a Boot Device on an EMC Storage Array local disks in the system, you may be forced (depending on how Linux discovers the drives) to put your boot block onto a local disk. ◆ Some servers present the option of disabling the local drive. If the local drive is disabled, then verify that the boot block is installed on the first EMC storage array device seen by the Linux host.
Configuring a Boot Device on an EMC Storage Array ◆ During the installation procedure, it is recommended that only one LUN be assigned to the Storage Group for ease of use. Once the installation has completed, additional LUNs may be added to the Storage Group. ◆ Booting from the SAN requires the use a Unisphere/Navisphere Management station with the Unisphere/Navisphere Manager or NaviCLI. The station must be separate from the boot server, but networked to the VNX series or CLARiiON system.
Configuring a Boot Device on an EMC Storage Array Configuring a Symmetrix boot device This section describes how to install an EMC-qualified version of Linux onto a Symmetrix system connected to an Intel-based x86 and x86_64 class systems and AMD Opteron-based x86_64 class systems. Preparing the Symmetrix system ◆ It is recommended that Volume Logix be enabled on the Symmetrix system for LUN-masking purposes.
Configuring a Boot Device on an EMC Storage Array ◆ Disable the BIOS on any other HBAs in the system other than the Emulex HBA designated for booting. Configuring the Emulex boot BIOS for SAN boot After the boot BIOS is installed and enabled, it must be configured for use for booting from the SAN. The following procedure describes how to configure the boot BIOS: 1. Connect the Symmetrix Fibre Channel port to the adapter in the lowest-numbered PCI slot in the server.
Configuring a Boot Device on an EMC Storage Array 13. The next screen displays a list of devices available for use as a boot LUN. Initially, the values are set to 0. However, once the information regarding the boot LUN is obtained, it is logged in this field. At the prompt, select the boot entry of 1. This entry will be used to fill in the information regarding the target boot device. 14. On the next screen, select 00 to clear the selected priority slot, or select the device from which the host will boot.
Configuring a Boot Device on an EMC Storage Array ◆ If the OS to be installed is RHEL 3.0 U6, U7, U8, or RHEL 3.9, proceed to “Installing RHEL 3.0 U6 through RHEL 3.9 onto the EMC storage device with the Emulex v7.x-Series driver” on page 138. ◆ If the OS to be installed is SLES 8 SP4 (32-bit only), proceed to “Installing SLES 8 SP4 for the i386 architecture onto the EMC storage device with the Emulex v7.x-Series driver diskette” on page 139.
Configuring a Boot Device on an EMC Storage Array Configuring a VNX series or CLARiiON boot device This section describes how to install an EMC-qualified version of Linux onto an EMC VNX series or CLARiiON storage system connected to an Intel-based x86 and x86_64 class systems and AMD Opteron-based x86_64 class systems. Preparing the VNX series or CLARiiON system ◆ It is recommended that Access Logix be enabled on the VNX series or CLARiiON system for LUN masking purposes.
Configuring a Boot Device on an EMC Storage Array ◆ SCSI hard disks are allowed in SAN boot configurations. However, the BIOS for the disk's SCSI adapters must be disabled. Any SCSI disks attached to the host should be disconnected during the operating system installation. ◆ For servers with SCSI CD-ROM drives, ensure that the BIOS is enabled on the SCSI channel that includes the CD-ROM. Disable the BIOS on any other integrated SCSI channels.
Configuring a Boot Device on an EMC Storage Array 9. Press ESC to return to the Parameters menu. 10. Press ESC twice to return to the list of installed adapters. Note: It is recommended, but not required, that the boot BIOS be disabled on the other adapters in the system. 11. From the adapter list, select 1 for the adapter to be used for boot, and press ENTER. 12. From the Options menu, select Option 1 - Configure Boot Devices, and press ENTER. 13.
Configuring a Boot Device on an EMC Storage Array What next? Select the appropriate section depending upon the Linux operating system to be installed: ◆ If the OS to be installed is any version of RHEL 2.1 or RHEL 3.0 U2 and U3, proceed to “Installing supported RHEL 2.1 and 3.0 releases onto the EMC storage device with the Emulex v7.x-Series driver” on page 136. ◆ If the OS to be installed is 3.0 U4 or U5, proceed to “Installing RHEL 3.0 U4 and U5 onto the EMC storage device with the Emulex v7.
Configuring a Boot Device on an EMC Storage Array Installing an operating system on a Symmetrix, VNX series, or CLARiiON This section includes the following information for installing an operating system on a Symmetrix, VNX series, or CLARiiON: ◆ “Installing supported RHEL 2.1 and 3.0 releases onto the EMC storage device with the Emulex v7.x-Series driver,” next ◆ “Installing RHEL 2.1 and 3.0 onto the EMC storage device with the Emulex v7.x-Series driver diskette” on page 137 ◆ “Installing RHEL 3.
Configuring a Boot Device on an EMC Storage Array contains the necessary kit to build driver diskettes, as well as detailed information on the function and development of driver diskettes. The entire build process must be performed on a Red Hat host that contains the kernel source tree for the target version of RHEL for which the driver diskette is being built. To ensure that the necessary source tree exists, the most straightforward method is to install the kernel source RPMs from the RHEL 2.1 or 3.
Configuring a Boot Device on an EMC Storage Array Although EMC does not make recommendations on the installation selections, it is worthwhile to create a boot diskette for the installation to aid in troubleshooting possible connectivity problems. On completion of the installation, the system will boot into the RHEL OS selected. Installing RHEL 3.0 U4 and U5 onto the EMC storage device with the Emulex v7.x-Series driver The Emulex v7.1.14 driver ships as the default in both RHEL 3.0 U4 (v2.4.21-27.
Configuring a Boot Device on an EMC Storage Array Since the Emulex v7.3.x driver is integrated into the kernel of RHEL 3.0 U6 through RHEL 3.9, there is no need to create a driver diskette image. Before beginning the installation, perform the steps described in the appropriate section for the EMC storage array being used: ◆ “Configuring a Symmetrix boot device” on page 128 ◆ “Configuring a VNX series or CLARiiON boot device” on page 132 Using the CDs for your Red Hat Enterprise Linux 3.
Configuring a Boot Device on an EMC Storage Array Do not remove the SLES 8 SP4 CD from the CD-ROM driver until the installation program prompts for the SLES 8 CD 1. Note: A separate /boot partition should be created during the installation. When the SLES 8 SP4 kernel is upgraded to a newer kernel version, the Emulex driver will revert to the default 1.23a driver that is integrated into the kernel. As a result, the Emulex v7.x-series driver must be reinstalled.
Configuring a Boot Device on an EMC Storage Array Installing supported RHEL 4, RHEL 5, OEL 4, OEL 5, Asianux 2.0, Asianux 3.0, SLES 9, SLES 10, and SLES 11 operating systems onto the boot device with the Emulex v8.x-series driver RHEL 4 or newer updates, RHEL 5, and newer updates, OEL 4 U4 and newer updates, OEL 5 and newer updates, Asianux 2.0 and newer service packs, Asianux 3.
Configuring a Boot Device on an EMC Storage Array Installing RHEL 5.2 and SLES 10 SP2 operating systems onto a boot device using FCoE Adapters EMC supports booting from an array device in FCoE environments with RHEL 5.2 and SLES 10 SP2 operating systems. Because RHEL 5.
Configuring a Boot Device on an EMC Storage Array 5. Insert the disk (described in Step 1) into either the floppy disk drive or the CD drive, depending on the option selected in Step 4. 6. Click OK, then press ENTER. The SCSI driver is loaded automatically. 7. The Disk Driver window displays, prompting for more drivers to install. Click NO, then press ENTER. 8. Insert the current Linux Red Hat product CD #1 in the CD drive (remove the iso-dd-kit CD first if necessary), then press ENTER. 9.
Configuring a Boot Device on an EMC Storage Array The following message displays: Make sure that CD number 1 is in your drive. 8. Put SLES 10 CD 1 in the drive and press OK. 9. Follow the on-screen instructions to complete the installation.
Configuring a Boot Device on an EMC Storage Array Upgrading the Linux kernel After successfully completing the installation and rebooting your host, you may want to upgrade the kernel to newer errata to take advantage of fixes and features incorporated into the kernel errata. EMC recommends installing the kernel packages, rather than upgrading, so that either kernel version may be used for boot. Also, if you have created a boot diskette, an updated diskette should be created after the upgrade.
Configuring a Boot Device on an EMC Storage Array 146 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the Linux Environment
8 Invisible Body Tag Additional Notes This chapter provides additional notes to consider. ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ Ethernet connectivity over the CNA ............................................. Using the Emulex lun-scan script .................................................. Device reconfiguration procedures ............................................... Device reconfiguration: Device numbering ................................. HPQ server-specific note ...................................................
Additional Notes Ethernet connectivity over the CNA The Emulex FCoE CNA delivers lossless 10 Gb/s Enhanced Ethernet support with dynamic allocation of networking and storage bandwidth that may be used for either system ethernet or iSCSI traffic, as well as FCoE. The Linux driver that supports the ethernet and iSCSI traffic for this device is ixgbe. The driver will automatically be installed and loaded by your supported Linux distribution.
Additional Notes Using the Emulex lun-scan script Contained on the EMC approved web page of the Emulex website is a script called lun_scan.sh, which allow you to dynamically (without a reboot or driver unload/reload) discover new targets and LUNs (on new or existing targets) that have come online. This script is located at the following URL: http://www.emulex.com/ts/downloads/linuxfc/linux_tools.
Additional Notes To scan on a single Emulex adapter, run the command plus the desired host number. For example: [root@l82bi250 /]# lun_scan.sh 2 Scanning lpfc2, scsi host number : 2 ◆ If the system contained multiple Emulex adapters, the command could be run with specific SCSI host numbers.
Additional Notes Device reconfiguration procedures The Linux v2.6.x kernel lacks a command built into the kernel that allows for a dynamic SCSI bus reconfiguration such as drvconfig or ioscan. The SCSI bus may be rescanned with a system reboot, or with a removal and reinsertion of the modular driver. In either case, all I/O must be stopped and all other mounted filesystems must be unmounted before rebooting or removing the modular driver. Note: In the Linux v2.6.
Additional Notes • For the Navisphere Host Agent: /etc/init.d/naviagent stop or service naviagent stop 4. If PowerPath is installed and enabled on the host, stop the PowerPath service by typing one of the following commands: /etc/init.d/PowerPath stop or service PowerPath stop 5. If the Emulex Applications Helper Module is installed and enabled on the host, remove the lpfcdfc module from the currently running kernel prior to the lpfc driver.
Additional Notes The unloading of the module can be accomplished with the modprobe (with the -r switch) command or the rmmod command. These commands are used to unload the loadable modules from the running kernel if they are not in use and if other modules are not dependent upon them. As mentioned previously, if the Emulex Application Helper Module is installed and loaded, you must remove it before the standard Emulex lpfc driver.
Additional Notes Device reconfiguration: Device numbering In the Linux v2.6.x kernel, SCSI addresses are not used in the device file names as they are in other types of UNIX (Sun, SGI, HP-UX, and BSD, for example). Block device filenames take the form /dev/sdln, where l is the letter denoting the physical drive and n is the number denoting the partition on that physical drive.
Additional Notes HPQ server-specific note When using Compaq systems, it is highly recommended that the Compaq SmartStart CD be run to configure the Compaq server prior to installing the Linux operating system. The SmartStart CD is shipped by Compaq with their systems, and is a bootable CD that is used to configure Compaq servers. If another operating system is selected other than Linux, there may be problems installing the operating system or using the drivers installed in the kernel.
Additional Notes LUN 000 requirement for the Linux v2.6.x kernel LUN 000 must exist on the EMC storage array in order for the SCSI layer to scan all devices on RHEL 4 (all releases), SLES 9 SP1, and Asianux 2.0. In these distributions, the v2.6.x kernel does not allow scanning past a nonexistent LUN 000. Note: This issue has been resolved with SLES 9 SP2 and RHEL 4 U3. Note: It is recommended that SPC-2 and the SC3 bit be set on the Symmetrix FA ports attached to the host.
Additional Notes Linux device-naming convention The four high-level device drivers in Linux are: sd sg sr st - Direct access (disks) - SCSI generic interface - Data CD-ROMs - Tapes The sg driver is a character-based device while the other three drivers are block-based devices. The sg driver is used primarily for scanners, CD writers, and printers. The sg device files are dynamically mapped to SCSI IDs/LUNs on the SCSI bus starting with the first SCSI controller.
Additional Notes SuSE SLES 7 support a maximum of 128 SCSI devices as opposed to the SuSE SLES 8 distribution, which supports all 256 SCSI devices. The 256 total of SCSI devices includes any non-EMC storage array devices, such as local disks. Red Hat distributions Red Hat Linux distributions can scan LUN numbers from 0 through 255; however, only the first 128 SCSI devices discovered by the system will be mapped to an sd device.
Additional Notes Therefore, if you have allocated LUNs from 0 through 255 to the SLES 8 Linux host, the host can scan and use all 256 logical units. However, just as with Red Hat distributions, if you present logical units over 256 to a Linux host, the host will neither see nor be able to access those devices with an address of over 256. PowerPath examples When EMC PowerPath® is installed, keep in mind that there is a limitation of 128 SCSI devices total per system, not per adapter.
Additional Notes device file names. Not all Linux distributions provide corresponding sg device file names for the sd device file names. Additionally, since the alphabetic sg device file names are deprecated, some distributions, such as SuSE SLES 8 do not provide them at all. For example, on Red Hat releases there are generally 128 sd device file names, 16 sg numeric file names, and 16 sg alphabetic file names.
Additional Notes Device reconfiguration procedures Linux currently lacks an actual kernel command that allows for a dynamic SCSI channel reconfiguration like drvconfig or ioscan. The three methods of reconfiguring devices on a Linux host are: ◆ Rebooting the host ◆ Unloading and reloading the modular Emulex driver ◆ Echoing the /proc filesystem Rebooting the host Rebooting the host allows reliable detection of newly added devices.
Additional Notes • For the Navisphere Host Agent: /etc/init.d/naviagent stop or service naviagent stop 4. If PowerPath is installed and enabled on the host, stop the PowerPath service with one of the following commands: /etc/init.d/PowerPath stop or service PowerPath stop 5. If the Emulex Application Kit has been installed, stop the Discovery Server and Remote Management Server processes with the following commands: /etc/init.d/ElxDiscSrv stop /etc/init.d/ElxRMSrv stop 6.
Additional Notes The unloading of the module can be done with either the modprobe (with the -r switch) or rmmod command. These commands are used to unload the loadable modules from the running kernel if they are not in use and if other modules are not dependent on them. Echoing the /proc filesystem If the /proc filesystem is enabled in the kernel, echoing the /proc can be used to add and/or remove single SCSI devices.
Additional Notes Rescanning the SCSI bus The SCSI bus may be rescanned with a system reboot, or with a removal and reinsertion of the modular driver. In either case, all I/O must be stopped and all other mounted file systems must be unmounted before rebooting or removing the modular driver. If the SCSI bus is not rescanned or the Unisphere/Navisphere Agent service not stopped, attempts to make a new filesystem with mkfs on the newly created partition will fail.
Additional Notes Incorporating EMC Fibre Channel After the EMC storage array has devices assigned with device addresses and the directors have been switched on line, the host can see all the target EMC devices assigned to that host interface. Devices are presented to the host in the same manner as devices accessed through a standard SCSI interface.
Additional Notes 166 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the Linux Environment
A Invisible Body Tag Special Instructions This appendix contains special instructions for the following: ◆ ◆ ◆ ◆ ◆ ◆ CLARiiON “Ghost” LUN reported with the Emulex v7.1.14 and v7.3.2 drivers (PQB filter) .......................................... 168 (CLARiiON only): Required /etc/modules.conf edit for RHEL 2.1, RHEL 3.0, and SLES 8.0 default modular kernels .... 171 (CLARiiON only): Linux Navisphere Agent/CLI limitation with fdisk on CLARiiON ............................................................
Special Instructions CLARiiON “Ghost” LUN reported with the Emulex v7.1.14 and v7.3.2 drivers (PQB filter) EMC supports the Emulex v7.1.14 driver that is integrated into the RHEL 3.0 U4 (v2.4.21-27.EL) and U5 (v2.4.21-32.EL) kernels and the Emulex v7.3.2 driver that is integrated into RHEL: 3.0 U6. When using these quarterly updates and the integrated Emulex driver, parameter value modifications must be made via an edit to /etc/modules.conf, rather than lpfc.conf. When running these RHEL 3.
Special Instructions Host: scsi2 Channel: 00 Id: 00 Vendor: DGC Model: RAID Type: Direct-Access Host: scsi2 Channel: 00 Id: 00 Vendor: DGC Model: RAID Type: Direct-Access Host: scsi2 Channel: 00 Id: 00 Vendor: DGC Model: RAID Type: Direct-Access Host: scsi2 Channel: 00 Id: 01 Vendor: DGC Model: Type: Direct-Access Lun: 02 5 Lun: 03 5 Lun: 04 5 Rev: 0207 ANSI SCSI revision: 04 Rev: 0207 ANSI SCSI revision: 04 Rev: 0207 ANSI SCSI revision: 04 Lun: 00 Rev: 0207 ANSI SCSI revision: 04 Due to the existence
Special Instructions For example: cd /boot cp initrd-2.4.21-27.0.4.ELsmp.img initrd-2.4.21-27.0.4.ELsmp.img.orig mkinitrd –v initrd-2.4.21-27.0.4.ELsmp.img 2.4.21-27.0.4.ELsmp 3. Reboot the host. After the host reboots, the Emulex driver should report all devices accurately.
Special Instructions (CLARiiON only): Required /etc/modules.conf edit for RHEL 2.1, RHEL 3.0, and SLES 8.0 default modular kernels When a Linux host is attached to both SPs in a CLARiiON storage system, the driver will report a disconnected LUN 0 on SPB and a failure to read the capacity of the device. The Unisphere/Navisphere Host Agent requires that disconnected LUN 0 be reported properly.
Special Instructions A kernel compilation is not required when appending the scsi_allow_ghost_devices parameter to /etc/modules.conf. Follow these steps to enable the new parameter in /etc/modules.conf: 1. Change to the appropriate directory and edit the file: vi /etc/modules.conf 2.
Special Instructions • For SuSE distributions use: cd /boot cp initrd-$1 initrd-$1.orig mkinitrd -k vmlinuz-$1 -i initrd-$1 Example: cd /boot cp initrd-2.4.21-266 initrd-2.4.21-266.orig mkinitrd –k vmlinuz-2.4.21-266 –i initrd-2.4.21-266 4. Reboot the host. (CLARiiON only): Required /etc/modules.conf edit for RHEL 2.1, RHEL 3.0, and SLES 8.
Special Instructions (CLARiiON only): Linux Navisphere Agent/CLI limitation with fdisk on CLARiiON There is a known issue with the Linux Naviagent/CLI in which the Naviagent does not close the sd devices before becoming a daemon and, as a result, the devices remain open. This may cause issues when attempting to access devices.
Special Instructions Disabling the Navisphere Agent service After the Naviagent registers the host with the CLARiiON system, the service may be disabled so that it does not start upon reboot. This may be performed manually or by using the curses-based setup tool.
Special Instructions In order to remove the RPM, stop the Naviagent service and then remove the RPM package. Example: /etc/init.d/naviagent stop rpm -ev naviagentcli-6.20.0.2.
Special Instructions (CLARiiON only): Linux Navisphere Naviagent/CLI limitation with VERITAS Volume Manager v3.2 on CLARiiON The Linux Navisphere Agent holds open the devices to be used by VERITAS Volume Manager v3.2 during the initialization and installation of the CLARiiON devices. When this occurs, the message Devices busy or in use is reported. The issue with the Navisphere Agent and VERITAS Volume Manager v3.2 can occur on initialization and installation of the VxVM, as well as on subsequent reboots.
Special Instructions (CLARiiON only): Linux Navisphere Agent requirement for RHEL 2.1 Intel Itanium systems The Navisphere Agent/CLI RPM for Linux is a 32-bit executable. Therefore, for RHEL 2.1 Itanium-based systems, the x86-compat-libs RPM package must be installed in order for the Linux Navisphere Agent/CLI RPM to install properly.
Special Instructions (CLARiiON Only): Linux Navisphere Agent requirement for RHEL 3.0 Intel Itanium systems The Navisphere Agent/CLI RPM is supported on RHEL 3.0 IA64 hosts. However, glibc-2.3.2-95.27 or later must be installed on the system in order for the Navisphere Agent/CLI RPM pkacage to install successfully on RHEL 3.0 IA64. Note that the x86-compat-libs RPM package that was required for RHEL 2.1 [IA64] is not available or necessary for RHEL 3.0 [IA64].
Special Instructions 180 EMC Host Connectivity with Emulex FC HBAs and FCoE CNAs for the Linux Environment
Index A Asianux 1.0 64 Asianux 2.0 hosts, editing Emulex driver parameters on 92 B binding, persistent 111 BIOS, boot 32, 102 boot BIOS 32, 102 device support 15 boot BIOS update 32 boot device, Symmetrix, configuring 128 boot device, VNX/CLARiiON, configuring 132 C cable, HBA, connecting 25 CEE/Menlo firmware 102 CNA 14 CNA, Ethernet connectivity 148 converged network adapter 14 D device drivers 157 E edit /usr/src/lpfc/lpfc.conf 112 Emulex Fibre Channel controller 32, 102 v7.1.
Index utilities 17 LP6DUTIL 37 lp6dutil 37 lpfc_2.6_driver_kit-8.0.16.6_x2-1.tar.gz 50 lpfc-install script 46, 48, 56, 113 M Master Boot Record (MBR) 15 Menlo (CEE) firmware 102 message url http //www.emulex.com/emc/support/index.js p 20, 21, 45 modules.conf 114 P persistent binding 111 S single-initiator/target zone 16 U uninstall, drivers 56 upgradable firmware 32, 102 utilities, Linux 17 V v2.10f driver, Emulex 47 v8.0.16.
