HP 3PAR Ubuntu Operating System Implementation Guide Abstract This implementation guide provides the information you need to configure an HP 3PAR StoreServ Storage system with Ubuntu 12.04.x LTS. General information is also provided on the basic steps required to allocate storage on the HP 3PAR StoreServ Storage that can then be accessed by the Ubuntu host.
© Copyright 2014 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Introduction...............................................................................................4 Supported Configurations..........................................................................................................4 HP 3PAR OS Upgrade Considerations.........................................................................................4 Audience................................................................................................................................
1 Introduction This implementation guide provides the information you need to configure an HP 3PAR StoreServ Storage system with the Ubuntu operating system. General information is also provided on the basic steps required to allocate storage on the HP 3PAR StoreServ Storage that can then be accessed by the Ubuntu host.
configuration. However, the appropriate HP documentation must be consulted in conjunction with the Ubuntu host and Host Bus Adapter (HBA) vendor documentation for specific details and procedures. NOTE: This implementation guide is not intended to reproduce or replace any third-party product documentation. For details about devices such as hosts, HBAs, fabric switches, and non-HP 3PAR software management tools, consult the appropriate third-party documentation.
2 Configuring the HP 3PAR StoreServ Storage for iSCSI Configuring Ports for an iSCSI Connection To configure an iSCSI target port on the HP 3PAR StoreServ Storage 7000 or HP 3PAR StoreServ Storage 10000 Storage for connection to an iSCSI Initiator, complete the following steps: 1. A 10 Gb iSCSI connection requires a one-time configuration using the controlport command. Check the current CNA configuration using the showport and showport -i commands.
4. Verify the changed settings by issuing showport -iscsi. # showport -iscsi N:S:P State IPAddr Netmask Gateway TPGT MTU Rate DHCP iSNS_Prim iSNS_Sec iSNS_Port 0:3:1 ready 10.100.0.101 255.0.0.0 0.0.0.0 31 1500 1Gbps 0 0.0.0.0 0.0.0.0 3205 0:3:2 loss_sync 0.0.0.0 0.0.0.0 0.0.0.0 32 1500 n/a 0 0.0.0.0 0.0.0.0 3205 1:3:1 ready 10.101.0.201 255.0.0.0 0.0.0.0 131 1500 1Gbps 0 0.0.0.0 0.0.0.0 3205 1:3:2 loss_sync 0.0.0.0 0.0.0.0 0.0.0.0 132 1500 n/a 0 0.0.0.0 0.0.0.
Ubuntu iscsiadm Utility Usage This section provides examples of a few commands using the Ubuntu iscsiadm utility to set up the iSCSI sessions. See man iscsiadm for more details. • Discover targets using SendTargets iSCSI Discovery: # iscsiadm -m discovery -t sendtargets -p 10.101.4.220 • iSCSI login: # iscsiadm -m node -T iqn.2000-05.com.3pardata:22220002ac00011b -p 10.101.4.220:3260 –l • iSCSI login all: # iscsiadm -m node --loginall=all • iSCSI logout: # iscsiadm -m node -T iqn.2000-05.com.
• Display iSCSI node configuration: # iscsiadm -m node • Show all records in discovery database: # iscsiadm -m discovery • Show discovery record setting: # iscsiadm -m discovery -p 10.0.0.
3205 # createhost -iscsi … # showhost ubuntuhost Id Name Persona ----------WWN/iSCSI_Name----------- Port 24 ubuntuhost Generic iqn.1996-04.de.suse:01:5198e46d85d5 --> iscsiadm -m discovery … > iscsiadm -m node -l # showiscsisession N:S:P --IPAddr--- TPGT TSIH Conns ------------iSCSI_Name-------------------StartTime------0:4:2 10.101.1.24 42 15 1 iqn.1996-04.de.suse:01:5198e46d85d5 2013-09-10 14:54:32 PDT 1:4:2 10.101.1.24 142 15 1 iqn.1996-04.de.suse:01:5198e46d85d5 2013-09-10 14:54:32 PDT 2:4:2 10.101.2.
Setting Up iSCSI for Ubuntu iSCSI timers and session parameters are specified in the /etc/iscsi/iscsid.conf file. You can adjust the iSCSI timers for better SAN performance. The replacement_timeout iSCSI timeout parameter prevents I/O errors from propagating to the application by controlling how long the iSCSI layer should wait for a timed-out path/session to re-establish itself before failing any commands on it. The default replacement_timeout value is 120 seconds.
/etc/rc1.d/S41open-iscsi /etc/rc6.d/S41open-iscsi /etc/rcS.d/S25open-iscsi 6. To prevent one host from using up all the I/O buffers, you will need to throttle the host I/O by reducing the node.session.cmds_max and node.session.queue_depth values. The following values were determined based on the multihost-to-single HP 3PAR StoreServ Storage port testing completed at the HP test labs and may require further tuning for your particular configuration. node.session.cmds_max = 32 node.session.
NOTE: In order for the parameters changes to take effect, re-run the iSCSI service after the change. 9. Enable any other configuration changes such as CHAP authentication. For details, see “Setting the Host CHAP Authentication on the HP 3PAR StoreServ Storage” (page 16). Configuring Ubuntu iSCSI Settings with Device-mapper Multipathing The /etc/multipath.conf file is used by Device-mapper where the multipathing parameters can be set. The default installed /etc/multipath.
Enabling Multipath 1. Use cat /etc/init.d/multipath-tools to show the start/stop run levels for the multipath daemon: # cat /etc/init.d/multipath-tools #! /bin/sh ### BEGIN INIT INFO # Provides: multipath-tools # Required-Start: udev $local_fs$syslog # Required-Stop: udev $local_fs$syslog # Should-Start: iscsi # Should-Stop: iscsi # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: multipath daemon # Description: ### END INIT INFO <...> 2.
2. Check state of the iSCSI service. # /etc/init.d/open-iscsi status * iscsid is running * Current active iSCSI sessions: iscsiadm: No active sessions. 3. Verify that the iSCSI module is loaded.
4. Now examine the iSCSI session and content session of the node information by issuing iscsiadm --mode session. For example: # iscsiadm --mode session tcp: [1] 10.102.2.2:3260,31 iqn.2000-05.com.3pardata:20310002ac0000b1 tcp: [2] 10.103.3.3:3260,131 iqn.2000-05.com.3pardata:21310002ac0000b1 For more information about Ubuntu iscsiadm command usage, see “Ubuntu iscsiadm Utility Usage” (page 8).
NOTE: The OutgoingUsername variable can be set to anything you want, but the OutgoingPassword has to be the same as the host CHAP secret configured on the HP 3PAR StoreServ Storage. 3. Perform discovery and login as described in “Creating the iSCSI Connection in Ubuntu Using the iscsiadm Command” (page 15). If the targets have been discovered previously, you must logout of the iSCSI sessions, delete the node and send target records before performing discovery and login by completing the following steps: a.
2. Set the host CHAP secret. # sethost initchap -f host_secret0 ubuntuhost 3. Set the target CHAP secret. # sethost targetchap -f target_secret0 ubuntuhost 4. Verify the host and target CHAP secret. # showhost -chap Id Name -Initiator_CHAP_Name- -Target_CHAP_Name6 ubuntuhost ubuntuhost S121 5. By default, the target Chap_Name matches the system name as shown in the showsys output, unless the -chapname option of the sethost command was used.
node.session.auth.password = host_secret0 #To set a discovery session CHAP username and password for target(s) # authentication by the initiator, uncomment the following lines: discovery.sendtargets.auth.username_in = S121 discovery.sendtargets.auth.password_in = target_secret0 # To set a CHAP username and password for target(s) # authentication by the initiator, uncomment the following lines: node.session.auth.username_in = S121 node.session.auth.
Target Port Limits and Specifications for iSCSI To avoid overwhelming a target port and ensure continuous I/O operations, observe the following limitations on a target port: • I/O queue depth on each HP 3PAR StoreServ Storage HBA model: QLogic 10G: 2048 (HP 3PAR StoreServ 10000 and HP 3PAR StoreServ 7000 systems only) • The I/O queues are shared among the connected host HBA ports on a first-come, first-served basis.
3 Allocating Storage for Access by the Ubuntu Host Creating Storage on the HP 3PAR StoreServ Storage This section describes the general steps and commands that are required to create the virtual volumes (VV) that can then be exported for discovery by the Ubuntu host. For additional information, see the HP 3PAR Command Line Interface Administrator’s Manual, which is available on the HP SC website: HP Support Center Creating Virtual Volumes Virtual volumes (VVs) are the only data layer visible to hosts.
Creating Thinly-provisioned Virtual Volumes To create thinly-provisioned virtual volumes (TPVVs), see the following documents: • HP 3PAR StoreServ Storage Concepts Guide • HP 3PAR Command Line Interface Administrator’s Manual • HP 3PAR Command Line Interface Reference These documents are available on the HP SC website: HP Support Center Exporting LUNs to the Host This section explains how to export LUNs to the host server as VVs, referred to as virtual LUNs (VLUNs).
3 TESTLUNS.3 ubuntuhost -----------------host 4 TESTLUNS.4 ubuntuhost -----------------host --------------------------------------------------------------------------5 total In the example above, there are five paths to the HP 3PAR StoreServ Storage from each HBA port. NOTE: Exporting VLUNs to an iSCSI host is the same as exporting VLUNs to a host that has a Fibre Channel connection to the HP 3PAR StoreServ Storage. The showvlun output is different for hosts with iSCSI connection.
3. Verify that the device is seen. ls -ltr /dev/dm-* (The link has a new time stamp) 4. Verify that the new WWN of the device appears. # tail /var/log/messages 5. Create a partition table for the device. fdisk /dev/dm- or parted /dev/dm- 6. Trigger udev by issuing the following command: # echo 'add' > /sys/block/dm-/uevent 7. Verify that the new partition is recognized by multipathd. multipath -ll If this method fails, proceed with the manual methods described below.
--color: use coloured prefixes OLD/NEW/DEL --hosts=LIST: Scan only host(s) in LIST --channels=LIST: Scan only channel(s) in LIST --ids=LIST: Scan only target ID(s) in LIST --luns=LIST: Scan only lun(s) in LIST --sync/nosync: Issue a sync / no sync [default: sync if remove] --attachpq3: Tell kernel to attach sg to LUN 0 that reports PQ=3 --reportlun2: Tell kernel to try REPORT_LUN even on SCSI2 devices --largelun: Tell kernel to support LUNs > 7 even on SCSI2 devs --sparselun: Tell kernel to support sparse L
If the device has changed its size, then issue the following command to obtain the new disk size: Method 2 - Adding Single Devices To add LUNs by using the echo scsi add statement, run the following commands: # echo "scsi add-single-device 0 1 2 3" >/proc/scsi/scsi where: • 0 specifies the host • 1 specifies the channel • 2 specifies the ID • 3 specifies the LUN (The SCSI midlayer will re-scan.
. . . Host: scsi0 Channel: 00 Id: 02 Vendor: 3PARdata Model: VV Type: Direct-Access . . . Host: scsi0 Channel: 00 Id: 03 Vendor: 3PARdata Model: VV Type: Direct-Access . . . Host: scsi0 Channel: 00 Id: 00 Vendor: 3PARdata Model: VV Type: Direct-Access . . . Type: Enclosure Host: scsi0 Channel: 00 Id: 01 Vendor: 3PARdata Model: VV . . . Type: Enclosure Host: scsi0 Channel: 00 Id: 02 Vendor: 3PARdata Model: VV Type: Direct-Access . . .
4 Modifying HP 3PAR Devices on the Host Server Creating Device-mapper Devices Complete the following steps to create Device-mapper devices. 1. Run the multipath command to create new Device-mapper nodes under the /dev/mapper directory. # multipath 2. Verify the Device-mapper devices have been created by issuing multipath -11.
May 17 13:18:32 | sda: uid = 360002ac00000000000000020000185e3 (callout) May 17 13:18:32 | sdb: not found in pathvec May 17 13:18:32 | sdb: mask = 0x1f ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/p 360002ac00000000000000020000185e3 7:0:0:0 sda 8:0 1 undef ready 3PARda Displaying Detailed Device-mapper Node Information Use multipath -l for listing of devices and dmsetup to get detailed dm node information. NOTE: With no_path_retry set to a value other than 0 in the /etc/multipath.
State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 0 Event number: 1 Major, minor: 253, 0 Number of targets: 1 UUID: mpath-350002ac001dc0031 # dmsetup table --target multipath 350002ac001dc0031: 0 4194304 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:0 1000 8:16 1000 Partitioning Device-mapper Nodes The following section provides guidelines for partitioning Device-mapper nodes. When partitioning a Device-mapper node, do not use fdisk on the /dev/mapper/XXX nodes.
where 350002ac001b40031p1 is a partition device of whole disk 350002ac001b40031. WARNING! While using fdisk, make sure the correct underlying device is used. Use the multipath command to identify the underlying device. WARNING! All I/O creating the file system and mount points needs to be done using the Device-mapper device nodes /dev/mapper/XXX. Data corruption will occur if any I/O is attempted on /dev/sdX device nodes.
# echo "1" > /sys/class/scsi_host//device// //delete For example: # kpartx -d /dev/mapper/350002ac001b40031 # dmsetup remove 350002ac001b40031 # echo "1" > /sys/class/scsi_host/host0/device/target0\:0\:0/0\:0\:0\:0//delete //delete NOTE: When using the echo command, make sure the devices are removed from each of the host HBA instances.
WARNING! While removing the device, make sure the correct underlying device is used. Use the multipath command to identify the underlying device. CAUTION: For iSCSI devices, do not remove the last iSCSI device in /proc/scsi/scsi without first stopping the iSCSI daemon /etc/init.d/open-iscsi stop, otherwise data corruption can occur and the host will hang. Any change to the /etc/multipath configuration file requires running the multipath command to be effective.
5 Known Issues and Examples This section contains known issues as of the release date of this document. Issues may have been addressed since the release date. Issue Static IP address entered during Ubuntu installation (IP address, gateway, etc.) are not in the /etc/ network/interfaces file.
SUPPORT COMMUNICATION - CUSTOMER ADVISORY Document ID: c03241282 Version: 2 Advisory: (Revision) Red Hat Enterprise Linux 6.2 and SUSE Linux Enterprise Server 11 SP2 - Linux Kernel May Be Tainted on HP ProLiant Servers Configured With an Intel Xeon E5-2600-Series Processor RESOLUTION HP will enable x2apic on select future platforms. The workaround listed below must be used in cases where x2apic support is not available.
# `GRUB_CMDLINE_LINUX'. # update-grub Issue iSCSI sessions cannot login after reboot of host. Workaround Use iscsiadm -m -node -l to log in to all sessions: /etc/rc.local # iscsiadm -m node -l Issue The /tmp directory is cleared after reboot of host. Workaround The parameters to consider are located in /etc/default/rcS. /etc/default/rcS # delete files in /tmp during boot older than x days.
Issue How to set up two Ethernet interfaces for iSCSI. Workaround Choose either option 1 or option 2. Option 1—Use Bonding Example of bonding two Ethernet interfaces: /etc/network/interfaces Modified to ipv4, two iSCSI initiator ports with static IP address on two subnets: # bnx2 - iSCSI auto eth1 iface eth1 inet address netmask network static 10.101.1.79 255.255.255.0 10.101.1.0 # bnx2 - iSCSI auto eth3 iface eth3 inet address netmask network static 10.101.2.79 255.255.255.0 10.101.2.
NOTE: A# indicates an HP 3PAR command, and > indicates an Ubuntu host command. Not all output from commands are shown.
10.101.2.243,3260] (multiple) Login to [target: iqn.2000-05.com.3pardata:21420002ac00011b, successful. Login to [target: iqn.2000-05.com.3pardata:22420002ac00011b, successful. Login to [target: iqn.2000-05.com.3pardata:20420002ac00011b, successful. Login to [target: iqn.2000-05.com.3pardata:23420002ac00011b, successful. # showiscsisession | grep ".24 " N:S:P --IPAddr--- TPGT TSIH Conns -------StartTime------0:4:2 10.101.1.24 42 15 1 14:54:32 PDT 1:4:2 10.101.1.24 142 15 1 14:54:32 PDT 2:4:2 10.101.2.
Issue be2net-4.2.389.0-ubuntu12u04.x86_64.run fails to install driver to /lib/ modules/3.5.0-23. Workaround Manually create the be2net module. be2net-4.2.389.0.tar.gz -> README gcc package required # cd # make Save the original driver: # mv /lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/emulex/benet/be2net.ko /lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/emulex/benet/be2net.ko.orig Replace with new driver: # cp be2net.ko /lib/modules/3.5.
Workaround Turn on verbose error logging, available with qla2xxx and lpfc. 'QUEUE FULL detected' or 'failed: x28' indicates a queue-full condition. ubuntuhost # echo 1 > /sys/module/qla2xxx/parameters/ql2xextended_error_logging ubuntuhost # tail –f /var/log/syslog Jun 21 08:30:05 dl380pg8-24 kernel: [139703.505821] qla2xxx [0000:0a:00.1]-3820:8: QUEUE FULL detected. Jun 21 08:30:05 dl380pg8-24 kernel: [139703.505833] qla2xxx [0000:0a:00.
6 Support and Other Resources Contacting HP For worldwide technical support information, see the HP support website: http://www.hp.
For information about: See: Migrating data from one HP 3PAR storage system to another HP 3PAR-to-3PAR Storage Peer Motion Guide Configuring the Secure Service Custodian server in order to monitor and control HP 3PAR storage systems HP 3PAR Secure Service Custodian Configuration Utility Reference Using the CLI to configure and manage HP 3PAR Remote Copy HP 3PAR Remote Copy Software User’s Guide Updating HP 3PAR operating systems HP 3PAR Upgrade Pre-Planning Guide Identifying storage system components
For information about: See: Planning for HP 3PAR storage system setup Hardware specifications, installation considerations, power requirements, networking options, and cabling information for HP 3PAR storage systems HP 3PAR 7200, 7400, and 7450 storage systems HP 3PAR StoreServ 7000 Storage Site Planning Manual HP 3PAR StoreServ 7450 Storage Site Planning Manual HP 3PAR 10000 storage systems HP 3PAR StoreServ 10000 Storage Physical Planning Manual HP 3PAR StoreServ 10000 Storage Third-Party Rack Physic
Typographic conventions Table 2 Document conventions Convention Element Bold text • Keys that you press • Text you typed into a GUI element, such as a text box • GUI elements that you click or select, such as menu items, buttons, and so on Monospace text • File and directory names • System output • Code • Commands, their arguments, and argument values • Code variables • Command variables Bold monospace text • Commands you enter into a command line interface • System
7 Documentation feedback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback (docsfeedback@hp.com). Include the document title and part number, version number, or the URL when submitting your feedback.
