HP StorageWorks Scalable File Share Release Notes Version 2.3 Product Version: HP StorageWorks Scalable File Share Version 2.
© Copyright 2005, 2008 Hewlett-Packard Development Company, L.P. Lustre® is a registered trademark of Cluster File Systems, Inc. Linux is a U.S. registered trademark of Linus Torvalds. Quadrics® is a registered trademark of Quadrics, Ltd. Myrinet® and Myricom® are registered trademarks of Myricom, Inc. InfiniBand® is a registered trademark and service mark of the InfiniBand Trade Association. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation.
Contents About these notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 1 New and changed features in HP SFS Version 2.3 1.1 Read the Release Notes for HP SFS Version 2.2-1 ........................................................................ 1-2 1.2 New and changed features in HP SFS server software .................................................................. 1-3 1.2.1 Lustre 1.4.11...................................................
1.8.1.7 Section 7.2.3 iLO settings — iLO browser tasks .................................................................1-19 1.8.1.8 Section 7.10 Step 10: Scanning MAC addresses ..............................................................1-19 1.8.1.9 Appendix F SSSU script examples....................................................................................1-19 1.8.2HP StorageWorks Scalable File Share System User Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-19 1.8.2.1 Section 1.
About these notes This document describes the new and changed features in the HP StorageWorks Scalable File Share (HP SFS) Version 2.3 server and client software release.
Documentation conventions This section lists the documentation conventions used in these notes. Italic type Italic (slanted) type indicates variable values, placeholders, and function argument names. Italic type is also used to emphasize important information. Courier font This font denotes literal items such as command names, file names, routines, directory names, path names, signals, messages, and programming language structures.
1 New and changed features in HP SFS Version 2.3 This chapter describes the new and changed features in the HP StorageWorks Scalable File Share (HP SFS) Version 2.3 product since HP SFS Version 2.2-1. The chapter is organized as follows: • Read the Release Notes for HP SFS Version 2.2-1 (Section 1.1) • New and changed features in HP SFS server software (Section 1.2) • Limitations (Section 1.3) • Known issues and workarounds in HP SFS Version 2.3 (Section 1.4) • Upgrade/downgrade notes (Section 1.
1.1 Read the Release Notes for HP SFS Version 2.2-1 In addition to reading this document, you should also read the HP StorageWorks Scalable File Share Release Notes for HP SFS Version 2.2-1 (dated November 2006). The Release Notes for HP SFS Version 2.2-1 contain information that also applies to the HP SFS Version 2.
1.2 New and changed features in HP SFS server software This section describes all new and changed features in the HP SFS Version 2.3 server software since Version 2.2-1, and is organized as follows: • Lustre 1.4.11 (Section 1.2.1) • NFS reexport on Linux 2.6 (Section 1.2.2) • OFED 1.2 on servers (Section 1.2.3) • OFED 1.2, 1.2.5, 1.3 on clients (Section 1.2.4) • Support for multiple OFED InfiniBand interfaces (Section 1.2.5) • Support for more InfiniBand HCA types (Section 1.2.
1.2.6 Support for more InfiniBand HCA types The following table lists the HCAs that have been tested with SFS V2.3. 1.2.
1.2.14 New shorter syscheck output format syscheck now always merges identical result lines for multiple nodes. This makes the reports much shorter, specially on large systems. Also this helps to point out abnormal results. For the SFS 2.2 syscheck format, use the new "format=2.2" option. 1.2.15 New vsp and remcons scripts These are easy-to-use scripts for accessing a node Virtual Serial Port or Remote Console.
1.3 Limitations This section describes limitations of the HP SFS Version 2.3 software, and is organized as follows: 1.3.1 • OFED1.2 does not support Red Hat 4 update 6 (Section 1.3.1) • OFED1.2.5 does not support SLES10 SP1 i686 SFS clients (Section 1.3.2) • OFED1.3 does not support i686 SFS clients (Section 1.3.3) • Voltaire ibhost stack not supported on SLES10 SP1 (Section 1.3.4) • Performance reduction for mirrored OST (Section 1.3.5) OFED1.
1.4 Known issues and workarounds in HP SFS Version 2.3 This section describes known issues and workarounds in the HP SFS Version 2.3 software, and is organized as follows: 1.4.1 • "bad dst nid" events in the log are harmless (Section 1.4.1) • When both power supplies of a node fail, then its peer node may reboot repeatedly (Section 1.4.2) • NFS clients kernel panic trying to create files larger than physical mem size (Section 1.4.
sfsmgr disable server N force=yes # Disable the dead server (N=1=admin, N=2=mds) sfsmgr show server # Make sure the new server state is recorded # Now return the machine to its normal state service mysqld stop # Stop the system database umount /var/hpls # Unmount the admin LUN service cluster start # Start the cluster service (which indirectly starts the admin service). chkconfig cluster on # Make sure the cluster service restarts again.
If your XC systemimage is standard/unedited, you can simply remove the current systemimage contents and re-create a new golden image using the following commands: Command Purpose /usr/bin/si_lsimage List available images /usr/sbin/si_rmimage base_image Delete the base_image /opt/hptc/sbin/updateimage --gc n0 --init Create a completely new base_image With the systemimage suitably updated, you can now reimage your clients as described in the XC documentation. 1.4.
1.4.9 Server management commands fail when targeting more than 24 servers at a time sfsmgr server management commands, such as boot/shutdown server, scan mac, and so on, may fail on some servers when targeting more than 24 servers at a time. On very large systems, HP strongly recommends to split such commands in groups targeting at most 24 servers. Example: For a 48-node server: sfs> shutdown server 2-24 wait=0 ... Command has finished: atlas[2-24] -- ... sfs> shutdown server 25-48 wait=0 ...
Also the command evlview -t 1 would fail to access the event log.
1.5 Upgrade/downgrade notes This section provides information about upgrading an existing system from an earlier version of the HP SFS software to HP SFS Version 2.3. The section is organized as follows: 1.5.1 • HP SFS server and client software versions (Section 1.5.1) • Upgrade/downgrade process (Section 1.5.2) • Starting file systems after upgrading (Section 1.5.
1.6 HP XC and HP SFS version interoperability Table 1-1 shows the interoperability between HP XC and HP SFS versions. Table 1-1 HP SFS and HP XC interoperability HP XC Version HP SFS Client Version HP SFS Server Version HP XC Version 3.2.1 HP SFS Version 2.3 HP SFS Version 2.3 HP SFS Version 2.2-1 HP SFS Version 2.2-1 HP XC Version 3.2 HP SFS Version 2.3 HP SFS Version 2.3 HP SFS Version 2.2-1 HP SFS Version 2.
1.7 Problems fixed in HP SFS Version 2.3 This section describes problems that were present in earlier versions of the HP SFS software, but have been fixed in HP SFS Version 2.3. Related bug numbers are shown where possible. 1–14 • (8106) A race condition occurred in ksocknal_terminate_conn leading to the following assertion logged in the client system logs: ASSERTION(list_empty(&conn>ksnc_tx_queue)).
• (7759) LustreError: Found inode with zero generation or link An inode generation of zero is perfectly legal and Lustre should not complain about it. • (7756) Client Enabler: When specifying the --prebuilt_ofed option on the command line, the path specified by --ofed_src was not passed through the build_SFS_client.sh script correctly. • (7747) sfsconfig shouldn't need -s flag when the server is present and up. • (7726) sfsconfig was incorrectly converting ldap entries to http.
1.8 Documentation updates and corrections This section provides updates for new features and corrections for errors in the HP SFS Version 2.3 documentation. 1.8.1 HP StorageWorks Scalable File Share System Installation and Upgrade Guide This section contains the following affected sections: • Section 2.4.3.1a Configuring multiple InfiniBand interfaces with OFED (Section 1.8.1.1) • Section 2.4.3.1b Configuring multiple o2ib LNETs per IB interface with OFED (Section 1.8.1.2) • Section 2.4.3.
At interconnect configuration time, the sfsmgr cli will detect all IPoIB interfaces corresponding to a connected port (see the previous section). It also gives an option for creating new alias interfaces based on one of these connected interfaces. For each new alias interface it will perform the following tasks: • Automatically create an alias IPoIB interface name. For example: ib0:0. • Prompt for the required IPoIB address information. • Automatically create one LNET. For example: o2ib2.
• show disk_group is now ls disk_group. See install guide sections: 5.2.4.4 Step 1. The o/p of ls disk_group has also changed. It now looks like: SCHSV01> ls disk_group "\Disk Groups\diskgroup0" \Disk Groups\diskgroup0 information: object objectid .............................: 00010710B40805606CC00000004001000000B802 objectname ...........................: \Disk Groups\diskgroup0 objecttype ...........................: diskgroupfolder objectwwn ............................: objecthexuid ..................
1.8.1.6 Section 6.13 Step 13: Scanning MAC addresses The scan mac command may not scale when applied to more than 20 servers at a time. HP recommends proceeding by increments of 20 servers or less in such cases. 1.8.1.7 Section 7.2.3 iLO settings — iLO browser tasks The same changes apply as for chapter "6.4.3 iLO settings — iLO browser tasks" above. 1.8.1.8 Section 7.10 Step 10: Scanning MAC addresses The scan mac command may not scale when applied to more than 20 servers at a time.
1.8.2.2 Section 5.7.1 Understanding quota tuning 1.8.2.2.1 OST block quota BTUNE and BUNIT values With HP-SFS V2.2-1 the per OST block quota allocation bunit default value is 100 MB, and the btune default value is 50 MB (50 %). With HP-SFS V2.3, the bunit and btune values are function of the number of ll_ost and ll_ost_io processes running on OSS servers.
The following table shows the quota allocation observed for a particular device when testing inserts/deletes with a block unit of 100MB: Inserts: Amount of data on OST device Units allocated to Reason device 0 1 Get initial allocation of 1 blockunit 396 MB 200 MB 2 Filled by > btune(50%) of bunit so this device now acquires 2 units 600 MB 3 Filled by > 1.5 of bunit (396 MB * 1.5 = 594) 1000 MB 4 Filled by > 2.5*blockunit (396 MB *2.
sfs> show network Name Device Role Lnet ---- ------ ---- ---- ib0 ib0 interconnect o2ib0 mp0 n/a console nic1 eth0 management nic2 eth1 interconnect tcp0 To create these alias interfaces, enter sfsmgr commands and menu choices...
Once confirmed and saved, the IB configuration for the above example will be: Client Server Interface ib0 ib0 Type Physical Physical Inet o2ib0 o2ib0 IP address 172.23.184.1 172.23.184.100 NetMask 255.255.255.0 255.255.255.0 Part. key 0xFFFF 0xFFFF Interface ib1 ib0:0 Type Physical Alias Inet o2ib1 o2ib1 IP address 172.23.185.1 172.23.185.100 NetMask 255.255.255.0 255.255.255.0 Part.
1. When two distinct IB fabrics are to be connected to the same SFS system, and if the servers are configured with dual port HCAs or multiple HCAs, one distinct o2ib LNET (o2ib0, o2ib1) will be associated to each IB interface. 2. To increase the per client node bandwidth, client nodes can distribute the SFS traffic across multiple IB interfaces. This requires that multiple LNETs are defined. On the client side, one LNET will be associated to each IB interface.
1.8.2.5.2 • The LNET ordering in the LNET module networks option at the server side does not matter. • A same client node does not necessarily use the same LNET to communicate with all SFS servers Section 7.11.1a.
The LNETS configured on each node would be as follows: Nodes LNET #1 LNET #2 Client node o2ib1(ib1) o2ib0(ib0) SFS node #1 o2ib0(ib0) SFS node #2 o2ib0(ib0) SFS node #3 o2ib0(ib0) SFS node #4 o2ib0(ib0) SFS node #5 o2ib0(ib0) SFS node #6 o2ib0(ib0) o2ib1(ib0:0) o2ib1(ib0:0) o2ib1(ib0:0) Consequently, the client node would use the o2ib1(ib1) LNET to communicate with even ranked server nodes [2,4,6] since o2ib1 appears first in the LNET options on the client side.
1.8.2.8 Section 9.32.1.1 Using the remote console Ignore the initial caution paragraph: (sfs management tools now use the ssh protocol for all communications with the iLO, so all passwords now travel encrypted.) Append to the step 1 Note: But only iLO2 Advanced have the remote console feature. If you have an iLO2 Standard, only the console virtual serial port is available, you cannot use the --remote option.
# hpaducli -f /tmp/adu.txt 1.8.2.13 Section B.10 Multiple IB interfaces bandwidth The graphs in figures B12 and B-13 show how the single client bandwidth evolve when using an additional Infiniband interface. The configuration is: • DDR Infiniband • 64 TB SFS system • 8 OSS servers, 4 SFS20s each (total 32 OSTs) • DL 380 client node • 2 x 2.
1.8.3 HP StorageWorks Scalable File Share Client Installation and User Guide This section contains the following affected sections: 1.8.3.1 • Section 1.3.3.1 Tested client configurations (Section 1.8.3.1) • Section 1.3.3.2 Untested client configurations (Section 1.8.3.2) • Section 3.2.1 Prerequisites for the SFS client enabler (Section 1.8.3.3) • Section 3.2.2 Building an HP SFS client kit using the sample script (Section 1.8.3.4) • Section 3.2.
Architecture Distribution Kernel version Interconnect • i686 RHEL4.0 U5 2.6.9-55 Gigabit Ethernet interconnect RHEL4.0 U5 2.6.9-55 Quadrics interconnect (QsNetII)from Quadrics, Ltdx Version 5.42.1 • x86_64 • ia64 • i686 • x86_64 OFED 1.2 InfiniBand Interconnect OFED 1.2.5 InfiniBand Interconnect OFED 1.3 InfiniBand Interconnect Voltaire InfiniBand Interconnect Version 3.5.5 Gridstack InfiniBand Interconnect Version 4.3.0_11 • i686 RHEL4.0 U5 2.6.
Architecture Distribution Kernel version Interconnect • x86_64 SLES10 SP1 2.6.16.46-0.12 & 2.6.16.46-0.14 OFED 1.2.5 InfiniBand Interconnect OFED 1.3 InfiniBand Interconnect 1.8.3.2 Section 1.3.3.2 Untested client configurations Update to Table 1-2 "Untested client configurations": Architecture Distribution Kernel version Interconnect • ia64 RHEL4.0 U4 2.6.9-42 OFED 1.2 InfiniBand Interconnect OFED 1.2.5 InfiniBand Interconnect Voltaire InfiniBand Interconnect Version 3.5.
Architecture Distribution Kernel version Interconnect • ia64 RHEL4.0 U6 2.6.9-67 OFED 1.2 InfiniBand Interconnect OFED 1.2.5 InfiniBand Interconnect Voltaire InfiniBand Interconnect Version 3.5.5 Gridstack InfiniBand Interconnect Version 4.3.0_11 Quadrics interconnect (QsNetII)from Quadrics, Ltd x Version 5.42.1 • x86_64 1.8.3.3 RHEL4.0 U6 2.6.9-67 Myrinet interconnect (Myrinet XP and Myrinet 2XP) (from Myricom, Inc.) Version 2.1.26 Section 3.2.
RHEL4 Update 5 and Update 6 Uses the following when the previous --no_infiniband build had been started from /build/SFS_client_V2.3 # /mnt/cdrom/client_enabler/build_SFS_client.sh --config auto \ --kernel /build/SFS_client_V2.3/build/linux/ \ --prebuilt_kernel \ --o2ib_src /usr/local/ofed/src/openib/drivers/infiniband \\ --allow_root\ SLES10 SP1 Uses the following as we have to build in /usr/src/packages/BUILD on a SLES system. # /mnt/cdrom/client_enabler/build_SFS_client.
OFED Distribution Software Installation Menu 1) 2) 3) 4) 5) View OFED Installation Guide Install OFED Software Show Installed Software Configure IPoIB Uninstall OFED Software Q) Exit Select Option [1-5]: (press 2) OFED Distribution Software Installation Menu OFED Distribution Software Installation Menu 1) Basic (OFED modules and basic user level libraries) 2) HPC (OFED modules and libraries, MPI and diagnostic tools) 3) All packages (all of Basic, HPC) 4) Customize Q) Exit Select Option [1-4]: (press 3) P
9. The generated Lustre and hpls RPMS will be places under the following directories for SLES and RHEL respectivley. RHEL4 output/RPMS/`uname -i` SLES10 /usr/src/packages/RPMS/`uname -i` See section 3.2.2 for more details on installing these RPMS. 1.8.3.9 New Section 3.2.2.5 Cisco IB build process for SLES10 1. If the system where you are building the kit is an x86_64 or em64t architecture, ensure that the 64-bit and 32-bit GCC (GNU Compiler Collection) build tools are installed on the system.
# ./build_SFS_client.sh --config auto \ --kernel /usr/src/linux/ \ --prebuilt_kernel \ --config ofed \ --ofed /usr/src/openib \ --prebuilt_ofed \ --private_ofed \ --allow_root \ See section 3.2.2 for more information on installing these RPMS. 1.8.3.10 Section 3.2.7 Additional OFED patches There are addition OFED patches provided under the client_enabler/src/common/ofed/ patches/ directory on the HP StorageWorks Scalable File Share Client Software DVD-ROM.
# cp kernel-smp-version_number.rpm /tmp/ # cp lustre-modules-version_number.rpm /tmp/ # cp lustre-version_number.rpm /tmp/ # cp hpls-lustre-client-version_number.rpm /tmp/ # cp gm-version_number.rpm /tmp/ In the case that you are running a relatively old install that has not been fully updated, you may receive the following message when trying to install the kernel rpm: error: Failed dependencies: perl-Bootloader >= 0.4.16 is needed by kernel-smp-version_number.
1.8.3.12 Section 3.3.3 Step 3: Running the sfsconfig command after installing the software The following warning was added: WARNING! The sfsconfig command must be executed on any new host where SFS is installed. Unless nodes are strictly identical it is not safe to duplicate configuration files such as the /etc/modprobe.conf.lustre files from one host to another. The following table describes the configuration settings handled by sfsconfig: Configuration File Setting Description /etc/hosts nnn.nnn.
Configuration File Setting Description /etc/modprobe.conf.lustre install osc /sbin/modprobe lquota ; /sbin/modprobe --ignoreinstall osc Required for quota support, if the server supports it. /etc/modprobe.conf.lustre remove mdc /sbin/modprobe -r - Required for quota support, if the -ignore-remove mdc ; server supports it. /sbin/modprobe -r lquota /etc/sfstab All lines Updates server and filesystem LNET addresses and options, based on the up-to-date server capabilities fetched on the server web site.
1.8.3.14 Section 5.1.1 Supported configurations for NFS servers and client systems Replace Table 5-1 (HP SFS client node qualified for use as NFS server) with the following: Distribution Kernel Version Red Hat Enterprise Linux 4 Update 5 2.6.9-55 Red Hat Enterprise Linux 4 Update 6 2.6.9-67 Remove the following note: NOTE: You cannot configure a system that is running a Version 2.6 kernel as an NFS server.
• If using the fsid capability in an NFS export entry, this fsid must be the same across all of the NFS servers that are exporting the same file system. • HP SFS Version 2.2-1 supports a maximum of four NFS servers. Using more than four servers is not supported at this time. are replaced with the following: • It is mandatory to specify the fsid parameter in the /etc/exports file.
1–42 New and changed features in HP SFS Version 2.
A Firmware versions This appendix provides details of the firmware versions that are required for use with HP SFS Version 2.3. Table A-1 Firmware component versions Component Version number HP StorageWorks Scalable File Share 20 Enclosure 2.02 or later This is the minimum supported version. Contact your HP Customer Support representative to check if a later version is available. On the HP IT Resource Center Web site http://www.itrc.hp.com: 1. Click the patch/firmware database link. 2.
Table A-1 Firmware component versions Component Version number BIOS - System ROM for ProLiant DL380 G4 2006.04.26 9 May 06 Download available at: http://www.hp.com/support/ Component Version number BIOS - System ROM for ProLiant DL360 G3 2005.03.03 26 Mar 05 Download available at: http://www.hp.com/support/ Component Version number BIOS - System ROM for ProLiant DL380 G3 2004.09.15 15 Oct 04 Download available at: http://www.hp.
B Product support information To ensure the best possible warranty or contract support of your HP StorageWorks Scalable File Share product, please contact HP Services using the information provided below.
B–2 Product support information
