5.6.x HP X9000 Series Network Storage System Installation Guide (TA768-96056, December 2011)

Installing the driver
CAUTION: OFED 1.5.2 overwrites pre-existing /lib/modules vital to X9000 Software and
NFS protocol. The InfiniBand driver exchanges these modules on install as InfiniBand wants RPC
and NFS to go over RDMA instead of IP.
You will need to manually move mv(7) updated kernel module files after the installation of the
OFED stack. To identify the module files location, run modinfo locally on the file serving node
against each of the parenthesized values to note the installed or updated path that you will need
to rename. For example:
modinfo {sunrpc | nfsd | nfs | lockd | nfs_acl | auth_rpcgss | exportfs}
To install the driver, complete the following steps:
1. Install the IB OFFED stack without 32-bit and without ib-bonding. Run the following command:
mlnxofedinstall --without-32bit - -without-ib-bonding
2. Check the affected kernel files.
/lib/modules/2.6.18-194.el5/updates/kernel/net/sunrpc/sunrpc.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfsd/nfsd.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs/nfs.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/lockd/lockd.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs_common/nfs_acl.ko
/lib/modules/2.6.18-194.el5/updates/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/exportfs/exportfs.ko
3. Rename all of the above files to use the following suffix: /path/name.ofed. For example:
mv /lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs/nfs.ko
/lib/modules/2.6.18-194.el5/updates/kernel/fs/nfs/nfs.ko.ofed
4. Clean up the modules with the depmod -a command and reboot the nodes. A reboot is
necessary for the changes to take effect.
"depmod -a" , "reboot"
5. Execute the following commands on each node to ensure that the modules are loaded on
startup:
chkconfig openibd on
service openibd start
6. The Subnet Manager opensmd must be running on at least one file serving node. Run the
command /usr/sbin/sminfo as root to determine whether opensmd is running on the
IB network.
If opensmd is not running, issue the following commands:
chkconfig opensmd on
service opensmd start
7. Verify the status of the HCA.
NOTE: If you are using Host Based SM, by default it is tied to Port1 of the HCA.
Run the following checks:
* ofed_info
* ibstat
* ibclearcounters
* ibdiagnet -lw 4x -ls 10 -r
8. Verify that the link is up and the state is active. If the state is initializing, there is no
subnet manager running on the fabric. See step 6.
Troubleshooting the InfiniBand network
Force connected mode for a file serving node:
Installing the driver 85