Q Simplify Fast Fabric Users Guide D000006-000 Rev.
Fast Fabric Users Guide Page ii Q D000006-000 Rev.
Q Fast Fabric Users Guide Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only.
Fast Fabric Users Guide Q © 2008 QLogic Corporation. All rights reserved worldwide. First Published: March, 2007 Printed in U.S.A. QLogic Corporation, 26650 Aliso Viejo Parkway, Aliso Viejo, CA 92656, (800) 662-4471 or (949) 389-6000 Page iv D000006-000 Rev.
Q Fast Fabric Users Guide Section 1 Introduction 1.1 1.2 1.3 1.3.1 1.3.2 Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . License Agreements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contact Information . .
Fast Fabric Users Guide Q 4.1.13 4.1.14 4.1.15 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.2.5 4.2.6 4.2.7 4.2.8 4.2.9 4.2.10 4.2.11 4.3 4.3.1 4.3.2 4.3.3 4.3.4 4.3.5 4.3.6 4.3.7 4.3.8 4.4 4.4.1 4.4.2 4.4.3 4.4.4 4.4.5 Run a command on all hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Copy a file to all hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . View ibtest result files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Q Fast Fabric Users Guide 5.1.8 5.1.9 5.2 5.2.1 5.2.2 5.2.3 5.2.4 5.2.5 5.3 5.3.1 5.3.2 5.3.3 5.3.4 5.3.5 5.4 5.4.1 5.4.2 5.4.3 5.4.4 5.5 5.5.1 5.5.2 5.5.3 5.5.4 5.6 5.6.1 5.6.2 5.6.3 5.6.4 5.6.5 5.6.6 5.6.7 5.6.8 Selection of Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Selection of local Ports (subnets) . . . . . . . . . . . . . . . . . . . . . . . . . . . . Basic Setup and Administration Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . pingall . . . .
Fast Fabric Users Guide Q 6.6 6.7 High Performance Linpack (HPL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pallas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix A Fast Fabric Quick Install Checklist A.1 A.2 A.3 A.4 A.5 A.6 A.7 A.8 Setup The Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing and verifying Firmware on the IB Chassis . . . . . . . . . . . . . . . .
Section 1 Introduction This manual describes installation, configuration and administration task information for the Fast Fabric Toolset. This manual is organized as follows: Section 1 describes the intended audience and technical support. Section 2 describes the Fast Fabric Toolset. Section 3 describes getting started with Fast Fabric. Section 4 describes the Fast Fabric Textual User Interface (TUI) menu. Section 5 describes the Fast Fabric command tools and test tools.
Q 1 – Introduction Technical Support 1.3 Technical Support Customers should contact their authorized maintenance provider for technical support of their QLogic products. QLogic-direct customers may contact QLogic Technical Support; others will be redirected to their authorized maintenance provider. Visit the QLogic support Web site listed in Contact Information for the latest firmware and software updates. 1.3.
Section 2 Fast Fabric Overview 2.1 Feature Overview The Fast Fabric Toolset is designed to both simplify and expedite common InfiniBand (IB) cluster management tasks. Fast Fabric can assist in generic management tasks as well as InfiniBand installation, upgrade, configuration and verification tasks.
Q 2 – Fast Fabric Overview Fast Fabric Architecture ❥ ❥ ❥ ❥ Aids in ongoing fabric status and configuration monitoring ❥ Automated fabric health checks and configuration baseline compare ❥ Automated chassis health checks and configuration baseline compare ❥ Automated SM health checks and configuration baseline compare Provides tools to accelerate common host administration tasks ❥ Executes commands across many hosts ❥ Copies files to and from many hosts ❥ Edits host-specific files across ma
Q 2 – Fast Fabric Overview Fast Fabric is typically installed on one or more IB Management Nodes. The IB Management Node must be connected to the rest of the cluster via both InfiniBand and a management network. The management network may be the primary InfiniBand network (IPoIB) or Ethernet. The management network will be used for Fast Fabric host setup and administration tasks. It may also be used for other aspects of server administration or operation.
2 – Fast Fabric Overview Fast Fabric Architecture Q the tools to operate with minimal user interaction and hence reduce the time to perform operations against many hosts or chassis. After initial installation, Fast Fabric can be configured to use IPoIB instead of the management network. NOTE: 2-4 Any reconfigurations that affect IPoIB or involve installing new IB hosts will not be able to use IPoIB.
Section 3 Getting Started Before using the Fast Fabric toolset, the Site Implementation Engineer must perform the tasks described in the sections which follow. To aid in keeping track of steps performed a checklist is provided (see appendix A). During the setup procedure, the Fast Fabric configuration files which must be edited or created are described throughout the procedure. For more information about the configuration files used by Fast Fabric see appendix B.
Q 3 – Getting Started Set Up the Fabric 3.2 Set Up the Fabric 1. (All) The first step in any installation is to physically install the hardware: ❥ Servers ❥ Core and leaf InfiniBand switches, such as the SilverStorm 9024 and 9000 Multi-Protocol Fabric Directors (9020, 9040, 9080, 9120 and 9240). ❥ Virtual I/O systems, such as the EVIC and FVIC cards for the SilverStorm 9000 Multi-Protocol Fabric Directors Series.
Q 3 – Getting Started ssh is used, no additional manual steps are require at this stage (typically Linux OS installation will enable ssh) Alternatively, if its desired to use rsh during fabric installation and/or operation, the following steps must be performed on each node such that the IB Management Node can login using rsh as user root. a. Each node must be configured such that the IB management node can rsh into it. The IB management node must also be able to rsh into itself.
Q 3 – Getting Started Using Fast Fabric The /etc/hosts file should not have any node-specific data (the following section will step through the task of copying this file to all the nodes). If using DNS: Consult the documentation for the DNS server being used. Make sure to edit the /etc/resolv.conf configuration on the IB Management Node to use the proper DNS server. Consult the Linux OS documentation for more information on configuring /etc/resolv.conf.
Q 3 – Getting Started SilverStorm Technologies Inc. InfiniBand 4.1.1.0.15 Software 1) 2) 3) 4) 5) 6) 7) 8) 9) a) Show Installed Software Reconfigure IP over IB Reconfigure Driver Autostart Update HCA Firmware Generate Supporting Information for Problem Report Host Setup via Fast Fabric Host Admin via Fast Fabric Chassis Admin via Fast Fabric Externally Managed Switch Admin via Fast Fabric Uninstall Software X) Exit In the above menu, items 6-9 represent the Fast Fabric menus.
3 – Getting Started Installing and Verifying Firmware on the SilverStorm IB Chassis Q items have been selected, press P. To unselect all items, press N. Pressing X or ESC will exit this menu and return to the Main Menu. If more than 1 item is selected, the items will be performed in the order shown in the menu. This is the typical order desired during fabric setup. If it's desired to perform items in a different order, select a single item and press P to perform it by itself. Then repeat.
Q 3 – Getting Started a. When Virtual I/O controllers (VIC) are installed in a chassis, each VIC should also be assigned a unique name. 4. (Switch) Configure the administrator password on each SilverStorm Chassis NOTE: Newer versions of SilverStorm chassis firmware permit SSH keys to be configured within the chassis for secure password-less login. In this case it is recommended to configure SSH keys in the chassis at this point such that the IB Management Node can login as admin without a password.
Q 3 – Getting Started Installing and Verifying Firmware on the SilverStorm IB Chassis and FF_CHASSIS_ADMIN_PASSWORD. Consult appendix B for more information about fastfabric.conf. NOTE: Fast Fabric will provide the opportunity to enter the chassis password interactively when needed. Hence it's not necessary to place it within fastfabric.conf. If it is desired to instead keep the QLogic Chassis admin password in fastfabric.conf, its recommended to change the fastfabric.conf permissions to be 0x600 (eg.
Q 3 – Getting Started NOTE: The chassis must be running firmware version 4.0.0.4.3 or later to perform this function. If the chassis is not up to this level, it will need to be manually updated via the chassis GUI. See the SilverStorm 9000 Users Guide for more information. NOTE: Consult the relevant chassis firmware release notes to ensure any prerequisites for the upgrade to the new firmware level have been met prior to performing the upgrade via Fast Fabric.
Q 3 – Getting Started Installing and Verifying Firmware on the IB Switches The steps which follow will require that an SM be operational within the fabric. 3.6 Installing and Verifying Firmware on the IB Switches If the fabric contains SilverStorm 9024FC series externally managed switches, Fast Fabric may be used to aid the installation and configuration of the switches. Prior to using Fast Fabric the following minimal steps need to be performed: 1.
Q 3 – Getting Started review all the settings. Refer to appendix B for more information about fastfabric.conf. When placed in the editor for ibnodes, create the file with a list of the switch node guids and desired switch names, one entry per line. Such as: 0x00066a00d9000138,edge1 0x00066a00d9000139,edge2 NOTE: Do not list Internally managed chassis, such as the SilverStorm 9000 chassis in this file. Those were covered in the previous section.
3 – Getting Started Installing InfiniBand on the Remaining Servers Q If any switch fails to be updated, use the "View ibtest result files" option to review the result files from the update. Refer to the section “Interpreting the ibtest log files” on page 5-68 for more details.
Q 3 – Getting Started SilverStorm Technologies Inc. IB Host Setup Menu (4.1.1.0.
Q 3 – Getting Started Installing InfiniBand on the Remaining Servers When placed in the editor for hosts, create the file with a list of the hosts names (the TCP/IP management network names) except the IB Management node from which Fast Fabric is presently being run, one entry per line. Such as: host1 host2 NOTE: Do not list the IB Management Node itself (i.e., the node where Fast Fabric is currently running).
Q 3 – Getting Started NOTE: If DNS is being used, this step should be skipped. NOTE: Typically, /etc/resolv.conf is setup as part of OS installation for each host. However, if /etc/resolv.conf was not setup on all the hosts during OS installation, the Fast Fabric "Copy a file to all hosts" operation could be done at this time to copy /etc/resolv.conf from the IB Management Node to all the other nodes. 8. (Linux) "Show uname -a for all hosts" will show the OS version on all the hosts.
Q 3 – Getting Started Verifying InfiniBand on the Remaining Servers Each time this is executed a Linux shell command (or sequence of commands separated by semicolons) may be specified to be executed against all selected hosts. NOTE: It is recommended at this time to run the "date" command to verify the the date and time is consistent on all hosts. If needed "Copy a file to all hosts" may be used to copy the appropriate files to all hosts to enable and configure NTP. 3.
Q 3 – Getting Started mgmthost below for example) and include the hosts file previously created, one entry per line. Such as: mgmthost include /etc/sysconfig/iba/hosts For further details about the file format refer to section “Selection of Hosts” on page 5-3. 4. (All) "Verify Hosts via Ethernet ping" will ping each selected host over the management network. If all hosts were found, continue to the next step.
Q 3 – Getting Started Complete Installation of additional IB Management Nodes 7. (Host) "Verify Hosts see each other" will verify that each host can see all the others via queries to the Subnet Administrator and the SA replica on each host has been fully populated. 8. (Host) "Verify Hosts ping via IPoIB" will verify that IPoIB is properly configured and running on all the hosts. This is accomplished via the IB management node pinging each host via IPoIB. 9.
Q 3 – Getting Started Repeat the following steps on each additional IB Management Node: 1. (All) Install the additional Fabric Access Software components using the procedure documented in the Fabric Access Software Users Guide. The IB Management Node must have at least Fast Fabric, the IB Stack and IPoIB installed and configured.
Q 3 – Getting Started Running HPL Create the file with a list of the chassis names (the TCP/IP Ethernet management port names assigned above) or IP addresses (Use of names is recommended). One entry per line. Such as: Chassis1 Chassis2 For further details about the file format refer to the section “Selection of Chassis” on page 5-4. 3. (All) Perform a health check using: all_analysis -e. If any errors are encountered resolve the errors and rerun all_analysis -e until a clean run occurs. 4.
Q 3 – Getting Started At this point the user is ready to move onto full scale HPL runs. Assorted sample HPL.dat files are provided in /opt/iba/src/mpi_apps/hpl-config. These files are a good starting point for most clusters and should get within 10-20% of the optimal performance for the cluster. The problem sizes used assume a cluster with 1GB of physical memory per processor (e.g., for a 2 processor node, 2GB of node memory is assumed).
Q 3 – Getting Started Upgrading IB software IB Management Nodes should also include the MPI Runtime and MPI Development packages, and if the user desires to rebuild MPI itself, the IB Development package and MPI Source packages will also be required. After completing the install, reboot each of the IB Management Nodes to ensure they are running the new IB software. NOTE: Ensure that existing configuration is appropriately upgraded, especially Fast Fabric and IPoIB related configuration files.
Q 3 – Getting Started NOTE: Do not list any of IB Management Nodes (eg. The nodes which have fast fabric installed) NOTE: The file may list the Management Network or IPoIB hostnames for the selected hosts 5. (Host) "Install/Upgrade InfiniServ Software" will upgrade the IB software on all the selected hosts. By default it will look in the current directory for the $FF_PRODUCT..tgz file.
3 – Getting Started Upgrading IB software 3-24 Q D000006-000 Rev A
Section 4 Fast Fabric TUI Menu Fast Fabric is easiest to use from the textual user interface (TUI) menu system. The menu system provides a way to perform all common tasks and presents common options. Additional less common options are available directly via the Command Line Tools documented in the next section. In the sections that follow, the menu system will be discussed. The majority of menu items directly invoke various Fast Fabric command tools.
Q 4 – Fast Fabric TUI Menu QuickSilver Fabric Access Software Users Guide. Selecting items 1-9 will display the given submenu. Pressing X will exit the menu system. Selection of a Fast Fabric menu (6-9) will present a submenu such as below: SilverStorm Technologies Inc. IB Host Setup Menu (4.1.1.0.
Q 4 – Fast Fabric TUI Menu At the top of each Fast Fabric menu, the file listing the components to operate on is shown. For example: Fast Fabric Host List: /etc/sysconfig/iba/hosts On each Fast Fabric menu, item 0 will permit a different file to be selected and will permit the editing of the file (using the editor selected via the EDITOR environment variable). In addition it will also permit review and editing of the fastfabric.conf file. The fastfabric.
Q 4 – Fast Fabric TUI Menu Host Setup via Fast Fabric 0) 1) 2) 3) 4) 5) 6) 7) 8) 9) a) b) c) d) e) Edit Config and Select/Edit Hosts Files Verify Hosts via Ethernet ping Verify rsh/rcp Configured Setup Password-less ssh/scp Copy /etc/hosts to all hosts Show uname -a for all hosts Install/Upgrade InfiniServ Software Configure IPoIB IP Address Build MPI Test Apps and Copy to Hosts Reboot Hosts Refresh ssh Known Hosts Rebuild MPI Library and Tools Run a command on all hosts Copy a file to all hosts View ibt
Q 4 – Fast Fabric TUI Menu NOTE: It is recommended that SSH be used in place of the check_rsh command. 4.1.4 Setup Password-less SSH/SCP (Linux) This will run the setup_ssh -i "" command. This will setup secure password-less SSH such that the IB Management Node can securely login to all the other hosts as root via the management network without requiring a password. Password-less SSH is required by Fast Fabric, MPI test applications and most versions of MPI (including QuickSilver MPI). 4.1.
4 – Fast Fabric TUI Menu Host Setup via Fast Fabric Q If any hosts fail to be updated, use the View ibtest result files option to review the result files from the update. For more details, see “Interpreting the ibtest log files” on page 5-68. 4.1.8 Configure IPoIB IP Address (Host) This will run the ibtest configipoib command to create the ifcfg-ib1 files on each host. The file will be created with a statically assigned IP address.
Q 4 – Fast Fabric TUI Menu 4.1.13 Run a command on all hosts (Linux) This will run the cmdall command. A Linux shell command (or sequence of commands separated by semicolons) may be specified to be executed against all selected hosts. 4.1.14 Copy a file to all hosts (Linux) This will run the scpall command. A file on the local host may be specified to be copied to all selected hosts. 4.1.15 View ibtest result files (All) This permits viewing of the test.log and test.
Q 4 – Fast Fabric TUI Menu Host Admin via Fast Fabric 4.2 Host Admin via Fast Fabric This menu is focused on verifying hosts and the fabric as well as administration of all the hosts. SilverStorm Technologies Inc. IB Host Admin Menu (4.1.1.0.
Q 4 – Fast Fabric TUI Menu 4.2.3 Summary of Fabric Components (All) This will run the fabric_info command to provide a brief summary of the counts of components in the fabric including how many switch chips, hosts, and links are in the fabric. It will also indicate if any 1x links were found (that could indicate a poorly seated or bad cable). Review the results against the expected configuration of the cluster. NOTE: The link count includes some internal links within the switch boxes.
4 – Fast Fabric TUI Menu Host Admin via Fast Fabric Q 4.2.8 Check MPI Performance (Host) This will run the ibtest mpiperf command to do a quick check of PCI and MPI performance. This displays the MPI latency and bandwidth between pairs of hosts (1-2, 3-4, 5-6, etc). The results are also written to the test.res file which may be viewed via the View ibtest result files. Refer to the section “Interpreting the ibtest log files” on page 5-68 for more details.
Q 4 – Fast Fabric TUI Menu 4.3 QLogic IB Chassis Admin via Fast Fabric This menu is focused on administration of QLogic 9000 series internally managed IB chassis. SilverStorm Technologies Inc. IB Chassis Admin Menu (4.1.1.0.
4 – Fast Fabric TUI Menu QLogic IB Chassis Admin via Fast Fabric Q 4.3.3 Update Chassis Firmware (Switch) This will run the ibtest -C update command to permit the chassis firmware version to be verified and updated as needed. NOTE: The chassis must be running firmware version 4.0.0.4.3 or later to perform this function. If the chassis is not up to this level, it will need to be manually updated via the chassis GUI. See the SilverStorm 9000 Users Guide for more information.
Q 4 – Fast Fabric TUI Menu 4.3.5 Reboot Chassis (Switch) This will run the ibtest -C reboot command to reboot all the selected chassis and ensure they go down and come back up (as verified via ping over the management network). 4.3.6 Generate all Chassis Problem Report Information (Switch) This will run the captureall -C command to collect configuration and status information from all chassis and generate a single *.tgz file that can be sent to the Support Representative. 4.3.
Q 4 – Fast Fabric TUI Menu SilverStorm Externally Managed IB Switch Administration via Fast Fabric 4.4 SilverStorm Externally Managed IB Switch Administration via Fast Fabric This menu is focused on administration of SilverStorm 9024FC externally managed switches. SilverStorm Technologies Inc. IB Switch Admin Menu (4.1.1.0.
Q 4 – Fast Fabric TUI Menu NOTE: Consult the relevant switch firmware release notes to ensure any prerequisites for the upgrade to the new firmware level have been met prior to performing the upgrade via Fast Fabric.
4 – Fast Fabric TUI Menu SilverStorm Externally Managed IB Switch Administration via Fast Fabric 4-16 Q D000006-000 Rev A
Section 5 Detailed Descriptions of Command LineTools Some of the commands are only applicable when Linux is being used. They will be marked with (Linux). Similarly some of the commands are only applicable when QuickSlver Linux IB software is being used on the hosts. Those will be marked with (Host). All commands which are applicable only when SilverStorm IB Switches or IB Chassis are being used will be marked with (Switch).
5 – Detailed Descriptions of Command LineTools Common Tool Options Q 5.1.3 -S Prompt for password for admin on chassis. By default Fast Fabric operations against SilverStorm chassis (such as cmdall, captureall, showallports, and ibtest) obtain the chassis admin password from the FF_CHASSIS_ADMIN_PASSWORD environment variable which may be directly exported or part of fastfabric.conf.
Q 5 – Detailed Descriptions of Command LineTools 5.1.6 Selection of Hosts For operations that are performed against a set of hosts, there are multiple ways to specify the hosts on which to operate: 1. Small sets of hosts can be easily specified on the command line via the -h option discussed below. 2. When multiple commands are performed against the same small set of hosts, the environment variable HOSTS can be used to specify a space separated lists of hosts. 3.
Q 5 – Detailed Descriptions of Command LineTools Common Tool Options If a relative path is specified for the -f option or HOSTS_FILE, the current directory will be checked first, followed by /etc/sysconfig/iba/ 5.1.6.1.1 Host List File Format Below is a sample host list file: # this is a comment 192.168.0.
Q 5 – Detailed Descriptions of Command LineTools 1. Small sets of chassis can .be easily specified on the command line via the -H option discussed below 2. When multiple commands will be performed against the same small set of chassis, the environment variable CHASSIS can be used to specify a space separated lists of chassis. 3. For groups of chassis which will be used often, a file may be created listing the chassis.
5 – Detailed Descriptions of Command LineTools Common Tool Options Q 5.1.7.1.1 Chassis List File Format Below is a sample chassis file: # this is a comment 192.168.0.5# chassis IP address edge1 # chassis resolvable TCP/IP name include /etc/sysconfig/iba/corechassis # included file Each line of the chassis list file may specify a single chassis, a comment or another chassis that list file to include. A chassis may be specified by chassis management network IP address or a resolvable TCP/IP name.
Q 5 – Detailed Descriptions of Command LineTools For example: i9k229:0 i9k229:0,1,5 192.168.0.5:0,1,5 NOTE: There must be no spaces within the chassis name and/or slot list. This format is used by cmdall and chassis firmware update. This format may be used anyplace a chassis name or IP address is valid, such as the -H option, the CHASSIS environment variable or chassis list files. The slot number specified is ignored on some operations (such as pingall).
5 – Detailed Descriptions of Command LineTools Common Tool Options Q 4. IBNODES_FILE environment variable 5. /etc/sysconfig/iba/ibnodes file For example if the -N option is used and the IBNODES_FILE environment variable is also exported, the command will operate only on switches specified via the -N option. 5.1.8.1 Switch List Files The -L option or the IBNODES_FILE environment variable may be used to provide the name of a file containing the list of SilverStorm IB switches on which to operate.
Q 5 – Detailed Descriptions of Command LineTools It is recommended that a unique node description be specified for each switch. This name should follow typical naming rules and use the characters a-z, A-Z, 0-9, and underscore. No spaces are allowed in the node description. Additionally, names should not start with a digit. For 9024FC switches, the node GUID can be found on a label on the bottom of the switch.
Q 5 – Detailed Descriptions of Command LineTools Common Tool Options 1. -p option 2. PORTS environment variable 3. -t option 4. PORTS_FILE environment variable 5. /etc/sysconfig/iba/ports file 6. default of the first active port on system (0 :0 port specification) For example, if the -p option is used and the PORTS_FILE environment variable is also exported, the command will operate only on ports specified via the -p option. 5.1.9.
Q 5 – Detailed Descriptions of Command LineTools Ports are specified as hca:port. No spaces are permitted. The first HCA is 1 and the first Port is 1. The value 0 for HCA or Port has special meaning. The allowed formats are: 0:0 = 1st active port in system 0:y = port y within system x:0 = 1st active port on HCA x x:y = HCA x, port y Files to be included may be specified via an include directive followed by a file name. File names specified should generally be absolute pathnames.
5 – Detailed Descriptions of Command LineTools Basic Setup and Administration Tools Q Example: pingall pingall -h 'arwen elrond' HOSTS='arwen elrond' pingall pingall -C pingall -C -H 'chassis1 chassis2' CHASSIS='chassis1 chassis2' pingall -C Environment Variables: The following environment variables are also used by this command: HOSTS, HOSTS_FILE - see discussion on selection of hosts above CHASSIS, CHASSIS_FILE - see discussion on selection of chassis above FF_MAX_PARALLEL - when -p option is used maxi
Q 5 – Detailed Descriptions of Command LineTools Environment Variables The following environment variables are also used by this command: HOSTS, HOSTS_FILE - see discussion on selection of hosts above 5.2.3 setup_ssh (Linux): creates ssh keys and configures them on all hosts so the system can ssh and scp into all other hosts without a password prompt. Typically, during cluster setup this tool is used to enable the root user on the IB Management node to login to the other hosts via password-less ssh.
5 – Detailed Descriptions of Command LineTools Basic Setup and Administration Tools Q Fast Fabric provides additional flexibility in the translation between IPoIB and management network hostnames. Refer to appendix C for more information. Setup_ssh provides an easy way to create ssh keys and distribute them to the hosts in the cluster. Many of the Fast Fabric tools (as well as many versions of MPI) require ssh be set up for password-less operation. Therefore, setup_ssh is an important setup step.
Q 5 – Detailed Descriptions of Command LineTools If hosts have IP addresses added (for example by installing IB software and enabling IPoIB), IP addresses changes, MAC addresses changed or other aspects have changed (such as server OS reinstallation), the local hosts ssh known_hosts file can be refreshed by running setup_ssh with the -C option. This option will not transfer the keys, but rather will connect to each host (management network and IPoIB) in order to refresh the ssh keys.
Q 5 – Detailed Descriptions of Command LineTools Basic Setup and Administration Tools Host Examples: cmdall date cmdall 'uname -a' cmdall -h 'elrond arwen' date HOSTS='elrond arwen' cmdall date Chassis Examples: cmdall -C 'ismPortStats' cmdall -C -H 'chassis1 chassis2' ismPortStats CHASSIS='chassis1 chassis2' cmdall ismPortStats Environment Variables The following environment variables are also used by this command: HOSTS, HOSTS_FILE - see discussion on selection of hosts above CHASSIS, CHASSIS_FILE - se
Q 5 – Detailed Descriptions of Command LineTools For operations against chassis use of the -S option is recommended. This avoids the need to keep the password in configuration files. 5.2.
Q 5 – Detailed Descriptions of Command LineTools Basic Setup and Administration Tools The above example creates a hostcapture directory in ./uploads// for each host in /etc/sysconfig/iba/hosts then creates hostcapture.all.tgz. captureall mycapture The above example creates a mycapture directory in ./uploads// for each host in /etc/sysconfig/iba/hosts then creates mycapture.all.tgz.
Q 5 – Detailed Descriptions of Command LineTools For operations against chassis use of the -S option is recommended. This avoids the need to keep the password in configuration files. NOTE: The resulting host capture files can require significant amounts of space on the Fast Fabric host. Actual size will vary, but sizes can be multiple megabytes per host. As such it is recommended to ensure adequate space is available on the Fast Fabric system.
Q 5 – Detailed Descriptions of Command LineTools File Management Tools -f hostfile - file with hosts in cluster, default is /etc/sysconfig/iba/hosts. -u user - user to perform copy to, default is current user code source_file: the name of files to copy from this system, relative to the current directory. Multiple files may be listed. source_dir: the name of directory to copy from this system, relative to the current directory.
Q 5 – Detailed Descriptions of Command LineTools 5.3.2 uploadall (Linux): Copies one or more files from a group of hosts to this system. Since the file name will be the same on each host, a separate directory on this system is created for each host and the file is copied to it. This is a convenient way to upload log files or configuration files for review.
5 – Detailed Descriptions of Command LineTools File Management Tools Q Example: # upload two files from 2 hosts uploadall -h 'arwen elrond' capture.tgz /etc/init.d/ipoib.cfg . # upload two files from all hosts uploadall capture.tgz /etc/init.d/ipoib.cfg . # upload network config files from all hosts uploadall -r -p /etc/sysconfig/network-scripts network-scripts # upload two files to a specific subdirectory of upload_dir uploadall capture.tgz /etc/init.d/ipoib.
Q 5 – Detailed Descriptions of Command LineTools -f hostfile - file with hosts in cluster. The default is /etc/sysconfig/iba/hosts. -h hosts - the list of hosts to download files to -u user - the user to perform the copy. The default is current user code -d download_dir - the directory to download files to. The default is ./downloads. If not specified, the environment variable DOWNLOADS_DIR will be used. If that is not exported the default (./downloads) will be used.
5 – Detailed Descriptions of Command LineTools File Management Tools Q 5.3.4 Simplified Editing of Node-Specific Files (Linux): The combination of uploadall and downloadall provide a powerful yet simple to use mechanism for reviewing and/or editing node-specific files without the need to login to each node. This is best explained with an example. Assume the file /etc./sysconfig/network-scripts/ifcfg-ib1 needs to be reviewed and possibly edited for each host.
Q 5 – Detailed Descriptions of Command LineTools 5.4 Fabric Analysis Tools 5.4.1 Fabric_info Fabric_info provides a brief summary of the components in the fabric. Fabric_info uses the first active IB port on the given local host to perform its analysis. Example output: Fabric_info Fabric_info has no options and uses no environment variables.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Number of 1x Ports - number of ports in the fabric running at 1x speed. Typically such ports represent a bad cable connection, a bad cable, too long a cable or perhaps faulty hardware on one side of the link. Fabric_info can be very useful as a quick assessment of the fabric state.
Q 5 – Detailed Descriptions of Command LineTools HOSTS - a list of hosts, used if the -h option is not supplied CHASSIS - a list of chassis, used if the -C is used and the -h option is not supplied HOSTS_FILE - a file containing the list of hosts, used in absence of -f and -h CHASSIS_FILE - a file containing the list of chassis, used in absence of -F and -H Example: showallports showallports -h 'elrond arwen' HOSTS-'elrond arwen' showallports showallports -C showallports -H 'chassis1 chassis2' CHASSIS-'ch
5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Q For operations against chassis use of the -S option is recommended. This avoids the need to keep the password in configuration files. When performing showallports against externally-managed switches it requires an IB-enabled management node with Fast Fabric installed. Typically this will be the Fast Fabric node from which showallports is being run.
Q 5 – Detailed Descriptions of Command LineTools output in the same order even if components have been rebooted. This is useful for comparison using simple tools like diff. iba_report permits multiple reports to be requested for a single run (i.e., 1 of each report type). By default iba_report uses the first active port on the local system. However, if the IB management node is connected to more than one fabric (e.g., a subnet), the HCA and port may be specified to select the fabric to analyze.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools -D/--dest point - destination for trace route -Q/--quietfocus - do not include focus description in report Report Types: comps - summary of all systems and SMs in fabric brcomps - brief summary of all systems and SMs in fabric nodes - summary of all node types and SMs in fabric brnodes - brief summary of all node types and SMs in fabric ious - summary of all IO units in the fabric links - summary of all links extlinks - summary of link
Q 5 – Detailed Descriptions of Command LineTools nodeguid:value1:port:value2 - value1 is numeric node GUID, value2 is port # iocguid:value - value is numeric IOC GUID iocguid:value1:port:value2 - value1 is numeric IOC GUID, value2 is port # systemguid:value - value is numeric system image GUID systemguid:value1:port:value2 - value1 is numeric system image GUID value2 is port # ioc:value - value is IOC Profile ID String (IOC Name) ioc:value1:port:value2 - value1 is IOC Profile ID String (IOC Name) value2 i
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Examples: iba_report can generate hundreds of different reports.
Q 5 – Detailed Descriptions of Command LineTools Identify the routes between this server and another server: iba_report -o route -D node:goblin Analyze a single switch for any high error counts: iba_report -o errors -F 'node:i9k156' Identify the routes between a server and an IOC: iba_report -o route -S node:duster -D 'ioc:Chassis 0x00066A005000010C, Slot 2, IOC 2' Clear all the port counters in the fabric: iba_report -C -o none Clear all the port counters on a multi-HCA server: iba_report -C -F node:
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Here is a sample of iba_report for a small fabric: [root@duster root]# iba_report Node Type Brief Summary 14 Connected CAs in Fabric: NodeGUID Port LID Type Name PortGUID Width Speed 0x0002c9020020e0d4 CA coyote1 1 0x000d 0x0002c9020020e0d5 4x 2.5Gb 0x00066a00580001e0 CA VEx in Chassis 0x00066a005000010c, Slot 2 2 0x0014 0x00066a02580001e0 4x 2.5Gb 4x 2.5Gb 4x 2.5Gb 4x 2.5Gb 1 0x0005 0x00066a00a0000384 1x 2.
Q 5 – Detailed Descriptions of Command LineTools 1 0x0011 0x00066a00a000447b 4x 2.5Gb 2 0x0012 0x00066a01a000447b 4x 2.5Gb 4x 2.5Gb 0x00066a0098004a73 CA erik 1 0x0009 0x00066a00a0004a73 3 Connected Switches in Fabric: NodeGUID Port LID Type Name PortGUID Width Speed 0x00066a00280002cd SW InfiniCon Systems InfiniFabric (Sw A Dev A) 0 0x0013 0x00066a00280002cd Noop Noop 3 4x 2.5Gb 5 4x 2.5Gb 0x00066a00d8000123 SW InfiniCon Systems InfinIO9024 0 0x0001 0x00066a00d8000123 4x 2.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools B) 0 0x0010 0x00066a10280002cd Noop Noop 2 4x 2.5Gb 4 4x 2.5Gb 1 Connected SMs in Fabric: State GUID Name Master 0x00066a00d8000123 InfiniCon Systems InfinIO9024 Each iba_report allows for various levels of detail. Increasing detail is shown as further indentation of the additional information. The -d option to iba_report controls the detail level. The default is 2.
Q 5 – Detailed Descriptions of Command LineTools permitted. The maximum detail per report varies, but most have less than 5 detail levels. For example, the above report when run at detail level 0 outputs: [root@duster root]# iba_report -d 0 Node Type Brief Summary 14 Connected CAs in Fabric: 3 Connected Switches in Fabric: 1 Connected SMs in Fabric: You will notice this is a nice summary of fabric components and is very similar to fabric_info.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 0x00066a00d8000123 SW InfiniCon Systems InfinIO9024 0x00066a10280002cd SW InfiniCon Systems InfiniFabric (Sw A Dev B) 1 Connected SMs in Fabric: State GUID Name Master 0x00066a00d8000123 InfiniCon Systems InfinIO9024 The above examples were all performed with a single report, the brnodes (Brief Nodes) report.
Q 5 – Detailed Descriptions of Command LineTools However, iba_report does not stop there. Additionally, iba_report has reports that will help to analyze the operational characteristics of the fabric and help to identify bottlenecks and faulty components in the fabric. To assist in this area, iba_report also supports the following reports: ❥ slowlinks - identifies links which are running slower than expected.
5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Q 5.4.3.2 Topology Verification iba_report provides a flexible way to identify changes to the fabric or the appropriate reassembly of the fabric after a move (for example after staging and testing the fabric in a remote location before final installation at a customer site). In this mode of operation, all the above reports are available, however the types of information output can be filtered.
Q 5 – Detailed Descriptions of Command LineTools When focusing a report, it can sometimes be helpful to also use a detail level of 0 or 1. In this case the report will show only a count of number of matches (for detail 0) and just the highest level of the entity which matches (for detail 1). 5.4.3.3.1 Advanced Focus The node name, node name pattern, node guid, node type, IOC name, IOC name pattern, IOC GUID, IOC type and system image GUID also allow for a port number specifier.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 5.4.3.3.
Q 5 – Detailed Descriptions of Command LineTools 5.4.3.4.1 Using iba_report to monitor for fabric changes iba_report can easily be used in other scripts. For example the following simple script could be run as a cron job to identify if the fabric has changed as compared to the initial design: #!/bin/bash # specify some filenames to use expected_config=/usr/local/report.master # master copy of config previously created config=/tmp/report$$ # where we will generate new report diffs=/tmp/report.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 5.4.3.5 Sample Output 5.4.3.5.1 Analysis of all ports in fabric for errors, inconsistent connections, bad cables [root@duster root]# iba_report -o errors -o slowconnlinks Links running slower than faster port Summary Links running slower than expected: 20 of 20 Links Checked, 0 Errors found Links configured to run slower than supported: Rate NodeGUID Port Type Name Enabled Supported 2.5g 0x00066a0098000384 1x <-> 2.
Q 5 – Detailed Descriptions of Command LineTools ExcessiveBufferOverrunErrors 3 VL15Dropped 100 Rate NodeGUID 10g 0x00066a0098000001 <-> 0x00066a00d8000123 Port Type Name 1 CA julio 8 SW InfiniCon Systems InfinIO9024 LinkDownedCounter: 5 Exceeds Threshold: 3 10g 0x00066a00980001b8 <-> 0x00066a00d8000123 1 CA orc 10 SW InfiniCon Systems InfinIO9024 LinkDownedCounter: 5 Exceeds Threshold: 3 10g 0x00066a0098000380 1 CA goblin SymbolErrorCounter: 65535 Exceeds Threshold: 100 LinkErrorRecoveryCou
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 5.4.3.5.2 Identification of the route between 2 nodes in the fabric [root@duster root]# .
Q 5 – Detailed Descriptions of Command LineTools 5.4.3.5.3 Analysis of the route between 2 nodes for errors, inconsistent connections, etc [root@duster root]# .
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Obtain very detailed information about nodes NOTE: To shorten the length of the output, the following example focuses on only 1 node.
Q 5 – Detailed Descriptions of Command LineTools Link Error Recovery Link Downed Port Rcv Errors Port Rcv Rmt Phys Err Port Rcv Sw Relay Err Port Xmit Discards Port Xmit Constraint Port Rcv Constraint Local Link Integrity Exc.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 5.4.3.5.5 Obtain very detailed information about IOUs NOTE: To shorten the length of the output, the following example focuses on only 1 IOC.
Q 5 – Detailed Descriptions of Command LineTools 10g 0x00066a00980001b8 <-> 0x00066a00d8000123 10g 0x00066a0098000380 <-> 0x00066a00d8000123 1 CA goblin 15 SW InfiniCon Systems InfinIO9024 2.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 5.4.3.5.
Q 5 – Detailed Descriptions of Command LineTools 5.4.3.5.
5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools Q 5.4.3.5.9 Generate reports in a "comparible manner" so topology verification can be performed against a known good configuration NOTE: To shorten the length of the output, the following example focuses on only 1 node.
Q 5 – Detailed Descriptions of Command LineTools LID: xxxxxx LMC: 0 Subnet: 0xfe8000000000000 SMLID: xxxxxx SMSL: 0 RespTimeout: 33 ms SubnetTimeout: 56 ms M_KEY: 0x0000000000000000 Lease: 0 s Protect: Readonly MTU: Active: 2048 Supported: 2048 VL Stall: 0 LinkWidth: Active: 4x Supported: 1-4x Enabled: 1-4x LinkSpeed: Active: 2.5Gb Supported: 2.5Gb Enabled: 2.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools 5.4.4 saquery (All): saquery can perform various queries of the subnet manager/subnet agent and provide detailed fabric information. In many cases iba_report provides a more powerful tool, however in some cases saquery is preferred, especially when dealing with service records and multicast. Saquery is installed on all hosts as part of the IB stack, but it is also included in Fast Fabric.
Q 5 – Detailed Descriptions of Command LineTools -A/--gidlist 'sgid ...;dgid ...': query by a list of Gids -o/--output type: output type for query (default is node) Node Types: ca - channel adapter sw - switch rtr - router GIDs: Specify a 64 bit subnet and 64 bit interface ID as: subnet:interface.
Q 5 – Detailed Descriptions of Command LineTools Fabric Analysis Tools mcfdb: list of switch multicast FDB records trace: list of trace records The following combinations of input (assorted query by options) and output (-o) are permitted: Table 5-1.
Q 5 – Detailed Descriptions of Command LineTools Table 5-1.
Q 5 – Detailed Descriptions of Command LineTools Advanced Initialization and Verification - ibtest 5.5 Advanced Initialization and Verification - ibtest (Switch and Host) : Ibtest performs a number of multi-step operations. In general operations performed by ibtest involve a login to one or more target systems (hosts or SilverStorm IB chassis depending on options used).
Q 5 – Detailed Descriptions of Command LineTools mpi mpidev mpisrc udapl sdp rds, or for a chassis upgrade, filenames/directories of firmware images to install. For directories specified, all .pkg files in directory tree will be used. shell wildcards may also be used within quotes, or for a switch upgrade, filename/directory of firmware image to install. For the directory specified, .emfw files in the directory tree will be used. shell wildcards may also be used within quotes.
Q 5 – Detailed Descriptions of Command LineTools Advanced Initialization and Verification - ibtest Ibtest provides detailed logging of its results. During each run the following files are produced: test.res: appended with summary results of run test.log: appended with detailed results of run save_tmp/: contains a directory per failed test with detailed logs test_tmp*/: intermediate result files while test is running The -c option will remove all of the above.
Q 5 – Detailed Descriptions of Command LineTools FF_PASSWORD: password to use to login as FF_USERNAME. Used in absence of -S option. FF_ROOTPASS: password to use when su to root (if FF_USERNAME is not root). Used in absence of -S option. FF_LOGIN_METHOD: how to login to hosts (Telnet, RSH or SSH), default is SSH FF_TIMEOUT_MULT: multiplier for response timeouts. Default is 2.
5 – Detailed Descriptions of Command LineTools Advanced Initialization and Verification - ibtest Q to select different install packages, the defaults are iba ipoib mpi (i.e., IB Stack, IPoIB and MPI). The default is the typical configuration for an MPI cluster compute node. The -r option can be used to specify a release to install other than the one that this host is presently running. The $FF_PRODUCT..tgz file (e.g., InfiniServPerf.2.1.
Q 5 – Detailed Descriptions of Command LineTools 5.5.1.5 sacache This verifies the given hosts have properly communicated with the SA and cached paths to each other. To run this command, InfiniBand must be installed and running on the given hosts. The subnet manager and switches must be up. If this test fails, cmdall 'cat /proc/driver/ics_dsc/gids' can be run against any problem nodes to see what they have cached. 5.5.1.
5 – Detailed Descriptions of Command LineTools Advanced Initialization and Verification - ibtest Q numbers representative of what most servers can achieve. Some server models may have 10-20% higher results. A result 5-10% below the above numbers is typically not cause for serious alarm, but may reflect limitations in the server design or the chosen BIOS settings. For more details about BIOS settings, consult the documentation from the server supplier and/or the server PCI chipset manufacturer. 5.5.
Q 5 – Detailed Descriptions of Command LineTools 5.5.2.2 reboot This reboots the given chassis and ensures they go down and come back up by pinging them during the reboot process. By selecting the proper FF_MAX_PARALLEL value a rolling reboot or a parallel reboot may be accomplished. In most cases a parallel upgrade is recommended for expediency. 5.5.
Q 5 – Detailed Descriptions of Command LineTools Advanced Initialization and Verification - ibtest 5.5.4 Interpreting the ibtest log files Each run of ibtest will create test.log and test.res files in the current directory. When ibtest indicates that some or all of the test cases failed, the test.res and test.log files should be reviewed. test.res will summarize which tests have failed. Using the test.res file for servers that failed can be quickly identified.
Q 5 – Detailed Descriptions of Command LineTools match the intended host names. Also make sure than when IPoIB host names are used, that the correct name was formed based on the ibtest -i '' argument. This applies a suffix to host names to create IPoIB host names. The default is -ib. Use -i '' to indicate no suffix. 5.6 Health Check and Baselining Tools (All): These tools help to rapidly identify if the fabric has a problem or if its configuration has changed since the last baseline.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools ❥ ❑ Compares present HW/SW/config to baseline ❑ Can be scheduled in hourly cron jobs As needed rerun "baseline" when expected changes occur ❑ Fabric upgrades ❑ Hardware replacements/changes ❑ Software configuration changes ❑ Etc. 5.6.
Q 5 – Detailed Descriptions of Command LineTools present. Also verify the appropriate links between servers and switches are present. If the fabric is not correctly configured, correct the configuration and rerun the baseline. Once a good baseline has been established, use the tools to compare the present fabric against the baseline and check its health. For example: fabric_analysis Will check the present fabric links and components against the previous baseline.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools previous baseline which is to be compared to (except when -e option is used). The FF_ANALYSIS_DIR option in fastfabric.conf can be changed to provide a customer specific alternate directory which will be used whenever the -d option is not specified. Under FF_ANALYSIS_DIR subdirectories will be created as follows: ❥ baseline - baseline snapshot from each analysis tool. ❥ latest - latest snapshot from each analysis tool.
Q 5 – Detailed Descriptions of Command LineTools ❥ fabric components (nodes, links, SMs, systems, and their SMA configuration) ❥ fabric PMA error counters and link speed mismatches Note that the comparison includes components on the fabric. Therefore operations such as shutting down a server will cause the server to no longer appear on the fabric and will be flagged as a fabric change or failure by fabric_analysis.
5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools Q options and reports to be used for the health analysis. It also can specify the PMA counter clearing behavior (-i seconds, -C, or none at all). The thresholds for PMA counter analysis default to /etc/sysconfig/iba/iba_mon.conf. However, an alternate configuration file for thresholds can be specified using the -c option. All files generated by fabric_analysis will start with fabric in their file name.
Q 5 – Detailed Descriptions of Command LineTools latest/fabric.0:0.links.diff - diff of baseline and latest fabric internal and external links. The .diff files are only created if differences are detected. If the -s option is used and failures are detected, files related to the checks that failed are also copied to the timestamped directory name under FF_ANALYSIS_DIR, such as: FF_ANALYSIS_DIR/2007-11-22-09:53:04 5.6.3.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools 5.6.3.2 IB fabric Items that are also checked during healthcheck Based on iba_report -o errors -o slowlinks: ❥ ❥ ❥ PMA error counters on all IB ports (HCA, switch external and switch internal) checked against configurable thresholds. ❑ Counters are cleared each time a healthcheck is run, each healthcheck reflects a counter delta since last healthcheck.
Q 5 – Detailed Descriptions of Command LineTools Environment Variables The following environment variables are also used by this command: CHASSIS, CHASSIS_FILE - see the discussion on the selection of chassis above. FF_TIMEOUT_MULT - multiplier for response timeouts. The default is 2. this typically does not need to be set, but in the event of unexpected timeouts or extremely slow chassis or management network, a larger value can be used. FF_CHASSIS_LOGIN_METHOD - how to login to a chassis.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools All files generated by fabric_analysis start with chassis. in the file name. The chassis_analysis tool generates files such as the following within FF_ANALYSIS_DIR. The actual file names reflect the individual chassis commands that have been configured via the FF_CHASSIS_HEALTH and FF_CHASSIS_CMDS parameters: Health Check: latest/chassis.
Q 5 – Detailed Descriptions of Command LineTools latest/chassis.fwVersion - the output of the fwVersion command for all selected chassis. latest/chassis.fwVersion.diff - diff of the baseline and latest fwVersion. latest/chassis.ismChassisSet12x - the output of the ismChassisSet12x command for all selected chassis. latest/chassis.ismChassisSet12x.diff - the diff of the baseline and latest ismChassisSet12x. latest/chassis.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools latest/chassis.timeDSTConf - the output of the timeDSTConf command for all selected chassis. latest/chassis.timeDSTConf.diff - diff of the baseline and latest timeDSTConf. latest/chassis.timeZoneConf - the output of the timeZoneConf command for all selected chassis. latest/chassis.timeZoneConf.diff - diff of the baseline and latest timeZonfConf. The .diff files are only created if differences are detected.
Q 5 – Detailed Descriptions of Command LineTools ❥ changes to SNMP persistent configuration within the chassis The following Chassis items will not be checked against baseline: ❥ changes to the chassis configuration on the management LAN (e.g., showChassisIpAddr, showDefaultRoute). Such changes will typically result in the chassis not responding on the LAN at the expected address that is detected by failures that will perform other chassis checks. 5.6.4.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools The hostsm_analysis tool performs analysis against the local server only. It is assumed that both the host SM and Fast Fabric are installed on the same system.
Q 5 – Detailed Descriptions of Command LineTools 5.6.5.1 Host SM items checked against the baseline ❥ SM configuration file ❥ The version of the SM rpm installed on the system 5.6.5.2 Host SM items also checked during healthcheck - The SM is in the running state 5.6.6 esm_analysis (Switch): The esm_analysis command has the following usage: esm_analysis [-b|-e] [-s] [-d dir] [-G esmchassisfile] [-E 'esmchassis'] -b - baseline mode. The default is the compare/check mode. -e - evaluate health only.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools or extremely slow chassis or management network, a larger value can be used. FF_CHASSIS_LOGIN_METHOD - how to login to a chassis. Can be SSH or Telnet. FF_CHASSIS_ADMIN_PASSWORD - the password for administrator on all chassis. Used in absence of the -S option. FF_CURTIME - The time stamp to use on the directory created in FF_ANALYSIS_DIR. The default is the current date and time.
Q 5 – Detailed Descriptions of Command LineTools Full analysis: latest/esm.smstatus - the output of the smControl status command for all selected chassis. latest/esm.smShowDefBcGroup - the output of the smShowDefBcGroup command for all selected chassis. latest/esm.smShowDefBcGroup.diff - diff of baseline and latest smShowDefBcGroup. latest/esm.smShowSMParms - the output of the smShowSMParms command for all selected chassis latest/esm. smShowSMParms.diff - diff of the baseline and the latest smShowSMParms.
Q 5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools 5.6.7 all_analysis (All):The all_analysis command has the following usage: all_analysis [-b|-e] [-s] [-d dir] [-c file] [-t portsfile] [-p ports] [-F chassisfile] [-H 'chassis'] [-G esmchassisfile] [-E 'esm chassis'] -b - baseline mode. The default is the compare/check mode. -e - evaluate health only. The default is the compare/check mode. -s - a save history of failures (i.e., errors and differences).
Q 5 – Detailed Descriptions of Command LineTools Note that the all_analysis command has options which are a superset of the options for all other analysis commands. The options will be passed along to the respective tools (e.g., the -c file option will be passed on to fabric_analysis if it is specified in FF_ALL_ANALYSIS). The output files will be all the output files for the FF_ALL_ANALYSIS selected set of analysis. See the previous sections for the specific output files.
5 – Detailed Descriptions of Command LineTools Health Check and Baselining Tools ❥ Q automated In both cases the user should follow the initial setup procedure outlined above to create a good baseline of the configuration. In the manual method, the user would run the tools manually when trying to diagnose problems, or when there is a concern or need to validate the configuration and health.
Section 6 MPI Sample Applications As part of a MPI Development installation, some sample MPI applications and benchmarks are installed to /opt/iba/src/mpi_apps. These can be used to perform basic tests and performance analysis of MPI and its performance.
Q 6 – MPI Sample Applications OSU Latency For all but the mandel and tachyon demos, when MPI applications are run with the scripts provided, the results of the run will be logged to a file in /opt/iba/src/mpi_apps/logs. The file name will include the date and time of the run for uniqueness. 6.1 OSU Latency This is a simple benchmark of end-to-end latency for various MPI message sizes. The values reported are one-direction latency.
Q 6 – MPI Sample Applications This will run assorted bandwidths from 4K to 4Mbytes. To run a different set of message sizes an optional argument specifying the maximum message size can be provided. This benchmark will only use the first two nodes listed in mpi_hosts. During this benchmark the /opt/iba/src/mpi_apps/mpi.param.pallas config file is used. 6.4 OSU Bandwidth2 This is a simple benchmark of maximum unidirectional bandwidth.
Q 6 – MPI Sample Applications Pallas Prior to running this application, a HPL.dat file must be installed into /opt/iba/src/mpi_apps/hpl/bin/ICS/HPL.dat on all nodes. The config_hpl script and some sample configurations are included. The config_hpl script can select from one of the assorted HPL.dat files in hpl-config. For assorted cluster sizes (by number of CPUs). Assorted sample HPL.dat files are provided in /opt/iba/src/mpi_apps/hpl-config.
Q 6 – MPI Sample Applications Pallas has known scalability limitations, especially in its AllToAll phase. This phase can simultaneously perform up to 4MB transfers to-and-from all nodes at once. The downside is a system must have approx 10*NP MB of memory available per process for Pallas data to run this benchmark. Therefore, for a small cluster (approximately 16 processors or less), it is modest at 160MB. However, for a larger cluster (approximately 256 processors or greater), it is rather large at 2.
6 – MPI Sample Applications Pallas 6-6 Q D000006-000 Rev A
Appendix A Fast Fabric Quick Install Checklist The sections below provide a checklist to aid tracking the steps have been completed for Fabric Setup, Installation and verification. Check off each step as its performed. Refer to the section “Getting Started” on page 3-1 for a more detailed explanation of each step. A.1 Setup The Fabric 1. Hardware installed, cabled, powered. 2. HCA installed in each server. 3. Desired Linux OS installed on each server with the following options: a.
Q A – Fast Fabric Quick Install Checklist Installing and verifying Firmware on the IB Chassis A.2 Installing and verifying Firmware on the IB Chassis 1. All Chassis connected to management network. 2. Unique IP address configured for each chassis. 3. Unique name configured for each chassis. 4. Administrator passwords configured the same on all chassis. 5. Fastfabric.conf file reviewed. 6. /etc/sysconfig/iba/chassis file created listing all chassis. 7. Chassis verified via Ethernet ping. 8.
Q A – Fast Fabric Quick Install Checklist 5. Setup password-less SSH/SCP. 6. TCP/IP Name Resolution configured on all hosts. a. if using /etc/hosts - Copy /etc/hosts to all hosts. b. if using DNS - /etc/resolv.conf copied or configured on all hosts. 7. Show uname -a for all hosts and verify OS version installed. 8. Install QuickSilver software. 9. Configure IPoIB IP Addresses. 10. Build MPI Test Apps and copy to hosts. 11. Reboot hosts. 12. Any additional host configuration completed. A.
Q A – Fast Fabric Quick Install Checklist Configure and initialize health check tools A.8 Configure and initialize health check tools This procedure should be followed on each IB management node from which the health check tools will be used. 1. Edit fastfabric.conf and review the health check tools parameters. 2. If using embedded SM(s), create /etc/sysconfig/iba/esm_chassis. 3. Perform a health check using all_analysis -e. 4. Create a cluster configuration baseline using all_analysis -b. 5.
Appendix B Fast Fabric Configuration Files The following configuration files are used by Fast Fabric:. Table B-1. Fast Fabric Configuration Files Configuration File Description /etc/sysconfig/fastfabric.conf Overall configuration file /etc/sysconfig/iba/iba_mon.
Q B – Fast Fabric Configuration Files fastfabric.conf NOTE: Do not edit /etc/sysconfig/fastfabric.conf-sample. The use of various configuration variables are discussed in the Environment Variables section for each command. #!/bin/bash # [ICS VERSION STRING: unknown] # This is a bash sourced config file which defines variables used in # fast fabric tools. Command line arguments will override these settings.
Q B – Fast Fabric Configuration Files # The shell functions below are only defined if no existing function/command # with given name, hence allowing use of shell functions or creation of a # command for this operation # shell Function to convert a basic hostname into an IPoIB hostname # if FF_IPOIB_SUFFIX is "", this should return $1 unmodified # such that commands can be used with -i "" to skip IPoIB operations # (special value of NONE will be converted to "" before this is called) if ! type ff_host_base
B – Fast Fabric Configuration Files fastfabric.
Q B – Fast Fabric Configuration Files export FF_PASSWORD="${FF_PASSWORD:-}" # if FF_USERNAME is not root, what is the root password needed when # suing to root export FF_ROOTPASS="${FF_ROOTPASS:=}" # How to login to chassis # can be ssh or telnet export FF_CHASSIS_LOGIN_METHOD="${FF_CHASSIS_LOGIN_METHOD:-telnet}" # password for admin (not needed if password-less ssh configured) export FF_CHASSIS_ADMIN_PASSWORD="${FF_CHASSIS_ADMIN_PASSWORD:-adminpass} " #base device number for IPoIB devices - IPoIB devices
Q B – Fast Fabric Configuration Files iba_mon.
Q B – Fast Fabric Configuration Files NOTE: D000006-000 Rev A Do not edit /etc/sysconfig/iba/iba_mon.conf-sample.
Q B – Fast Fabric Configuration Files iba_mon.conf # This file controls the iba_mon Port Counter monitoring Thresholds. # [ICS VERSION STRING: unknown] # # Error Counters are specified in absolute number of errors over Interval. # All Data Movement thresholds are specified in terms of average data/second # over the monitoring interval. # # Setting a threshold to 0 disables monitoring of the given counter # # Output is generated when a threshold is exceeded.
Q B – Fast Fabric Configuration Files B.3 Host List Files The /etc/sysconfig/iba/hosts and /etc/sysconfig/iba/allhosts files are used to specify the hosts which Fast Fabric will operate against for many operations. If desired alternate filenames may be specified in fastfabric.conf, via environment variables or on the command line. Refer to the section “Selection of Hosts” on page 5-3 for more information. Below is a sample host list file: # this is a comment 192.168.0.
Q B – Fast Fabric Configuration Files Selection of slots within a chassis # this is a comment 192.168.0.5# chassis IP address edge1# chassis resolvable TCP/IP name include /etc/sysconfig/iba/corechassis# included file Each line of the chassis list file may specify a single chassis, a comment or another chassis list file to include. Chassis may be specified by chassis management network IP address or a resolvable TCP/IP name. Typically, names are used for readability.
Q B – Fast Fabric Configuration Files 9000 series chassis, slot 0 is always an alias for the presently active management card for the chassis. For the remainder of slot usages in the chassis, the chassisQuery command can be executed against a given chassis to identify which slots have management, EVIC or FVIC cards. NOTE: For any operation, care should be taken that a given chassis is listed only once with all relevant slots as part of that single specification.
Q B – Fast Fabric Configuration Files Port List Files For externally-managed switches, the node GUID can be found on a label on the bottom of the switch. Alternately the node GUIDs for switches in the fabric can be found use a command such as: saquery -t sw -o nodeguid NOTE: The above command will report all switch node GUIDs, including those of internally-managed chassis such as the SilverStorm 9120. GUIDs for internally-managed chassis cannot be specified for use in the ibnodes file. B.
Q B – Fast Fabric Configuration Files Comments may be placed on any line. By using a # to precede the comment. On lines with a port or include directive, the # must be white-space separated from any preceding port or included file name.
B – Fast Fabric Configuration Files Port List Files B-14 Q D000006-000 Rev A
Appendix C Configuration of IPoIB Name Mapping The Fast Fabric tools support the concept of a management network and an IPoIB network. For some clusters the management network will be a low speed network such as 10/100 Ethernet. For other clusters IPoIB may serve double duty as the host management network. NOTE: When using IPoIB as the management network, the initial installation of IB software cannot be done via Fast Fabric.
C – Configuration of IPoIB Name Mapping C-2 Q D000006-000 Rev A
Appendix D Multi-Subnet Fabrics Fast Fabric is designed primarily to manage a single subnet fabric. However many powerful functions of FastFabric are also available when installing and operating multi-subnet fabrics. When operating a multi-subnet fabric, an subnet manager (SM) is required for each subnet. An SM may be run within switches within each subnet, or a host-based SM may be run. A host-based SM can manage multiple subnets (assuming the host server is connected to more than one subnet).
D – Multi-Subnet Fabrics Primarily Independent Subnets Q typically with different IP subnets. Consult the QuickSilver Fabric Access Software Users Guide for more information on configuring IPoIB. NOTE: When managing a cluster where compute nodes are not running the QuickSilver host stack or where the IPoIB settings on the compute nodes are incompatible with the IB Management node (e.g., when a 4K MTU is used on the compute nodes), it is recommended not to run IPoIB on the IB management node(s).
Q D – Multi-Subnet Fabrics ❥ (All): However instead it is recommended to run: iba_report -i 10 -o errors -o slowlinks -h x -p y where x and y specify the applicable HCA and port to select the desired subnet. Repeat for each subnet. ❥ (Host): “Verify Hosts see each other” on page 4-9 can be run for each subnet by using the allhosts files specific to each subnet (i.e., those listing only hosts in a single subnet).
D – Multi-Subnet Fabrics Overlapping Subnets Q fabric operation. Follow the installation instructions outlined in “Getting Started” on page 3-1 with the following adjustments: From “Design the Fabric” on page 3-1, design the cabling such that the Fast Fabric node will be connected to each IB subnet it will manage. The Fast Fabric node must also have a management network path to all the nodes in all the subnets it will manage.
Q D – Multi-Subnet Fabrics ❥ (All): Create the allhosts file per the instructions. Next, create additional files per subnet that list all the hosts in each subnet including the IB management node. ❥ (All): “Verify Hosts via Ethernet ping” on page 4-4 can be performed per the instructions. ❥ (All): “Summary of Fabric Components” on page 4-9 is not presently supported for IB management nodes connected to multiple fabrics.
D – Multi-Subnet Fabrics Overlapping Subnets Q variable may be used to specify all the HCAs and ports on the IB management node such that all subnets are checked. Similarly, the esm_chasssis and chassis files used should list all relevant SilverStorm IB chassis in all subnets. “Running HPL” on page 3-20 can be run for each subnet by creating mpi_hosts files specific to each subnet (i.e., only listing hosts in a single subnet). “Upgrading IB software” on page 3-21 can be performed per the instructions.
