HP XC System Software Installation Guide Part Number: AA-RWJXC-TE June 2005 Product Version: HP XC System Software Version 2.1 This manual describes how to install and configure HP XC System Software Version 2.1 on HP Cluster Platforms 3000, 4000, and 6000.
© Copyright 2003–2005 Hewlett-Packard Development Company, L.P. AMD and AMD Opteron are trademarks or registered trademarks of Advanced Micro Devices, Inc. FLEXlm is a trademark of Macrovision Corporation. InfiniBand is a registered trademark and service mark of the InfiniBand Trade Association. Intel, the Intel logo, Itanium, Xeon, and Pentium are trademarks or registered trademarks of Intel Corporation in the United States and other countries. Linux is a U.S. registered trademark of Linus Torvalds.
Contents About This Document 1 Overview 1.1 1.2 1.3 2 Task 1: Read Related Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Task 2: Prepare the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Task 3: Verify Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.7.6 4.8 4.9 4.10 4.11 4.12 5 Run the Operation Verification Program (OVP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Take a Snapshot of the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create a Baseline Report of Your System Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . You Are Done . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H Customizing Client Node Disks Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dynamic Disk Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example 1: Changing Default Partition Sizes and Swap Space For All Client Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.4.4 L.4.5 L.5 Step 4: Edit the SLURM Configuration File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 5: Configure the Maui Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verifying Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L-4 L-4 L-4 Sample XC.lic File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
About This Document This document describes the procedures and tools that are required to install and configure HP XC System Software Version 2.1 on HP Cluster Platforms 3000, 4000, and 6000. An HP XC system is integrated with several open source software components. Some open source software components are being used for underlying technology, and their deployment is transparent.
• Appendix C provides guidelines for various pieces of user-defined information that you are asked to define or supply during the installation and configuration process. • Appendix D provides guidelines for installing optional software from third-party vendors. • Appendix E provides a description of the sequence of events generated by the automated discovery process as well as troubleshooting techniques.
QuickSpecs for HP XC System Software Provides a product overview, hardware requirements, software requirements, software licensing information, ordering information, and information about commercially available software that has been qualified to interoperate with the HP XC System Software. The QuickSpecs are located at the following URL: http://www.hp.com/techservers/clusters/xc_clusters.
Home page for Platform Computing, the developer of the Load Sharing Facility (LSF HPC). LSF HPC, the batch system resource manager used on an XC system, is tightly integrated with the HP XC and SLURM software. For your convenience, the following Platform LSF documents are shipped on the HP XC documentation CD in PDF format. The Platform LSF documents are also available on the XC Web site.
Manpages Manpages provide online reference and command information from the command line. Manpages are supplied with the HP XC system for standard HP XC components, Linux user commands, LSF commands, and other software components that are distributed with the HP XC system. Manpages for third-party vendor software components may be provided as a part of the deliverables for that component.
A comprehensive site containing general information, such as the specification and FAQs, and pointers to a variety of other resources, including tutorials, implementations, and other MPI-related sites. Related Compiler Web Sites • http://www.intel.com/software/products/compilers/index.htm Web site for Intel® compilers. • http://support.intel.com/support/performancetools/ Web site for general Intel software development information. • http://www.pgroup.
[ ] In command syntax and examples, brackets ([ ]) indicate that the contents are optional. If the contents are separated by a pipe character ( | ), you must choose one of the items. { } In command syntax and examples, braces ({ }) indicate that the contents are required. If the contents are separated by a pipe character (|), you must choose one of the items. ...
1 Overview This chapter contains the following information: • How to navigate this manual (Section 1.1) • How to upgrade your system from HP XC System Software Version 1.0, 2.0, or 2.0A to Version 2.1 (Section 1.2) • A summary of system and host naming conventions and sample IP addresses used in this manual (Section 1.3) 1.1 How to Use This Manual This manual is designed to lead you through the mandatory installation and configuration tasks in a predefined sequence.
Table 1-1: Naming Conventions Item Value System host name (also known as the cluster alias or LVS alias) penguin Head node host name n16 (n[maximum number of nodes]) Head node alias nh Host names of nodes with connections to the external network through an Ethernet port penguin1, penguin2, penguin3, and so on. Node name prefix Internal node names are assigned using the default node naming prefix, the letter n. Fully qualified domain name southpole.com External IP address of the system 192.
2 Before You Begin This chapter describes the following preinstallation tasks; perform them in the following sequence: 1. Read related documentation (Section 2.1). 2. Prepare the hardware (Section 2.2). 3. Verify the firmware revision on the console devices (Section 2.3). 4. Arrange for IP address assignments (Section 2.4). 5. Obtain and save the license key file (Section 2.5). 6. Purchase and license additional software products from third-party vendors (Section 2.6). 2.
• The external IP address of the head node, if it is to be connected to an external network • The IP address of any other node that will have a connection to an external network When you contact your network administrator about assigning IP addresses, you must also provide the host names for any system that will have a connection to an external network, therefore, consider the following: • If you assign a login role to one or more nodes, what name will you use as the cluster alias? This name is the hos
2.6 Task 6: Purchase and License Additional Software From Third-Party Vendors An HP XC system supports the use of several third-party software products. Use of these products is optional; the purchase and installation of these components is your decision and depends on your site’s requirements. The XC software does not bundle, nor does HP resell, the TotalView debugger or the Intel and The Portland Group compilers, which are required by the HP Mathematical Library (MLIB).
3 Installing Software on the Head Node This chapter contains an overview of the installation process (Section 3.1) and describes installation tasks, which must be performed in the following order: 1. Gather items before you begin (Section 3.2). 2. Prepare for the installation question-and-answer session (Section 3.3). 3. Install the XC software stack on the head node (Section 3.4). 4. Install additional RPMs from the XC DVD (optional) (Section 3.5). 5. Install software patches (Section 3.6). 6.
Table 3-1: The XC Software Stack (cont.) Software Product Description LSF HPC with SLURM on XC The Load Sharing Facility (LSF HPC) developed by Platform Computing, Inc. provides both direct job allocation and policy-scheduled interactive and batch queue management on an XC system. LSF HPC provides a rich set of policies for scheduling and prioritizing jobs based on combinations of static and dynamic system attributes.
3.1.3 Kickstart File The ks.cfg Kickstart file is supplied on the HP XC System Software DVD, and it contains HP-supplied default values to reduce the number of answers you have to provide during the installation session. The Kickstart file differs depending on your cluster platform architecture, and the appropriate version of the file is included on your distribution media. 3.1.
Table 3-4: Distribution Media • Cluster Platform Model Software DVD Title CP3000 HP XC System Software for Intel Xeon Systems Version 2.1 CP4000 HP XC System Software for AMD Opteron Systems Version 2.1 CP6000 HP XC System Software for Intel Itanium 2 Processors Version 2.1 Know how to use a text editor (such as vi, emacs, or gedit) because at some points in the system installation and configuration process, you are required to modify text files. 3.
Table 3-5: Software Installation Questions and Answers (cont.) Item Description and User Action Head node IP address Provide the IP address that identifies the head node’s connection to the external network. This is the address assigned by your site’s network administrator. The IP address must be in your site’s Domain Name Service (DNS) configuration. Network mask Provide the network mask for your site.
upon your cluster platform architecture and how the boot device was set. Enter one of the following commands: • On Opteron- and Xeon-based systems, enter this command: boot: linux ks=cdrom:/ks.cfg • On Itanium-based systems, if the boot device was set through the preconfigured boot option, and the ELILO boot prompt is displayed, enter this command: ELILO boot: linux ks=cdrom:/ks.
2. Change to the following directory: # cd /mnt/cdrom/LNXHPC/RPMS 3. Find the RPM you want to install and issue the appropriate command to install it. For example, issue a command similar to the following: # rpm -ivh kde-i18n-French-3.1.3-1.noarch.rpm 3.6 Install Patches or RPM Updates Additional RPMs containing software updates and fixes may be available for XC Version 2.1.
9. Download all patches that are available for your cluster platform into the temporary directory you created. If you are unable to retrieve the files from this site, contact the XC support team at xc_support@hp.com. 10. Follow the patch installation instructions in the README file. 3.7 Install HP StorageWorks Scalable File Share (Optional) HP StorageWorks Scalable File Share (HP SFS) is based on Lustre™ technology developed by Cluster File Systems, Inc.
4 Configuring and Imaging the System This chapter contains an overview of the system configuration and imaging process (Section 4.1) and describes system configuration tasks, which must be performed in the following order: 1. Gather information before you begin (Section 4.2). 2. Prepare for the system configuration question-and-answer session (Section 4.3). 3. Place the license key file in the correct location (Section 4.4). 4.
When nodes are instructed to PXE boot from their network adapter attached to the administrative network, the nodes automatically proceed through an installation process that replicates the golden client and does the following: • Partitions the local disk. By default, the first disk is used because disks other than the first disk are not supported due to restrictions in the image replication environment for this release.
Table 4-1: Configuration Questions and Answers (cont.) Question Description and User Action Number of nodes that are workstations Enter the number of nodes that are workstations, that is, nodes that do not have console ports. There is no default response. Enter 0 (zero) if your system does not contain workstations. You can include the ws= keyword on the discover command line to bypass this question during the discovery process.
Table 4-1: Configuration Questions and Answers (cont.) Question Description and User Action Nagios® configuration Nagios is an open source tool that provides the monitoring capabilities on an XC system; it watches specified hosts and services and issues alerts when problems occur and when problems are resolved. You have the option to enable web access to the Nagios monitoring application.
The default contents of this file is similar to the following: Base = 172 nodeBase = 172.20 cpBase = 172.21 swBase = 172.20.65 icBase = 172.22 otherBase = 172.23.0 netMask = 255.224.0.0 The following describes the parameters in this file: Base The common first octet that is used for all base IP addresses. nodeBase The base IP address of all nodes. cpBase The base IP address of the console branch. swBase The base IP address of the Ethernet switches. icBase The base IP address of the interconnect.
3. Enter the following command to configure the administrative network interface on the head node, set the password for the MySQL database administrator, configure the interconnect, start the MySQL server, and initialize the system configuration database: # ./cluster_prep Command output looks similar to the following: The cluster_prep utility prepares the head node for the automated discovery process. This utility is run during the initial cluster configuration or any time your cluster is reconfigured.
5. Consider using the script command to capture the output from the discover command into a file if your system contains a large number of nodes (more than 32). This step is optional. Enter the following command to capture command output into a file: # script your_filename 6. Enter Start the automatic discovery of all network components in your system; this step uses the simplest form of the command: # .
Discovering Console Ports... running port_discover on 172.20.65.2 running port_discover on 172.20.65.2 Discovering 172.20.65.2 port 42 ... Console Port OK Discovering 172.20.65.2 port 41 ... Console Port OK Discovering 172.20.65.2 port 40 ... Console Port OK Restarting dhcpd discovered_cps is 3 Checking if all console ports are reachable ... number of cps to check, 3 pinging 172.21.0.14 pinging 172.21.0.16 pinging 172.21.0.15 .done Opening /etc/hosts Opening /etc/hosts.new.XC Opening /etc/powerd.
2 Enter the password for the Root Administration Switch that you previously defined when you prepared the hardware. If you did not preset a password, press the Enter key. 3 Enter the common user name and password that you previously defined for the console port management devices (the iLO, LO-100i, or MP devices) when you prepared the hardware. If you did not predefine these items, press the Enter key. 4 This is normal behavior.
Appendix G describes how to configure the line monitoring cards for each interconnect type. Return to this chapter when you are finished. 4.7.2 Configure sendmail (Required) LSF HPC requires a mail program to send job output to users submitting jobs and to send administrative messages to the LSF administrator. By default, LSF HPC uses the mail program /usr/lib/sendmail.
5. Use the text editor of your choice to edit the /etc/yp.conf file and add the following line, where domain_name is the NIS domain name and NIS_slave_name is either the IP address or host name of the slave NIS server (the head node): domain domain_name server NIS_slave_name 6. Save your changes and exit the file. 7. Use the text editor of your choice to edit the /etc/sysconfig/iptables.proto file to open ports for the yp client, yp server, and yp transfer service on the various network adapters.
_______________________ Note _______________________ If the master NIS server is on a different architecture (for example Tru64 UNIX®), you might see messages similar to the following, which is caused by an incompatibility in the architecture and program ID: Trying ypxfrd ... not running You can safely ignore these messages and proceed to the next step. 13. Use the text editor of your choice to add nis entries to the end of each line in the /etc/nsswitch.
18. Start the cluster_fstab service to merge the changes from the fstab.proto file into the /etc/fstab file on the head node: # service cluster_fstab start 19. Ensure that NIS logins are accepted, and recycle the sshd service on the head node: # service sshd restart 20. When the head node is configured as a slave NIS server, you may notice that the head node does not accept NIS logins after it has been rebooted. If this happens, check the status of the indicated services by issuing the following commands.
3. Save your changes and exit the file. 4.7.6 Customize Client Node Disk Partitioning (Optional) The XC client node imaging process requires a single system disk on each client node for the operating system installation. Client node disk devices are partitioned automatically as part of the automated client imaging process. Table 4-3 lists the default disk partition layout on client nodes. Partition sizes are created as a percentage of available disk space after subtracting the total swap size.
done by the cluster_config command will fail to properly export the /hptc_cluster directory from the head node, which will cause a failure. 2. Enter the following command to apply the default configuration, change default node role assignments, or modify network Ethernet interfaces on any node except the head node: # .
Before deciding whether or not you want to accept the default configuration, consider the following: • A compute role is assigned to the head node by default. Thus, LSF users will submit jobs that run on the head node and will obtain less than optimal performance if interactive users are also on the head node. You may want to consider removing the compute role from the head node to prevent it from being configured as a SLURM compute node.
See Appendix J for information about how to determine the QsNetII network type. 8. Supply the name of the LVS alias if you assigned a login role to one or more nodes, this example uses the alias penguin. This is the name by which users will log into the system. If you did not assign a login role to any node, you will not be asked to supply an LVS alias. Running C20gmmon Myrinet interconnect not being used on this system.
partition. If other partitions are desired they must be configured manually in /hptc_cluster/slurm/etc/slurm.conf Here is the current Node Partition configuration: PartitionName=lsf RootOnly=YES Shared=FORCE Nodes=n[14-16] The SLURM configuration is finished.
Creating /opt/hptc/lsf/top ... ... Done checking the LSF TOP directory /opt/hptc/lsf/top ... You have specified to LSF_LICENSE, lsfinstall will continue without installing a license. Checking LSF Administrators ... LSF administrator(s): "lsfadmin" Primary LSF administrator: "lsfadmin" Searching hpc distribution tar files in /opt/hptc/lsf/files Please wait ... 1) linux2.4-glibc2.
... Done creating LSF configuration directories and files ... Creating a new cluster "hptclsf" ... Adding entry for cluster hptclsf to /opt/hptc/lsf/top/conf/lsf.shared. Installing lsbatch directories and configurations ... Creating /opt/hptc/lsf/top/conf/lsbatch/hptclsf ... Creating /opt/hptc/lsf/top/conf/lsbatch/hptclsf/configdir ... Creating /opt/hptc/lsf/top/work/hptclsf ... Creating /opt/hptc/lsf/top/work/hptclsf/logdir ... Creating /opt/hptc/lsf/top/work/hptclsf/lsf_indir ...
Sanitizing services in the golden image Image replication environment configuration complete.
Initially, all nodes with the compute role are listed as SLURM compute nodes, and these nodes are configured statically with a processor count of two. A sample default compute node entry in the slurm.conf file looks like this: NodeName=n[1-64] Procs=2 SLURM provides the ability to set several other compute node characteristics. At a minimum, you should ensure that the processor count is accurate.
You can improve the responsiveness on such systems by limiting the number of nodes that are imaged simultaneously, which is accomplished by including the --max_at_once= option on the startsys command line. The recommended values for the --max_at_once option are shown in Table 4-4.
3. Complete this step for systems installed with an InfiniBand interconnect; skip this step for all other interconnect types. Create a list of all nodes in the XC system. The InfiniBand diagnostic tools requires this list of nodes in order to function properly: # shownode all > /usr/voltaire/scripts/HCA400-Checks/node-list Proceed to Section 4.11 to start LSF HPC. 4.11 Start LSF HPC Follow this procedure to start LSF HPC: 1. Begin this procedure as the root user on the head node. 2.
Refer to the troubleshooting information in the HP XC System Software Administration Guide if you do not receive a status of ok from the bhosts command. LSF HPC Documentation LSF HPC procedures and information supplied in the HP XC manuals supersedes the information supplied in the LSF manuals from Platform Computing Corporation. The Platform LSF manpages are installed by default. The HP-supplied lsf_diff(1) describes LSF command differences on an HP XC system.
5 Verifying the System Follow the procedures described in this chapter to verify the successful installation and configuration of the XC system components. HP recommends that you perform all tasks described in this chapter: 1. Run the operation verification program to verify the integrity of the installation and configuration of your XC system (Section 5.1). 2. Create a baseline snapshot of the system configuration database (Section 5.2). 3. Run the sys_check utility (Section 5.3). 5.
PING 172.22.0.14 ic-n14 Test completed successfully. ... OK All IP addresses were reachable. +++ PASSED +++ Verify client_nodes: Testing hptc_cluster_mount ... Starting the command ’/opt/hptc/bin/shownode servers hptc_cluster_fs_client’ Here is the output from the command: n[14-16] Starting the command ’/opt/hptc/bin/expandnodes n[14-16]’ Here is the output from the command: n14 n15 n16 3 nodes provide the ’hptc_cluster_fs_client’ service. Now checking if ’/hptc_cluster’ is mounted on those nodes.
Verify SLURM: Testing daemon_responds ... Starting the command ’/opt/hptc/bin/scontrol ping’ Here is the output from the command: Slurmctld(primary/backup) at n16/(NULL) are UP/DOWN Checking output from scontrol. +++ PASSED +++ Testing partition_state ... Starting the command ’/opt/hptc/bin/sinfo --all’ Here is the output from the command: PARTITION AVAIL TIMELIMIT NODES lsf up infinite 3 STATE NODELIST idle n[14-16] Checking output from command. +++ PASSED +++ Testing node_state ...
’/opt/hptc/bin/sinfo --partition lsf --exact -o "%5c %D"’ Here is the output from the command: CPUS 2 NODES 3 +++ PASSED +++ Testing hosts_status ... Starting the command ’/opt/hptc/lsf/top/6.0/linux2.4-glibc2.3-amd64-slurm/bin/bhosts -w’ Here is the output from the command: HOST_NAME STATUS lsfhost.localdomain ok JL/U - MAX 6 NJOBS 0 RUN 0 SSUSP 0 USUSP 0 RSV 0 Checking output from command.
Test failures and warnings are clearly reported, and the log file contains some troubleshooting information. In some cases, the errors might be obvious, and the test output is terse. For example, an LSF HPC test might fail, and the log file message might say that LSF HPC has not been configured. See the ovp(8) manpage and the HP XC System Software Administration Guide for more information about verifying individual cluster components on demand.
You have the option to perform additional configuration tasks depending upon your system or site requirements; proceed to Chapter 6.
6 Configuring SAN Storage Devices This chapter describes how to configure SAN storage devices; the following topics are described: • Installing and configuring the HP StorageWorks Enterprise Virtual Array 3000 or 5000 (EVA3000 or EVA5000) (Section 6.1.1) • Making and mounting the SAN storage file systems (Section 6.1.2) 6.
For more information about SAN devices, see the storage array systems home page at the following URL: http://h18006.www1.hp.com/storage/arraysystems.html 6.1.1 Install and Configure the EVA3000 or EVA5000 SAN Storage Device The EVA3000 or EVA5000 SAN storage device is configured through a SAN appliance that provides a browser interface for configuration. You do not need to install additional software on your cluster to allow for the configuration of these devices.
Host: scsi2 Channel: 00 Id: 01 Lun: 03 Vendor: COMPAQ Model: HSV110 (C)COMPAQ Type: Direct-Access Host: scsi2 Channel: 00 Id: 02 Lun: 00 Vendor: COMPAQ Model: HSV110 (C)COMPAQ Type: Unknown Host: scsi2 Channel: 00 Id: 02 Lun: 01 Vendor: COMPAQ Model: HSV110 (C)COMPAQ Type: Direct-Access Host: scsi2 Channel: 00 Id: 03 Lun: 00 Vendor: COMPAQ Model: HSV110 (C)COMPAQ Type: Unknown Host: scsi2 Channel: 00 Id: 03 Lun: 01 Vendor: COMPAQ Model: HSV110 (C)COMPAQ Type: Direct-Access 3.
First data block=0 313 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 20 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. 6.
A Installation and Configuration Checklist Table A-1 provides a checklist to ensure you complete all installation and configuration tasks in the correct order. Perform all tasks on the head node unless otherwise noted. Table A-1: Installation and Configuration Checklist Task Description Related Section Is Task Complete? Preparing for the Installation 1. Read related documents, especially the HP XC System Software Release Notes. Section 2.1 2.
Table A-1: Installation and Configuration Checklist (cont.) Task Description Related Section 18. Define and set up the system environment before the golden image is created: Section 4.7 • Configure interconnect switch line monitoring cards (required) • Configure sendmail (required) • Create user accounts (optional) • Configure NIS (optional) • Limit user access to compute nodes (optional) • Customize client node disk partitions (optional) 19.
B External Ethernet Connections During the Kickstart installation process, you are prompted to supply the Ethernet device to use as the head node’s connection to the external network. Use the information in Table B-1 to determine the correct port based on the type of system interconnect in use and the number of Ethernet ports on the head node. The appropriate choice for the external connection is shown in the column titled External Connection.
Table B-1: External Ethernet Connections (cont.) Head Node Server Model HP Integrity rx2620 System Interconnect Number of Ethernet Ports Admin Connection External Connection Any other 2 eth1 eth0 Any other More than 2 eth1 eth2 GigE 3 eth0 eth2 GigE More than 3 eth0 eth2 Any other More than 1 eth0 eth1 a Gigabit Ethernet b Any other means any other type of system interconnect besides GigE, such as Myrinet, Quadrics, or InfiniBand, that is supported for this HP Cluster Platform.
C Guidelines for User-Defined Information This appendix contains guidelines for some user-defined information you are asked to supply during the installation and configuration process. • Host names (Section C.1) • Passwords (Section C.2) C.1 Defining Host Names Follow these guidelines when deciding on a host name: • Names can contain from 2 to 63 alphanumeric uppercase or lowercase characters ( a-z, A-Z, 0-9 ). Fully qualified host names (for example, mysys.example.
• A password must contain a combination of uppercase and lowercase letters. • At least one of the first six characters in a password must be a number, a special character, or an uppercase letter. • HP recommends, but does not mandate, that you use a combination of numbers and special characters such as the dollar sign ($ ), the percent sign ( % ), the number sign ( # ), the period ( . ), the hyphen ( - ), the underscore ( _ ), or the at sign ( @ ) in a password.
D Installing Additional Software This chapter contains information about installing and using additional software products. The following topics are discussed: • Overview of third-party software on an XC system (Section D.1) • Intel and PGI Compilers (Section D.2) • MLIB dependency on compilers (Section D.3) D.1 Overview An HP XC system supports the use of several third-party software products.
must edit the corresponding modulefile to point to their new location or create a corresponding symbolic link. Modulefiles are located in the /opt/module/modulefiles directory.
E Discovery Flowchart and Troubleshooting This appendix provides the following information: • A flowchart of the system discovery process (Section E.1) • How to proceed if you encounter problems running the discover command (Section E.2) E.1 Discovery Flowchart Figure E-1 provides a high level flowchart that illustrates the processing performed by the discover command.
E.2 Troubleshooting the Discovery Process This section provides troubleshooting hints to help you solve some common problems that may occur during the discovery process. The following conditions are described: • ProCurve switches do not obtain their IP address (Section E.2.1) • ProCurve switches may take time to obtain their IP addresses (Section E.2.2) • All console ports are not discovered (Section E.2.
_________________________ Note _________________________ If the ---oldmp option was used on the discover command line, it is assumed that all management processors (MPs) have their IP addresses set statically, and therefore are not subject to this step in the discovery process. If some console ports are not configured to use DHCP, they will not be discovered. Therefore, the first item to verify is whether or not the non-discovered console ports are configured to use DHCP.
Switch 172.20.65.3 port 6 ... NO Node Found Switch 172.20.65.4 port 1 ... Node Found . . . In this case, there is a node plugged into port 6 of the Branch Root switch at address 172.20.65.3. To resolve the discovery problem, examine this node to see what actions it is taking during power-up. Is it network booting? Is the proper network interface plugged into the switch? After these issues are resolved, run the discover process again.
F Enabling telnet on iLO Devices The procedure described in this appendix applies only to XC systems with nodes that use Integrated Lights Out (iLO) as the console management device. New nodes that are managed with iLO console management connections that have never been installed with XC software may have iLO interfaces that have not been configured properly for XC operation. In order for the discover command to discover the console ports, the iLO devices must have telnet enabled.
2. 3. Do one of the following: • If you cannot find an entry corresponding to the new node, check the network connections. Make repairs and rerun the discover command. • If you do find an entry corresponding to the new node, note the IP address on the line that begins with the string fixed-address, and proceed to step 3. Open the Mozilla Web browser on the head node, and in the Web address field at the top of the window, enter the IP address you noted in step 2 appended with /ie_index.
G Configuring Interconnect Switch Line Monitoring Cards Configuring the Quadrics switch controller cards, the InfiniBand switch controller cards, and the Myrinet monitoring line cards on the system interconnect is required for diagnosing and debugging problems with the system interconnect. You must configure these cards just before running the cluster_config command.
such as HyperTerminal. Connecting to the switch controller displays the Quadrics Switch Control main menu, which looks similar to the following: Quadrics Switch Control -- (QR0N00) 1. 2. 3. 4. 5. 6. 7. 8. 9. Show network settings Change network settings Run jtest Set module mode Firmware upgrade Quit Reboot Access Settings Self Test 3. Choose the Change network settings menu option and choose BOOTP. 4. Choose the Show network settings to obtain the line card MAC address.
option xc-macaddress "00:11:0a:ea:ea:41"; # location "Level 2 Switch 172.20.65.3, Port 3"; } host QR0N00 { hardware ethernet your_MAC_address; option host-name "QR0N00"; fixed-address 172.20.66.1; filename "503-upgrade.tar"; } } } 10. Restart the DHCP service on the head node: # service dhcpd restart 11. Use the text editor of your choice to open the /etc/hosts file and add an entry for each switch, using the data in Table G-1 as a reference.
Follow this procedure to configure network access to the Myrinet switch monitoring line cards: 1. Begin this procedure as the root user on the head node. 2. Ensure that an Ethernet cable from the monitoring line card primary Ethernet port is connected to an Ethernet port on the admin root switch. This connection is described in the HP XC System Software Hardware Preparation Guide. 3. Obtain the line card MAC address from the label on the front panel of the Myrinet switch. 4.
Make your entries above the following line in the file because any entries that follow this line will be deleted if you reconfigure your system: #XC-CLUSTER Do Not Edit Below this Line 7. Save your changes and exit the file. 8. Repeat this entire procedure for each monitoring line card, modifying the switch name and switch address using the data in Table G-2 as a reference. 9.
8. Access the configuration mode: ISR-9024# config 9. Access the interface fast mode: ISR-9024(config)# interface fast 10. Set the IP address and netmask. Enter the IP address of the switch and the netmask, using the data in Table G-3 as a reference. The netmask must be set to 255.255.0.0 as shown: ISR-9024(config-if-fast)# ip-address-fast set IP_address 255.255.0.0 11. Confirm the settings: ISR-9024(config-if-fast)# ip-address-fast show 12.
H Customizing Client Node Disks Use the information in this appendix to customize the disk partition layout on client node disk devices. The following topics are described: • An overview of the client node imaging process and customizing disk partitions on client nodes (Section H.1) • Dynamic disk configuration process (Section H.2) • Static disk configuration process (Section H.3) H.
• Use the static disk configuration procedure (Section H.3) when you want to create partitions other than the default or if you want to create file system types other than the default on the client disks. H.2 Dynamic Disk Configuration Dynamic disk configuration identifies the actual client disk device type and size during the actual autoinstallation operation based on a list of possible disk architectures supported within an XC configuration.
for swap space), the /var partition is decreased to 29% of available disk space (after accounting for swap space), and the amount of swap space is decreased from 6 GB to 4 GB. 1. Use the text editor of your choice to open the /opt/hptc/systemimager/etc/make_partitions.sh file and search for the block of text that looks like this: BOOT_PERCENTAGE=".01" ROOT_PERCENTAGE=".60" VAR_PERCENTAGE=".39" 2.
Begin this procedure as the root user on the head node: 1. Change to the /var/lib/systemimager/scripts directory, and make a copy of the base_image.master.0 file: # cd /var/lib/systemimager/scripts # cp base_image.master.0 login.master.0 2. Use the text editor of your choice to open the new login.master.0 file, and search for the block of text that looks like this: BOOT_PERCENTAGE=".01" ROOT_PERCENTAGE=".60" VAR_PERCENTAGE=".39" 3.
In an ideal environment, client disks obtain their configuration directly from the golden client (head node). When the golden client and client nodes are of the same platform type, disk type, and disk size, this works fine. However, in many XC system configurations, there is a heterogeneous mix of platform types, disk types, and disk sizes.
H.3.2 Customizing Client Disk Configuration Customizing a client’s disk configuration involves either editing an existing .conf file or creating a new one. The format of the .conf file is explained in detail in the autoinstallscript.conf(5) manpage. If you must create a new .conf file, HP recommends that you copy an existing .conf file with the proper disk architecture (/dev/sd, /dev/hd, /dev/cciss) as a starting point, and make the necessary modifications to the copied file. H.3.2.
2. Use the text editor of your choice to modify the login_node.conf file with the necessary disk configuration changes. Refer to the autoinstallscript.conf(5) manpage for formatting guidelines. 3. Save your changes to the file. 4. Create a new autoinstallation script: # mkautoinstallscript --quiet \\ --no-listing \\ --image base_image \\ --script login_node \\ --force YES \\ --config-file /opt/hptc/systemimager/etc/login_node.
I Using the Configuration Menu This appendix describes how to use the text-based cluster configuration menu that is displayed by the cluster_config utility. The following topics are included in this appendix: • An overview of the cluster_config text-based menu interface (Section I.1) • Displaying the current system configuration (Section I.2) • Modifying node configurations (Section I.3) • Adding an Ethernet connection to any node other than the head node (Section I.3.
Please enter node name or [C]ancel : n15 Current Node: n15 [E]xternal Network Configuration, [R]oles, [H]elp, [B]ack: You now have two options: • Proceed to Section I.3.1 to configure an external Ethernet connection on any node except the head node, which will connect the node directly to your public network. • Proceed to Section I.3.2 modify node role assignments. I.3.
Enter the number that corresponds to the role or roles you want to assign to this node. Use a space-separated list to specify multiple roles, for example, 1 3 5. Or, enter c to cancel. Assume you want to add a login role to node n15. Not only do you enter the number 4, you must also enter the numbers 1 and 6 to retain the current compute and nat_client roles on node n15. Basically, you have to supply every role that you want to assign to the node.
The management database names of the services provided by this role are: • SLURM compute (slurm_compute) • SuperMon (mond) • Nagios remote plugin execution agent (nrpe) To enable monitoring of the nodes, nodes with the compute role runs the Nagios Remote Plugin Execution agent and the SuperMon Monitoring service. Any node with this role may be called upon to execute jobs scheduled by the users. Nodes with this role are also often called compute nodes. I.4.
_________________________ Note _________________________ HP recommends that you consider using one management hub for every 64-128 nodes. I.4.5 External Role The external role supplies the NAT server service, which does network address translation within the cluster. This enables applications to access nodes that do not have an external network connection. The management database name of the service supplied by this role is nat.
I.4.7 Resource Management Role Nodes with the resource management role provide the services necessary to support SLURM and LSF HPC. The management database names of the services provided by this role are: • LSF master node (lsf) • SLURM central control service (slurm_controller) You can assign this role to multiple nodes in the cluster to provide support for failover. Multiple nodes may have this role defined, but only one is active at a time.
J Determining the Network Type The information in this appendix applies only to cluster platforms with a Quadrics QsNetII interconnect. During the processing of the cluster_config command, the swmlogger gconfig script prompts you to supply the network type of your system. The network type describes the topology of your system, and it is used to create the qsnet diagnostics database. The Quadrics diagnostic database utilities will not work properly if you specify an incorrect network type.
K Reinstalling An XC System with Version 2.1 This appendix describes how to reinstall HP XC System Software Version 2.1 on a system that is already running Version 2.1. Reinstalling an XC system with the same release may be necessary if you participated as a field test site of an advance development kit. The procedure differs depending upon your cluster platform architecture: • Opteron- or Xeon-based (Section K.1) • Itanium-based systems (Section K.2) K.
3. Prepare the specified nodes to network boot instead of booting from local disk: # setnode --resync n[1-5] 4. Reboot nodes n1 through n5: # stopsys n[1-5] # startsys n[1-5] The nodes automatically reboot when the reimaging is complete. 5. Reset the job state on nodes n1 through n5: # scontrol update NodeName=n[1-5] State=IDLE K.2 Reinstalling Itanium-Based Systems The information in this section describes the following tasks: • How to reinstall the entire system (Section K.2.
c. • If all boot labels are consistently named Netboot, there is no need to change anything, proceed to step 4, • If the boot label names for all nodes are not consistently named Netboot, proceed to step 3c to change the label for all nodes that do not have the label Netboot. Enter a command similar to the following to remove the inconsistent or incorrect boot labels.
nodes to network boot from the DHCP server and automatically starts a reinstallation operation on the node or nodes by temporarily modifying the EFI environment to allow the node or nodes to network boot. This procedure assumes nodes 1 through 5 (n1–5) are to be reinstalled. 1. Begin this procedure as the root user on the head node. 2. Use the scontrol command to ensure that all jobs are drained from nodes n1 through n5: # scontrol update NodeName=n[1-5] State=DRAIN Reason="system shutdown" 3.
n1: Boot0000* Linux n1: Boot0001* Netboot n4: n4: n4: n4: n5: n5: n5: n5: 4. . . . BootCurrent: 0000 BootOrder: 0000,0001 Boot0000* Linux Boot0001* Netboot BootCurrent: 0000 BootOrder: 0000,0001 Boot0000* Linux Boot0001* Netboot . . .
L Installing the Maui Scheduler This appendix describes how to install and configure the Maui Scheduler to interoperate with SLURM on an HP XC system. L.1 Overview The Maui Scheduler is a software tool designed to allow flexible and near-optimal scheduling in a multiresource, high-performance computing environment.
Review the Maui Documentation Before installing and configuring the Maui Scheduler on your XC system, review the information available at the following Web sites: • http://www.chpc.utah.edu/docs/manuals/software/maui.html • http://supercluster.org/research/papers/xlinux99.shtml • http://www.clusterresources.com/products/maui/ Ensure That LSF HPC Is Not Activated HP does not support the use of the Maui Scheduler with the HP XC version of LSF HPC.
The following summarizes the steps to install and configure the Maui Scheduler to work with SLURM on an HP XC cluster: 1. Download the Maui Scheduler kit (Section L.4.1). 2. Compile the Maui Scheduler from its source distribution (Section L.4.2). 3. Edit the Maui configuration file (Section L.4.3). 4. Edit the SLURM configuration file (Section L.4.4). 5. Configure the Maui Scheduler (Section L.4.5). L.4.
4. Determine the SLURM partition name. In order for the Maui Scheduler to access your SLURM partition, you must define a partition in the maui.cfg file with the same name as your SLURM partition. To determine the SLURM partition name, view the /hpcd_cluster/slurm/etc/slurm.conf file, and find the following line: PartitionName=PARTA Default=YES Shared=FORCE Nodes=n[13-16] 5. Define a partition in the maui.cfg file with the same name as your SLURM partition. To do this, update the maui.
0 Idle Jobs BLOCKED JOBS---------------JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME Total Jobs: 1 Active Jobs: 1 Idle Jobs: 0 Blocked Jobs: 0 • Show a detailed status report for a specific job: root@n16: checkjob 116 checking job 116 State: Running Creds: user:root group:root qos:DEFAULT WallTime: 00:00:00 of 1:00:00 SubmitTime: Tue June 28 20:30:37 (Time Queued Total: 00:00:06 Eligible: 00:00:06) StartTime: Tue June 28 20:30:43 Total Tasks: 6 Req[0] TaskCount: 6 Partition: lsf Network: [NONE] Mem
Glossary A Administrative Network The private network within the XC system that is used for administrative operations. admin branch The half (branch) of the Administrative Network that contains all of the general-purpose admin ports to the nodes of the XC system. B base image The collection of files and directories that represents the common files and configuration data that are applied to all nodes in an XC system. branch switch A component of the Administrative Network.
extensible firmware interface See EFI external network node A node that is connected to a network external to the XC system. F fairshare An LSF HPC job-scheduling policy that specifies how resources should be shared by competing users. A fairshare policy defines the order in which LSF HPC attempts to place jobs that are in a queue or a host partition. FCFS First come first served.
image server A node specifically designated to hold images that will be distributed to one or more client systems. In a standard XC installation, the head node acts as the image server and golden client. Integrated Lights Out See iLO interconnect Provides high-speed connectivity between the nodes. It is used for message passing and remote memory access capabilities for parallel applications.
LSF master host The overall LSF coordinator for the system. The master load information manager (LIM) and master batch daemon (mbatchd) run on the LSF master host. Each system has one master host to do all job scheduling and dispatch. If the master host goes down, another LSF server in the system becomes the master host. LVS Linux Virtual Server. Provides a centralized login capability for system users. LVS handles incoming login requests and directs them to a node with a login role.
P parallel application An application that uses a distributed programming model and can run on multiple processors. An HP XC MPI application is a parallel application. That is, all interprocessor communication within an HP XC parallel application is performed through calls to the MPI message passing library. PXE Preboot Execution Environment. A standard client/server interface that allows networked computers that are not yet installed with an operating system to be configured and booted remotely.
symmetric multiprocessing See SMP Glossary-6
Index A adduser command, 4-10 B back up database, 5-5 /boot file system, 3-3 /boot/efi file system, 3-3 C checklist of installation tasks, A-1 client nodes default disk partition layout, 4-14 cluster configuration modifying, I-1 cluster platform CP3000, 3-3 CP4000, 3-3 CP6000, 3-3 cluster_config command, 4-14 text based menu, I-1 cluster_prep command, 4-5 commands cluster_config, 4-14 cluster_prep, 4-5 discover, 4-5, E-2 managedb, 5-5 startsys, 4-22 common role, I-6 compilers, D-1 MLIB dependency, D-2 com
guidelines, C-1 HP SFS, 3-8 HP StorageWorks Scalable File Share ( See HP SFS ) /hptc_cluster file system, 3-3 I I/O role, I-4 iLO settings on DL585, F-1 InfiniBand interconnect connecting to the switch monitoring line card, G-6 log into line monitoring card, G-6 node-list file, 4-24 switch controller card, G-5 installation additional RPMs, 3-6 additional software, 3-8 checklist, A-1 head node, 3-2 maui scheduler, L-1 overview, 3-2 preparing for, 2-1 procedure, 3-5 reinstalling an XC system, K-1 SAN stora
pam_slurm module, 4-13 partition layout for SLURM, 4-22 password guidelines, C-1 patches, 3-7 PGI compiler, 2-3, D-1 power on nodes, 4-22 processor architectures, 3-3 ProCurve switch, E-2 does not obtain IP address, E-2 program development tools, D-1 propagate golden image, 4-22 Q configuration file, 4-21 partition layout, 4-22 post-configuration script, 4-23 slurm.conf file, 4-21 slurm.
