HP XC System Software Administration Guide Part Number: AA-RWJYB-TE June 2005 Product Version: HP XC System Software Version 2.1 This manual describes the tools and procedures necessary to administer, monitor, and maintain an HP XC system.
© Copyright 2003-2005 Hewlett-Packard Development Company, L.P. AMD and AMD Opteron are trademarks or registered trademarks of Advanced Micro Devices, Inc. FLEXlm is a trademark of Macrovision Corporation. InfiniBand is a registered trademark and service mark of the InfiniBand Trade Association. Intel , the Intel logo, Itanium, Xeon, and Pentium are trademarks or registered trademarks of Intel Corporation in the United States and other countries. Linux is a U.S. registered trademark of Linus Torvalds.
Contents About This Document 1 HP XC Administrative Environment 1.1 1.2 1.3 1.4 1.4.1 1.4.2 1.4.3 1.4.3.1 1.4.3.2 1.5 1.6 1.6.1 1.6.2 1.6.3 1.6.4 1.7 1.8 1.8.1 1.8.2 1.8.3 1.8.4 1.9 1.10 1.11 1.12 2 1-1 1-2 1-3 1-3 1-3 1-5 1-5 1-5 1-6 1-7 1-8 1-8 1-8 1-9 1-9 1-9 1-9 1-10 1-11 1-11 1-12 1-12 1-12 1-13 1-13 Starting Up and Shutting Down the HP XC System 2.1 2.2 2.3 2.4 2.4.1 2.4.2 2.4.3 2.4.4 2.4.5 2.5 3 Understanding Nodes, Roles, and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.3 3.3 3.4 3.5 3.5.1 3.5.2 3.5.3 3.6 3.7 4 Accessing the Configuration and Management Database . . . . . . . . . . . . . . . . . . . . . . . . . Querying the Configuration and Management Database . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying Configuration Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying the Nodes That Provide a Specified Service . . . . . . . . . . . . . . . . . . . . .
7.2.1 7.2.2 7.2.3 7.3 7.3.1 7.3.2 7.3.3 7.4 7.4.1 7.4.2 7.4.3 7.5 8 8-1 8-1 8-1 8-2 8-2 8-3 Console Management Facility (CMF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Connecting to a Remote Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1 9-1 Administering Local User Accounts 10.1 10.2 10.3 10.4 10.5 10.6 11 Open Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12 LSF-HPC Administration Integration of LSF-HPC with SLURM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Job Starter Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.1 SLURM External Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.2 SLURM lsf Partition . . . . . . . . . . . . . . . . . . . . . .
15.4.2.1 15.4.2.2 15.4.2.3 15.4.2.4 15.4.3 16 The swmlogger Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The qselantestp Diagnostic Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The qsnet2_level_test Diagnostic Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The qsnet2_drain_test Diagnostic Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1-2 1-3 3-1 6-1 8-1 8-2 11-1 11-2 12-1 12-2 14-1 viii Contents Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recommended Administrative Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HP XC System Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
About This Document This manual describes the procedures and tools that are required to maintain the HP XC system. It provides an overview of the administrative environment and describes administration tasks, node maintenance tasks, Load Sharing Facility (LSF®) administration tasks, and troubleshooting information. An HP XC system is integrated with several open source software components. Some open source software components are being used for underlying technology, and their deployment is transparent.
• Chapter 12 provides a brief overview of the Platform Load Sharing Facility (LSF) application and describes common LSF administration tasks. • Chapter 13 describes how to manage modulefiles. • Chapter 14 provides information and procedures for mounting internal and external file systems. • Chapter 15 discusses diagnostic tools available for the HP XC system. • Chapter 16 provides troubleshooting information. • Chapter 17 describes administrative tasks for adding and replacing nodes.
HP XC Program Development Environment The following URL provides pointers to tools that have been tested in the HP XC program development environment (for example, TotalView® and other debuggers, compilers, and so on): ftp://ftp.compaq.com/pub/products/xc/pde/index.html HP Message Passing Interface HP Message Passing Interface (MPI) is an implementation of the MPI standard for HP systems. The home page is located at the following URL: http://www.hp.
• - Quick Reference Card - Running Jobs with Platform LSF http://www.llnl.gov/LCdocs/slurm/ Home page for the Simple Linux Utility for Resource Management (SLURM) , which is integrated with LSF to manage job and compute resources on an XC system. • http://www.nagios.org/ Home page for Nagios®, a system and network monitoring application. Nagios watches specified hosts and services and issues alerts when problems occur and when problems are resolved.
If you are not sure about a command you need to use, enter the man command with the -k option to obtain a list of commands that are related to the keyword. For example: # man -k keyword Related Information This section provides pointers to the Web sites for related software products and provides references to useful third-party publications. The location of each Web site or link to a particular topic is subject to change without notice by the site provider. Related Linux Web Sites • http://www.redhat.
Additional Publications For more information about standard Linux system administration or other related software topics, refer to the following documents, which must be purchased separately: • Linux Administration Unleashed, by Thomas Schenk, et al. • Managing NFS and NIS, by Hal Stern, Mike Eisler, and Ricardo Labiaga (O’Reilly) • MySQL, by Paul Debois • MySQL Cookbook, by Paul Debois • High Performance MySQL, by Jeremy Zawodny and Derek J.
| In command syntax and examples, a pipe character ( | ) separates items in a list of choices. discover(8) A cross-reference to a manpage includes the appropriate section number in parentheses. For example, discover(8) indicates that you can find information on the discover command in Section 8 of the manpages. Ctrl/x In interactive command examples, this symbol indicates that you hold down the first named key while pressing the key or button that follows the slash ( / ).
1 HP XC Administrative Environment This chapter introduces the HP XC Administrative Environment and discusses key components. The following topics are covered in this chapter: • A description of nodes and services (Section 1.1) • A discussion of Administrator passwords (Section 1.2) • A description of the secure shell as a means to access the HP XC system (Section 1.3) • An explanation of the HP XC administrative commands (Section 1.
• LVS Director Service The LVS Director is the node that controls the placement of the user logins on the LVS ring. • Login The login service is assigned to any node in the Linux Virtual Server (LVS) ring. LVS assigns a user login to one of these nodes whenever that user requests a login to the system through the system name or Internet Protocol (IP) address. A login node requires a connection to the external network. • I/O Service An I/O node provides I/O services for the system.
Root password Used to access the superuser account, to perform system administration, and to invoke management tools. Database administrator password Used for changes to the configuration management database. Administrator password for ProCurve switches Used for configuration of system administration switches. One password should be set for all ProCurve switches in the system. This password should have been set when the hardware was prepared before installation.
Table 1-1: HP XC System Commands 1-4 Command Description cexec The cexec command is a shell script that invokes the pdsh command to perform commands on multiple node in the system. Manpages: cexec(1), lsf_diff(1) cluster_config Thecluster_config allows you to view and modify the default role assignments and node configuration, modify the default role assignments on any node, and add, modify, or delete Ethernet connections to any node except the head node.
Table 1-1: HP XC System Commands (cont.) Command Description setnode Use this command to set the node settings in the configuration and management database. With this command, you can enable or disable nodes, or cause a re-imaging of specified nodes. Manpage: setnode(8) shownode This command has a variety of subcommands that provides information about nodes and node information.
given number of nodes, or only certain nodes to perform the command or commands passed as arguments to the pdsh command. The pdsh shell relies on the ssh transport mechanism. Ensure that ssh is properly installed on the system; changes to the ssh configuration might cause the pdsh shell to fail. Although the pdsh shell is capable of using several remote shell services, including rsh and ssh, the security settings of the HP XC system make ssh the shell of choice.
# cexec -a "users" n4: root db dal guest n3: cjg n2: rc wk n1: root tcr The following example runs the who --count command on nodes n12 and n25: # cexec -a "who --count" n12: root bg rmk n12: # users=3 n25: root wra guest spg n25: # users=4 See the cexec(1) manpage for additional information. 1.5 Understanding the Configuration and Management Database The HP XC system stores information about the nodes and system configuration in the configuration and management database (cmdb).
1.6 Networking The HP XC system uses the Linux Virtual Server (LVS), Network Time Protocol (NTP), Network Address Translation (NAT), and Network Information Service (NIS). 1.6.1 Linux Virtual Server (LVS) for HP XC System Alias The HP XC system uses the Linux Virtual Server (LVS) to present a single host name for user logins. LVS is a highly scalable virtual server built on a cluster of real servers.
1.6.3 Network Address Translation The HP XC system uses Network Address Translation (NAT) to allow nodes in the HP XC system that do not have direct external network connections to open outbound network connections to external addresses. NAT enables application nodes to access network-available resources without the additional network management, resources, or load of making application nodes part of an external public network. 1.6.
HP XC specific software /opt/hptc (Section 1.8.2) HP XC configuration data /opt/hptc/etc (Section 1.8.3) Systemwide file system /hptc_cluster (Section 1.8.4) 1.8.1 General File System Layout The HP XC system relies on key files. Interfering with these files may cause the system to fail. The best way to avoid this situation is to respect the placement of directories and files, especially when installing software packages.
1.8.2 HP XC File System Layout Software installed in the /opt/hptc directory is for the exclusive use of the HP XC System Software. Do not install or replace any other software in this directory, unless it is an officially supported patch. Software packages are installed in directories under the /opt/hptc directory under their own names. ________________________ Caution _______________________ Do not add, replace, or remove any files in the /opt/hptc directory.
1.8.4 Systemwide Directory A systemwide directory is mounted at /hptc_cluster. This directory contains configuration and log file information that is applicable across the system. Log files can be found under /hptc_cluster/adm/logs. ________________________ Caution _______________________ Do not add, replace, or remove any files in this directory. Doing so will cause the HP XC system to fail. 1.9 Log Files Components that comprise the HP XC system generate their own log files.
The module command features keywords that enable you to load, unload, and list modules, as described here: • Use the load keyword to load a module: # module load package-name • Use the list keyword to list all loaded modules: Be sure to unload a module before loading another to help avoid module versioning conflicts. See the HP XC System Software User’s Guide for additional information.
Table 1-3: Recommended Administrative Tasks (cont.) When Task Reference Frequently Check the Nagios Web interface to monitor the system status.
2 Starting Up and Shutting Down the HP XC System This chapter describes the procedures for starting up and shutting down an HP XC system: • Starting up the HP XC system (Section 2.1) • Shutting down the HP XC system (Section 2.2) • Shutting down individual nodes (Section 2.3) • Managing node power (Section 2.4) • Disabling and enabling nodes (Section 2.5) 2.
________________________ Notes ________________________ Be sure to use the --full option if you want to shut down the head node. By default, the stopsys command keeps the head node up. When you attempt to shut down the HP XC system from a node other than the head node, a warning message is displayed and the operation stops. The --force option of the stopsys command overrides this condition and enables you to continue the system shutdown.
You can also use the shownode status command instead of the power status command: # shownode status n2 ON 2.4.2 Turning Node Power On The power command allows you to turn on power to a set of nodes. You can turn on power to a node from any other node, but you must be superuser to do so. Invoke the power command with the --on option and designate the node or nodes: # power --on nodelist 2.4.3 Turning Node Power Off You can turn off power to a specific node by using the power command.
the power command with the --uidoff option instead of using the locatenode command with the --off option. 2.5 Disabling and Enabling a Node You can disable one or more nodes in the HP XC system. Disabled nodes are ignored when the HP XC system is started or stopped with the startsys and stopsys commands, respectively. The setnode --disable command updates the cmdb to indicate a specified node or a group of nodes as disabled.
3 Managing System Services This chapter describes the HP XC system services and the procedures for their use. These topics are covered in this chapter: • A discussion on HP XC system services and the service command (Section 3.1) • Information on the HP XC system commands to display information on which services are present, which nodes provide a given service, and which services are provided on a given node (Section 3.2) • Restarting a service (Section 3.3) • Stopping a service (Section 3.
Table 3-1: HP XC System Services 3-2 Service Function Database Name Console Management Facility Master Communicates with node consoles. cmfd Console Management Facility Client Client for the Console Management Facility cmf_client Database Server Runs the management database server. dbserver DHCP Server Assigns IP addresses to nodes. dhcp Myrinet Switch Monitor Monitors the Myrinet Switch.
Table 3-1: HP XC System Services (cont.) Service Function Database Name Slurm Central Control Service Manages nodes, partitions, and jobs slurm_controller Slurm Launch Allows user to launch jobs on nodes with slurm_compute service. slurm_launch Supermon System Monitoring daemon mond Supermon aggregator Gathers information from subordinate nodes running Supermon. supermond Syslog-ng Next Generation system logging service.
slurm_compute: n[1-3] slurm_controller: n3 supermon: n3 supermond: n3 swmlogger: n3 syslogng: n[1-3] syslogng_forward: n3 See the shownode(8) manpage for more information. Alternatively, you can obtain an extensive list of all services running on a given node by invoking the following command: # service --status-all 3.2.2 Displaying the Nodes That Provide a Specified Service You can use the shownode servers command to display the node or nodes that provides a specific service to a given node.
The shownode services node client command does not display any output if there is no client. Another keyword, servers, allows you to determine the node that provides a specified node its services. In this example, the node n3 is found to supply itself with the ntp service: # shownode services n3 servers ntp: n3 The shownode services node server command displays no output if there is no server. See the shownode(8) manpage for more information. 3.
• The first stanza lists all the roles: [global] roles = <
. . . swmlogger newservice EOT b. Add the name of the new service to the stanza for the appropriate role. This example uses the common role. common = <
. . . common newrole EOT b. Add the name of the new service to the stanza that lists all the services: services = <
3.7 NTP Service The head node acts as the primary Network Time Protocol (NTP) internal server for all other nodes in an HP XC system. Other nodes are NTP clients of the head node. You can specify where the NTP server obtains its time reference during the configuration of the HP XC system; up to three external time servers are accepted if the internal server is connected to an external network. You can also specify the internal server’s clock as the time reference instead.
4 Managing Licenses This chapter describes the following topics: • A description of the license manager and the license file (Section 4.1) • A means to determine if the license manager is running (Section 4.2) • Instructions on how to start and stop the license manager (Section 4.3) 4.1 Introduction The license manager service runs on the head node and maintains licensing information for software on the HP XC system.
The output of the lmstat command depends on the licensed products and on the number of CPUs licensed for the HP XC system. 4.3 Starting and Stopping the License Manager The service command enables you to start, stop, and restart the license manager service from the head node. 4.3.1 Starting the License Manager Use the following command to start the license manager: # service hptc-lm start 4.3.2 Stopping the License Manager Use the following command to stop the license manager: # service hptc-lm stop 4.
5 Managing the Configuration and Management Database The configuration and management database, cmdb, is key to the configuration of the HP XC system. It keeps track of which nodes are enabled or disabled, the services that a node provides, the services that a node receives, and so on. This chapter contains the following information: • Accessing the cmdb (Section 5.1) • Querying the cmdb (Section 5.2) • Determining the value of a system attribute and setting it (Section 5.
5.2.1 Displaying Configuration Details The shownode command enables you to list the configuration details for nodes and other hardware. # shownode config all: cp_ports: cp-n1: cp_type: IPMI host_name: cp-n1 hwaddr: 00:00:1a:19:6f:53 ipaddr: 172.21.0.1 location: Level 1 Switch 172.20.65.2, Port 40 netmask: 255.0.0.0 cp-n2: cp_type: host_name: hwaddr: ipaddr: location: netmask: IPMI cp-n2 00:00:1a:19:70:bb 172.21.0.2 Level 1 Switch 172.20.65.2, Port 41 255.0.0.0 golden_client: n3 . . . 5.2.
5.4 Backing Up the Configuration Database The managedb backup command allows you to copy all the data from the configuration and management database into a backup file; the database remains intact during the backup. You can use this backup file to restore the database at a later time, if needed. You must be superuser to use this command. Here is an example of the managedb backup command that backs up the cmdb.
Time Parameter Archive (or Purge) All Metrics Data: nw Older than n weeks ny Older than n years The following example archives metrics data older than four days to an archive file named arfile_tuesday in the current directory: # managedb archive --archive ./arfile_tuesday 4d Be sure to archive the configuration and management database on a regular basis (at least monthly) to save disk space. See the managedb(8) manpage for more information. The archive.
6 Monitoring the System System monitoring can identify situations that can become problems later. This chapter discusses the following topics: • An overview of the monitoring tools (Section 6.2) • How to observe system environmental data (Section 6.3) • How to view system statistics (Section 6.4) • How to customize Nagios metrics gathering (Section 6.5) • Organization of the logging of node events and how to view those events (Section 6.6) 6.
• shownode metrics paging • shownode metrics sensors (supported only on the XC6000 only) • shownode metrics swap You can use either the pdsh command or the cexec command to execute these commands remotely on another node. See Section 1.4.3 for more information. See Section 6.4 and the shownode(8) manpage for more information. 6.2.2 Nagios The HP XC System Software uses the Nagios Open Source monitoring application to gather and display system statistics, such as processor load and disk usage.
Table 6-1: Monitored Services (cont.
Figure 6-2: Nagios Main Window You can choose any of the options on the left navigation bar, but you will be asked for a login and a password initially. This login and password are established when the HP XC system is configured. Figure 6-3: Nagios Login Screen The Nagios passwords are maintained in the /opt/hptc/nagios/etc/htpasswd.users file; use the htpasswd command to manipulate this file to add a user, to delete a user, or to change the user password. Nagios offers various views of the HP XC system.
For example, Nagios monitors the four hosts and two switches in an HP XC system, and reports on the status of six hosts. Keep this in mind when using the Nagios utility. The HP XC System Software provides plug-ins that monitor these and other system statistics. Ensure that the following services are available: • httpd The Nagios utility continues to run even if this service is stopped, but you will not be able to use the web interface.
Figure 6-4: Nagios Menu Additional information about the Nagios Service Status for All Hosts window and related topics are available by selecting Documentation and by visiting the Nagios Web site at following Web site: http://www.nagios.org . You can also use the shownode metrics sensors command to display some environmental data. See Section 6.4.1 for more information. The hpasm package is provided and enabled by default on all nodes with iLO management.
displayed in degrees Celsius, and fan speeds are noted in terms of revolutions per minute. Portions of this output were truncated.
6.4.4 Monitoring Paging and Swap Data from the Command Line The shownode metrics paging command displays the paging data for the node on which this command is issued. # shownode metrics paging Timestamp |Node |pgpgin |pgpgout |pswpin |pswpout ----------------------------------------------------------------date and time stamp |n5 |6098711 |21758456 |175 |1044 The shownode metrics swap command displays swap space total and usage for the node on which the command is issued.
contact_groups notification_interval notification_period notification_options check_command register } _____________________ admins 240 24x7 w,u,c,r getMetrics!paging!cpuinfo! cputype!btime!processes! netinfo!meminfo!swapinfo! time!switch!cputotals! sensors!avenrun 1 Note _____________________ The check_command entry of the service definitions is a one-line entry. It is displayed here on multiple lines to fit on the page. b.
define service{ use host_name name service_description is_volatile check_period max_check_attempts normal_check_interval retry_check_interval contact_groups notification_interval notification_period notification_options check_command register } i. 3. nagios %HOST% gatherSensors Gather supermond sensors 0 24x7 3 60 2 admins 240 24x7 w,u,c,r getMetrics!sensors 1 Save the file. Invoke the following command to reconfigure the Nagios service: # service nagios nconfigure 4.
Destinations Contains the devices and files where the messages are sent or saved. Logs Combines the sources, filters, and destination into specific rules to handle the different messages. The syslog-ng tool provides a basic set of filters that process messages; the following table indicates where specific message types are sent.
This example indicates that two nodes, n3 and n4, supply the syslogng_forward service. The global node, which is the head node, also provides this service. By using the headnode command to rule out the head node, you can identify the regional node or nodes. # headnode n4 In the previous example, the global node is n4, and there is only one regional node, n3. This three-tiered approach arranges the log files in order of importance as defined by the syslog-ng.conf rules file.
7 Distributing Software Throughout the System This chapter includes the following topics: • A general overview on the image replication and distribution environment (Section 7.1) • A discussion on managing software changes (Section 7.2) • Information on updating the HP XC system’s golden image (Section 7.3) • A description of the methods for propagating software updates to all nodes of the HP XC system (Section 7.
Client nodes synchronize to the golden client by way of distribution of the golden image. When you use the updateclient command to update nodes with a new golden image, only files that have changed in the golden image between the current installation and the new image are distributed; this saves time and network bandwidth.
7.2.2 Using File Overrides to the Golden Image File overrides overwrite files delivered in the golden image. File overrides are copied to the client nodes after the golden image is transferred, thus overriding files in the golden image itself. File overrides are organized in a subdirectory hierarchy rooted at /var/lib/systemimager/overrides. Each subdirectory of the ./overrides directory specifies a complete file override hierarchy that is copied to the root directory of the client’s file system.
As mentioned above, the database should be the destination for per-node specific data. However, some existing services may already place configuration data used by the service in files in known locations. The golden client file system and the cluster common storage are available to support such applications. Global service configuration scripts are located in the /opt/hptc/etc/gconfig.d directory.
The following command makes a copy of the default golden image, base_image, in the /var/lib/systemimager/images directory. The saved image name in this example is base_image.orig. This command must be run on the image server node, which is the head node. # cpimage --verbose --directory /var/lib/systemimager/images base_image base_image.orig If you are preserving multiple images, save earlier versions as compressed archives using your favorite compression utility to preserve disk space on the image server.
nodes must be operational. The updateimage command displays any nodes that could not be reached, and thus not set to network boot. This may be a problem only on XC6000 systems because each node’s EFI environment must be modified. You must resolve those nodes’ EFI environments manually. The -no-netboot option of the updateimage command overrides this default action and maintains the boot order for each client node.
_______________________ Note _______________________ The updateclient.local.exclude file is used specifically by the updateclient utility; it has no effect when nodes are reimaged as a result of a client autoinstall operation. The updateclient.local.exclude file is resident on each node and referenced locally during the updateclient operation. You should update this file as you would update other software throughout the HP XC system; that is, edit the /etc/systemimager/updateclient.local.
4. Run the updateclient utility in conjunction with the cexec command to distribute the golden image to all client nodes: # cexec -x ‘nodename‘ -a "updateclient \ --server nh --image base_image --no-bootloader" 5. Run the per-node configuration service on the head node, as follows, to update it: # service nconfig nconfigure # service nconfig nstart 6.
3. Install the software on the golden client. Note the names of the services that are created as a result. 4. Create global and node-specific configuration files (gconfig/nconfig) or both, if the services will run only on some specialized nodes. 5. Place the gconfig files in /opt/hptc/etc/gconfig.d, and the nconfig files in /opt/hptc/etc/nconfig.d, using an appropriate priority ordering based on the prefix of the file name. 6. Update the /opt/hptc/systemimager/etc/chkconfig.
8 Opening an IP Port in the Firewall This chapter includes the following topics: • A discussion on the IP ports that are open by default (Section 8.1) • Information on how to open IP ports (Section 8.2) 8.1 Open Ports Each node in an HP XC system is set up with an IP firewall, for security purposes, to block communications on unused network ports. External system access is restricted to a small set of externally exposed ports.
Table 8-2: Open Internal Ports in the Firewall Port Service Protocol Use 22 ssh tcp Secure user logins and file transfers 25 smtp tcp Mail server 69 tftp udp Trivial transfer protocol 111 sunrpc udp/tcp RPC-based code 123 ntp udp/tcp Network Time Protocol 443 https tcp Secure Hypertext Transfer Protocol 514 syslog tcp System logger 873 rsync tcp rsync utility 1024 to 65535 various tcp/udp Required for SLURM and LSF-HPC The default setup has all other ports on all inte
The list of interfaces specified by the --interface option must not contain any space characters. # openipport --port 44 --protocol udp \ --interface Admin,Interconnect,lo --verbose The following example also opens port 44 in the firewall on node n3; this example uses the same protocol and interface options as the previous example. The cexec command is used to update node n3 and ensure that a log file records this command.
This portion of the /etc/sysconfig/iptables.proto file should resemble the following: # set up port 389 on Interconnect interface: -A RH-Firewall-1-INPUT -i Interconnect -p tcp -m tcp --dport 389 -j ACCEPT # setup port 389 on admin interface -A RH-Firewall-1-INPUT -i Admin -p tcp -m tcp --dport 389 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited 3.
9 Connecting to a Remote Console This chapter contains the following topics: • Information on the Console Management Facility (Section 9.1) • A procedure for connecting to a remote console (Section 9.2) 9.1 Console Management Facility (CMF) The Console Management Facility (CMF) daemon runs on the head node and collects and stores console output for all other nodes on the system. This information is stored individually for every node and is backed up periodically.
4. Enter the user name and password for that node’s console: n16 login: username passwd: password _________________________ Note _________________________ Do not press Ctrl/B to return to the Management Processor (MP) console menu. Doing so stops the Console Management Facility (CMF) from logging the console data for the node. Use the telnet utility to access a node’s MP menu. The output of the console command is logged in the /hptc_cluster/adm/logs/cmf.dated/date/console-name.
10 Administering Local User Accounts This chapter describes how to add, modify, and delete a local user account on the HP XC system. It discusses how: • To administer local user accounts (Section 10.1) • To add a local user account (Section 10.2) • To modify a local user account (Section 10.3) • To delete a local user account (Section 10.4) • To change the root password (Section 10.5) • To synchronize the Network Information Service (NIS) database (Section 10.6) 10.
• User identifier number (UID) A default value is assigned if you do not supply this information; this value is usually the next available UID. If you assign a value, first make sure that it is not already in use. • Group identifier number (GID) A default value is assigned if you do not supply this value. • Home directory A default value based on the login name is assigned automatically if you do not provide this information.
10.5 Changing the Root Password The following procedure describes how to change the root password and propagate that change throughout the HP XC system: 1. Log in as superuser on the head node. 2. Use the passwd command to change the password locally on the head node. At this time, the root password is changed only on the head node. 3.
5. Remove the temporary file: # rm -f /tmp/root_crontab 6. Verify the changes you made to the root user’s crontab file: # # # # crontab -l DO NOT EDIT THIS FILE - edit the master and reinstall. (/tmp/root_crontab installed on date time year) (Cron version -- $Id: crontab.c,v 2.13 vixie Exp $) . . .
11 SLURM Administration The HP XC system uses the Simple Linux Utility for Resource Management (SLURM). This chapter discusses issues specific to the HP XC system. • An overview of SLURM on the HP XC system (Section 11.1) • A discussion on SLURM configuration (Section 11.2) • An overview of limiting user access (Section 11.3) • Information on job accounting (Section 11.4) • A discussion on how to monitor and manage SLURM (Section 11.
SLURM allows you to collect and analyze job accounting information. Section 11.4.3 describes how to configure job accounting information on the HP XC system. Section 16.2 provides SLURM troubleshooting information. 11.2 Configuring SLURM The HP XC provides global and local directories for SLURM files: • The /hptc_cluster/slurm directory is the sharable location for SLURM files that need to be shared among the nodes. SLURM state files, job logging files, and the slurm.conf configuration file reside there.
Obtain the MemTotal value from /proc/meminfo, then divide that value by 1024 to determine the correct amount of RealMemory on a node. Use the cexec command to gather this information from all the nodes in the HP XC system: # cexec -a ’grep MemTotal /proc/meminfo’ SLURM provides a great deal of flexibility in configuring the compute nodes.
11.2.2 Configuring SLURM Servers The default assignment for the SLURM server — that is the node to run the SLURM slurmctld — is the head node; this assignment occurred during installation unless you selected a different node. By default, no backup slurmctld daemon is configured. Refer to the HP XC System Software Installation Guide for information about changing the choice of primary and backup nodes for SLURM by using the cluster_config utility. 11.2.
This might be reset as follows: NodeName=DEFAULT Procs=2 TmpDisk=1024 RealMemory=4000 NodeName=n[1-15] RealMemory=1000 Weight=8 NodeName=n[16-127,129] RealMemory=2000 Weight=16 NodeName=n[128,130-512] Feature="noswap" Procs=4 Weight=50 This reset accomplishes the following after SLURM reconfiguration: • The assignments NodeName=DEFAULT Procs=2 TmpDisk=1024 specify that, by default, all nodes have two processors and that 1 GB (1,024 MB) of temporary disk space is available on each node, unless otherwise sp
Table 11-2: SLURM Partition Characteristics (cont.) Characteristic Description MaxTime Specifies the maximum time limit (in minutes) allowed for jobs in this partition. The default is unlimited or -1. MinNodes Specifies the minimum number of nodes that may be allocated to any single job. The default is 1. Shared A text string that indicates whether node sharing for jobs is allowed. The string values are: State YES The node may be shared or not, depending on the allocation.
You must perform the following to enable the pam_slurm module. 1. Open the /opt/hptc/slurm/etc/nconfig.d/munge_config.pl file. 2. Search for the line containing the string my $ENABLE_PAM_SLURM = in this file. If the associated value is 1, access is already restricted. Otherwise, continue with this procedure. 3. Replace the following line my $ENABLE_PAM_SLURM = 0; with this one: my $ENABLE_PAM_SLURM = 1; 4. Save the file. 5. Distribute this file to all nodes; see Chapter 7 for more information.
• First nonzero error code returned by jobstep • Sum of system CPU time and user CPU time _________________________ Note _________________________ These statistics are gathered after each jobstep completes. 11.4.1 Using the sacct Command The sacct command enables you to analyze the system accounting data collected in the job accounting log file. As the superuser, you can examine the accounting data for all jobs and job steps recorded in the job accounting log.
11.4.2 Disabling Job Accounting Job accounting is turned on by default. Note that job accounting is required if you are using LSF. Follow this procedure to turn off job accounting: 1. Log in as the superuser on the SLURM server (see Section 11.2.2). 2. Use the text editor of your choice to edit the /hptc_cluster/slurm/etc/slurm.conf file as follows: a. Locate the parameter JobAcctType. JobAcctType=jobacct/log b.
• Any node from which you execute the sacct command _______________________ Note _______________________ Ensure that the log file is located on a file system with adequate storage to avoid file system full conditions. This example uses the file /hptc_cluster/slurm/job/jobacct.log, which is the default and recommended file. Ensure that the configured job accounting log directory exists. 3. Determine the value for the Job Account Polling Frequency parameter.
# for the data file # JobAcctLoc=/hptc_cluster/slurm/job/jobacct.log JobAcctParameters="Frequency=10" . . . g. 5. Save the file. Restart the slurmctld and slurmd daemons: # cexec -a "service slurm restart" 11.5 Managing SLURM The SLURM squeue, sinfo, and scontrol utilities and the Nagios system monitoring utility provide the means for monitoring and controlling SLURM on your HP XC system. 11.5.
Use the following command to return node n17 to service: # scontrol update nodename=n17 state=idle 11.6 Enabling or Disabling Quadrics QsNet II Support in SLURM SLURM has system interconnect switch support for Quadrics QsNet II. SLURM enables this support during installation if the Quadrics hardware is detected on the head node. Setting the SwitchType variable in the /hptc_cluster/slurm/etc/slurm.
12 LSF-HPC Administration The Platform Load Sharing Facility for High Performance Computing (LSF-HPC) product is installed and configured as an embedded component of the HP XC system during installation. This product has been integrated with SLURM to provide a comprehensive high-performance workload management solution for the HP XC system.
# controllsf show current LSF is currently up, and assigned to node n16 All LSF administration must be done from the LSF-HPC Execution Host. The lsadmin and badmin commands can only be run on this host; they are not intended to be run on any other nodes in the HP XC system and may produce false results if they are. When the LSF-HPC scheduler determines that it is time to dispatch a job, it requests an allocation of nodes from SLURM.
to make specific topology-based allocation requests. See the HP XC System Software User’s Guide for more information. 12.1.3 SLURM lsf Partition An lsf’ partition is created in SLURM; this partition contains all the nodes that LSF-HPC manages. This partition must be configured such that only the superuser can make allocation requests (RootOnly=YES). This configuration prevents other users from directly accessing the resources that are being managed by LSF-HPC.
virtual IP is an internal IP on the HP XC Administration network, and the virtual hostname is lsfhost.localdomain. The LSF-HPC Execution Host is configured to host the vIP, then the LSF-HPC daemons are started on that node. The Nagios infrastructure contains a module that monitors the LSF-HPC virtual IP.
• HP OEM licensing is configured. HP OEM licensing is enabled in LSF-HPC by adding the following string to the LSF-HPC configuration file, /opt/hptc/lsf/top/conf/lsf.conf. This tells LSF-HPC where to look for the shared object to interface with HP OEM licensing. XC_LIBLIC=/opt/hptc/lib/libsyslic.so • Access to LSF-HPC from every node in the cluster is configured. Configuring all nodes in the HP XC system as LSF-HPC floating client nodes makes available access to LSF-HPC from all nodes.
12.3.2 Shutting Down LSF-HPC At system shutdown, the /etc/init.d/lsf script also ensures an orderly shutdown of LSF-HPC, if LSF-HPC is not running on the head node. You can use the controllsf command, as shown here, to stop LSF-HPC regardless of where it is active in the HP XC system. # controllsf stop If LSF-HPC failover is enabled, LSF-HPC is restarted on the head node when the stopsys command is executed; this allows jobs to be queued.
This value represents the maximum value of disk space on the node with the least amount of disk space. Running a job on all nodes must account for this value. Here is an example of the LSF-HPC lshosts command. $ lshosts HOST_NAME type model lsfhost.loc SLINUX6 Opteron8 cpuf ncpus maxmem maxswp server RESOURCES 16.0 22 2048M Yes (slurm) The lshosts command reports a hyphen (-) for all the other load index and resource information.
Example 12-2 is a similar example but twenty processors are reserved. Example 12-2: Launching Another Job Without the JOB_STARTER Script Configured $ bsub -I -n20 hostname Job <21> is submitted to default queue . <> <> n120 In both the previous examples, processors were reserved but not used.
You can use the -l (long) option to obtain detailed information about a job, as shown in this example. $ bjobs -l 116 Job <116>, User , Project default, Status , Queue , Co mmand date time: Submitted from host , CWD <$HOME>, Ou tput File <./>, 8 Processors Requested; date time: Started on 8 Hosts/Processors <8*lsfhost.
12.9 LSF-HPC Failover The following sections discuss aspects of the LSF-HPC failover mechanism. 12.9.1 Overview of LSF-HPC Monitoring and Failover Support The LSF-HPC failover mechanism on the HP XC system requires two nodes with the resource_management role. One is selected as the primary LSF-HPC Execution Host and the others are considered backup nodes.
By default, the HP XC resource management system attempts to place the SLURM controller and the LSF execution host on the same node to constrain the use of system resources. If only one node has the resource management role, the LSF-HPC execution daemons and the SLURM control daemon both run on that node. If two nodes are assigned the resource management role, by default, the first node becomes the primary resource management node, and the second node is the backup resource management node.
REQUEUE_EXIT_VALUES=122 12.10 LSF-HPC Monitoring LSF-HPC is monitored and controlled by Nagios using the check_lsf plug-in. When LSF-HPC is down, the response of the check_lsf plug-in depends on whether LSF-HPC failover is enabled or disabled. When LSF-HPC failover is disabled The check_lsf plug-in returns an immediate failure notification to Nagios. When LSF-HPC failover is enabled The check_lsf plug-in decides if LSF-HPC is supposed to be running.
12.11 Enhancing LSF-HPC You can set environment variables to influence the operation of LSF-HPC in the HP XC system. These environment variables affect the operation directly and set thresholds for LSF-HPC and SLURM interplay. 12.11.1 LSF-HPC Enhancement Settings Table 12-2 describes the environment variables that may be used to enhance LSF-HPC.
Environment Variable Description LSF_HPC_NCPU_INCREMENT=increment This is the upper limit for the number of CPUs that changed since the last checking cycle. The default value is 0 (zero). LSF_HPC_NCPU_INCR_CYCLES=incr_cycles The incr_cycles parameter is the minimum number of consecutive cycles where the number of CPUs changed does not exceed LSF_HPC_NCPU_INCREMENT. The default value is 1. LSF-HPC checks total usable CPUs every two minutes.
SLURM affinity is enabled. The virtual hostname is "xclsf". 6. Edit the $LSF_ENVDIR/lsf.cluster.cluster_name file and change the LSF-HPC virtual hostname to the new one in the HOSTS section. 7. Edit $LSF_ENVDIR/hosts file to remove the old LSF-HPC virtual hostname entry. 8.
13 Managing Modulefiles This section describes how to load, unload, and examine modulefiles. 13.1 Managing Modulefiles Modulefiles provide a mechanism for accessing software commands and tools, particularly for third-party software. The HP XC System Software does not use modules for system-level manipulation. A modulefile contains the information that alters or sets shell environment variables, such as PATH and MANPATH.
14 Mounting File Systems This chapter provides information and procedures for performing tasks to mount file systems that are internal and external to the HP XC system. • A discussion of NFS on the HP XC system (Section 14.1) • A description of the global fstab file (Section 14.2) • A procedure for mounting file systems from a disk attached to one node of the HP XC system. to other nodes. (Section 14.3) • A procedure for mounting file systems from a remote system (Section 14.4 ) 14.
The fstab.proto file is comprised of individual portions that correlate to the nodes. The first portion is designated as ALL; all the lines in this portion are copied to the /etc/fstab file of all the nodes. Subsequent portions are designated for a specified node or nodes. Subsequent portions must begin with the lowest numbered node in the HP XC system and progress to the head node, which is the highest numbered node. The lines in each portion are copied to the corresponding node.
• Standard mounting instructions, in the manner of an /etc/fstab entry • Comment lines, each beginning with the # character • A node designator Lines that begin with the characters #% followed by a node designator indicate that the following mounting instructions pertain only to the specified nodes. For example, the node designator #% n2 specifies that all subsequent lines apply to the node n2, until either another node designator or the end of file is reached.
Figure 14-1: Internal File System Mounting Example HP XC Cluster . . . n59 n60 /scratch /dev/sdb1 n61 /scratch n62 /scratch n63 /scratch n64 . . . 14.3.1 Understanding the csys Utility in the Mounting Instructions The csys utility provides a facility for managing file systems on a clusterwide basis. It works in conjunction with the mount and umount commands by providing a pseudo file system type. The csys utility is documented in the csys(5) manpage.
options Specifies a comma-separated list of options, as defined in the csys(5) manpage. The hostaddress and _netdev options are mandatory. • The hostaddress argument specifies the node that serves the file system to other nodes and specifies the network (administration or system interconnect) to be used. The hostaddress is specified either by its node name or by its IP address. The following entry uses the node name for n60 over the administration network.
_______________________ Note _______________________ The node that exports the file system to the other nodes in the HP XC system must have the disk_io role. 3. Determine whether you need to mount this file system over the administrative network or over the system interconnect. As a general rule, specify the administration network for administrative data and the system interconnect on behalf of application data. 4. Edit the /hptc_cluster/etc/fstab.proto file as follows: a.
Example 14-2: The fstab.proto File Edited for Internal File System Mounting # This file contains additional file system information # for all # the nodes in the XC cluster. When a node # boots, this file will be parsed and from it a new # /etc/fstab will be created. # # Do not edit /etc/fstab! Edit this file # (/hptc_cluster/etc/fstab) instead.
Figure 14-2: Remote File System Mounting Example HP XC Cluster . . . n21 n22 External Server n23 xeno /extra n24 n25 . . . 14.4.1 Understanding the Mounting Instructions The syntax of the fstab entry for remote mounting using NFS is as follows: exphost: expfs mountpoint fstype options exphost Specifies the external server that is exporting the file system. The exporting host can be expressed as an IP address or as a fully qualified domain name.
xeno:/extra /workarea nfs rw,_netdev This means that the node with this fstab file entry mounts the file system named /extra, which is exported from the system named xeno, and mounts it at /workarea. The file system can be read from and written to. The _netdev option means that the file system is not mounted until the network is up. 14.4.2 Mounting Remote File Systems Use the following procedure to mount a remote file system to one or more nodes in an HP XC system: 1.
If the remote file mounting is successful, the output contains a line like the following line for each node that mounts the remote file system: nodename:exphost:expfs on mountpoint type nfs options In the example used here, the output resembles the following: n22: xeno:/extra on /workarea type nfs (rw,_netdev,addr=192.168.9.22) n23: xeno:/extra on /workarea type nfs (rw,_netdev,addr=192.168.9.23) n24: xeno:/extra on /workarea type nfs (rw,_netdev,addr=192.168.9.24) Example 14-3: The fstab.
15 Using Diagnostic Tools The HP XC system provides several diagnostic tools. These tools are discussed in this chapter, as follows: • A description of the sys_check utility and its use (Section 15.1) • A discussion on the ovp utility (Section 15.2) • A description of the dgemm utility (Section 15.3) • Individual discussions of diagnostics for the system interconnects (Section 15.4) Troubleshooting procedures are described in Chapter 16. 15.
15.2 Using the ovp Utility for System Verification The ovp utility ensures that the HP XC system is operational; it is recommended that you run this utility at the end of the initial installation process. By default, this utility includes the following tests to verify that: • The license key file is installed and licensing is working. • Network connectivity has been established. • The administrative network is operational. • All application nodes are responding and available to run applications.
Test list for ’license’: file_integrity server_status Test list for ’SLURM’: daemon_responds sinfo Test list for ’LSF’: identification Test list for ’interconnect’: quadrics/qsnet_database quadrics/swmlogger quadrics/network Test list for ’cluster’: xring (X) You can select the appropriate test by pattern. Also, you can enter multiple selections by separating them with commas.
n m l inner-iteration-value outer-iteration -value cpus-per-node cut-off detailed-output-flag The following table describes these parameters and indicates their default values.
InfiniBand http://www.voltaire.com Other tools have been written specifically for use with the HP XC system. To use the diagnostic tools, the system interconnect must be properly configured. The IP addresses must be configured and the /etc/hosts file must be updated with the switch names, for example MR0N00 for Myrinet system interconnect and QR0N00 for Quadrics system interconnect. These topics are discussed in the HP XC System Software Installation Guide.
This command is configured to run once each hour by a crontab file in the /etc/cron.hourly directory. 15.4.1.2 The gm_drain_test Diagnostic Tool This diagnostic tool runs five tests for the Myricom switches in an HP XC system. It must be launched from the head node and it should be run only during allocated preventive maintenance. The five tests are as follows: gm_prodmode_mon Checks environmental operating parameters. gm_allsize Tests network links. gm_debug Tests PCI bandwidth.
temperature, and link errors. It also handles alerts from the monitoring daemons on each node that generate messages. The MySQL database for qsnet is created automatically during the configuration phase of the HP XC system installation and the swmlogger daemon is configured to run on the head node. Use the qsdiagadm and the qsneterr utilities to report information from the qsnet database.
-clean Clears out the log file directory. This ensures that if the file already exists, the old data is deleted before running the new test to ensure that the data is fresh from the current run. The use of this option is recommended. -N nodes Specifies that you want to run this test only on a subset of nodes; the nodes parameter is a comma-separated list of nodes. The default is to run this test on all nodes.
-noparse Allows you to run the qsnet2_level_test to record the log files into the chosen directory, but it does not parse the log files to find and report errors automatically. This is particularly useful during drain-time testing, when there is only a 20- or 30-minute period in which to perform preventive maintenance testing. You can use the -parse option to verify the results at a later time. -v Specifies verbose output, which is required to identify which component or location is causing errors.
example, the test is run on rail 0 with the -clean option to remove data from a previous test that can corrupt the test results. # qsnet2_level_test level3 -d \ /hptc_cluster/adm/logs/diag/quadrics \ -N n1,n2 -r 0 -clean -v -t 180 Example 3 The following example displays the output of this diagnostic test, run on three nodes, when the timeout expires.
qsnet2_drain_test [-help] [-d logdirectory] The -help option displays the command line options. The -d option allows you to specify a log directory. The output from the qsnet2_drain_test utility and from individual tests are bundled in a tar file (compressed with the gzip utility) and placed in the specified log directory; the directory/var/log/diag/quadrics/qsnet2_drain_test is used by default if the -d option is not specified.
16 Troubleshooting This chapter provides information that helps you troubleshoot problems with your HP XC system: • A discussion on troubleshooting the system Interconnect (Section 16.1) • A list of issues regarding SLURM (Section 16.2) • A discussion on LSF-HPC issues (Section 16.3) See also Chapter 15 for information on available diagnostic tools, which can be also used to locate the source of the failure. 16.
5. Make sure all the Myrinet RPMs are installed. # rpm -q -a . . . gm-2.1.7_Linux-2.1hptc m3-dist-1.0.14-1 mute-1.9.6-1 . . . The version numbers may differ. 6. Run the lsmod command to display loaded modules. There should be one Myrinet GM loadable module installed. The size can vary. # lsmod | grep -i gm gm 7. 589048 3 The Myrinet myri0 interface should be up. Use the ifconfig command to display the interface network configuration.
5. Run the lsmod command to display loaded modules. There should be eight Quadrics loadable modules installed. The reported sizes can vary. # lsmod jtag eip ep rms elan4 elan3 elan qsnet 6. 7.
port_lid=0x0002 port_lmc=0x00 max_mtu=2048 port=2 port_state=PORT_DOWN sm_lid=0x0000 port_lid=0x0000 port_lmc=0x00 max_mtu=2048 2. Run the ib-setup command to verify that the configuration is correct. The output should be similar to that in the following example. # ib-setup ====== Voltaire HCA400 InfiniBand Stack Setup ====== Version: ibhost-v2.1.5_5_itapi: date on amt152. domain. System: kernel version: 2.4.21-15.10hp. XCsmp, memory 3595MB. Hostname: n3. IB configuration: AutoStart: on, SM: VFM.
gsi adaptor-tavor vverbs-base cm_2 gsi a daptor-tavor] mlog 64304 6 [ipoib-ud ats ibat cm_2] 148496 1 54576 0 [sdp ipoib-ud ats devucm q_mng ibat 9792 0 [sockets sdp ipoib-ud ats devucm q_mng ibat cm_2 gsi adaptor-tavor vverbs-base] 82208 0 [sockets sdp ipoib-ud ats q_mng ibat repository cm_2 gsi adaptor-tavor vverbs-base mlog] hadump 2240 0 [gsi] mod_ib_mgt 76992 0 (unused) mod_vapi 137824 0 [adaptor-tavor mod_ib_mgt] mod_vipkl 235776 0 [mod_vapi] mod_thh 248064 0 [mod_vapi] mod_hh 26896 0 [mod_vipkl mo
• Node partitions (only one by default) • Authentication (MUNGE is used by default ) SLURM uses the MUNGE package to authenticate users between nodes in the system. Both MUNGE and SLURM require files that contain encrypted keys. The names of the SLURM files are configured in the /hptc_cluster/slurm/etc/slurm.conf file; the MUNGE key file is /opt/hptc/munge/etc/keys/.munge_key.
Checking node and partition status Use the following command to check the status of your nodes and partitions: # sinfo --all 16.3 LSF Troubleshooting Take the following steps if you are have trouble submitting jobs or controlling LSF-HPC: • Ensure that the date is synchronized throughout the HP XC system. • Check that the /hptc_cluster directory (file system) is properly mounted on all nodes. SLURM relies on this file system. • Ensure that SLURM is configured, up, and running properly.
LSF-HPC failover attempts to ensure that a different node is used after it removes control from the present node. However, if all other options are exhausted, LSF-HPC failover tries the current node again before giving up. If you are trying to perform load balancing, log in to the primary LSF execution host node and execute the controllsf start here command from that node.
17 Servicing the HP XC System This section describes procedures for servicing the HP XC system. For more information, refer to the Service Guide for your system. The procedures discussed in this section are: • Adding a node (Section 17.1) • Replacing a node (Section 17.2) • Replacing a system interconnect card in an XC6000 system (Section 17.3) 17.1 Adding a Node The following procedure describes how to add one or more nodes in the HP XC system.
Modify the default role assignments depending upon your system requirements. When you are finished with your modifications, enter P to proceed: The cluster configuration is ready to be applied. Do it? [y/n] y b. When prompted, enter y to regenerate ssh keys: Root ssh keys for the cluster already exist (Warning: you will not be able to ssh/pdsh to other nodes until you reimage them) Would you like to regenerate them? ([n]/y) y c.
1. • Number of CPUs • Memory size • Number of ports Prepare the replacement node hardware as described in the HP XC System Software Hardware Preparation Guide. Note the following requirements for each replacement node: • The node must be set to boot from the network. • The console port (cp) is set to request IP addresses through DHCP. • The node’s power must be turned off. 2. Log in as superuser on the head node. 3.
17.3 Replacing a System Interconnect Card in an XC6000 System Use the following procedure to replace a Myrinet system interconnect card, InfiniBand system interconnect card, or a Quadrics system interconnect card in an XC6000 system. The example commands in the procedure use node n3. ________________________ Caution _______________________ The replacement system interconnect card must be the same as the system interconnect card to be replaced. 1. Log in as superuser on the head node. 2.
Glossary A Administrative Network The private network within the XC system that is used for administrative operations. admin branch The half (branch) of the Administrative Network that contains all of the general-purpose admin ports to the nodes of the XC system. B base image The collection of files and directories that represents the common files and configuration data that are applied to all nodes in an XC system. branch switch A component of the Administrative Network.
extensible firmware interface See EFI external network node A node that is connected to a network external to the XC system. F fairshare An LSF job-scheduling policy that specifies how resources should be shared by competing users. A fairshare policy defines the order in which LSF attempts to place jobs that are in a queue or a host partition. FCFS First come first served.
image server A node specifically designated to hold images that will be distributed to one or more client systems. In a standard XC installation, the head node acts as the image server and golden client. Integrated Lights Out See iLO interconnect The private network within the XC system that is used primarily for user file access and for communications within applications. interconnect Provides high-speed connectivity between the nodes.
LSF master host The overall LSF coordinator for the system. The master load information manager (LIM) and master batch daemon (mbatchd) run on the LSF master host. Each system has one master host to do all job scheduling and dispatch. If the master host goes down, another LSF server in the system becomes the master host. LVS Linux Virtual Server. Provides a centralized login capability for system users. LVS handles incoming login requests and directs them to a node with a login role.
P parallel application An application that uses a distributed programming model and can run on multiple processors. An HP XC MPI application is a parallel application. That is, all interprocessor communication within an HP XC parallel application is performed through calls to the MPI message passing library. PXE Preboot Execution Environment. A standard client/server interface that allows networked computers that are not yet installed with an operating system to be configured and booted remotely.
symmetric multiprocessing See SMP Glossary-6
Index A adding a local user account, 10-1 adding a node, 17-1 adding a service, 3-5 archive.cron script, 5-4 B bacct command, 12-9 base_image, 7-4 updating, 7-4 base_exclude_file, 7-6 bhist command, 12-9 /bin directory, 1-10 bjobs command, 12-8 bkill command, 12-9 bresume command, 12-9 bstop command, 12-9 C cexec command, 1-4t syslog-ng filter, 6-11 changing the root password, 10-3 check_lsf plug-in, 12-12 chkconfig.
disabling a node, 2-4 distributing file images to nodes, 7-1 procedure, 7-2 HP XC system booting, 2-1 shut down, 2-1 startup, 2-1 /hptc_cluster directory, 1-12, 3-2t, 7-3, E 16-6, 16-7 email contact by, 6-3 enabling a node, 2-4 environmental data, 6-5 /etc directory, 1-10 /etc/fstab.proto file, 14-1 /etc/systemimager/updateclient.local.
LSF-HPC controlling LSF-HPC service, 12-6 default user environment, 12-5 enhancing, 12-13 implementation, 12-1 installation details, 12-4 job accounting, 12-9 job submission, 12-7 load indexes, 12-6 resource information, 12-6 shutting down, 12-6 starter script, 12-2, 12-7 starting up, 12-5 troubleshooting, 16-7 LSF-HPC failover, 12-6, 12-10, 12-12, 16-7 running jobs, 12-11 LSF-HPC jobs controlling, 12-8 monitoring, 12-8 LSF-HPC monitoring, 12-12 LSF-HPC-SLURM integration, 12-1 LSF-HPC-SLURM thresholds, 12-1
console, 1-2 database administrator, 1-2 Nagios, 1-2 ProCurve, 1-2 root, 1-2, 10-3 pdsh command, 1-5 per-node service configuration, 7-3 ports, 1-4t NFS, 14-1 open external, 8-1 open internal, 8-1 opening, 8-2 privileged, 12-13 power command, 1-4t, 2-2, 2-3 power daemon service, 1-2 power management, 2-2 propagating file images to nodes, 7-1 Q qsdiagadm utility, 15-7 qselantestp utility, 15-7 qsnet2_drain_test utility, 15-10 qsnet2_level_test utility, 15-8 Quadrics system interconnect diagnostic tools, 15-
SystemImager, 1-2, 7-4n T /tmp directory, 1-10 troubleshooting InfiniBand, 16-3 LSF-HPC, 16-7 Myrinet system interconnect, 16-1 Quadrics system interconnect, 16-2 SLURM, 16-5 system interconnect, 16-1 U unit identifier LED, 2-3 updateclient utility, 7-6 updateimage command, 7-4, 7-5 updateimage utility, 1-5t updgi_exclude_file, 7-6 user accounts ( See local user accounts ) user authentication, 11-6 /usr directory, 1-10 /usr/bin directory, 1-10 /usr/local directory, 1-10 /usr/sbin directory, 1-10 utility
