HP XC System Software Administration Guide Version 3.1

lsf up infinite 122 idle n[5-16,18-127]
lsf up infinite 1 down n17
swaptest up infinite 4 idle n[1-4]
In this example, node n17 is down.
The squeue utility reports the state of jobs currently running under the SLURM's control. For more
information about the squeue utility, see squeue(1).
The SLURM log files on each node in /var/slurm/log are helpful for diagnosing specific problems.
The log files slurmctld.log and slurmd.log log entries from their respective daemons. Both these
log files have the following format:
[ date and time stamp] Log Entry
14.6 Draining Nodes
Use the SLURM scontrol command to change a node's state. SLURM provides DRAIN and DOWN states
for taking nodes out of service. Draining a node means that the current job is allowed to finish on that
node while no other jobs are scheduled for that node.
There are a variety of reasons why a node must be drained. For example, you may want exclusive use of
a node to perform diagnostics on it or you may need to replace it.
To drain one or more nodes use the scontrol command as follows:
# scontrol update NodeName=nodelist State=drain Reason="describe reason here"
See “The nodelist Parameter” (page 33) for a discussion on the use of the nodelist parameter.
The reason that you provide for the node draining is displayed by the sinfo command. Be brief but
descriptive.
Here, node n17 is drained so that it can be removed from service for maintenance:
# scontrol update nodename=n17 state=drain reason="maintenance"
After the node has drained, use the scontrol command to remove a node from service. The following
shows the command to remove the drained node in the example, node n17.
# scontrol update nodename=n17 state=down
The scontrol command returns nodes to an IDLE state so that they can be reused. The following command
places n17 in the IDLE state to return it to service:
# scontrol update NodeName=nodelist State=resume
When returning a node to service, HP recommends that you set the state to DRAIN, even if no jobs are
currently running. This has two advantages:
• It is easier to recognize nodes that are down unexpectedly when skimming the output of the sinfo
command.
• If the node is rebooted accidentally or a as part of the maintenance procedure, the DRAIN state persists.
The DOWN state may or may not persist, pending on the setting of the NodeName/State parameter
in the slurm.conf file.
Table 14-4 shows the corresponding meaning of the output of the sinfo command for various transitions:
170 Managing SLURM