Running Jobs with Platform LSF™ Version 7 Update 3 Release date: May 2008 Last modified: May 12, 2008 Support: support@platform.com Comments to: doc@platform.
Copyright © 1994-2008, Platform Computing Inc. We’d like to hear from You can help us make this document better by telling us what you think of the content, you organization, and usefulness of the information. If you find an error, or just want to make a suggestion for improving this document, please address your comments to doc@platform.com. Your comments should pertain only to Platform documentation. For product support, contact support@platform.com.
Contents 1 About Platform LSF . . Cluster Concepts Job Life Cycle 2 Working with Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Running Jobs with Platform LSF
C H A P T E R 1 About Platform LSF Contents ◆ ◆ “Cluster Concepts” on page 6 “Job Life Cycle” on page 16 Running Jobs with Platform LSF 5
Cluster Concepts Clusters, jobs, and queues Cluster A group of computers (hosts) running LSF that work together as a single unit, combining computing power and sharing workload and resources. A cluster provides a single-system image for disparate computing resources. Hosts can be grouped into clusters in a number of ways.
Job slot A job slot is a bucket into which a single unit of work is assigned in the LSF system. Hosts are configured to have a number of job slots available and queues dispatch jobs to fill job slots. Commands ◆ bhosts —View job slot limits for hosts and host groups ◆ bqueues —View job slot limits for queues ◆ busers —View job slot limits for users and user groups Configuration ◆ Define job slot limits in lsb.resources.
Hosts Host An individual computer in the cluster. Each host may have more than 1 processor. Multiprocessor hosts are used to run parallel jobs. A multiprocessor host with a single process queue is considered a single machine, while a box full of processors that each have their own process queue is treated as a group of separate machines.
Master host Where the master LIM and mbatchd run. An LSF server host that acts as the overall coordinator for that cluster. Each cluster has one master host to do all job scheduling and dispatch. If the master host goes down, another LSF server in the cluster becomes the master host. All LSF daemons run on the master host. The LIM on the master host is the master LIM. Commands ◆ lsid —View the master host name Configuration ◆ The master host is the first host listed in the lsf.cluster.
Commands ◆ lsadmin resstartup —Starts res ◆ lsadmin resshutdown —Shuts down res ◆ lsadmin resrestart —Restarts res Configuration ◆ Port number defined in lsf.conf lim Load Information Manager (LIM) running on each server host. Collects host load and configuration information and forwards it to the master LIM running on the master host. Reports the information displayed by lsload and lshosts. Static indices are reported when the LIM starts up or when the number of CPUs (ncpus) change.
◆ lsadmin limshutdown —Shuts down LIM ◆ lsadmin limrestart —Restarts LIM ◆ lsload —View dynamic load values ◆ lshosts —View static host load values Configuration ◆ Port number defined in lsf.conf. ELIM External LIM (ELIM) is a site-definable executable that collects and tracks custom dynamic load indices. An ELIM can be a shell script or a compiled binary program, which returns the values of the dynamic resources you define. The ELIM executable must be named elim and located in LSF_SERVERDIR.
Tasks are run without using the batch processing features of LSF but still with the advantage of resource requirements and selection of the best host to run the task based on load. Commands Local task ◆ lsrun —Submit an interactive task ◆ lsgrun —Submit an interactive task to a group of hosts ◆ See also LSF utilities such as ch, lsacct, lsacctmrg, lslogin, lsplace, lsload, lsloadadj, lseligible, lsmon, lstcsh An application or command that does not make sense to run remotely.
◆ Host model Mapped to hosts in lsf.cluster.cluster_name The combination of host type and CPU speed (CPU factor) of the computer. All hosts of the same relative speed are assigned the same host model. The CPU factor is taken into consideration when jobs are being dispatched. Commands ◆ lsinfo -m —View a list of currently running models ◆ lsinfo -M —View all models defined in lsf.shared Configuration ◆ Defined in lsf.shared ◆ Mapped to hosts in lsf.cluster.
Total resident memory usage in KB of all currently running processes in a job ◆ Total virtual memory usage in KB of all currently running processes in a job ◆ Currently active process group ID in a job ◆ Currently active processes in a job On UNIX, job-level resource usage is collected through PIM. ◆ Commands ◆ lsinfo —View the resources available in your cluster ◆ bjobs -l —View current resource usage of a job Configuration ◆ SBD_SLEEP_TIME in lsb.
◆ bjobs -l —View suspending conditions for a particular job and the scheduling thresholds that control when a job is resumed Configuration Runtime resource usage limits ◆ lsb.hosts —Configure thresholds for hosts ◆ lsb.queues —Configure thresholds for queues Limit the use of resources while a job is running. Jobs that consume more than the specified amount of a resource are signalled or have their priority lowered. Configuration ◆ Hard and soft limits lsb.
Job Life Cycle 1 Submit a job You submit a job from an LSF client or server with the bsub command. If you do not specify a queue when submitting the job, the job is submitted to the default queue. Jobs are held in a queue waiting to be scheduled and have the PEND state. The job is held in a job file in the LSF_SHAREDIR/cluster_name /logdir/info/ directory. Job ID Job name LSF assigns each job a unique job ID when you submit the job. You can also assign a name to the job with the -J option of bsub.
4 Run job sbatchd handles job execution.
Running Jobs with Platform LSF
C H A P T E R 2 Working with Jobs Contents ◆ ◆ ◆ ◆ ◆ ◆ ◆ ◆ “Submitting Jobs (bsub)” on page 20 “Modifying a Submitted Job (bmod)” on page 24 ❖ “Modifying Pending Jobs (bmod)” on page 25 ❖ “Modifying Running Jobs” on page 27 “Controlling Jobs” on page 28 ❖ “Killing Jobs (bkill)” on page 29 ❖ “Suspending and Resuming Jobs (bstop and bresume)” on page 30 ❖ “Changing Job Order Within Queues (bbot and btop)” on page 32 ❖ “Controlling Jobs in Job Groups” on page 33 “Submitting a Job to Specific Hosts”
Submitting Jobs (bsub) In this section ◆ ◆ ◆ ◆ ◆ ◆ ◆ “bsub command” on page 20 “Submitting a job to a specific queue (bsub -q)” on page 20 “Submitting a job associated to a project (bsub -P)” on page 21 “Submitting a job associated to a user group (bsub -G)” on page 22 “Submitting a job with a job name (bsub -J)” on page 22 “Submitting a job to a service class (bsub -sla)” on page 22 “Submitting a job under a job group (bsub -g)” on page 23 bsub command You submit a job with the bsub command.
Viewing available queues To see available queues, use the bqueues command. Use bqueues -u user_name to specify a user or user group so that bqueues displays only the queues that accept jobs from these users. The bqueues -m host_name option allows users to specify a host name or host group name so that bqueues displays only the queues that use these hosts to run jobs. You can submit jobs to a queue as long as its STATUS is Open. However, jobs are not dispatched unless the queue is Active.
Submitting a job associated to a user group (bsub -G) You can use the bsub -G user_group option to submit a job and associate it with a specified user group. This option is only useful with fairshare scheduling. For more details on fairshare scheduling, see Administering Platform LSF. You can specify any user group to which you belong as long as it does not contain any subgroups. You must be a direct member of the specified user group.
See Administering Platform LSF for more information about service classes and goaloriented SLA driven scheduling. Submitting a job under a job group (bsub -g) Use bsub -g to submit a job into a job group. The job group does not have to exist before submitting the job. For example: bsub -g /risk_group/portfolio1/current myjob Job <105> is submitted to default queue. Submits myjob to the job group /risk_group/portfolio1/current.
Modifying a Submitted Job (bmod) In this section ◆ ◆ ◆ “Modifying Pending Jobs (bmod)” on page 25 “Modifying Running Jobs” on page 27 “Controlling Jobs” on page 28 24 Running Jobs with Platform LSF
Modifying Pending Jobs (bmod) If your submitted jobs are pending (bjobs shows the job in PEND state), use the bmod command to modify job submission parameters. You can also modify entire job arrays or individual elements of a job array. See the bmod command in the Platform LSF Command Reference for more details. Replacing the job command-line To replace the job command line, use the bmod -Z "new_command " option.
Modifying a job submitted to a job group Use the -g option of bmod and specify a job group path to move a job or a job array from one job group to another. For example: bmod -g /risk_group/portfolio2/monthly 105 moves job 105 to job group /risk_group/portfolio2/monthly. Like bsub -g, if the job group does not exist, LSF creates it. bmod -g cannot be combined with other bmod options. It can only operate on pending jobs. It cannot operate on running or finished jobs.
Modifying Running Jobs Modifying resource reservation A job is usually submitted with a resource reservation for the maximum amount required. Use bmod -R to modify the resource reservation for a running job. This command is usually used to decrease the reservation, allowing other jobs access to the resource.
Controlling Jobs LSF controls jobs dispatched to a host to enforce scheduling policies, or in response to user requests. The LSF system performs the following actions on a job: Suspend by sending a SIGSTOP signal ◆ Resume by sending a SIGCONT signal ◆ Terminate by sending a SIGKILL signal On Windows, equivalent functions have been implemented to perform the same tasks.
Killing Jobs (bkill) The bkill command cancels pending batch jobs and sends signals to running jobs. By default, on UNIX, bkill sends the SIGKILL signal to running jobs. Before SIGKILL is sent, SIGINT and SIGTERM are sent to give the job a chance to catch the signals and clean up. The signals are forwarded from mbatchd to sbatchd, which waits for the job to exit before reporting the status.
Suspending and Resuming Jobs (bstop and bresume) The bstop and bresume commands allow you to suspend or resume a job. A job can also be suspended by its owner or the LSF administrator with the bstop command. These jobs are considered user-suspended and are displayed by bjobs as USUSP. When the user restarts the job with the bresume command, the job is not started immediately to prevent overloading. Instead, the job is changed from USUSP to SSUSP (suspended by the system).
Resuming a job bresume command To resume a job, use the bresume command. Resuming a user-suspended job does not put your job into RUN state immediately. If your job was running before the suspension, bresume first puts your job into SSUSP state and then waits for sbatchd to schedule it according to the load conditions. For example, to resume job 3421, enter: bresume 3421 Job <3421> is being resumed You cannot resume jobs suspended by another user; you can only resume your own jobs.
Changing Job Order Within Queues (bbot and btop) By default, LSF dispatches jobs in a queue in the order of arrival (that is, first-come-first-served), subject to availability of suitable server hosts. Use the btop and bbot commands to change the position of pending jobs, or of pending job array elements, to affect the order in which jobs are considered for dispatch. Users can only change the relative position of their own jobs, and LSF administrators can change the position of any users’ jobs.
Controlling Jobs in Job Groups Stopping (bstop) Use the -g option of bstop and specify a job group path to suspend jobs in a job group bstop -g /risk_group 106 Job <106> is being stopped Use job ID 0 (zero) to suspend all jobs in a job group: bstop -g /risk_group/consolidate 0 Job <107> is being stopped Job <108> is being stopped Job <109> is being stopped Resuming (bresume) Use the -g option of bresume and specify a job group path to resume suspended jobs in a job group: bresume -g /risk_group 106 Job <1
bkill -g /risk_group/consolidate 0 Job <116> is being terminated Deleting (bgdel) Use bgdel command to remove a job group. The job group cannot contain any jobs. For example: bgdel /risk_group Job group /risk_group is deleted. deletes the job group /risk_group and all its subgroups. For more information See Administering Platform LSF for more information about using job groups.
Submitting a Job to Specific Hosts To indicate that a job must run on one of the specified hosts, use the bsub -m "hostA hostB ..." option. By specifying a single host, you can force your job to wait until that host is available and then run on that host. For example: bsub -q idle -m "hostA hostD hostB" myjob This command submits myjob to the idle queue and tells LSF to choose one host from hostA, hostD and hostB to run the job.
Submitting a Job and Indicating Host Preference When several hosts can satisfy the resource requirements of a job, the hosts are ordered by load. However, in certain situations it may be desirable to override this behavior to give preference to specific hosts, even if they are more heavily loaded.
Submitting a job with different levels of host preference You can indicate different levels of preference by specifying a number after the plus sign (+). The larger the number, the higher the preference for that host or host group. You can also specify the + with the keyword others. For example: bsub -m "groupA+2 groupB+1 groupC" myjob In this example, LSF gives first preference to hosts in groupA, second preference to hosts in groupB and last preference to those in groupC.
Using LSF with Non-Shared File Space LSF is usually used in networks with shared file space. When shared file space is not available, use the bsub -f command to have LSF copy needed files to the execution host before running the job, and copy result files back to the submission host after the job completes. LSF attempts to run the job in the directory where the bsub command was invoked.
If the submission and execution hosts have different directory structures, you must ensure that the directory where remote_file and local_file will be placed exists. LSF tries to change the directory to the same path name as the directory where the bsub command was run. If this directory does not exist, the job is run in your home directory on the execution host.
Reserving Resources for Jobs About resource reservation When a job is dispatched, the system assumes that the resources that the job consumes will be reflected in the load information. However, many jobs do not consume the resources they require when they first start. Instead, they will typically use the resources over a period of time. For example, a job requiring 100 MB of swap is dispatched to a host having 150 MB of available swap.
Submitting a Job with Start or Termination Times By default, LSF dispatches jobs as soon as possible, and then allows them to finish, although resource limits might terminate the job before it finishes. You can specify a time of day at which to start or terminate a job. Submitting a job with a start time If you do not want to start your job immediately when you submit it, use bsub -b to specify a start time. LSF will not dispatch the job before this time.
Running Jobs with Platform LSF
C H A P T E R 3 Viewing Information About Jobs Use the bjobs and bhist commands to view information about jobs: bjobs reports the status of jobs and the various options allow you to display specific information. ◆ bhist reports the history of one or more jobs in the system. You can also find jobs on specific queues or hosts, find jobs submitted by specific projects, and check the status of specific jobs using their job IDs or names.
Viewing Job Information (bjobs) The bjobs command has options to display the status of jobs in the LSF system. For more details on these or other bjobs options, see the bjobs command in the Platform LSF Command Reference. Unfinished current jobs The bjobs command reports the status of LSF jobs. When no options are specified, bjobs displays information about jobs in the PEND, RUN, USUSP, PSUSP, and SSUSP states for the current user.
Viewing Job Pend and Suspend Reasons (bjobs -p) When you submit a job, it may be held in the queue before it starts running and it may be suspended while running. You can find out why jobs are pending or in suspension with the bjobs -p option. You can combine bjob options to tailor the output. For more details on these or other bjobs options, see the bjobs command in the Platform LSF Command Reference.
Viewing suspend reasons only The -s option of bjobs displays reasons for suspended jobs only.
Viewing Detailed Job Information (bjobs -l) The -l option of bjobs displays detailed information about job status and parameters, such as the job’s current working directory, parameters specified when the job was submitted, and the time when the job started running. For more details on bjobs options, see the bjobs command in the Platform LSF Command Reference.
Viewing Job Resource Usage (bjobs -l) LSF monitors the resources jobs consume while they are running. The -l option of the bjobs command displays the current resource usage of the job. For more details on bjobs options, see the bjobs command in the Platform LSF Command Reference.
Viewing Job History (bhist) Sometimes you want to know what has happened to your job since it was submitted. The bhist command displays a summary of the pending, suspended and running time of jobs for the user who invoked the command. Use bhist -u all to display a summary for all users in the cluster. For more details on bhist options, see the bhist command in the Platform LSF Command Reference.
Examples bhist bhist bhist bhist -n -n -n -n 1 2 3 0 searches the current event log file lsb.events searches lsb.events and lsb.events.1 searches lsb.events, lsb.events.1, lsb.events.2 searches all event log files in LSB_SHAREDIR Viewing chronological history of jobs By default, the bhist command displays information from the job event history file, lsb.events, on a per job basis.
Viewing Job Output (bpeek) The output from a job is normally not available until the job is finished. However, LSF provides the bpeek command for you to look at the output the job has produced so far. By default, bpeek shows the output from the most recently submitted job. You can also select the job by queue or execution host, or specify the job ID or job name on the command line. For more details on bpeek options, see the bpeek command in the Platform LSF Reference.
Viewing Information about SLAs and Service Classes Monitoring the progress of an SLA (bsla) Use bsla to display the properties of service classes configured in lsb.serviceclasses and dynamic state information for each service class. Examples ◆ One velocity goal of service class Tofino is active and on time. The other configured velocity goal is inactive.
bsla Kyuquot SERVICE CLASS NAME: Kyuquot -- Daytime/Nighttime SLA PRIORITY: 23 USER_GROUP: user1 user2 GOAL: VELOCITY ACTIVE WINDOW: (9:00-17:30) STATUS: Active:On time VELOCITY: 8 CURRENT VELOCITY: 0 GOAL: DEADLINE ACTIVE WINDOW: (17:30-9:00) STATUS: Inactive NJOBS 0 ◆ PEND 0 RUN 0 SSUSP 0 USUSP 0 FINISH 0 The throughput goal of service class Inuvik is always active.
OPTIMUM NUMBER OF RUNNING JOBS: 5 NJOBS PEND RUN SSUSP USUSP FINISH 111 94 5 0 0 12 -------------------------------------------------------------SERVICE CLASS NAME: Tuktoyaktuk -- throughput 3 PRIORITY: 15 GOAL: THROUGHPUT ACTIVE WINDOW: Always Open STATUS: Active:On time SLA THROUGHPUT: 4.00 JOBs/CLEAN_PERIOD THROUGHPUT: 3 OPTIMUM NUMBER OF RUNNING JOBS: 4 NJOBS 104 PEND 96 RUN 4 SSUSP 0 USUSP 0 FINISH 4 These two service classes have the following historical performance.
bacct -sla Tuktoyaktuk Accounting information about jobs that are: - submitted by users user1, - accounted on all projects. - completed normally or exited - executed on all hosts. - submitted to all queues. - accounted on service classes Tuktoyaktuk, -----------------------------------------------------------------------------SUMMARY: ( time unit: second ) Total number of done jobs: 87 Total number of exited jobs: 0 Total CPU time consumed: 18.0 Average CPU time consumed: 0.2 Maximum CPU time of a job: 0.
Viewing Jobs in Job Groups Viewing job group information (bjgroup) Use the bjgroup command to see information about jobs in job groups.
Viewing Information about Resource Allocation Limits Your job may be pending because some configured resource allocation limit has been reached. Use the blimits command to show the dynamic counters of resource allocation limits configured in Limit sections in lsb.resources. blimits displays the current resource usage to show what limits may be blocking your job.
End Limit Begin Limit NAME = limit_ext1 PER_HOST = all RESOURCE = ([user1_num, 30] [hc_num, 20]) End Limit blimits displays the following: blimits INTERNAL RESOURCE LIMITS: NAME limit1 limit1 limit1 USERS QUEUES HOSTS user1 q2 hostA@cluster1 user1 q3 hostA@cluster1 user1 q4 hostC PROJECTS - SLOTS - MEM 10/25 - TMP 30/2953 40/590 SWP 10/258 - EXTERNAL RESOURCE LIMITS: NAME USERS limit_ext1 limit_ext1 - QUEUES - HOSTS hostA@cluster1 hostC@cluster1 PROJECTS - user1_num 1/30 hc_num 1/20 1/20 In lim
Index A automount option, /net CPU time limit, small jobs 38 D B bacct, collecting project information bacct command 53 batch jobs accessing files file access killing 29 38 directories, remote access 21 G goal-oriented scheduling.
L limits, modifying for running jobs resource usage, viewing for jobs 27 logs, viewing jobs not listed in active event log lsb.resources file, viewing limit configuration (blimits) 57 lsrcp command for remote file access 38 N O order of job execution P pending reasons 49 RUN job state sbatchd (slave batch daemon), remote file access 22 38 bacct command 53 bsla command 52 service level agreement.
