LSF Version 7.3 - Administering Platform LSF

Administering Platform LSF 483
C HAPTER
30
Chunk Job Dispatch
Contents
◆ About Job Chunking on page 483
◆ Configure Chunk Job Dispatch on page 484
◆ Submitting and Controlling Chunk Jobs on page 486
About Job Chunking
LSF supports job chunking, where jobs with similar resource requirements
submitted by the same user are grouped together for dispatch. The
CHUNK_JOB_SIZE parameter in
lsb.queues and lsb.applications specifies
the maximum number of jobs allowed to be dispatched together in a chunk job.
Job chunking can have the following advantages:
◆ Reduces communication between sbatchd and mbatchd, and scheduling
overhead in
mbatchd
◆ Increases job throughput in mbatchd and more balanced CPU utilization on
the execution hosts
All of the jobs in the chunk are dispatched as a unit rather than individually. Job
execution is sequential, but each chunk job member is not necessarily executed in
the order it was submitted.
Chunk job candidates
Jobs with the following characteristics are typical candidates for job chunking:
◆ Take between 1 and 2 minutes to run
◆ All require the same resource (for example a software license or a specific
amount of memory)
◆ Do not specify a beginning time (bsub -b) or termination time (bsub -t)
Running jobs with these characteristics without chunking can underutilize
resources because LSF spends more time scheduling and dispatching the jobs than
actually running them.