Platform LSF Administration Guide Version 6.2

About Job Chunking
Administering Platform LSF
410
About Job Chunking
LSF supports job chunking, where jobs with similar resource requirements submitted
by the same user are grouped together for dispatch. The CHUNK_JOB_SIZE
parameter in
lsb.queues specifies the maximum number of jobs allowed to be
dispatched together in a chunk job.
Job chunking can have the following advantages:
◆ Reduces communication between sbatchd and mbatchd, and scheduling
overhead in
mbatchd
◆ Increases job throughput in mbatchd and more balanced CPU utilization on the
execution hosts
All of the jobs in the chunk are dispatched as a unit rather than individually. Job
execution is sequential, but each chunk job member is not necessarily executed in the
order it was submitted.
Chunk job candidates
Jobs with the following characteristics are typical candidates for job chunking:
◆ Take between 1 and 2 minutes to run
◆ All require the same resource (for example a software license or a specific amount
of memory)
◆ Do not specify a beginning time (bsub -b) or termination time (bsub -t)
Running jobs with these characteristics in normal queues can under-utilize resources
because LSF spends more time scheduling and dispatching the jobs than actually
running them.
Configuring a special high-priority queue for short jobs is not desirable because users
may be tempted to send all of their jobs to this queue, knowing that it has high priority.