User guide

3–InfiniBand
®
Cluster Setup and Administration
Performance Settings and Management Tips
IB0054606-02 A 3-27
Increasing the number of kernel receive queues allows more CPU cores to be
involved in the processing of verbs traffic. This is important when using parallel file
systems such as Lustre or IBM's GPFS (General Parallel File System). The
module parameter that sets this number is krcvqs. Each additional kernel
receive queue (beyond the one default queue for each port) takes user contexts
away from PSM and from the support of MPI or compute traffic. The formula
which illustrates this trade-off is:
PSM Contexts = 16 - (krcvqs-1)x num_ports
Where number_ports is the number of ports on the HCA
For example, on a single-port card with krcvqs=4 set in modprobe.conf:
PSM Contexts = 16 - (4-1)x 1 = 16 - 3 = 13
If this were a 12-core node, then 13 is more than enough PSM contexts to run an
MPI process on each core without making use of context-sharing. An example ,
ib_qib options line in the modprobe.conf file, for this 12-core node case is:
options ib_qib singleport=1 krcvqs=4
Table 3-2 can be used as a guide for setting the krcvqs parameter for the
number of cores in the system supporting PSM processes and the number of
ports in the HCA. The table applies most readily to nodes with 1 HCA being used
to support PSM (for example, MPI or SHMEM) processes. For nodes with multiple
HCAs that are being used for PSM, the table decide the maximum number of
cores that will be assigned on each HCA to support PSM (MPI or SHMEM)
processes, then apply the table to each HCA in turn.
Table 3-2. krcvqs Parameter Settings
Cores per Node (to
be used for
MPI/PSM on 1
HCA):
1-port, Set krcvqs=
2 active ports in the HCA,
Set krcvqs=
61-64
1
a
1
a
57-60 2
1
a
53-56 3 2,1 (2 for port 1, 1 for one
port)
12-52 4 2
8-11 3 2,1 (2 for port 1, 1 for one
port)
4-7 2
1
a
1-3
1
a
1
a