User guide

3InfiniBand
®
Cluster Setup and Administration
Performance Settings and Management Tips
IB0054606-02 A 3-33
This command lets the driver automatically decide on the allocation behavior
and disables this feature on platforms with AMD and Intel
Westmere-or-earlier CPUs, while enabling it on newer Intel CPUs.
Tunable options:
option ib_qib numa_aware=0
This command disables the NUMA awareness when allocating memories
within the driver. The memory allocation requests will be satisfied on the
node's CPU that executes the request.
option ib_qib numa_aware=1
This command enables this feature with the driver allocating memory on the
NUMA node closest to the HCA.
recv_queue_size, Tuning Related to NAKs
The Receiver Not Ready Negative Acknowledgement (RNR NAKs) can slow
IPoIB down significantly. IB is fast enough to overrun IPoIB's receive queue before
the post receives can occur.
The counter to look for on the sending side in this file is RC RNR NAKs as shown
in the following example:
# cat /sys/class/infiniband/qib0/stats
Port 1:
RC timeouts 0
RC resends 0
RC QACKs 0
RC SEQ NAKs 0
RC RDMA seq 0
RC RNR NAKs 151 <---------
RC OTH NAKs 0
. . .
Ctx:npkts 0:170642806
Check the RC RNR NAKs before and after running the IPoIB test to see if that
counter is increasing. If so, then increasing IPoIB's recv_queue_size to 512 in
the ib_ipoib.conf file should eliminate RNR NAKs.