User guide

IB0054606-02 A 8-1
8 Dispersive Routing
Infiniband
®
uses deterministic routing that is keyed from the Destination LID
(DLID) of a port. The Fabric Manager programs the forwarding tables in a switch
to determine the egress port a packet takes based on the DLID.
Deterministic routing can create hotspots even in full bisection bandwidth (FBB)
fabrics for certain communication patterns if the communicating node pairs map
onto a common upstream link, based on the forwarding tables. Since routing is
based on DLIDs, the IB fabric provides the ability to assign multiple LIDs to a
physical port using a feature called Lid Mask Control (LMC). The total number of
DLIDs assigned to a physical port is 2^LMC with the LIDS being assigned in a
sequential manner. The common IB fabric uses a LMC of 0, meaning each port
has 1 LID assigned to it. With non-zero LMC fabrics, this results in multiple
potential paths through the fabric to reach the same physical port. For example,
multiple DLID entries in the port forwarding table that could map to different
egress ports.
Dispersive routing, as implemented in the PSM, attempts to avoid congestion
hotspots described above by “spraying” messages across these paths. A
congested path will not bottleneck messages flowing down the alternate paths
that are not congested. The current implementation of PSM supports fabrics with
a maximum LMC of 3 (8 LIDs assigned per port). This can result in a maximum of
64 possible paths between a SLID, DLID pair ([SLID, DLID],[SLID, DLID+1],
[SLID,DLID+2]…..[SLID,DLID+8],[SLID+1, DLID],[SLID+1, DLID+1]…..[SLID+7,
DLID+8]). Keeping state associated with these many paths requires large amount
of memory resources, with empirical data showing not much gain in performance
beyond utilizing a small set of multiple paths. Therefore PSM reduces the number
of paths actually used in the above case to 8 where the following paths are the
only ones considered for transmission — [SLID, DLID], [SLID + 1, DLID + 1],
[SLID + 2, DLID + 2] ….. [SLID + N, DLID + N]. This makes the resource
requirements manageable while providing most of the benefits of dispersive
routing (congestion avoidance by utilizing multiple paths).