User Manual

OpenSM – Subnet ManagerRev 2.2-1.0.1
Mellanox Technologies
180
root list is provided, the closer the topology to a pure and symmetrical fat-tree, the more optimal
the routing will be.
The algorithm also dumps compute node ordering file (
opensm-ftree-ca-order.dump) in
the same directory where the OpenSM log resides. This ordering file provides the CN order
that may be used to create efficient communication pattern, that will match the routing tables.
8.5.4.1 Routing between non-CN Nodes
The use of the io_guid_file option allows non-CN nodes to be located on different levels in the
fat tree. In such case, it is not guaranteed that the Fat
Tree algorithm will route between two non-
CN nodes. In the scheme below, N1, N2 and N3 are non-CN nodes. Although all the CN have
routes to and from them, there will not necessarily be a route between N1,N2 and N3. Such
routes would require to use at least one of the switches the wrong way around.
Spine1 Spine2 Spine 3
/ \ / | \ / \
/ \ / | \ / \
N1 Switch N2 Switch N3
/|\ /|\
/ | \ / | \
Going down to compute nodes
To solve this problem, a list of non-CN nodes can be specified by \'-G\' or \'--io_guid_file\'
option. These nodes will be allowed to use switches the wrong way around a specific number of
times (specified by \'-H\' or \'--max_reverse_hops\'. With the proper max_reverse_hops and
io_guid_file values, you can ensure full connectivity in the Fat Tree. In the scheme above, with a
max_reverse_hop of 1, routes will be instanciated between N1<->N2 and N2<->N3. With a
max_reverse_hops value of 2, N1,N2 and N3 will all have routes between them.
Using max_reverse_hops creates routes that use the switch in a counter-stream way.
This option should never be used to connect nodes with high bandwidth traffic
between them! It should only be used to allow connectivity for HA purposes or similar.
Also having routes the other way around can cause credit loops.
8.5.4.2 Activation through OpenSM
• Use ‘-R ftree’ option to activate the fat-tree algorithm
LMC > 0 is not supported by fat-tree routing. If this is specified, the default routing
algorithm is invoked instead.
8.5.5 LASH Routing Algorithm
LASH is an acronym for LAyered SHortest Path Routing. It is a deterministic shortest path rout-
ing algorithm that enables topology agnostic deadlock-free routing within communication net-
works.