User Manual

Features Overview and ConfigurationRev 2.3-1.0.1

Mellanox Technologies

/|\ /|\

/ | \ / | \

Going down to compute nodes

To solve this problem, a list of non-CN nodes can be specified by \'-G\' or \'--io_guid_file\'

option. These nodes will be allowed to use switches the wrong way around a specific number of

times (specified by \'-H\' or \'--max_reverse_hops\'. With the proper max_reverse_hops and

io_guid_file values, you can ensure full connectivity in the Fat Tree. In the scheme above, with a

max_reverse_hop of 1, routes will be instanciated between N1<->N2 and N2<->N3. With a

max_reverse_hops value of 2, N1,N2 and N3 will all have routes between them.

Using max_reverse_hops creates routes that use the switch in a counter-stream way.

This option should never be used to connect nodes with high bandwidth traffic

between them! It should only be used to allow connectivity for HA purposes or similar.

Also having routes the other way around can cause credit loops.

Activation through OpenSM

• Use ‘-R ftree’ option to activate the fat-tree algorithm

LMC > 0 is not supported by fat-tree routing. If this is specified, the default routing

algorithm is invoked instead.

3.2.2.5.4 LASH Routing Algorithm

LASH is an acronym for LAyered SHortest Path Routing. It is a deterministic shortest path rout-

ing algorithm that enables topology agnostic deadlock-free routing within communication net-

works.

When computing the routing function, LASH analyzes the network topology for the shortest-path

routes

between all pairs of sources / destinations and groups these paths into virtual layers in

such a way as to avoid deadlock.

LASH analyzes routes and ensures deadlock freedom between switch pairs. The link

from HCA between and switch does not need virtual layers as deadlock will not arise

between switch and HCA.

In more detail, the algorithm works as follows:

1. LASH determines the shortest-path between all pairs of source / destination switches. Note,

LASH ensures the same SL is used for all SRC/DST - DST/SRC pairs and there is no guar-

antee that the return path for a given DST/SRC will be the reverse of the route SRC/DST.

2. LASH then begins an SL assignment process where a route is assigned to a layer (SL) if the

addition

of that route does not cause deadlock within that layer. This is achieved by main-

taining and analysing a channel dependency graph for each layer. Once the potential addition

of a path could lead to deadlock, LASH opens a new layer and continues the process.

3. Once this stage has been completed, it is highly likely that the first layers processed will

contain more paths than the latter ones.

To better balance the use of layers, LASH moves

paths from one layer to another so that the number of paths in each layer averages out.