User guide

34 Implementing Cisco InfiniBand on IBM BladeCenter
As a packet traverses the subnet, a service level (SL) is defined to ensure its QoS level. Each
link along a path can have a different VL, and the SL provides each link a desired priority of
communication. Each switch/router has an SL-to-VL mapping table that is set by the subnet
manager to keep the proper priority with the number of VLs supported on each link.
Therefore, the InfiniBand Architecture can ensure end-to-end QoS through switches, routers,
and over the long haul.
Credit-based flow control
Link-level flow control is used to manage data flow between two point-to-point links. Flow
control is handled on a per-VL basis, allowing separate virtual fabrics to maintain
communication utilizing the same physical media. Each receiving end of a link supplies
credits to the sending device on the link to specify the amount of data that can be received
without loss of data. Credit passing between each device is managed by a dedicated link
packet to update the number of data packets the receiver can accept. Data is not transmitted
unless the receiver advertises credits indicating that receive buffer space is available.
Data integrity
At the link level there are two CRCs per packet, variant CRC (VCRC) and invariant CRC
(ICRC), that ensure data integrity. The 16-bit VCRC includes all fields in the packet and is
recalculated at each hop. The 32-bit ICRC covers only the fields that do not change from hop
to hop. The VCRC provides link-level data integrity between two hops and the ICRC provides
end-to-end data integrity. In a protocol such as Ethernet, which defines only a single CRC, an
error can be introduced within a device that then recalculates the CRC. The check at the next
hop would reveal a valid CRC even though the data has been corrupted. InfiniBand includes
the ICRC so that when a bit error is introduced, the error will always be detected.
3.1.5 Physical layer
The physical layer transforms the packet into the signal that will be passed over the physical
interconnect. The InfiniBand interconnect uses bidirectional serial links between nodes. The
base speed of an InfiniBand link is 2.5 Gbps. This base speed is referred to as
1X. Bandwidth
can be increased by increasing the number of channels per link with 4X and 12X connections
for link speeds of 10 Gbps and 30 Gbps respectively. The InfiniBand Architecture also
supports dual and quad data rates for speeds of up to 120 Gbps (12X link at quad data rate).
The InfiniBand specification also defines electrical and mechanical characteristics for copper
and fibre optic cables and connectors.