Specifications
DATA CENTER BEST PRACTICES
SAN Design and Best Practices 20 of 84
Recommendations for avoiding frame congestion (when the number of frames is the issue rather than bandwidth
utilization) include:
•Use more and smaller trunks.
•Storage ports should follow the array vendor-suggested fan-in ratio for ISLs into the core. Follow vendor-
suggested recommendations when implementing a large number of low-capacity LUNs.
•Bandwidth through the core (path from source/host to destination/target) should exceed storage
requirements.
•Host-to-core subscription ratios should be based on both the application needs and the importance of
the application.
•Plan for peaks, not average usage.
•For mission-critical applications, the ratio should exceed peak load enough such that path failures do not
adversely impact the application. In other words, have enough extra bandwidth to avoid congestion if a
link fails.
DATA FLOW CONSIDERATIONS
Congestion in the Fabric
Congestion is a major source of poor performance in a fabric. Sufciently impeded trafc translates directly into
poor application performance.
There are two major types of congestion: trafc-based and frame-based. Trafc-based congestion occurs when
link throughput capacity is reached or exceeded and the link is no longer able to pass more frames. Frame-
based congestion occurs when a link has run out of buffer credits and is waiting for buffers to free up to
continue transmitting frames.
Traffic versus Frame Congestion
Once link speeds reach 4 Gbps and beyond, the emphasis on fabric and application performance shifts from
trafc -level issues to frame congestion. It is very difcult with current link speeds and Brocade features such
as Brocade ISL Trunking or UltraScale ICLs to consistently saturate a link. Most infrastructures today rarely
see even two-member trunks reaching a sustained 100 percent utilization. Frame congestion can occur when
the buffers available on a Fibre Channel port are not sufcient to support the number of frames the connected
devices wish to transmit. This situation can result in credit starvation backing up across the fabric. This
condition is called back pressure, and it can cause severe performance problems.
One side effect of frame congestion can be very large buffer credit zero counts on ISLs and F_Ports. This is not
necessarily a concern, unless counts increase rapidly in a very short period of time. Brocade has added a new
feature, Bottleneck Detection, to more accurately assess the impact of a lack of buffer credits.
The sources and mitigation for trafc are well known and are discussed at length in other parts of this
document. The remainder of this section focuses on the sources and mitigation of frame-based congestion.
Sources of Congestion
Frame congestion is primarily caused by latencies somewhere in the SAN—usually storage devices and
occasionally hosts. These latencies cause frames to be held in ASICs and reduce the number of buffer credits
available to all ows traversing that ASIC. The congestion backs up from the source of the latency to the other
side of the connection and starts clogging up the fabric. Back pressure can be created from the original source
of the latency to the other side and all the way back (through other possible paths across the fabric, to the
original source again. Once this situation arises the fabric is very vulnerable to severe performance problems.