Specifications

DATA CENTER BEST PRACTICES
SAN Design and Best Practices 31 of 84
– Size of fabrics: If the size of edge fabrics is likely to grow, and the inter-fabric trafc is expected to grow
accordingly, provision additional IFLs and ISLs such that the capacity of available paths stays well ahead
of current usage. That way, incremental growth on the edge can be accommodated without the need to
immediately upgrade the backbone.
– Amount of trafc between fabrics: If the inter-fabric trafc is expected to grow even without growth in the
individual edge fabrics, then provision additional IFLs and ISLs such that the capacity of available paths
stays ahead of current usage. That way, incremental increases in data ow across the backbone can be
accommodated without the need to immediately upgrade the backbone. Make sure that you allow for
plenty of room for backbone expansion.
Avoiding Congestion
Just as with a at Layer 2 fabric, a routed SAN needs to be evaluated for trafc bandwidth and potential
bandwidth utilization between all end-points. For routed topologies, this means calculating trafc owing in
and out of every edge fabric and providing enough links into and across the backbone to accommodate that
trafc. Use the same guidelines that apply to ISLs when connecting fabrics through IFLs for improved utilization
and resiliency. As often happens as fabrics evolve, an edge fabric can be of higher performance versus the
backbone, resulting in a completely oversubscribed backbone. This can lead to congestion at peak loads and
high latency due to slow device response. Prior to upgrading the edge fabric, consider increasing the number of
ISLs or upgrading the backbone to avoid congestion.
Available Paths
The best approach is to have multiple trunked paths between edge fabrics so that trafc can be spread across
available resources; however, it is never good practice to attach both A and B fabrics to the same backbone
router. From the perspective of FC, you should adhere to the concept of an “air gap” all the way from host to
storage. A common device connected to both A and B fabrics can cause a SAN-wide outage. If an air gap is
implemented, faults on one fabric cannot affect the other fabric. These faults can manifest from defects in host,
fabric, or storage hardware and software, as well as human error. It is not relevant that FCR keeps the fabrics
separate, because these types of faults can transcend FCR and cause the entire SAN to fail.
Design Guidelines and Constraints for Routed SANs
Some of the key metrics and rules of thumb for routed SAN topologies are:
•Keep A and B fabrics separated all the way from host to storage from a FC perspective. This is referred to
as the “air gap.” This does not include IP networks passing FCIP trafc, although FCIP ISL end points should
never cross-connect A and B fabrics, as this is the same as a traditional ISL cross connecting A and B fabrics.
•Localize trafc within an edge fabric as much as possible.
•Have a plan for predening the domains in the fabric (for example, edge switches with a certain range,
translate domains in a certain range that connect to the backbone fabric, and unique backbone fabric IDs to
avoid domain overlap).
•Consider upgrading the backbone to higher performance prior to upgrading the edge fabric.
•When sharing devices using FCR between a core with B-Series switches running Brocade FOS and an edge
fabric with M-Series switches running M-EOS, connect only one McDATA edge switch to the Brocade Backbone,
or keep the number of EX_Ports from each FCR roughly balanced in edge fabric.
•No more than one long-distance hop between source and destination.
•Place long-distance links within the backbone (as opposed to between edge and backbone), as edge fabrics
can then be isolated from disruption on the long-distance links. An edge fabric that contains a long-distance
link is referred to as a remote edge. Remote edges can be the product of VEX_Ports and EX_Ports that
connect to FC-based DWDM. Remote edges are not considered best practice.
•Use Logical SAN (LSAN) zones only for devices that will actually be communicating across the backbone. In
other words, do not make every zone an LSAN zone for ease.