DATA CENTER SAN Design and Best Practices Version 2.3 A high-level guide focusing on Fibre Channel Storage Area Network (SAN) design and best practices, covering planning, topologies, device sharing in routed topologies, workload monitoring, and detecting server and storage latencies—to help with decisions required for successful SAN design.
DATA CENTER BEST PRACTICES CONTENTS INTRODUCTION........................................................................................................................................... 6 Audience and Scope............................................................................................................................. 6 Approach.............................................................................................................................................. 6 Overview.............
DATA CENTER BEST PRACTICES Misbehaving Devices .......................................................................................................................... 25 DESIGN GUIDELINES................................................................................................................................. 26 Monitoring.......................................................................................................................................... 26 IOPS and VMs.......................
DATA CENTER BEST PRACTICES Design Guidelines............................................................................................................................... 53 Monitoring.......................................................................................................................................... 54 SCALABILITY AND PERFORMANCE.............................................................................................................. 54 SUPPORTABILITY............................
DATA CENTER BEST PRACTICES Storage Traffic Patterns....................................................................................................................... 69 Server Traffic Patterns......................................................................................................................... 69 Backup Traffic Patterns........................................................................................................................ 70 Tape Library.............................
DATA CENTER BEST PRACTICES INTRODUCTION This document is a high-level design and best practices guide based on Brocade products and features, focusing on Fibre Channel SAN design. Covered topics include the early planning phase, understanding possible operational challenges, and monitoring and improving an already implemented SAN infrastructure. Emphasis is given to in-depth information on how the data flows between devices affect a SAN infrastructure at a design and operational level.
DATA CENTER BEST PRACTICES This document does not consider physical environment factors such as power, cooling, and rack layout. Rather, the focus is on network connectivity (both inter-switch and edge device) and software configurations. Note: The scope of this document is switch-centric and does not discuss end-device setup, configuration, as well as maintenance. Fabric monitoring, management, and diagnostics and McDATA and Brocade interoperability and migration are covered in separate documents.
DATA CENTER BEST PRACTICES Backup and disaster tolerance • Is there a centralized backup? (This will determine the number of ISLs needed to minimize congestion at peak loads.
DATA CENTER BEST PRACTICES The ClearLink diagnostic capability provides an opportunity to measure and thoroughly test ISLs before they are put into production. It is recommended that diagnostics be conducted prior to deployment or when there are CRC errors that could be caused by physical layer issues.
DATA CENTER BEST PRACTICES The difference between these four scenarios is device placement (where devices are attached to the network) and the associated traffic flow, which is discussed further in the “Data Flow Considerations” section later in this document. • Scenario A has localized traffic, which can have small performance advantages but does not provide ease of scalability or manageability.
DATA CENTER BEST PRACTICES Servers and storage devices should be connected to both networks utilizing some form of Multi-Path I/O (MPIO) solution, such that data can flow across both networks seamlessly in either an active/active or active/ passive mode. MPIO ensures that if one path fails, an alternative is readily available. Ideally, the networks would be identical, but at a minimum they should be based on the same switch architecture. In some cases, these networks are in the same location.
DATA CENTER BEST PRACTICES In addition to redundant fabrics, redundant links should be placed on different blades, different ASICs, or at least different port groups whenever possible, as shown in Figure 3. (Refer to Appendix C to determine trunk groups for various port blades. For more details, see the Brocade Fabric OS Administrator’s Guide.) Whatever method is used, it is important to be consistent across the fabric.
BEST PRACTICES fig05_SAN_Design DATA CENTER Figure 5. Example of three-chassis ICL configuration for 8 Gbps platform. Note: Refer to the Brocade DCX Hardware Reference Manual for detailed ICL connectivity. ICLs can be used instead of ISLs for a Brocade DCX/DCX-4S core-edge fabric—taking into account that the ICL cable length is 2 meters or less. Also, an ICL connection is considered a “hop of no concern” in a FICON environment.
DATA CENTER BEST PRACTICES on each core blade must reside within the same ICL trunk boundary on the core blades. If more than four ICL connections are required between a pair of Brocade DCX 8510 chassis, additional ICL connections should be added in pairs (one on each core blade). Domain 1 Brocade DCX 8510-8 fig07_SAN_Design Domain 2 Brocade DCX 8510-8 UltraScale ICL Trunk Boundary Trunked UltraScale ICLs Figure 7. Minimum connections needed between Brocade DCX 8510 chassis.
DATA CENTER BEST PRACTICES Brocade recommends core-edge or edge-core-edge as the primary SAN design methodology, or mesh topologies used for small fabrics (under 2000 ports). As a SAN design best practice, edge switches should connect to at least two core switches with trunks of at least two ISLs each. Each of those trunks should be attached to a different blade/port group.
DATA CENTER BEST PRACTICES However, this approach does not scale well. Given the high-speed, low-latency nature of Fibre Channel, attaching these host-target pairs on different switches does not mean that performance is adversely impacted. Though traffic congestion is possible, it can be mitigated with proper provisioning of ISLs/UltraScale ICLs. With current generation switches, locality is not required for performance or to reduce latencies.
DATA CENTER BEST PRACTICES Fan-In Ratios and Oversubscription Another aspect of data flow is the “fan-in-ratio” or “oversubscription”, in terms of source ports to target ports and device to ISL’s. This is also referred to as the “fan-out-ratio” if viewed from the storage array perspective. The ratio is the number of device ports that share a single port, whether ISL, UltraScale ICL, or target.
DATA CENTER BEST PRACTICES Another method is to assign host ports to storage ports based on capacity. The intended result is a small number of high-capacity hosts and a larger number of low-capacity servers assigned to each storage port, thus distributing the load across multiple storage ports. Figure 14 shows the impact of the two different LUN provisioning strategies described above.
DATA CENTER BEST PRACTICES 4 × 8 Gbps ISLs fig15_SAN_Design 1:1 Oversubscription for Initiator to the Core 4 × 8 Gbps Control Ports Figure 15. One-to-one oversubscription for targets into the core. The realized oversubscription ratio of host-to-ISL should be roughly the same as the host-to-target ratio, taking into account the bandwidth (that is, if there are four hosts accessing a single 4 Gbps storage port, then those four hosts should have a 4 Gbps pipe into the core.
DATA CENTER BEST PRACTICES Recommendations for avoiding frame congestion (when the number of frames is the issue rather than bandwidth utilization) include: • Use more and smaller trunks. • Storage ports should follow the array vendor-suggested fan-in ratio for ISLs into the core. Follow vendorsuggested recommendations when implementing a large number of low-capacity LUNs. • Bandwidth through the core (path from source/host to destination/target) should exceed storage requirements.
DATA CENTER BEST PRACTICES Sources of high latencies include: • Storage devices that are not optimized or where performance has deteriorated over time • Distance links where the number of allocated buffers has been miscalculated or where the average frame sizes of the flows traversing the links has changed over time • Hosts where the application performance has deteriorated to the point that the host can no longer respond to incoming frames in a sufficiently timely manner • Incorrectly configured HBA’s •
DATA CENTER BEST PRACTICES Bottleneck detection can also serve as a confirmation to host information when storage latencies are suspected in poor host performance. The reverse (eliminating the storage as the source of poor performance) is also true. Beginning with Brocade FOS v6.4, Bottleneck Detection can also be applied to ISLs (E_Ports) and will highlight issues on those links. The sampling interval and number of notifications are configurable, as well as the alerting mechanisms. With Brocade FOS v6.
DATA CENTER BEST PRACTICES Fabric Watch tracks a variety of SAN fabric elements and events. Monitoring fabric-wide events, ports, and environmental parameters enables early fault detection and isolation as well as performance measurement. You can configure fabric elements and alert thresholds on an individual port basis, and you can also easily integrate Fabric Watch with enterprise system management solutions. Fabric Watch provides customizable monitoring thresholds.
DATA CENTER BEST PRACTICES • Brocade platforms are capable of generating SNMP traps for most error conditions. Consider implementing some sort of alerting mechanism via SNMP. Monitoring and Notifications Error logs should be looked at regularly. Many end users use combinations of syslog and SNMP with the Brocade Fabric Watch and the logs to maintain a very close eye on the health of their fabrics.
DATA CENTER BEST PRACTICES Solution 1: Make the B-Series act more like the M-Series by using PBR versus EBR on the B-Series so that it will have the same single flow per ISL, just like the M-Series. Solution 2: Migrate sooner to a B-Series core and B-Series edge from B-Series core and M-Series edge (or vice versa). Most customers find themselves in Interop Fabrics temporarily. Solution 3: Move the heavy hitters (microbursts) to the core (in a core-edge topology).
DATA CENTER BEST PRACTICES Typically, a storage port services many applications, and the loss of that storage can severely impact all the applications connected to it. One of the advantages of a core-edge design is that it is very simple to isolate servers from their storage and ensure that any action applied to host port for a given behavior can be very different than the action applied to a storage port for the same behavior.
DATA CENTER BEST PRACTICES IOPS and VMs Another method for determining bandwidth and/or oversubscription is to use the IOPS between host and storage devices. If the typical I/O size is known, along with the typical number of IOPS, then the administrator can calculate both average and estimated peak loads in terms of Megabytes per second (MB/sec). Next, look at the paths through the network for these I/Os, along with I/Os from other devices using the same network paths.
DATA CENTER BEST PRACTICES Target Edge Fabric 1 E_Port Fiber Channel Switch Target Edge Fabric 2 Edge Fabric 3 E_Port E_Port Fiber Channel Switch IFL IFL Long Distance IFL fig17_SAN_Design Host EX Ports FC Router Figure 17. Typical MetaSAN topology.
DATA CENTER BEST PRACTICES Edge fabric A1 Core-edge backbone A Edge fabric A3 Edge fabric B1 Edge fabric A4 Core-edge backbone B Edge fabric B2 Edge fabric B4 fig20_SAN_Design Edge fabric B3 Edge fabric A2 Figure 19. Example of an edge-core-edge backbone in a redundant routed fabric topology. Backbone Considerations There are many factors to consider when designing backbone fabrics. As mentioned above, the general SAN topology recommendations are applicable to backbone fabrics.
DATA CENTER BEST PRACTICES –– M-Series (legacy McDATA) switches: Similar to legacy switches, the presence of M-Series switches anywhere in the SAN environment impacts the features that are supported and, depending on the platform and firmware version, may have other impacts as well.
DATA CENTER BEST PRACTICES –– Size of fabrics: If the size of edge fabrics is likely to grow, and the inter-fabric traffic is expected to grow accordingly, provision additional IFLs and ISLs such that the capacity of available paths stays well ahead of current usage. That way, incremental growth on the edge can be accommodated without the need to immediately upgrade the backbone.
DATA CENTER BEST PRACTICES • As edge fabrics and the routed network grow, the use of “filters” such as LSAN zone binding and LSAN tagging can improve topology convergence time and efficient usage of FCR resources. • Make the backbone fabrics redundant to improve resiliency. This means redundancy for each fabric; therefore, fabric A would be redundant and so would fabric B. Fabric B would never be used as the redundancy for fabric A, and vice-versa.
DATA CENTER BEST PRACTICES Use Case: FICON and Open Systems (Intermix) Virtual Fabrics enable customers to share FICON and FCP traffic on the same physical platform. As chassis densities increase, this is a viable option for improved hardware utilization while maintaining director class availability.
DATA CENTER BEST PRACTICES of latency at the receiving Condor3-based switch in order to decrypt and uncompress the frame. This results in a total latency time of 12.4 microseconds, again not counting the link transit time. Virtual Fabric Considerations (Encryption and Compression) The E_Ports in the user-created Logical Switch, base switch, or default switch can support encryption and compression. Both encryption and compression are supported on XISL ports, but not on LISL ports.
DATA CENTER BEST PRACTICES On the Brocade DCX 8510 with Gen 5 Fibre Channel platform, 4K buffers are available per ASIC to drive 16 Gbps line rate to 500 KM at 2KB frame size. Brocade FOS v7.1 provides users additional control when configuring a port of an LD or LS link, allowing users to specify the buffers required or the average frame size for a long-distance port. Using the frame size option, the number of buffer credits required for a port is automatically calculated.
DATA CENTER BEST PRACTICES at all. On most large scale arrays, the FC port that has been assigned to RDR is dedicated to only RDR and no host traffic. Considering that the RDR port on the array can only communicate RDR traffic, there is no need to run that port into the production fabric.
DATA CENTER BEST PRACTICES A Brocade 7800 Back-Up Site Brocade 7800 FC FC A FCIP B FC FC Brocade 7800 Storage Array WAN IP Network B Brocade 7800 Storage Array fig24_SAN_Design Production Site Figure 23. Four-box solution connected to production fabrics. In environments that require production fabric attached channel extenders, it is not best practice to connect the same channel extender to both “A” and “B” fabrics.
DATA CENTER BEST PRACTICES When using a four-box solution, it is inappropriate to make ISL cross-connections between the two channel extenders within a data center site and both the “A” and “B” FC fabrics, because of the reasons discussed above. However, it is permissible to do so on the Ethernet/WAN side (see Figure 25).
DATA CENTER BEST PRACTICES Using EX_Ports and VEX_Ports If an FCR architecture is indicated, an “X” port is needed. An “X” port is a generic reference for an EX_Port or a VEX_Port. The only difference between an EX_Port and a VEX_Port is that the “V” indicates that it is FCIP-facing. The same holds true for E_Ports and VE_Ports; VE_Ports are E_Ports that are FCIP-facing.
DATA CENTER BEST PRACTICES Brocade DCX with FX8-24 N_Port FCIP Tunnel F_Port VE_Port IP Cloud VEX_Port EX_Port E_Port Edge Fabric Isolated Remote Edge Fabric Brocade 7800 fig28_SAN_Design Isolated Edge Fabric Figure 28. Edge-remote edge architecture. Another design consideration with “X” ports is: How many can be in a path? This is indeed limiting.
DATA CENTER BEST PRACTICES IBM does not consider a connection to a channel extender to be a hop of concern. Only one hop is supported by IBM, and a connection to a channel extender is not considered a hop. Brocade UltraScale ICLs are also not considered hops. The key here is—what is considered a channel extender? A channel extender is a Brocade 7800 with no end devices connected to any of the FC ports. Only ISLs are allowed to be connected to the FC ports.
DATA CENTER BEST PRACTICES Mode 3, Deflate, also known as GZIP, is entirely a software-based algorithm and not suitable for synchronous applications. Deflate takes the tradeoff between compression ratio and compression rate further. The maximum rate per FCIP complex is 2.5 Gbps ingress from the FC side. Mode 3 has been designed to work efficiently with an OC-12 WAN connection. Mode 3 typically gets about a 4:1 compression ratio.
DATA CENTER BEST PRACTICES In a shared link situation, if you think of the bandwidth as separated into three areas, black (0 Ò x bps), gray (x Ò y bps), and white (y Ò maximum bps), ARL can help manage the bandwidth usage. Black is the floor value for ARL. This is the amount of bandwidth that is reserved exclusively for FCIP. White is the ceiling value, and it is reserved exclusively for other shared traffic. Gray is the area in between, which FCIP may use if other shared traffic is not using it.
DATA CENTER BEST PRACTICES DSCP and L2CoS are configured on a per-FCIP circuit basis. It is recommended that you not alter the QoS markings for F-Class traffic unless it is required to differentiate and expedite F-Class traffic across the IP network between the sites. Failure of F-class traffic to arrive in a timely manner will cause instability in the FC fabric. This is less of an issue with directly connected separate RDR networks.
DATA CENTER BEST PRACTICES Gaussian or “normal” distribution .0214 .00135 -3σ fig33_SAN_Design fg(x) .0214 .1359 -2σ .3413 -σ .3413 0 X .00135 .1359 σ 2σ 3σ Figure 32. Gaussian curve. For remote tape/Virtual Tape Library (VTL) backup, the extension systems can be used to extend the data center fabric from the location where backup servers are attached to the remote fabric and where the remote tape/VTL is located, as shown by the blue arrow in Figure 33.
DATA CENTER BEST PRACTICES FCIP Trunking The Brocade 7800 and FX8-24 have an exclusive feature called FCIP Trunking. FCIP Trunking offers the ability to perform the following functions: • Bandwidth aggregation • Lossless failover/failback • Granular load balancing • In-order delivery • Prevention of IFCC on mainframes A single tunnel defined by a VE_Port or VEX_Port may have one or more circuits associated with it.
DATA CENTER BEST PRACTICES ARL is used to manage the bandwidth going into the cores based on the available WAN bandwidth. There may be a single WAN connection or separate WAN connections between the sites. ARL is used to manage the BW from the Brocade 7800s to the WAN connection. This example has a single WAN connection, although you could just as well use more than one WAN connection.
DATA CENTER BEST PRACTICES state machines at any one time. A state machine lives for the life of an exchange, and then it is deleted. FICON Emulation operates somewhat differently, and it is not discussed here in detail. State machines live in the VE_Port and require the outbound and return of an exchange’s sequences to pass through the same VE_Port; otherwise, the state of the protocol errors, and a failure occurs.
DATA CENTER BEST PRACTICES Here is a list of configuration differences between FICON and Open System environments that require VF LS when mixing environments on the same switch or Director: • The APTpolicy setting for FICON is not the same as Open Systems. FICON typically uses Port-Based Routing (PBR) or Domain-Based Routing (DBR) with Lossless enabled. Open Systems uses Exchange-Based Routing (EBR) without Lossless enabled. • In-Order Delivery (IOD) is used in FICON environments.
DATA CENTER BEST PRACTICES Brocade FICON Director n tio ec n on Ec b 0G IP Core A gateway LS 50 VE12 LS 60 VE22 LS 128 fig36_SAN_Design 01 xge IP Core B gateway xge1 10 GbE connection Figure 36. Two Virtual Fabric Logical Switches deployed on a Brocade FICON Director Figure 37 shows in more detail the anatomy of the circuits from the VE_Ports in each LS. The iproute that is configured has the destination subnet/mask and associates that destination with a local gateway on the IP core router.
DATA CENTER BEST PRACTICES • Per-Packet Load Balancing (PPLB) is not supported by Brocade. PPLB is often the cause of chronic and sometimes severe Out-Of-Order Packets (OOOP). To some degree, OOOP can easily be handled by TCP; however, PPLB will cause TCP to go into overdrive and put a dramatic strain on the system. PPLB also causes large latencies in delivering sequences to the upper layer protocols, because TCP is waiting for packets that are arriving out of order.
DATA CENTER BEST PRACTICES • They create separate LUNs for each data store and allow VMs to access data directly through N_Port ID Virtualization (NPIV). The advantage of this approach is that VMs can access data more or less directly through a virtual HBA. The disadvantage is that there are many more LUNs to provision and manage. Most VMs today tend to do very little I/O—typically no more than a few MB/sec per VM via very few IOPS.
DATA CENTER BEST PRACTICES Consider the IBM AIX VIO platform as an example to explain Unix workload virtualization. (Other vendor systems such as Oracle/Sun Solaris and HP HP-UX behave somewhat differently.) NPIV came late to Unix, with IBM recently adopting NPIV in AIX VIO 2.1 to improve traffic through the SCSI I/O abstraction layer. The difference is illustrated in Figure 38. FC HBA VIO 2.
DATA CENTER BEST PRACTICES Monitoring • Bottleneck Detection becomes very important here. Use it to monitor latencies on both the hypervisor and storage ports to identify high latencies as soon as you can. Address the latencies as soon as possible. • Brocade Fabric Watch is essential in early notification of potential issues in the fabric. Given the much higher concentration of I/O due to the server consolidation, you should closely monitor traffic levels.
DATA CENTER BEST PRACTICES Reviewing performance requirements: • Host-to-storage port fan-in/out ratios • Oversubscription ratios: –– Host to ISL –– Edge switch to core switch –– Storage to ISL • Size of trunks • Routing policy and currently assigned routes; evaluate actual utilization for potential imbalances Watching for latencies such as these: • Poor storage performance • Overloaded hosts or applications • Distance issues, particularly changes in usage (such as adding mirroring or too much workload) •
DATA CENTER BEST PRACTICES • Implement serial console server: Implement serial remote access so that switches can be managed even when there are network issues or problems during switch boot or firmware upgrades. • Use aliases: Use “aliases,” which give switch ports and devices meaningful names. Using aliases to give devices meaningful names can lead to faster troubleshooting. • Configure supportftp: Configure supportftp for automatic file transfers.
DATA CENTER BEST PRACTICES much smaller number (56 is recommended in Brocade fabrics). The domain count limit typically comes into play when a large number of small-port-count switches are deployed. Large-bladed server deployments, for example, can easily push the domain count up over recommended limits when embedded blade switches are part of the implementation. FC switches in blade server enclosures typically represent fewer than 32 ports.
DATA CENTER BEST PRACTICES • Error isolation and management: Most initiator errors are not propagated through to the fabric. Disconnecting an upstream port, for example, does not cause a fabric rebuild. Most management activities on the Brocade Access Gateway are also isolated from the fabric. One possible scenario is server administrators managing the Access Gateways and storage administrators simply providing LUNs and zoning support for the servers using NPIV.
DATA CENTER BEST PRACTICES Maintenance There is usually no need to keep the Brocade Access Gateway firmware levels synchronized with the firmware levels deployed in the fabrics to which it is connected (and Brocade supports connections from other vendors’ NPIV-enabled devices, where firmware synchronization is impossible). This can be significant for very large fabrics with many devices, including many Access Gateways.
DATA CENTER BEST PRACTICES fig11_SAN_Design Host Tier Storage Tier Tape Tier Figure 40. The same edge-core-edge tiered topology.
DATA CENTER BEST PRACTICES STORAGE Storage arrays have evolved significantly over the last few years. Performance has increased, capacities have exploded, and more LUNs are supported than ever before. The performance and capacity of low-end arrays have also improved. New features include the following: • Some arrays time out and reset their ports if they do not receive acknowledgements from the connected host after specific intervals.
DATA CENTER BEST PRACTICES Figure 41 shows a typical implementation of an in-line virtualized storage solution. The host or VM accesses storage via a storage controller (shown on top) through the storage network. The orange arrows indicate data access to and from the storage controller. The storage controller typically controls all access to the physical storage, shown on the right (and indicated by the blue arrows).
DATA CENTER BEST PRACTICES SECURITY There are many components to SAN security in relation to SAN design, and the decision to use them is greatly dependent on installation requirements rather than network functionality or performance. One clear exception is the zoning feature used to control device communication. The proper use of zoning is key to fabric functionality, performance, and stability, especially in larger networks.
DATA CENTER BEST PRACTICES Zoning recommendations include the following: • Always enable zoning. • Create zones with only one initiator (shown in Figure 42) and target, if possible. • Define zones using device WWPNs (World Wide Port Names). • Default zoning should be set to No Access. • Use FA-WWN if supported by Brocade FOS (v7.0 or later) and Brocade HBA driver (3.0 or later).
DATA CENTER BEST PRACTICES Use the SCC policy in environments where there is a need for strict control of fabric members. Since the SCC policy can prevent switches from participating in a fabric, it is important to regularly review and properly maintain the SCC ACL. DCC Policy The DCC policy restricts the devices that can attach to a single FC Port. The policy specifies the FC port and one or more WWNs allowed to connect to the port.
DATA CENTER BEST PRACTICES Together, the policy distribution and fabric-wide consistency settings provide a range of control on the security policies from little or no control to very strict control. For a detailed discussion of SAN security concepts and issues, see Securing Fibre Channel Fabrics, by Roger Bouchard, on Brocade Bookshelf (www.brocade.com/bookshelf).
DATA CENTER BEST PRACTICES • What is the average disk drive utilization (the greater the utilization, the longer the response times)? Contact your driver vendor to identify response times based on utilization for sizing workloads. Utilization 25% 50% 50% Disk drive response (milliseconds) • What is the raid level used? This will determine available disk space and performance for the application.
DATA CENTER BEST PRACTICES Some topics to consider for backup and restore as you plan for SAN expansion or a new design are these: • If the backup site is local, what is the window to complete the backup? If the backup site is remote, what is the window to complete the backup? How much of the bandwidth pipe is available? • Is there a dedicated backup server, or do other applications share the server? Is the backup SAN on a separate SAN or a shared network? • How often are full backups completed, and how l
DATA CENTER BEST PRACTICES TOOLS FOR GATHERING DATA Brocade SAN Health Brocade SAN Health is a free tool that allows SAN administrators to securely capture, analyze, and report comprehensive information about Brocade fabrics with switches running Brocade FOS and M-EOS operating systems and Cisco MDS fabrics running SANOS/NXOS.
DATA CENTER BEST PRACTICES • MB/s writes • Worst-case latency (ms) • HBA queue depth This is an example of a guideline for determining the queue depth for HBAs attached to an EMC array: Queue depth value = 8*n/h (where n= number of members in a metavolume group of disks, where within in the disk contiguous blocks are allocated; h = number of HBAs that can see the metavolume).
DATA CENTER BEST PRACTICES SUMMARY Once the initial discussions with key stakeholders are complete, data should be analyzed to support an optimized SAN design given business drivers, funding, and available resources. Sometimes it can be difficult to analyze the requirements from various organizations, and creating a radar chart may help to visually analyze competing requirements from internal groups (see Appendix B).
DATA CENTER BEST PRACTICES APPENDIX A: IMPORTANT TABLES The following table shows the support distance based on cable type and data rates. OM1 Link Distance OM2 Link Distance OM3 Link Distance OM4 Link Distance OS1 Link Distance Speed Name 62.
DATA CENTER BEST PRACTICES APPENDIX B: MATRICES This section provides example checklists and tables you can use to identify dominant factors, including facilities that will have an impact on the SAN design.
DATA CENTER BEST PRACTICES Metrics and Impact on SAN Design and Performance The following table details the metrics that need to be collected and their impact on SAN design and performance.
DATA CENTER BEST PRACTICES Consolidated SAN Snapshot SAN Requirements Data (Complete for each SAN) Fabric Information Target # of user ports per fabric Target # of total ports per fabric Target # of switches per fabric (# switches/switch type, total switches) Number of fabrics Number of sites in environment Topology (core-edge, ring, mesh, other) Maximum hop count Expected growth rate (port count) Fabric licenses SAN Device Information Number/types of hosts and OS platforms Number/types of storage devices
DATA CENTER BEST PRACTICES Application-Specific Details Backup/Restore Infrastruction Servers System OS Version, Patch Level HBA Driver Version Server 1/HBA Server 2/HBA Server 3/HBA Backup Software Vendor Version Patch FC Switch Vendor Model Firmware Brocade Storage Vendor Model Firmware Array 1 Array 2 Tape Libray Vendor Model Firmware Library Note: Keep a similar table for each application.
DATA CENTER BEST PRACTICES Quantitative Analysis: Radar Maps SAN Admin Radar Map Rank (1 is low, 10 is high) SAN/Storage Admin Concerns Notes ISL utilization 8 Is traffic balanced across ISLs during peaks? Switch outage 1 Have there been switch outages? If so what was the cause? Zoning policy 6 Is the zoning policy defined? Number of switches in the fabric 10 Is the current number of switches a concern for manageability? Scalability 6 Can the existing design scale to support additional sw
DATA CENTER BEST PRACTICES Facilities Radar Map Facility Rank (1 is low, 10 is high) Concern for physical real estate 8 What is the total available space for all the hardware? Support racks 10 How many racks are needed? Power 10 Is there adequate power? Air conditioning 9 Is there adequate air conditioning? Physical location 8 How important is it to have all the equipment in the same physical location or aisle? Cable labeling 10 Are cables labeled for easy identification? Switch label
DATA CENTER BEST PRACTICES APPENDIX C: PORT GROUPS A port group is a group of eight ports, based on the user port number, such as 0–7, 8–15, 16–23, and up to the number of ports on the switch or port blade. Ports in a port group are usually contiguous, but they might not be. Refer to the hardware reference manual for your product for information about which ports can be used in the same port group for trunking.
DATA CENTER BEST PRACTICES Gen 5 Fibre Channel Platforms The Brocade FC16-48 and FC16-32 blades for the Brocade DCX 8510 Backbone and the Brocade 6520, 6510, and 6505 Switches provide trunk groups with a maximum of 8 ports per trunk group. The trunking octet groups are in the following blade port ranges: 0-7, 8-15, 16-23, 24-31, 32-39, and 40-47. (Trunk groups 32-39 and 4047 are obviously not applicable to FC16-32). Trunk boundary layout is on the faceplate of the blade.
DATA CENTER BEST PRACTICES Brocade 6520 Trunk Groups 1 2 3 4 5 6 7 8 9 10 11 12 Figure 48. Brocade 6520 Front Port Groups Brocade 6510 Trunk Groups 1 2 3 4 5 6 Figure 49. Brocade 6510 Front Port Groups Brocade 6505 Trunk Groups 1 2 3 Figure 50.
DATA CENTER BEST PRACTICES APPENDIX D: TERMINOLOGY Term Brief Description Base switch Base switch of an enabled virtual fabric mode switch ClearLink Diagnostics Diagnostics tool that allows users to automate a battery of tests to verify the integrity of all 16 Gbps transceivers in the fabric Default switch Default switch of an enabled virtual fabric mode switch E_Port A standard Fibre Channel mechanism that enables switches to network with each other Edge Hold Time Enables the switch to time ou
DATA CENTER BEST PRACTICES APPENDIX E: REFERENCES Software and Hardware Product Documentation • Brocade Fabric OS v7.1 Release Notes • Brocade Fabric OS Administrator’s Guide, supporting Brocade Fabric OS v7.1 • Brocade Fabric OS Command Reference Manual, supporting Brocade Fabric OS v7.1 • Brocade Fabric Watch Administrator’s Guide, supporting Brocade Fabric OS v7.1 • Brocade Access Gateway Administrator’s Guide, supporting Brocade Fabric OS v7.
DATA CENTER BEST PRACTICES © 2013 Brocade Communications Systems, Inc. All Rights Reserved. 05/13 GA-TB-329-04 ADX, AnyIO, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, and Vyatta are registered trademarks, and HyperEdge, The Effortless Network, and The On-Demand Data Center are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
