Summary of Best Practices for Storage Area Networks IBM This document can be found on the web, www.ibm.com/support/techdocs Version: 1.2 Date: January 26, 2012 IBM SAN Central Jim Blue jeblue@us.ibm.
Notices: This white paper reflects the IBM SAN Central support team concerns from customer feedback about the lack of a single source of best practices related to storage area networks (SANs) and storage systems. It has been produced and reviewed by members of numerous IBM support teams and is presented “As-Is”.
Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or TM), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries.
Table of Contents 1 Introduction .................................................................................................................. 5 2 Zoning .......................................................................................................................... 6 3 Fabric and Switch ......................................................................................................... 9 4 Disk Storage .....................................................................................
A Summary of SAN Best Practices for Storage Area Networks 1 Introduction Whether a SAN environment is simple or complex, many corporate SAN administrators and their management, want to have a collection of general "rules of thumb" on best practices in a single document. In general, a vast amount of best practices information and “rules of thumb” currently exists, but these informational nuggets are scattered across numerous documents, web sites and other sources.
2 Zoning Zoning is the way Storage Area Networks (SANs) keep devices isolated from each other; this has implications for security, fabric stability, and resource management. Without zoning, it would be very common for a problem on one host to be able to affect every host in the fabric; without zoning the problems this can cause are difficult or impossible to troubleshoot.
a WWPN for a device, switches interpret the WWNN to designate all associated ports for the device. Using the WWNN in a zone can cause multipathing issues where there are too number of paths between a server and storage. A possible refinement is to keep destinations isolated from each other in “single-initiator, single-target” zoning. This is sometimes used in environments where it is common for a single host to access multiple destination devices.
with multiple ports will help prevent a single storage port from being the source or cause a performance bottleneck due to high latency or outfight congestion. One overall theme for SAN best practice guidelines is consistency, and this concept is equally applicable to zone element naming conventions. A zone name should indicate which devices are enabled to communicate by the specific zone along with enough info to determine which fabric the zone can be found.
development environments compared to the desired goal of stable operations for the production side. Switch vendors offer management tools and/or commands which can assist a SAN administrator with planning and changing a zone configuration. When such tools are available, the SAN administrator should make use of them as a validity check if for no other reason. Switches will attempt to keep the impact of zone changes as local as possible to just the edge devices that may be impacted by the changes.
• Port Statistics Counters (link reset, loss of sync, loss of signal, detection of faulty frames, dropped frames) • Event Logs (Fabric reconfiguration, fabric rebuild, device leaving fabric, hardware failure) Since ‘Event Logs’ can be configured to show (or not) different types of informational, warning and error messages, the need to routinely clear event logs will vary.
configured into multiple virtual switches and each virtual instance has a different Domain ID, the inclusion of a Domain ID in the switch name is not suggested. Switch Domain IDs within redundant fabrics should be unique between the fabrics if at all possible. This practice allows administrators and product support specialists to review data from edge devices (servers or storage systems) and be able to quickly determine which fabric and switch a specific device port is a member.
• • ISL to switch Edge-1-port_13 SVC1-node 1-port 10 In SAN environments with redundant fabrics and switches from the same vendor are deployed, use the same topology in the redundant fabrics. Using the same topology will be easier for administrators to keep the workload balanced between fabrics. Some exceptions are to be expected, such as single data paths to certain tape drives.
one inter-switch link (ISL) between the two switches. Since this configuration is repeated for multiple storage ports, there can be more and more servers communicating across a single common ISL. The ratio of total server bandwidth to ISL bandwidth is known as the ISL oversubscription. For example, there are eight servers connecting to a common storage port traversing an ISL. Each server has a 2 Gbps port connection while the ISL is running at 4 Gbps.
4 Disk Storage 4.1 General topics One theme which is repeated throughout best practice statements centers on one word: consistency. And consistency with disk storage systems is very applicable across many areas of operation. One area which is often overlooked is consistency of code version among multiple storage systems of the same type. The primary reasoning for this suggestion is the code level on a storage system is increasing playing a role in the overall interoperability within the SAN environment.
The above values are for 3.5 inch hard disk drives (HDD), and can be increased by approximately 15% when using the smaller 2.5 inch HDD. Thus, an eight 15K RPM, 3.5” drive RAID 5 array (7 + P) should be capable of between 1155 to 1400 IOps. Another generic factor for disk-based storage systems involves separation of different types of workload over storage system ports.
extent pools with a pool to be managed by each storage controller to fully exploit the DS8000’s resources. There is a minimum of one rank per extent pool. Although it is possible to configure many ranks in an extent pool, for SAN Volume Controller performance optimization, best practice guidelines recommend a configuration of one extent pool per rank per array. This configuration allows for directing storage allocations to a known location within the DS8000.
• Ensure that you have an equal number of extent pools and volumes, equally spread on the Disk Adapters and the two servers of the DS8000 storage subsystem. • Ensure that managed disk groups contain managed disks with similar characteristics and approximately the same capacity. Consider the following factors: The number of DDMs in the array site (for example 6 + P + S or 7 + P) and the physical disk type (for example, 10K/15K rpm).
4.3 XIV The main goal for the host connectivity is to create a balance of the resources in the XIV Storage System. Balance is achieved by distributing the physical connections across the Interface Modules. A host usually manages multiple physical connections to the storage device for redundancy purposes by a SAN connected switch. It is ideal to distribute these connections across each of the Interface Modules.
the host workload can be tailored to use multiple threads or spread the work across multiple volumes. The XIV Storage architecture was designed to perform under real-world customer production workloads, with lots of I/O requests at the same time. Queue depth is an important host bus adapter (HBA) setting because it essentially controls how much data is allowed to be “in flight” onto the SAN from the HBA. A queue depth of 1 requires that each I/O request be completed before another is started.
4.4 DS3000, DS4000 and DS5000 Storage Systems With DS3000, DS4000 or DS5000 arrays, the number of physical drives to put into an array always presents a compromise. On one hand striping across a larger number of drives can improve performance for transaction based workload and on the other it can have a negative effect on sequential workload.
turns out to be less critical based on the caching it provides, and less variation in its I/O profile used to access back-end disks. For the SVC the maximum destage size is 32k and thus it is not possible to achieve full stride writes for random workload. For the SVC the only opportunity for full stride writes occurs with large sequential workloads, and in that case the larger the segment size, the better. Larger segment sizes can have an adverse effect on random I/O however.
b) All MDisks within an MDisk Group should be of the same capacity and RAID level, and in keeping with rule 1, made up of the same kind of disk. c) Never spread MDisk Groups across multiple back-end disk units, such as two different IBM 2107 units. d) If an array has multiple segment sizes to choose from, choose the largest if a particular MDisk Group will be used for multiple kinds of applications. (This rule may not apply to some disk arrays, which do not allow the selection of disk segment size.
mechanisms in Fibre Channel for dealing with congestion in the fabric itself are not very effective. The problems caused by fabric congestion can range anywhere from dramatically slow response time all the way to storage access loss. These issues are common with all high-bandwidth SAN devices and are inherent in Fibre Channel; they are not unique to the SVC. When an Ethernet network becomes congested, the Ethernet switches simply discard frames for which there is no room.
unable to communicate to your disk arrays or mirror write cache because you have a single congested link leading to an edge switch. If at all possible, plan and design for the maximum size configuration the SVC cluster could eventually reach. The design of the SAN can change radically for larger numbers of hosts. Modifying the SAN later to accommodate a larger-than-expected number of hosts will either produce a poorly-designed SAN or be very difficult, expensive, and disruptive.
Within a particular adapter, the ports should be selected based on the port group (0/1 and 2/3). The 4-port adapter in the DS8300 are oversubscribed on port group boundaries • • 4-adapter/8-port configuration, 4 port-0’s and 4 port 2’s 8-adapter/16 ports, 8 port-0’s and 8 port-2’s If two SVC clusters share a common DS8000 storage system, they should use separate paths per SVC cluster per DS8000 host adapter. (E.g. SVC A uses ports 0 and 2, while SVC B uses ports 1 and 3.
• • • • • • • • • • • • • • • Do not route the cables near unprotected steam or refrigeration lines. Do not coil or bend the cable to less than a 96.0-mm (3.78 in.) diameter. If any strain is induced on the cable, the minimum diameter will need to be increased. Do not pull cables into position; place them. Do not grasp the cable with pliers. Do not attach a pull rope or wire to the connectors. Always clean the connectors before attaching them.
When installing fiber cabling, use labels on each end of the cable which provides sufficient information to identify the appropriate connection ports. Use a logical naming scheme to uniquely identify every cable along with its source and destination points. Good labeling reduces potential confusion as well as easing the task of finding a specific cable in a bundle. Cable labels should always be included as a key step for routine change management documentation and standard operating procedures.
6 High Availability In general, the size of the SAN as measured in the number of physical switches will normally determine the design that is most effective to employ. A single switch operates as a stand-alone with no connections to other switches. Two switches are cascaded, or simply connected to each other with a sufficient number of ISLs to meet oversubscription ratios. As the switch count increases, mesh or partial mesh designs can be considered until six or seven total switches.
of the safety factor for traffic expansion, implement a spare ISL or ISL trunk. The fabric still needs to be able to avoid congestion if an ISL were to go down due to issues such as switch line card failure. As stated earlier, a SAN design can exceed the best practice guidelines for ISL oversubscription ratios of 7:1 with a clear understanding of the traffic patterns.
For systems in high availability SAN environments, individual hosts will have multiple paths to their respective storage volumes. Without multipathing utilities, many operating systems would treat each path as a completely separate and unique resource even though the end storage resource was the same in all cases.
Change control processes should be designed to make the execution of change go as smoothly and quickly as possible, not merely slow down change (through endless checks, verifications, and meetings) in an attempt to reduce risk. The delay imposed by inefficient change controls lead to rush jobs that actually increase net risk.
In addition, a completed record should contain a complete list detailing how the changes were made. This can be as simple as copying a checklist out of the Standard Operating Procedures, checking off each step as it is completed, and pasting the completed list into the record. The total information on implementation should allow the change to be backed out later if needed. For every common change, there should be a documented standard operating procedure (SOP) implementing the change.
documentation templates is beyond the scope of this white paper. Certainly, the example “procedure” for choosing an MDisk Group is inadequate for most complex SAN environments. The easiest way to develop a Standard Operating Procedure is to integrate it with the ensuing checklist to implement the operation. The procedural notes that assist the Storage Administrator in developing the ticket are in italics. Those notes would obviously not be part of the ticket template, just the SOP document.
Switch: __48000_1_ _ Port: __39__ WWPN: __00:11:22:33:44:55 :66:77__ Port Name:__XYZ123_A__ Slot/Port: __5__ __Clust er 2, Group 1, Ports 1__ HBA B: Switch: __48000_2 __ Port: WWPN: Port Slot/Por __39_ __00:11:22:33:44:55:66:8 Name:__XYZ123_B t: __6__ _ 8__ __ __Cluster 2, Group 1, Ports 2__ 4.
Name: Size: __XYZ123_1__ __200GB__ ioGroup: __2__ mDisk Group: Mode: __Striped__ __DS8300_300_1__ 11.___ Map the new vDisk to the Host 12.___ If this is the last ticket of the evening, obtain a config dump and attach it to this ticket under __new.zip If this is not the last ticket, record the ticket where this information may be found here: __ __ 13.___ Map the new volume(s) to the Host 14.
Key factors to consider are how dynamic is the business’ SAN environment and the typical degree of urgency for changes. The business management and SAN administrators will need to determine a schedule for routine changes (such as new storage allocations or re-zoning operations) deemed appropriate and prevents the deployment of massive numbers of changes being implemented in a single maintenance window.
information that is needed for the configuration. The configuration interfaces will not keep track of which project a given storage allocation was intended for. Lastly, there may be a great many things that need to be kept track of that a SAN device will never record, such as the name of the project/application that owns a server; contact names, in-service date, etc. NOTE: For obvious reasons, none of this documentation should be stored on a SANattached server.
1) An environmental dependencies checklist should be created and used. As an example; if upgrading an HBA driver on a server, what are the implications for other server utilities and hardware (such as multipathing driver, HBA firmware, etc.) and will support be maintained attaching to another device. These types of questions must be answered prior to upgrading code on any of the devices. 2) IBM recommends that a proactive code maintenance schedule be put into place for all devices.
per week. Each update item will have a very brief description as well as a link to browse directly to the information. 9 Monitoring One of the most often overlooked things that need monitoring in a SAN is host path monitoring. A very common reason for server outages in general is missing paths that turn what should be non-disruptive maintenance (or failures) into a real problem. For example: For whatever reason, a server loses connectivity over one of its HBAs.
general guidelines or suggested threshold levels to determine when a connection and/or device is performing well or poorly. Since each network is unique, meaningful thresholds will need to be determined via a trial and error method. One alternate method involves using TPC to collect performance metrics over a period of time, such as a full week, and then use the various peaks as a starting point for setting thresholds.
The key question of what needs to be monitored is also the most difficult question to answer. “It depends.” There are a number of questions which have to be answered first. • • What monitoring tools are available? How much of the SAN environment is appropriate and/or would benefit from proactive monitoring? • How often will monitoring be consulted and how will it be used? Depending on the capability of available tools, the following items are a good starting point for proactive monitoring.
9.2 Tools Management is one of the key issues behind the concept of infrastructure simplification. The ability to manage heterogeneous systems at different levels as though they were a fully-integrated infrastructure offering the system administrator a unified view of the entire SAN environment is a goal that many vendors and developers have been striving to achieve.
10 Solution-centric There are a number of best practice items that deserve some amount of coverage which do not readily lend themselves to one of the previous topic headings. Thus, this section is called solution-centric and not miscellaneous since these items can have an impact of overall SAN operations. 10.1 Security Security has always been a major concern for networked systems administrators and users.
access which LUN by means of the storage device control program. Whenever the host accesses a particular LUN, the storage device will check its access list for that LUN, and it will allow or disallow access to the LUN. Server-level access control is called persistent binding. Persistent binding uses configuration information stored on the server, and is implemented through the server’s HBA driver.
connectivity. The last thing that we want to do with security is to create SAN islands, as that would destroy the essence of the SAN. True cross-platform data sharing solutions, as opposed to data partitioning solutions, are also a requirement. Security and access control also need to be improved to guarantee data integrity. Encryption is the translation of data into a secret code and is the most effective way to achieve data security.
connected to the management LAN, and allows only traffic from the management stations and certain protocols that you define. At a high level, some of the security best practices include the following: • • • • • • • Default configurations and passwords should be changed. Configuration changes should be checked and double checked to ensure that only the data that is supposed to be accessed can be accessed. Management of devices usually takes a “telnet” form—with encrypted management protocols being used.
A SAN is a simple thing, a path from a server to a common storage resource. So, where do the complexity and interoperability concerns originate? The first key point to remember is that interoperability is based on the lowest common denominator for features, functionality and available services. Limited budgets and too few skilled people have pushed many organizations into looking for short term solutions.
There is an additional complication with certain types of applications, such as backup solutions. In these cases, the application vendor will also produce a matrix of tested, supported hardware. 10.4 Separation of Production and Test Segments A test environment is useful in many instances. A key factor for this testbed is that it should contain at least one of the production environment’s critical devices.
Appendix A - References and additional materials The following listing has been used in the development of this paper. These publications and materials were not the sole reference source used for this paper. They are not presented in any particular order. Some of these references may have multiple revisions, and the author has strived to utilize the most recent version.
IBM Redbooks. SG24-7521. SAN Volume Controller: Best Practices and Performance Guidelines Brocade Data Center Best Practices Guide. GA-BP-300-00. Fabric Resiliency Best Practices Brocade Data Center Best Practices Guide. GA-BP-329-00. SAN Design and Best Practices Brocade Data Center Fabric Best Practices Guide. GA-BP-036-02. Cabling the Data Center Brocade Publications. 53-1000672-03. Performance User Manual Brocade Publications. 53-0000350-04. LAN Guidelines For Brocade SilkWorm Switches.
(This page is intentionally blank) © IBM Copyright, 2012 www.ibm.
