McDATA® Products in a SAN Environment Planning Manual P/N 620-000124-500 REV A
Record of Revisions and Updates Revision Date Description 620-000124-000 5/2002 Initial release of the manual. 620-000124-100 9/2002 Revision of the manual to describe the Intrepid 6140 Director, Sphereon 4500 Switch, and Release 6.3 of the Enterprise Fabric Connectivity Manager application. 620-000124-200 2/2003 Revision of the manual to include additional information and describe Release 7.1 of the Enterprise Fabric Connectivity Manager application.
Contents Preface .......................................................................................................................... xiii Chapter 1 Introduction to McDATA Multi-Protocol Products Product Overview.............................................................................1-2 Multi-Protocol Hardware .........................................................1-2 SAN Management Applications..............................................1-5 Directors .......................................
Contents Chapter 2 Product Management Product Management.......................................................................2-2 Out-of-Band Management .......................................................2-2 Inband Management .................................................................2-5 Management Interface Summary............................................2-6 Management Server Support ..........................................................2-7 Management Server Specifications .............
Contents General Fabric Design Considerations ....................................... 3-29 Fabric Initialization ................................................................. 3-30 Fabric Performance ................................................................. 3-31 Fabric Availability ................................................................... 3-37 Fabric Scalability ..................................................................... 3-39 Obtaining Professional Services...................
Contents Management Server, LAN, and Remote Access Support ...........5-9 Management Server ..................................................................5-9 Remote User Workstations..................................................... 5-11 SNMP Management Workstations........................................5-13 SANpilot Interface...................................................................5-14 Security Provisions .........................................................................
Contents Task 12: Assign Port Names and Nicknames............................. 6-13 Task 13: Complete the Planning Worksheet ............................... 6-14 Task 14: Plan AC Power ................................................................ 6-28 Task 15: Plan a Multiswitch Fabric (Optional) ........................... 6-29 Task 16: Plan Zone Sets for Multiple Products (Optional) ....... 6-30 Task 17: Plan SAN Routing (Optional)........................................
Contents viii McDATA Products in a SAN Environment - Planning Manual
Tables 2-1 Out-of-Band and Inband Product Support Summary ............................ 2-6 3-1 ISL Transfer Rate Versus Fabric Port Availability (Two-Director Fabric) 3-22 4-1 4-2 4-3 4-4 mSAN Routing Domain ............................................................................ mSAN Supported Limits ........................................................................... mFCP Versus iFCP .....................................................................................
Tables x McDATA Products in a SAN Environment - Planning Manual
Figures 1-1 1-2 1-3 1-4 1-5 1-6 1-7 1-8 1-9 Cabinet-Mount McDATA Products ........................................................... 1-4 Intrepid 6064 Director .................................................................................. 1-9 Intrepid 6140 Director ................................................................................ 1-11 Intrepid 10000 Director .............................................................................. 1-13 Sphereon 3232 Fabric Switch .................
Figures xii 3-10 3-11 3-12 3-13 3-14 3-15 3-16 3-17 3-18 2-by-14 Core-to-Edge Fabric ...................................................................... Example Multiswitch Fabric ...................................................................... ISL Oversubscription .................................................................................. Device Locality ............................................................................................ Device Fan-Out Ratio .....................
Preface This publication is part of a documentation suite that supports McDATA® multi-protocol switching and routing products, including the: • Intrepid® 6064 Director. • Intrepid 6140 Director. • Intrepid 10000 Director. • Sphereon™ 3232 Fabric Switch. • Sphereon 4300 Fabric Switch. • Sphereon 4500 Fabric Switch. • Eclipse™ 1620 SAN Router. • Eclipse 2640 SAN Router.
Preface Chapter 1, Introduction to McDATA Multi-Protocol Products This chapter provides an overview of McDATA multi-protocol products, and describes product performance and connectivity, security, and serviceability features. Chapter 2, Product Management - This chapter describes out-of-band and inband product management; the management server; product firmware; backup and restore features; and software. Overviews of the graphical user interfaces (GUIs) and command line interface (CLI) are included.
Preface Related Publications Other publications that provide additional information about McDATA products include: • Intrepid 6064 and 6140 Directors: — McDATA Intrepid 6064 and 6140 Directors Element Manager User Manual (620-000172). — McDATA Intrepid 6064 Director Installation and Service Manual (620-000108). — McDATA Intrepid 6140 Director Installation and Service Manual (620-000157). • Intrepid 10000 Director: — McDATA Intrepid 10000 Director Element Manager User Manual (620-000227).
Preface • Eclipse 2640 SAN Router: — McDATA Eclipse 2640 SAN Router Administration and Configuration Manual (620-000203). — McDATA Eclipse 2640 SAN Router Installation and Service Manual (620-000202). • General Support Publications: — McDATA SANavigator Software User Manual (621-000013). — McDATA EFC Manager Software User Manual (620-000170). — McDATA SANvergence Manager User Manual (620-000189). — McDATA SANpilot User Manual (620-000160). — McDATA E/OS SNMP Agent User Manual (620-000168).
Preface Where to Get Help For technical support, contact the McDATA Solution Center. The center provides a single point of contact for assistance and is staffed 24 hours a day, seven days a week, including holidays. Contact the center at the phone number, fax number, or e-mail address listed below. Please have the product serial number (printed on the service label attached to the director or switch) available. Phone: (800) 752-4572 or (720) 558-3910 Fax: (720) 558-3851 E-mail: support@mcdata.
Preface All other trademarked terms, indicated by a registered trademark symbol (®) or trademark symbol (™) on first use in this publication, are trademarks of their respective owners in the United States or other countries or both. Laser Compliance Statement Product laser transceivers are tested and certified in the United States to conform to Title 21 of the Code of Federal Regulations (CFR), Subchapter J, Parts 1040.10 and 1040.11 for Class 1 laser products.
Preface International Safety Conformity Declaration (CB Scheme) A certification bodies (CB) test report supporting a product indicates safety compliance with the International Electrotechnical Commission (IEC) system for conformity testing and certification of electrical equipment (IECEE) CB scheme. The CB scheme is a multilateral agreement among participating countries and certification organizations that accepts test reports certifying the safety of electrical and electronic products.
Preface • Products conform with all protection requirements of EU directive 73/23/EEC (Low-Voltage Directive) in accordance with the laws of the member countries relating to electrical safety. • Products conform with all protection requirements of EU directive 93/68/EEC (Machinery Directive) in accordance with the laws of the member countries relating to safe electrical and mechanical operation of the equipment.
Preface People’s Republic of China CCC Mark The China Compulsory Certification mark (CCC mark) on a product indicates compliance with People’s Republic of China regulatory requirements for safety and EMC (for information technology equipment) as set forth by the National Regulatory Commission for Certification and Accreditation.
Preface Korean MIC Mark Mexican NOM Mark NOM Russian GOST Certification Danger and Attention Statements xxii The Korean Ministry of Information and Communications mark (MIC mark) on a product indicates compliance with regulatory requirements for safety and EMC (for information technology equipment) as authorized and certified by the Korean Radio Research Institute (RRI).
Preface DANGER Use the supplied power cords. Ensure the facility power receptacle is the correct type, supplies the required voltage, and is properly grounded. DANGER Utiliser les câbles d’alimentation fournis. S’assurer que la prise de courant du local est du type correct, délivre la tension requise et est correctement raccordée à la terre. GEFAHR Die mitgelieferten Netzkabel verwenden.
Preface PERICOLO Usare il cavo di alimentazione in dotazione. Assicurarsi che la presa di corrente a disposizione sia del tipo corretto, eroghi la tensione richiesta e sia dotata di messa a terra idonea. PERIGO Use os cordões elétricos fornecidos. Certifique-se de que o tipo de receptor de energia da facilidade é apropriado, fornece a voltagem necessária, e está corretamente aterrado. PELIGRO Utilice los cables de alimentación proporcionados.
Preface The following ATTENTION statements appear in this publication and describe practices that must be observed while installing or servicing a product. An ATTENTION statement provides essential information or instructions for which disregard or noncompliance may result in equipment damage or loss of data.
Preface xxvi McDATA Products in a SAN Environment - Planning Manual
1 Introduction to McDATA Multi-Protocol Products The enterprise-level storage area network (SAN) of today is typically complex and managed at the device layer. These problems result in SANs that use storage assets inefficiently, and are complex, error prone, expensive, and time-consuming to manage.
Introduction to McDATA Multi-Protocol Products 1 Product Overview McDATA provides storage network solutions that are integrated across a variety of platforms, original equipment manufacturers (OEMs), and locations. Solutions are modular and support multiple technologies (current and future), protocols, and data transmission speeds. These solutions include: Multi-Protocol Hardware • Multi-protocol hardware. • SAN management applications.
Introduction to McDATA Multi-Protocol Products 1 — 32-port Sphereon 3232 Switch. — 12-port Sphereon 4300 Switch. The switch provides both switched fabric and Fibre Channel arbitrated loop (FC-AL) connectivity. — 24-port Sphereon 4500 Switch. The switch provides both switched fabric and FC-AL connectivity. Refer to Fabric Switches for detailed information about each product.
Introduction to McDATA Multi-Protocol Products 1 6. Sphereon 4500 Switch. 7. Intrepid 6140 Director. 8. Eclipse 1620 SAN Router. 9. Eclipse 2640 SAN Router. 10. Intrepid 10000 Director.
Introduction to McDATA Multi-Protocol Products 1 SAN Management Applications McDATA offers the following SAN management applications installed on the rack-mount management server: • SANavigator® application - The SANavigator application (Version 4.2 or later) is an integrated software package that provides management of an enterprise-wide, heterogeneous SAN (with multiple vendor applications) from a single console.
Introduction to McDATA Multi-Protocol Products 1 Refer to Chapter 2, Product Management for information about SAN management applications and the management server. Chapter 2 also describes switch management through: • The Internet using a product’s SANpilot® interface. • Inband (Fibre Channel) application clients. • Simple network management protocol (SNMP) workstations. • A command line interface (CLI) or PC-attached Telnet session.
Introduction to McDATA Multi-Protocol Products 1 Director Performance Directors provide the following general performance features: • High bandwidth - Ports on Intrepid-series directors provide full-duplex serial data transfer at a rate of 1.0625, 2.1250, or 10.2000 gigabits per second (Gbps).
Introduction to McDATA Multi-Protocol Products 1 — A multiswitch fabric topology provides the ability to connect directors (and other McDATA switch elements) through expansion ports (E_Ports) and interswitch links (ISLs) to form a Fibre Channel fabric. Director elements receive data from a device and, based on the destination N_Port address, route the data through the fabric (and possibly through multiple switch elements) to the destination device.
Introduction to McDATA Multi-Protocol Products 1 Figure 1-2 Intrepid 6064 Director The director supports McDATA’s non-blocking extendable open network (EON™) architecture and concurrent firmware downloads through hot code activation (HotCAT®) technology. The director also provides a modular design that enables quick removal and replacement of FRUs, including a: • Cable management assembly and front bezel with power (green) and system error (amber) light-emitting diodes (LEDs).
Introduction to McDATA Multi-Protocol Products 1 • A minimum of eight to a maximum of 16 Fibre Channel port cards as follows: — Fiber port module (FPM) cards. Each FPM card provides four 1.0625 Gbps Fibre Channel port connections through duplex small form factor pluggable (SFP) fiber-optic transceivers. — Universal port module (UPM) cards. Each UPM card provides four 2.1250 Gbps Fibre Channel port connections through duplex SFP fiber-optic transceivers. — Ten-gigabit port module (XPM) cards.
Introduction to McDATA Multi-Protocol Products 1 Figure 1-3 Intrepid 6140 Director The director supports McDATA’s non-blocking EON architecture and concurrent firmware downloads through HotCAT technology. The director also provides a modular design that enables quick removal and replacement of FRUs, including a: • Front bezel with power (green) and system error (amber) LEDs. • Redundant CTP (2.1250 and 10.2000 Gbps operation) logic cards. • Redundant SBAR assembly logic cards.
Introduction to McDATA Multi-Protocol Products 1 • Backplane. • A minimum of 16 to a maximum of 35 Fibre Channel port cards as follows: — UPM cards. Each UPM card provides four 2.1250 Gbps Fibre Channel port connections through duplex SFP fiber-optic transceivers. — XPM cards. Each XPM card provides one 10.2000 Gbps Fibre Channel port connection through a duplex XFP fiber-optic transceiver.
Introduction to McDATA Multi-Protocol Products 1 Figure 1-4 Intrepid 10000 Director The director supports McDATA’s non-blocking nScale™ architecture that allows the product to be flexibly partitioned into multiple (up to four) separate directors (FlexPar™ feature), each with its own management and Fibre Channel services subsystems. In addition, the director supports concurrent firmware downloads through HotCAT technology. FlexPar-specific software can also be independently and concurrently upgraded.
Introduction to McDATA Multi-Protocol Products 1 • Smaller fabrics or SAN islands built around the director (but separated through FlexPars) benefit from better resource utilization because port configurations are flexible to accommodate change and the hardware does not require over-provisioning for growth. The FlexPar feature enables additional fabric ports to be added to a partition on demand, without interrupting fabric traffic. Refer to Inter-FlexPar Routing for information.
Introduction to McDATA Multi-Protocol Products 1 Fabric Switches In similar fashion to directors, fabric switches also provide highperformance, dynamic connections between end devices in a Fibre Channel switched network. Fabric switches also support mainframe and OSI computing environments. Through non-blocking architecture and limited FRU redundancy, fabric switches also offer high availability and high-performance bandwidth.
Introduction to McDATA Multi-Protocol Products 1 Sphereon 3232 Fabric Switch Figure 1-5 The Sphereon 3232 Fabric Switch operates at 2.1250 Gbps, provides fabric connectivity for to up to 32 Fibre Channel devices, and supports FICON, EON architecture, and HotCAT technology. Figure 1-5 illustrates the switch. Sphereon 3232 Fabric Switch The switch provides a modular design that enables quick removal and replacement of FRUs, including: • Redundant power supplies and cooling fans.
Introduction to McDATA Multi-Protocol Products 1 The switch rear panel provides a 9-pin DSUB maintenance port for connection to a local terminal or remote terminal. Although the port is typically used by authorized maintenance personnel, operations personnel can use the port to configure switch network addresses. Sphereon 4300 Fabric Switch The Sphereon 4300 Fabric Switch operates at 1.0625 or 2.
Introduction to McDATA Multi-Protocol Products 1 The switch provides a modular design that enables quick removal and replacement of FRUs, including up to 12 duplex SFP fiber-optic port transceivers. Shortwave laser transceivers are available for transferring data over multimode fiber-optic cable. Longwave laser transceivers are available for transferring data over singlemode fiber-optic cable. Fiber-optic cables attach to switch port transceivers with duplex LC connectors.
Introduction to McDATA Multi-Protocol Products 1 Figure 1-7 illustrates the switch. Figure 1-7 Sphereon 4500 Fabric Switch The switch provides a modular design that enables quick removal and replacement of FRUs, including: • Redundant power supplies and cooling fans. The switch provides two power supplies, each with an AC power receptacle and three cooling fans (six fans total). • Up to 24 duplex SFP fiber-optic port transceivers.
Introduction to McDATA Multi-Protocol Products 1 The switch rear panel provides a 9-pin DSUB maintenance port for connection to a local terminal or remote terminal. Although the port is typically used by authorized maintenance personnel, operations personnel can use the port to configure switch network addresses. SAN Routers The Fibre Channel protocol was designed for high-performance channel and storage applications within the limited confines of a data center.
Introduction to McDATA Multi-Protocol Products 1 SAN Router Performance • Implement iSAN routing and BC/DR solutions - SAN routers provide TCP/IP-based (iFCP protocol) distance extension solutions that connect geographically-dispersed SANs into an internetworked storage area network (iSAN), perform iSAN routing, and run business continuance and disaster recovery applications over existing MAN or WAN infrastructures. Refer to iSAN Routing and Implementing BC/DR Solutions for detailed information.
Introduction to McDATA Multi-Protocol Products 1 • High-availability - To ensure an availability of 99.9%, multiprotocol SAN router design provides a redundant configuration of power supplies and cooling fans. High availability is also provided through concurrent firmware upgrades and spare or unused multi-protocol ports.
Introduction to McDATA Multi-Protocol Products 1 SAN router ports operate as follows: • Two user-configured FCP (FIBRE CHANNEL 1 and 2) ports provide 1.0625 Gbps Fibre Channel storage connectivity using SFP port connectors. FCP ports can be configured for: — Auto negotiation (FC-Auto) operation. This is the default selection.
Introduction to McDATA Multi-Protocol Products 1 The SAN router front panel provides two AC power receptacles, an Ethernet LAN connector (MGMT), port status LEDs, green power supply status (PS 1 and PS 2) LEDs, and a green system status (SYS) LED. The panel also provides a 9-pin DSUB maintenance port (MGMT) for connection to a local terminal or remote terminal.
Introduction to McDATA Multi-Protocol Products 1 — Fibre channel fabric loop port (FL_Port) for public loop device connectivity, fabric port (F_Port) for fabric device connectivity, loop port (L_Port) for private loop device connectivity, or router port (R_Port) for SAN routing operation. • Four user-configurable intelligent ports (13 through 16) provide IP network connectivity using SFP port connectors.
Introduction to McDATA Multi-Protocol Products 1 Connectivity Features McDATA directors, fabric switches, SAN routers, and their associated Element Manager applications support the following connectivity features. Products or product classes that do not support a connectivity feature are noted.
Introduction to McDATA Multi-Protocol Products 1 • Extended distance support - Fibre Channel ports are configured for extended distance operation (using repeaters) by changing the port buffer-to-buffer credit (BB_Credit) setting to a higher value. Refer to Distance Extension Through BB_Credit for detailed information. BB_Credits are configured as follows: — Intrepid 6000-series Directors - Ports configured at 1.0625 Gbps transmit data (1,800 bytes frames) up to 120 km by setting the BB_Credit value to 60.
Introduction to McDATA Multi-Protocol Products 1 — Eclipse-series SAN routers - Intelligent ports that support IP network connectivity are not assigned BB_Credits. However, the ports provide approximately 96 megabytes (MB) of Transmission Control Protocol (TCP) packet buffering per transmission direction. TCP output buffering absorbs fabric data to ensure Fibre Channel BB_Credits are not exhausted. Up to eight MB of buffering can be allocated to any single iFCP or iSCSI session.
Introduction to McDATA Multi-Protocol Products 1 • Audit log tracking - Configuration changes to a director or fabric switch are recorded in an audit log stored on the management server. Users can display the audit log through the Element Manager application. Log entries include the date and time of the configuration change, a description of the change, and the source of the change. NOTE: SAN routers do not support audit log tracking.
Introduction to McDATA Multi-Protocol Products 1 • LEDs provide visual indicators of hardware status or malfunctions. LEDs are provided on FRUs, operator panels, front panels, and bezels. • System alerts, event logs, audit logs, link incident logs, and hardware logs display the following director and fabric switch information at the management server or remote workstations: — Director status. — Fabric switch status. — Ethernet link status. — Fibre Channel link status.
Introduction to McDATA Multi-Protocol Products 1 • Automatic notification of significant system events (to support personnel or administrators) through e-mail messages or the call-home feature. NOTE: SAN routers do not support the e-mail message feature. The Sphereon 4300 Switch and SAN routers do not support the call-home feature. In addition, the call-home feature may not be available if the EFC Management applications (EFCM Lite) are installed on a customersupplied platform.
Introduction to McDATA Multi-Protocol Products 1 • Status monitoring of redundant FRUs and alternate data paths to ensure continued product availability in case of failover. The SAN management application queries the status of each backup FRU daily. A backup FRU failure is indicated by an illuminated amber LED. • Data collection through the product’s Element Manager application, SANpilot interface, or SANvergence Manager application (for SAN routers) to help isolate system problems.
Introduction to McDATA Multi-Protocol Products 1 • SNMP management for SAN routers using the following MIBs as defined by IETF working documents, RFC memorandums, and McDATA. All listed MIBs run on each SAN router. Up to eight authorized management workstations can be configured through the Element Manager application to send SNMP trap messages that indicate product operational state changes and failure conditions. Up to four workstations can be configured to receive unsolicited SNMP trap messages.
Introduction to McDATA Multi-Protocol Products 1 • Advanced fabric diagnostic features that include: — ISL port fencing - Any ISL that bounces (repeatedly attempts to establish a connection) causes disruptive fabric rebuilds. ISL fencing establishes a user-defined bounce threshold that when reached, automatically blocks the disruptive E_Port. — Digital SFP diagnostic support - This feature provides access to diagnostic data generated by newer SFP optical transceivers.
2 Product Management This chapter describes the management of McDATA multi-protocol products, including Intrepid-series directors, Sphereon-series fabric switches, and Eclipse-series SAN routers. The chapter specifically describes: • Product management, including out-of-band (non-Fibre Channel) methods, inband (fibre connection (FICON) or Fibre Channel) methods, and a management interface summary.
Product Management 2 Product Management Out-of-band (non-Fibre Channel) management server access to all McDATA products is provided through an Ethernet local area network (LAN) connection on a director control processor (CTP) card, fabric switch front panel, or SAN router front panel. As an optional feature, inband (Fibre Channel or FICON) management access to selected McDATA products is provided through a Fibre Channel port connection.
Product Management 2 • Management using simple network management protocol (SNMP). An SNMP agent is implemented through the Element Manager application that allows administrators on SNMP management workstations to access product management information using any standard network management tool.
Product Management 2 In contrast to the applications installed on the management server, EFCM Lite does not include support for the: — Call-home feature. — Ability to download remote clients from the server. Install clients on remote workstations from the software distribution disk provided with this management option. NOTE: The Sphereon 4300 Switch and SAN routers do not support product management through the EFCM Lite application. Figure 2-1 illustrates out-of-band product management.
Product Management 2 Inband Management The following inband management access methods are provided for directors and fabric switches as options: • Management through the product’s open-system management server (OSMS) that communicates with an application client. The application resides on an open-systems interconnection (OSI) device attached to a director or switch port, and communicates using Fibre Channel common transport (FC-CT) protocol.
Product Management 2 S/390 or zSeries 900 Parallel Enterprise Server Host-Attached Console FICON Channel OSI Server TM Fibre Channel Connection Intrepid 6064 Director Figure 2-2 Management Interface Summary Table 2-1 2-6 Inband Product Management Table 2-1 summarizes McDATA products and the out-of-band or inband management interfaces available to support the products.
Product Management 2 Table 2-1 Out-of-Band and Inband Product Support Summary (continued) Product SANavigator EFCM SANvergence Manager SNMP CLI SANpilot EFCM Lite OSMS FMS 4300 Fabric Switch NO NO NO YES YES YES NO YES NO 4500 Fabric Switch YES YES NO YES YES YES YES YES NO 1620 SAN Router NO NO YES YES YES NO NO NO NO 2640 SAN Router NO NO YES YES YES NO NO NO NO Management Server Support The management server is a one rack unit (1U) high, LAN-accessed,
Product Management 2 The server is dedicated to operation of the SAN management and associated Element Manager applications. These applications provide a GUI and implement web and other server functions. Refer to SAN Management Applications for additional information. NOTE: The server and SAN management application provide a GUI to monitor and manage products and are a dedicated hardware and software solution that should not be used for other tasks.
Product Management 2 Recommended Specifications • 24X read speed slim-type compact disk-rewritable (CD-RW) and 8X read speed digital video disk (DVD) combination drive, data only. • 56K peripheral component interconnect (PCI) internal data and fax modem, using the V .92 dial-up specification. • 16 MB graphics card. • Network interface card (NIC) with two 10/100 Mbps Ethernet adapters using RJ-45 connectors.
Product Management 2 Ethernet Hub The management server and managed directors, fabric switches, and SAN routers connect through a 10/100 Base-T Ethernet hub. Figure 2-4 illustrates the 24-port hub. .
Product Management 2 • Unix workstation with color monitor, keyboard, and mouse, using a: — Linux-based system using an Intel Pentium III processor with one GHz or greater clock speed, using the Red Hat® 7.3 or higher operating system. — Hewlett-Packard® PA-RISC® processor with 400 MHz or greater clock speed, using the HP-UX® 11 or higher operating system. — Sun® Microsystems UltraSPARC™ IIi or later processor, using Solaris™ Version 7.0 or higher operating system.
Product Management 2 • Firmware Services 2-12 E/OSi - The Enterprise Operating System (internetworking) performs system configuration, management, and Fibre Channel and IP-based routing functions for Eclipse-series SAN routers. Director and fabric switch firmware (E/OS and E/OSn) provides services that manage and maintain Fibre Channel connections between ports.
Product Management 2 • Fabric services - This function supports the fabric controller (login server) and name server. For redundant directors, fabric services also implement a replication manager that synchronizes node port (N_Port) registration databases between redundant CTP cards and allows CTP failover.
Product Management 2 — Operating parameters, such as buffer-to-buffer credit (BB_credit), error detect timeout value (E_D_TOV), resource allocation timeout value (R_A_TOV), switch priority, switch speed (1.0625 or 2.1250 Gbps), and preferred Domain_ID. — Active zoning configuration. — SNMP configuration parameters, such as trap recipients, community names, and write authorizations.
Product Management 2 SAN Management Applications This section describes SAN management applications that provide a GUI to monitor, manage, and control directors, fabric switches, and SAN routers. SAN management applications include SANavigator 4.2 (or later), EFCM 8.6 (or later), and SANvergence Manager 4.6 (or later). An associated Element Manager application is provided for each managed product.
Product Management 2 • Centralized configuration - Vendor-specific device management applications can be launched from the SAN management application, including McDATA Element Manager applications. The application also provides management of director and switch zoning across multiple vendors and product models. • Monitoring and notification - The application provides realtime monitoring and event notification for devices in the SAN. Informational, warning, and fatal events are recorded.
Product Management 2 The main window provides a: • Menu bar - Commands at the top of the window provide drop-down menu selections to perform functions for SAN devices, including editing, viewing, planning, discovery, configuration, and monitoring.
Product Management 2 • Master log - The master log at the lower left corner of the window displays a list of informational, warning, or fatal events. The log also includes the event source, type, description, time, and IP address of the device generating the event. • Utilization legend - The color-coded utilization legend explains percent utilization for links depicted on the physical map.
Product Management 2 Element Manager Application An Element Manager application is provided for each managed product (Intrepid 6064 Director, Intrepid 6140 Director, Intrepid 10000 Director, Sphereon 3232 Switch, and Sphereon 4500 Switch). NOTE: An Element Manager application is not supported for the Sphereon 4300 Switch. The Element Manager application works in conjunction with the SAN management application and is a Java-based GUI for managing and monitoring a director or switch.
Product Management 2 Figure 2-7 Hardware View When the mouse cursor is moved over a FRU in the product graphic, the FRU border highlights in blue and a pop-up identification label appears. Mouse selections (right or left click) open dialog boxes or menus that display FRU properties or allow users to perform operations and maintenance tasks.
Product Management 2 SANvergence Manager Application Application GUI This section describes the SANvergence Manager and Element Manager applications that provide a GUI to monitor and manage SAN routers, attached Fibre Channel elements, and metropolitan storage area network (mSAN) connectivity. An Element Manager application is provided for Eclipse 1620 and 2640 SAN Routers. The SANvergence Manager application is an intuitive GUI that communicates with an attached metropolitan simple name server (mSNS).
Product Management 2 The main window provides a: • Menu bar - Commands at the top of the window provide dropdown menu selections to access Actions (add a SAN, remove a SAN, configure SAN properties), Tools (configure parameters, generate reports, perform backup and restore operations), and Options (set general, viewing, and zoning preferences).
Product Management 2 Figure 2-9 Device Window (Element Manager) The graphical representation of the product emulates the hardware configuration and operational status of the corresponding real product. For example, if all router ports are connected and functional, this configuration is reflected in the device window. Mouse selections (right or left click) open dialog boxes or menus that display FRU properties or allow users to perform operations and maintenance tasks.
Product Management 2 SANpilot Interface With product firmware Version 1.2 (or later) installed, administrators or operators with a browser-capable PC and an Internet connection can monitor and manage the director or switch through the SANpilot interface. The interface provides a GUI similar to the Element Manager application and supports product configuration, statistics monitoring, and basic operation.
Product Management 2 — Enable port beaconing, reset ports, and perform port diagnostics. — Retrieve dump files, retrieve product information files, and perform product firmware upgrades. — Install optional feature keys. The SANpilot interface can be opened from a standard web browser running Netscape Navigator 4.6 or higher or Microsoft Internet Explorer 4.0 or higher. At the browser, enter the IP address of the product as the Internet uniform resource locator (URL).
Product Management 2 • View - At the View panel, the Director or Switch (default), Port Properties, FRU Properties, Unit Properties, Operating Parameters, and Fabric task selection tabs appear. • Configure - At the Configure panel, the Ports (default), Director or Switch, Management, Zoning, Security, and Performance task selection tabs appear. • Monitor - At the Monitor panel, the Port List (default), Port Stats, Logs, and Node List task selection tabs appear.
3 Planning Considerations for Fibre Channel Topologies A storage area network (SAN) is typically defined as a network of shared storage resources that can be allocated throughout a heterogeneous environment. This chapter describes planning considerations for incorporating McDATA switching products into Fibre Channel SAN topologies. The chapter specifically describes: • Fibre Channel topologies (arbitrated loop and multiswitch fabric). • Characteristics of arbitrated loop operation.
Planning Considerations for Fibre Channel Topologies 3 • Arbitrated loop - This topology uses a Sphereon 4300 or 4500 Fabric Switch to connect multiple device node loop ports (NL_Ports) in a Fibre Channel arbitrated loop (FC-AL) or hub configuration without benefit of a multiswitch fabric. Both switches support a switched mode topology that provides a single, logical connection between two device NL_Ports.
Planning Considerations for Fibre Channel Topologies 3 This section focuses on loop operation for Sphereon 4300 (12-port) and 4500 (24-port) Fabric Switches that operate at 1.0625 or 2.1250 gigabits per second (Gbps) and support FC-AL operation using FL_Ports and public and private device connectivity. Shared Mode Versus Switched Mode Legacy arbitrated loop switches (such as the McDATA ES-1000 Switch) are configured to operate in user-selectable shared or switched mode.
Planning Considerations for Fibre Channel Topologies 3 Part (A) of Figure 3-1 shows device D1 connected to server S1 through a pair of H_Ports. Although the remaining switch H_Ports (six ports) and device D2 are unavailable for connection, frame traffic between device D1 and server S1 travels through a loop that consists of all eight H_Ports, device D1, device D2, and server S1.
Planning Considerations for Fibre Channel Topologies 3 Part (A) of Figure 3-2 shows server S1 connected to device D1 through a switched pair of FL_Ports communicating at 1.0625 Gbps. Server S2 is connected to device D2 through a second switched pair of ports, also communicating at 1.0625 Gbps. Because of opportunistic bandwidth sharing, the two switched pairs effectively increase the switch bandwidth to 2.1250 Gbps.
Planning Considerations for Fibre Channel Topologies 3 Public Versus Private Devices Sphereon 4300 and 4500 Fabric Switches support connection of public and private arbitrated loop devices as follows: • Public device - A loop device that can transmit a fabric login (FLOGI) command to the switch, receive acknowledgement from the switch’s login server, register with the switch’s name server, and communicate with fabric-attached devices is a public device.
Planning Considerations for Fibre Channel Topologies 3 • Private device - A loop device that cannot transmit an FLOGI command to the switch nor communicate with fabric-attached devices is a private device. As shown in Figure 3-4, device D2 is a private loop device connected to a Sphereon 4500 Switch and cannot communicate with any fabric-attached device. However, device D2 can communicate with switch-attached server S2 (using private addressing mode).
Planning Considerations for Fibre Channel Topologies 3 Private devices only communicate with other devices on the same arbitrated loop, and interconnected public and private devices can communicate with each other. Such intermixed devices establish operating parameters and loop topology configuration through a port login (PLOGI) command exchange, rather than through the switch’s name server. Be aware that public device-to-private device communication may cause problems.
Planning Considerations for Fibre Channel Topologies 3 Figure 3-5 Public Loop Connectivity • Figure 3-6 Private loop - A private loop is not connected to a switched fabric and the switch’s embedded FL_Port is inactive. All devices attached to the loop can only communicate with each other. Private loop connectivity for a Sphereon 4500 Switch is illustrated in Figure 3-6.
Planning Considerations for Fibre Channel Topologies 3 FL_Port Connectivity Sphereon 4300 and 4500 Fabric Switches provide loop connectivity through GX ports that are active as FL_Ports. The ports provide port addressing, physical connectivity, and Fibre Channel frame routing. Each FL_Port (and the embedded FL_Port) has a 24-bit address identifier. The address identifier is expressed in hexadecimal format as DD AA PP, where: • DD is the domain identifier.
Planning Considerations for Fibre Channel Topologies 3 Planning for Fabric-Attached Loop Connectivity Public arbitrated loop topology supports the connection of workgroup or departmental FC-AL devices to a switched fabric through any 4300 and 4500 Fabric Switch port active as an E_Port. This topology is well-suited to: • Providing connectivity between a workgroup or departmental SAN and a switched fabric, thus implementing connectivity of FC-AL devices to fabric devices at the core of the enterprise.
Planning Considerations for Fibre Channel Topologies 3 • Improved resource manageability. Distributed resources are consolidated and managed through Fibre Channel connectivity instead of physical relocation. One management server manages the operation and connectivity of multiple fabric directors, fabricattached devices, arbitrated loop switches, and FC-AL devices. • Improved security of business applications and data.
Planning Considerations for Fibre Channel Topologies 3 Figure 3-7 Tape Device Consolidation Server Consolidation Providing fabric connectivity for multiple FC-AL tape drives by attaching them individually to a Fibre Channel director is likewise not a cost-effective solution. A practical solution is to consolidate the tape drives on an inexpensive loop switch, then connect the switch to a director E_Port.
Planning Considerations for Fibre Channel Topologies 3 Figure 3-8 Tape Drive Consolidation Fabric Topologies Several topologies exist from which to build a Fibre Channel fabric infrastructure. This section describes the most effective fabric topologies and provides guidance on when to deploy each topology. The topologies are effective for a wide variety of applications, are extensively tested by McDATA, and are deployed in several customer environments.
Planning Considerations for Fibre Channel Topologies 3 TM TM TM TM Interswitch Link Fabric Connection Figure 3-9 Full Mesh Fabric Full-mesh fabrics provide increased resiliency over cascaded or ring fabrics and are well suited for applications that require any-to-any connectivity. If a single ISL fails, traffic is automatically routed through an alternate path. Mesh fabrics also form effective backbones to which other SAN islands can be connected.
Planning Considerations for Fibre Channel Topologies 3 A modified or partial-mesh fabric is similar to a full-mesh fabric, but each switch does not have to be directly connected to every other switch in the fabric. The fabric is still resilient to failure but does not carry a cost premium for unused or redundant ISLs. In addition, partial-mesh fabrics scale easier than full-mesh fabrics.
Planning Considerations for Fibre Channel Topologies 3 Tier 2 Devices TM TM TM Edge Switches 10/100 RST 10/100 RST 31 29 27 25 10/100 23 TM 30 PWR ERR Tier 1 Device Core Director 28 26 24 22 31 29 27 25 23 TM 21 RST 20 TM 21 10/100 19 18 17 16 RST 15 14 13 12 30 TM 11 10 9 8 7 5 3 28 26 24 1 PWR ERR 6 4 2 22 20 19 18 17 16 15 14 13 12 11 10 9 8 7 5 3 1 PWR ERR 6 4 2 0 PWR 0 ERR Tier 1 Device Core Director TM TM TM TM T
Planning Considerations for Fibre Channel Topologies 3 SAN Islands • Tier 2 - A Tier 2 device connects to an edge switch and Fibre Channel traffic from the device must traverse only one ISL (hop) to reach a device attached to a core director or switch. • Tier 3 - A Tier 3 device connects to an edge switch and Fibre Channel traffic from the device can traverse two ISLs (hops) to reach a device attached to a core director or switch.
Planning Considerations for Fibre Channel Topologies 3 D3 D4 D5 D6 D7 N N N N N F F F F F F N D8 F N D9 F N D10 ISLS D2 N N D1 F E E E E E E E E TM TM F Di2 S1 E E S2 ISL ISL TM E E F F N D12 Figure 3-11 S3 Multiswitch Fabric N D11 D = Device ISL = Interswitch Link S = Fabric Switch (Director) Example Multiswitch Fabric A multiswitch fabric is typically complex and provides the facilities to maintain routing to all device N_Ports attached to the fab
Planning Considerations for Fibre Channel Topologies 3 • Heterogeneous fabric - Vendor interoperability in the fabric environment is supported; therefore, fabric elements can include directors, fabric switches, and open-fabric compliant products supplied by original equipment manufacturers (OEMs). To determine if interoperability is supported for a product or if communication restrictions apply, refer to the supporting publications for the product or contact McDATA.
Planning Considerations for Fibre Channel Topologies 3 • Distance requirements - The distance between elements in a fabric affects the type of optical port transceiver and cabling required. In addition, variables such as the number of connections, grade of fiber-optic cable, device restrictions, application restrictions, buffer-to-buffer credit limits, and performance requirements can affect distance requirements.
Planning Considerations for Fibre Channel Topologies 3 • Bandwidth - ISL connections can be used to increase the total bandwidth available for data transfer between two directors or switches in a fabric. Increasing the number of ISLs between elements increases the corresponding total ISL bandwidth but decreases the number of port connections available to devices. Table 3-1 illustrates ISL transfer rate versus port availability for a fabric consisting of two Intrepid 6064 Directors.
Planning Considerations for Fibre Channel Topologies 3 • Preferred path - Preferred path is an option that allows a user to configure an ISL data path between multiple fabric elements (directors and fabric switches) by configuring the source and exit ports of the origination fabric element and the Domain_ID of the destination fabric element. Each participating director or switch must be configured as part of a desired path. For information about the feature, refer to Preferred Path.
Planning Considerations for Fibre Channel Topologies 3 • Zoning - For multiswitch fabrics, zoning is configured on a fabric-wide basis. Changes to the zoning configuration apply to all directors and switches in the fabric. To ensure the zoning configuration is maintained, certain rules are enforced when two or more elements are connected through ISLs to form a fabric or when two or more fabrics are joined. For additional information, refer to Configuring Zones.
Planning Considerations for Fibre Channel Topologies 3 In the audit log, note the Principal setting maps to a number code of 1, Default maps to a number code of 254, and Never Principal maps to a number code of 255. Number codes 2 through 253 are not used. • Fabric WWN assignment - The Fabric Manager application identifies fabrics using a fabric WWN. The fabric WWN is the same as the WWN of the fabric’s principal switch.
Planning Considerations for Fibre Channel Topologies 3 — For the Intrepid 10000 Director and Eclipse-series SAN routers, the firmware does not add a base offset to the numerically-assigned preferred Domain_ID. — For non-McDATA directors and switches, the product firmware may not add a base offset to the numerical preferred Domain_ID or may add a different hexadecimal base offset (not 60).
Planning Considerations for Fibre Channel Topologies 3 When multiple minimum-hop paths (ISLs) between fabric elements are detected, firmware balances the data transfer load and assigns ISL as follows: — The director or switch assigns an equal number of device entry ports (F_Ports) to each E_Port connected to an ISL. For example, if a fabric element has two ISLs and six attached devices, the load from three devices is transferred through each ISL.
Planning Considerations for Fibre Channel Topologies 3 • E_Port segmentation - When an ISL activates, the two fabric elements exchange operating parameters to determine if they are compatible and can join to form a single fabric. If the elements are incompatible, the connecting E_Port at each director or switch segments to prevent the creation of a single fabric. A segmented link transmits only Class F traffic; the link does not transmit Class 2 or Class 3 traffic.
Planning Considerations for Fibre Channel Topologies 3 RSCNs are transmitted to all registered device N_Ports attached to the fabric if either of the following occur: — A fabric-wide event occurs, such as a director logging in to the fabric, a director logging out of the fabric, or a reconfiguration because of a director or ISL failure. — A zoning configuration change.
Planning Considerations for Fibre Channel Topologies 3 Fabric Initialization When multiple directors or switches are connected, E_Port (ISL) communication must be established between fabric elements and the fabric must be initialized. During fabric initialization, the fabric elements: • Establish the operating mode for connected E_Port pairs and exchange link parameters (E_Port names, timeout values, classspecific information, and flow control parameters).
Planning Considerations for Fibre Channel Topologies 3 Fabric Performance I/O Requirements During the design phase of a Fibre Channel fabric, performance requirements of the fabric and of component directors, fabric switches, and devices must be identified and incorporated. An effective fabric design can accommodate changes to performance requirements and incorporate additional directors, switches, devices, ISLs, and higher speed links with minimal impact to fabric operation.
Planning Considerations for Fibre Channel Topologies 3 • Read/write mixture - Although application I/O is typically a mixture of read and write operations, some applications are very biased. For example, video server applications are almost 100% read intensive, while real-time video editing applications are mostly write intensive. Read operations typically take less time than write operations, therefore storage devices for a read-intensive application usually wait for data transfer.
Planning Considerations for Fibre Channel Topologies 3 Storage 1 Gbps ISL NT Server 1 TM NT Server 2 Bandwidth (MBps) 1 Gbps ISL TM 200 150 100 50 0 1 NT Server 1 (100 MBps Max) Figure 3-12 NT Server 2 (100 MBps Max) 5 10 15 Time (Sec) 20 25 ISL Oversubscription Two NT servers, each with maximum I/O of 100 MBps, are contending for the bandwidth of a single ISL operating at 1.0625 Gbps. In addition to data, the ISL must also transmit Class F traffic internal to the fabric.
Planning Considerations for Fibre Channel Topologies 3 • Upgrade the existing ISL - Fabric element software, firmware, and hardware can be upgraded to support a 2.1250 or 10.2000 Gbps bandwidth traffic load between fabric elements. A 2.1250 or 10.2000 Gbps ISL is sufficient to support the bandwidth of both NT servers operating at peak load. • Deliberately employ ISL oversubscription - SANs are expected to function well, even with oversubscribed ISLs.
Planning Considerations for Fibre Channel Topologies 3 When designing a core-to-edge fabric, servers and storage devices that support such bandwidth-intensive applications should be attached to core directors as Tier 1 devices. As a best practices policy (assuming 1.0625 Gbps ISLs), devices that generate a sustained output of 35 MBps or higher are candidates for Tier 1 connectivity. FICON devices also must use Tier 1 connectivity. For additional information, refer to FCP and FICON in a Single Fabric.
Planning Considerations for Fibre Channel Topologies 3 • Device output in Gbps does not oversubscribe ISLs, leading to fabric congestion. • Device output in IOPS does not result in a connectivity scheme that exceeds fan-out ratios, leading to port congestion. Figure 3-15 illustrates performance tuning for a simple fabric using appropriate ISL connectivity, device locality, and fan-out regions for device connectivity. The fabric is comprised of one core director and six edge switches.
Planning Considerations for Fibre Channel Topologies 3 The fabric is divided into four performance regions as follows: Fabric Availability • Local Tier 1 devices - A video server application with I/O capabilities of 40 MBps and 2,000 IOPS must be connected to the fabric. Because the application is critical and high bandwidth (in excess of 35 MBps), the server and associated storage are directly attached to the core director as Tier 1 devices. No ISLs are used for server-to-storage connectivity.
Planning Considerations for Fibre Channel Topologies 3 Fibre Channel fabrics are classified by four levels of resiliency and redundancy. From least available to most available, the classification levels are: • Nonresilient single fabric - Directors and switches are connected to form a single fabric that contains at least one single point of failure (fabric element or ISL). Such a failure causes the fabric to fail and segment into two or more smaller fabrics.
Planning Considerations for Fibre Channel Topologies 3 10/100 10/100 RST 10/100 RST 10/100 RST TM RST TM TM TM PWR PWR PWR PWR ERR ERR ERR ERR Fabric “A” Fabric “B” TM TM 10/100 10/100 RST 10/100 RST RST 10/100 RST TM TM TM TM PWR ERR PWR ERR PWR PWR ERR ERR Interswitch Link Fabric Connection Figure 3-16 Redundant Fabrics Some dual-attached devices support active-active paths, while others support only active-passive paths.
Planning Considerations for Fibre Channel Topologies 3 A scalable fabric allows for nondisruptive addition of fabric elements (directors, fabric switches, and SAN routers) or ISLs to increase the size or performance of the fabric or SAN. Large, scalable fabrics and SANs are constructed by incorporating: • High-port count directors - Installing high-port count directors as SAN building blocks provides a larger number of non-blocking Fibre Channel ports per fabric element and reduces the need for ISLs.
Planning Considerations for Fibre Channel Topologies 3 FCP and FICON in a Single Fabric Fibre Channel Layer 4 (FC-4) describes the interface between Fibre Channel and various upper-level protocols. FCP and FICON are the major FC-4 protocols. FCP is the Fibre Channel protocol that supports the small computer system interface (SCSI) upper-level transport protocol.
Planning Considerations for Fibre Channel Topologies 3 • When a director or fabric switch is set to open systems management style, FCP connectivity is defined within a Fibre Channel fabric using WWNs of devices that are allowed to form connections. When connecting to the fabric, an FCP device queries the name server for a list of devices to which connectivity is allowed.
Planning Considerations for Fibre Channel Topologies 3 • The Domain_ID and physical port number of the director or fabric switch port to which a device is attached. FICON configuration attributes are implemented through logical port addressing. This concept is consistent with the address-centric nature of FICON and allows ports to be swapped for maintenance operations without regenerating a host configuration.
Planning Considerations for Fibre Channel Topologies 3 Figure 3-18 illustrates port numbering and logical port addressing for Intrepid 6140 Director ports accessed from the rear. UPM Cards UPM Cards 34 33 143 8F 142 8E 141 8D 140 8C 136 88 137 89 138 8A 139 8B 93 92 91 90 8C 8D 8E 8F 135 87 134 86 133 85 132 84 8B 8A 89 88 32 SBAR - 1 Module SBAR - 0 Module Figure 3-18 Intrepid 6140 Port Numbers and Logical Port Addresses (Rear) The figure shows: • SBAR positions (0 and 1).
Planning Considerations for Fibre Channel Topologies 3 • FICON port-to-port connectivity is hardware enforced, while FCP port-to-port connectivity is software or hardware enforced (depending on the director or switch firmware release level). — FICON architecture controls connectivity through a host-based HCD program, the CUP, and a director or switch-resident PDCM array.
Planning Considerations for Fibre Channel Topologies 3 • When employing inband (Fibre Channel) director or switch management, the open-systems management server (OSMS) is associated with the FCP protocol, and the FICON management server (FMS) is associated with the FICON protocol. Management server differences tend to complicate security and control issues. NOTE: The Intrepid 10000 Director and Sphereon 4300 and 4500 Fabric Switches do not support out-of-band management through FMS.
Planning Considerations for Fibre Channel Topologies 3 SANtegrity Binding McDATA offers a SANtegrity Binding feature (including both fabric binding and switch binding) that allows the creation of reliable SAN configurations and provides a mechanism for attached devices to query the user-configured security level employed in a SAN. The feature significantly reduces the impacts of accidental or operatorinduced errors.
Planning Considerations for Fibre Channel Topologies 3 However, the firmware and SAN management applications do not prevent FCP and FICON device configurations that may interfere with each other. A successful intermix environment requires a set of best practice conventions as follows: 1. Upgrade fabric element firmware to a common version - Ensure fabric elements are operating at a common firmware level. This reduces errors due to director or switch incompatibility. E/OS Version 4.
Planning Considerations for Fibre Channel Topologies 3 — When using inband director or switch management, either (or both) of the FMS or OSMS features can be enabled. When either (or both) features are enabled, the director or switch can be set to open systems or FICON management style. 3. Upgrade fabric elements to a common feature set - Ensure a common set of PFE-keyed optional features (refer to Optional Feature Keys) is installed on each fabric element.
Planning Considerations for Fibre Channel Topologies 3 6. Configure PDCM arrays - For each director or switch managed by the FICON management style, define the allow and prohibit settings for FICON device connectivity. Use the Element Manager application’s Configure Allow/Prohibit Matrix - Active dialog box. Port connectivity assignment (step 4) should be reflected in PDCM arrays for FICON connectivity management.
Planning Considerations for Fibre Channel Topologies 3 — All FICON devices must be included in the same zone to facilitate proper state change notification. This is achieved by creating a unique FICON zone or using the default zone. Disable the default zone and explicitly create a unique zone for all FICON devices. Regardless of the director or switch operating mode, FCP devices must be zoned in the traditional fashion, and FICON devices must be zoned to provide isolation from the FCP devices.
Planning Considerations for Fibre Channel Topologies 3 • Better fabric performance - As a connection between fabric switches, a 10.2000 Gbps ISL delivers significantly greater bandwidth. Fibre Channel devices that are not 10.2000 Gbpscapable benefit from a higher-speed ISL, because slower traffic is multiplexed and transmitted through the 10.2000 Gbps ISL. • Additional port count - If additional ISL bandwidth is not required for fabric performance, 10.
Planning Considerations for Fibre Channel Topologies 3 High-Integrity Fabrics Cascaded FICON directors and switches must support high-integrity fabrics. McDATA fabric elements must have the SANtegrity Binding feature installed and operational with Enterprise Fabric Mode enabled. High-integrity fabric architecture support includes: • Fabric binding - Only directors or switches with fabric binding installed are allowed to attach to specified fabrics in a SAN.
Planning Considerations for Fibre Channel Topologies 3 Minimum Requirements The following are minimum hardware, firmware, and software requirements to configure and enable a FICON-cascaded SAN: • A single-vendor switching environment with two or more of the following McDATA directors or switches: — Intrepid 6064 or 6140 Director. — Sphereon 3232 Fabric Switch. • E/OS Version 4.0 (or later) must be installed on all directors or switches. E/OS firmware Version 6.0 (or later) is recommended.
Planning Considerations for Fibre Channel Topologies 3 FICON Cascading Best Practices A successful FICON-cascaded SAN environment requires a set of best practice conventions as follows: 1. Connect fabric elements - Establish one or more ISLs between cascaded directors of fabric switches as follows: a. Ensure fabric elements are defined to the SAN management application. If the elements must be defined, refer to the appropriate switch or director installation manual for instructions. b.
Planning Considerations for Fibre Channel Topologies 3 b. Ensure the Link Incident field displays None and the Reason field is blank. If an ISL segmentation or other problem is indicated, go to MAP 0000: Start MAP in the product-specific Installation and Service Manual. c. Click Close to close the dialog box and return to the Hardware View. 4. Install SANtegrity Binding on fabric elements - Configure and enable the SANtegrity Binding feature at each director or switch as follows: a.
Planning Considerations for Fibre Channel Topologies 3 7. Verify FICON devices are still logged in - Maximize the Element Manager application. Inspect the Node List View and verify FICON devices (channels and control units inspected in step 5) are still logged in to each director or switch. 8.
Planning Considerations for Fibre Channel Topologies 3 NOTE: An cascaded switch is a fabric director or switch connected to a destination control unit and an entry switch. c. Run the IOCP to create an input/output configuration data set (IOCDS). The switch ID (CHPID macroinstructions) and 2-byte link address (control unit macroinstructions) are updated in the IOCDS. Refer to the IBM FICON Native Implementation and Reference Guide (SG24-6266) for additional information. 10.
4 Implementing SAN Internetworking Solutions Enterprise-level information technology (IT) departments often deploy storage configurations that include direct-attached storage, network-attached storage (NAS), and small, isolated storage area networks (SANs). These configurations often result in: • Isolated and inefficiently-used applications, data storage, and computing resources. • Costly, decentralized, and complex asset management. • Slow transactions and data access.
Implementing SAN Internetworking Solutions 4 SAN Island Consolidation SAN islands tend to be constructed along application (such as product test, finance, or engineering), operating system (OS), protocol, or geographical (site-based) boundaries. Because of application and OS segmentation, large data centers at single sites often consist of SAN islands constructed with relatively small Fibre Channel switches.
Implementing SAN Internetworking Solutions 4 Large Fabric Problems • Inability to consistently schedule maintenance downtime for each SAN. • Stranded resources. Unused ports in one SAN cannot be used by applications in another (port limited) SAN, and expensive resources (such as tape backup elements) cannot be easily shared across SAN boundaries. Fibre Channel fabrics configure and manage themselves, and require operator intervention only upon failure.
Implementing SAN Internetworking Solutions 4 Another persistent problem associated with large Fibre Channel fabrics is multi-vendor incompatibility. Due to lack of common communications standards and fabric shortest path first (FSPF) protocol, switch vendors may have to support multiple (standardscompliant and proprietary) interoperability modes. In addition, protocol enhancements may force vendors to support multiple firmware versions for the same product.
Implementing SAN Internetworking Solutions 4 This feature reduces unused ports and resources and consolidates the enterprise into a single infrastructure, while maintaining multiple independent application and fault isolation domains. Up to four partitions can be enabled for each director (0 though 3), where a partition consists of one or more line modules (LIMs). User access and Fibre Channel traffic (Class 2, Class 3, and Class F) are isolated within each partition.
Implementing SAN Internetworking Solutions 4 • Create up to three additional FlexPars and assign resources to those FlexPars. • Perform director firmware upgrades to all Flexpars. • Enable or disable the protocol subsystem for any Flexpar. • Enable or disable switch modules and control processor (CTP) cards. • Perform director shutdowns, restarts, and field-replaceable unit (FRU) switchovers. • Set the director Internet protocol (IP) address, gateway address, and subnet mask.
Implementing SAN Internetworking Solutions 4 Zone FlexPars implement an RSCN zone isolation feature that prevents fabric-format RSCNs from propagating to devices in zones not impacted by the RSCN. With zone FlexPars enabled, zoning change RSCNs are handled like device availability change RSCNs. Because the feature is device centric, zone FlexPars work in loop environments and with node port ID virtualization (NPIV) enabled.
Implementing SAN Internetworking Solutions 4 SAN Routing Connecting isolated, department-level, and application-specific Fibre Channel SANs is a requirement for most enterprises. Consolidating SAN islands: • Provides campus storage connectivity and interoperability between formerly-incompatible Fibre Channel fabrics (from the same or different vendors).
Implementing SAN Internetworking Solutions 4 Figure 4-2 • Tier 2 - To connect SAN islands without physically merging the fabrics, the second tier consists of metropolitan storage area networks (mSANs). SAN routers connect fabrics within a data center or campus to form an mSAN and transmit data between fabrics through router ports (R_Ports). Refer to mSAN Routing for additional information.
Implementing SAN Internetworking Solutions 4 Figure 4-3 SAN Routing Concepts The following sections discuss SAN routing concepts, including: 4-10 • R_Port operation. • Routed SAN zoning. • mSAN routing. • iFCP operation. • iSAN routing. • Inter-FlexPar routing. • Best practices.
Implementing SAN Internetworking Solutions 4 R_Port Operation To avoid building a large Fibre Channel fabric with its inherent reconfiguration issues, SAN Routing provides any-to-any connectivity (to maximize use of common assets across SAN islands), while retaining the fault isolation characteristics of smaller SANs.
Implementing SAN Internetworking Solutions 4 Instead of Class F frame transmission, routing communication is provided by Fibre Channel network address translation (FC_NAT) technology. This is similar to the technology used by IP networks to convert private addresses to public addresses. The principal switch in each router-connected fabric assigns the Domain_ID to the associated R_Port acting as an edge switch.
Implementing SAN Internetworking Solutions 4 NOTE: Proxy Domain_IDs 30 and 31 are reserved for routing domains and cannot be assigned to directors or switches in any router-attached fabric. The routing domain with proxy Domain_ID 30 represents Fibre Channel devices that are part of a router-attached fabric (part of a local mSAN). The routing domain with proxy Domain_ID 31 represents devices that are directly attached to the router’s Fibre Channel ports or connected through an iFCP link.
Implementing SAN Internetworking Solutions 4 For attached fabrics in which participating element’s Interop Mode is set to McDATA Fabric 1.0, Domain_IDs of 30 and 31 are recognized by SAN management applications and all attached devices. For attached fabrics in which participating element’s Interop Mode is set to Open Fabric 1.0, Domain_IDs of 30 and 31 are recognized by SAN management applications. Domain_IDs of 7E and 7F are recognized by all attached devices.
Implementing SAN Internetworking Solutions 4 R_Port Domain_ID Assignment The default preferred Domain_ID for each SAN router R_Port is 1. However, each port should be assigned a preferred Domain_ID (set at the R_Ports tab of the Fabric Configuration dialog box) that is unique within the attached fabric. The principal switch in the attached fabric then attempts to allocate this requested (preferred) Domain_ID to the R_Port.
Implementing SAN Internetworking Solutions 4 There is only one router fabric manager per fabric. If more than one R_Port (from the same or multiple routers) connects to a fabric, then the port with the lowest worldwide name (WWN) is elected router fabric manager for that fabric. Other R_Ports become subordinate ports. All R_Ports (router fabric manager or not) participate in FSPF routing protocol and traffic forwarding.
Implementing SAN Internetworking Solutions 4 A No Zone Synchronization policy is typically not suitable for larger SAN routing environments where many devices must be visible to numerous fabrics. Append IPS Zones When the zone policy is set to Append IPS Zones, Internet protocol storage (IPS) zone set information from the router is appended to the active zone set for every router-attached fabric in the mSAN.
Implementing SAN Internetworking Solutions 4 mSAN Routing An mSAN consists of one or two SAN routers that interconnect up to six Fibre Channel fabrics. These fabrics are typically dispersed within a data center or metropolitan campus. If two SAN routers are used, they are connected with multiple (one to four) Gigabit Ethernet (GbE) bandwidth IRLs. These GbE connections (using mFCP as the transport protocol) are characterized by low latency, high bandwidth, and negligible packet loss.
Implementing SAN Internetworking Solutions 4 During SAN router configuration, each R_Port is assigned (through the SANvergence manager application) a unique Fabric_ID between 1 and 12. Although the theoretical limit is 12 Fabric_IDs per mSAN, the supported limit is six. As shown in Table 4-1, four Area_IDs are available to each Fabric_ID. Therefore, the combination of domain, area, and fabric IDs creates a theoretical limit of 1,024 devices per fabric (although the supported number is far less).
Implementing SAN Internetworking Solutions 4 Router Connectivity through mFCP mFCP provides connectivity (through a GbE-bandwidth IRL) between two Eclipse 2640 SAN Routers. mFCP is similar to Fibre Channel protocol (FCP) but implements user datagram protocol (UDP) for open systems interconnection (OSI) Layer 4 transport. mFCP links are used for path failover in high-availability mSANs. NOTE: The Eclipse 1620 SAN Router does not support mFCP and must be deployed in mSANs as a single-router configuration.
Implementing SAN Internetworking Solutions 4 If a direct Fibre Channel connection exists between routed fabrics, storage traffic traverses the Fibre Channel ISL and not the routerto-router mFCP link. Only SNS traffic traverses the mFCP link. However, if a router-to-router mFCP link is the only path between two Fibre Channel devices, the link is traversed by storage traffic.
Implementing SAN Internetworking Solutions 4 iFCP Operation There are three protocols competing to transmit storage-related I/O traffic over long-distance transmission control protocol/Internet protocol (TCP/IP) links: • iSCSI is a TCP/IP-based protocol for establishing and managing connections between IP-based storage devices, hosts, and clients. iSCSI operates on top of TCP, moving block data (iSCSI packets) over an IP Ethernet network. Refer to iSCSI Protocol for additional information.
Implementing SAN Internetworking Solutions 4 From the standpoint of fabric build events, the only difference between a local and stretched E_Port connection is the latency introduced by the TCP/IP link and associated WDM or FCIP hardware. A disruptive build fabric event at one local site propagates to the connected site. Likewise, a disruption in the TCP/IP link may cause the extended SAN to segment into separate SAN islands.
Implementing SAN Internetworking Solutions 4 Figure 4-6 iSAN Routing iFCP WAN Extension An internetworked SAN (iSAN) is a network composed of multiple Fibre Channel fabrics or mSANs, connected by one or more SAN routers, where at least one fabric is remotely located and connected through a WAN. The WAN connection is an iFCP IRL, characterized by high latency and ranging in bandwidth from Digital Service 1 (DS1) at 1.544 Mbps to GbE at 1000 Mbps.
Implementing SAN Internetworking Solutions 4 SAN Routing also streamlines SAN connectivity by eliminating network address issues associated with duplicate Domain_IDs. Because fabric elements and devices at either end of an iFCP connection remain in separate mSANs, address conflicts between the mSANs do not occur. SAN Routing provides address translation (through FC_NAT) for zoned devices with authorization to communicate across the network.
Implementing SAN Internetworking Solutions 4 • Rate limiting - If ingress traffic enters the SAN router faster than egress traffic leaves, port buffers fill and cause dropped data packets. Dropped packets cause TCP to resort to internal (and inefficient) flow control, causing dramatic link throughput decrease. Rate limiting prevents this problem. Refer to Intelligent Port Speed for detailed information about rate limiting.
Implementing SAN Internetworking Solutions 4 and results in a lower compression bandwidth. The algorithm has an average compression ratio increase of approximately 30% over LZO. The algorithm is recommended when up to 8 TCP sessions are used and the available bandwidth is between 10 Mbps (thin Ethernet) and 45 Mbps (DS3 transport level). — Deflate - This algorithm incorporates a history cache with Huffman encoding. In addition, a hash table (saved in the compressed data) is used to perform string searches.
Implementing SAN Internetworking Solutions 4 mFCP to iFCP Comparison Table 4-3 compares mFCP to iFCP and summarizes the features of each protocol.
Implementing SAN Internetworking Solutions 4 Figure 4-7 illustrates inter-FlexPar routing. Flexpar B (tape backup fabric) is isolated from Flexpar C (product development fabric) as normally desired. However, development personnel occasionally perform tape backups that require access to Flexpar B devices. An E_Port from each FlexPar is physically connected to a SAN router R_Port, and Flexpar C servers are zoned to communicate with Flexpar B tape devices.
Implementing SAN Internetworking Solutions 4 2. Domain_ID assignment - Manually assign unique Domain_IDs to all Fibre Channel directors, fabric switches, and SAN router R_Ports. Ensure the Insistent Domain_ID option is enabled at the SAN management or SANvergence Manager application. Do not assign Domain_ID 30 or 31 to any fabric elements. These proxy Domain_IDs are reserved for routing domains within the SAN router. 3.
Implementing SAN Internetworking Solutions 4 7. Use redundant mFCP connections - For high availability (not increased bandwidth), use multiple mFCP connections between SAN routers to ensure the mSAN does not partition and connectivity to routing domains 30 and 31 remains intact. 8.
Implementing SAN Internetworking Solutions 4 Local mSAN Name: Boston Local mSAN_ID: 20 Date: 1/12/05 Exported Zone_ID Range: 1 to 100 Local Zone_ID Range: 101 to 512 Local mSAN Zone Summary Exported (Y/N) mSAN ID Y 2 DB_Replication_2 Remote site for disaster recovery. Y 3 Tape_Library_3 Remote site for data center tape library access. N 101 Nightly_Backup Local initiator for nightly tape backups. N 501 Test_Fabric Test fabric for the local mSAN.
Implementing SAN Internetworking Solutions 4 Local mSAN Name: Chicago Local mSAN_ID: 30 Date: 1/12/05 Exported Zone_ID Range: 1 to 100 Local Zone_ID Range: 101 to 512 Local mSAN Zone Summary Exported (Y/N) mSAN ID Y 2 N N mSAN Name Description DB_Replication_2 Replication site for disaster recovery. 101 Web_Server Local web server and associated hardware. 501 Local_Test Local test fabric.
Implementing SAN Internetworking Solutions 4 12. Configure R_Ports - For all configured SAN router R_Ports in the same fabric: a. The R_Port interconnect modes (McDATA Fabric 1.0 or Open Fabric 1.0) must be identical. This parameter is set at the SANvergence Manager application. The corresponding E_Port interoperability mode must also be identical. This parameter is set at the director or fabric switch Element Manager application. b.
Implementing SAN Internetworking Solutions 4 c. Port zoning can be confusing in a multi-vendor environment because OEMs implement zoning in different ways. When zoning with vendor-specific SAN management applications, zone through port WWNs, not node (device) WWNs. d. Ensure devices are physically connected before importing their node (device) WWNs to a router.
Implementing SAN Internetworking Solutions 4 Implementing BC/DR Solutions The post-9/11 business environment requires corporations to protect critical data by implementing cost-effective business continuity and disaster recovery (BC/DR) solutions. These BC/DR solutions drive the requirement to extend local data center SANs to geographically distant locations.
Implementing SAN Internetworking Solutions 4 • Reliability - Local storage traffic requires high-reliability communication and is intolerant of data loss, out-of-order packet receipt, or data retransmission. WANs typically provide besteffort communication service and rely on upper-level protocols for end-to-end transport. Because of these differences, a protocol conversion approach is usually required to integrate Fibre Channel SAN traffic over a geographically-dispersed network.
Implementing SAN Internetworking Solutions 4 • Synchronous data replication (SDR) - This operational mode ensures a remote data copy (identical to the primary copy) is created at the time the primary data is created. An update operation does not complete until confirmed at both the primary and mirrored sites. An incomplete operation rolls back at both locations, ensuring the remote copy is a mirror image of the primary copy. SDR is synonymous with disk mirroring.
Implementing SAN Internetworking Solutions 4 SAN Extension Transport Technologies Dark Fiber There are several extension transport technologies available to connect geographically-dispersed SAN islands, all of which differ in performance, latency, and implementation cost. The primary technologies include: • Dark fiber (repeated or unrepeated). • Wavelength division multiplexing (WDM). • Synchronous optical network (SONET) and synchronous digital hierarchy (SDH). • Internet protocol (IP).
Implementing SAN Internetworking Solutions 4 • Figure 4-8 Creates one logical Fibre Channel fabric through a stretched E_Port connection. The connection is vulnerable to disruptions caused by events at each site or to disruptions caused by problems with the extended-distance dark fiber link. Dark Fiber Extended-Distance Connectivity Due to the high cost of burying cables, dark fiber has limited physical availability relative to other WAN link options.
Implementing SAN Internetworking Solutions 4 Light wavelengths used are typically around 1,550 nanometers (nm). Optical fiber performs well in this wavelength region, with very little attenuation. For CWDM, differing wavelengths are separated by multiples of 20.0 nm. For DWDM, differing wavelengths are separated by multiples of 0.8 nm. The lower wavelength numbers provided by CWDM are due to lower accuracy (and price) of lasers.
Implementing SAN Internetworking Solutions 4 • Requires sufficient BB_Credits assigned to the link (such as credits available through the Intrepid 10000 Director buffer pool). Because WDM is a method to transmit multiple signals over the same fiber-optic cable, there is no BB_Credit limitation difference between WDM and dark fiber. Refer to Distance Extension Through BB_Credit for information about requirements. • Creates one logical Fibre Channel fabric through a stretched E_Port connection.
Implementing SAN Internetworking Solutions 4 SONET and SDH are globally standardized technologies, more widely deployed than dark fiber or WDM, and provide a protected connection between two locations. SONET and SDH rings are also self-healing. This means a link is usually restored within 50 ms of break detection without user intervention. This makes SONET and SDH highly-available services.
Implementing SAN Internetworking Solutions 4 • Does not require Fibre Channel BB_Credits assigned to the link because buffering is built in to the GFP function to enable long-distance transmission of storage traffic. • Creates a routed iSAN or one logical Fibre Channel fabric through a stretched E_Port connection, depending upon the protocol and if one or more SAN routers are deployed in the link.
Implementing SAN Internetworking Solutions 4 SONET or SDH service can be purchased on a monthly basis in accordance with a negotiated SLA. However, the transport links may require sufficient BB _Credits to use the purchased bandwidth. Because of BB_Credit limitations, GFP equipment must provide buffering and flow control for native FCP or FICON storage data. Without GFP equipment, iFCP must be used to transmit the data.
Implementing SAN Internetworking Solutions 4 • Figure 4-11 Creates a routed iSAN through an extended-distance iFCP interface. Eclipse 1620 SAN routers at each end of the link provide intelligent port connectivity and iFCP protocol conversion. The routed SAN connection ensures disruptions at one site are isolated and not allowed to propagate to other locations. This connection does not support native FCP or FICON operation.
Implementing SAN Internetworking Solutions 4 Figure 4-12 SAN Extension Technology Comparison • WDM - This technology supports high-bandwidth, low-latency applications with short RTO and RPO requirements. Applications include peer-to-peer computer clustering (grid computing) and real-time disk mirroring (SDR or ADR operational mode) over short to medium metropolitan distances. WDM scales to higher bandwidths at a lower relative cost than SONET or SDH.
Implementing SAN Internetworking Solutions 4 • IP - This technology supports low-bandwidth, high-latency applications with long RTO and RPO requirements. Applications include asynchronous disk backup or tape vaulting over metropolitan to extended (intercity) distances. SAN routers are included in the extended-distance link (iFCP only), so the technology isolates the connected SANS and prevents disruptions caused by fabric or link problems. Table 4-4 compares and contrasts the transport technologies.
Implementing SAN Internetworking Solutions 4 • Distance Extension Through BB_Credit SAN routing requirements - If a single logical Fibre Channel fabric (created through a stretched E_Port connection) is unacceptable because of the potential for disruptive fabric rebuilds, include one or more SAN routers in the extendeddistance link. Token-based buffer-to-buffer flow control governs transmission of data and link control frames in a Fibre Channel switched fabric.
Implementing SAN Internetworking Solutions 4 To support greater Fibre Channel transmission rates (long-link ports), the Intrepid 10000 Director provides a buffer pool that allocates user-defined BB_Credits to each port. This buffer pool is increased if the remote fabric PFE key is enabled (refer toRemote Fabric for information). Each director line module (LIM) contains two scalable packet processors, each supporting two optical paddles. A paddle pair provides 16 ports (1.0625 or 2.
Implementing SAN Internetworking Solutions 4 • Intelligent Port Speed Figure 4-13 10.2000 Gbps long link (mixed paddles) - A paddle pair with one 1.0625 or 2.1250 Gbps paddle and one 10.2000 Gbps paddle provides ten connections. 16 BB_Credits are assigned to the slowest eight short-link ports, and 96 BB_Credits are assigned to one 10.2000 Gbps short-link port (224 BB_Credits total). The remaining 1149 BB_Credits are assigned to the long-link port, supporting a repeated transmission distance of 190 km.
Implementing SAN Internetworking Solutions 4 Figure 4-14 WAN Link Performance (Rate Limiting Enabled) When configuring a SAN router for extended-distance operation over an IP WAN link, the peak available bandwidth must be determined or obtained from the network service provider, and storage traffic over the link must be rate-limited accordingly. If the IP WAN link is dedicated, the peak available bandwidth equals the total link bandwidth.
Implementing SAN Internetworking Solutions 4 • Digital Signal 1 (DS1) - A framing and formatting specification that transmits 24 digital data channels on a T1 synchronous line. Each channel transmits at 64 Kbps (full-duplex), providing an aggregate bandwidth of 1.544 Mbps. Typical T1 lines are long-distance, point-to-point connections used for private networks and corporate Internet communication. The iFCP protocol overhead for a clear-channel (private) T1 link is 4.4%, which provides a bandwidth of 1.
Implementing SAN Internetworking Solutions 4 • Fast Ethernet - A transmission medium specified by IEEE 802.3 that carries information at 100 Mbps (full-duplex) in baseband form using Category-5 copper cable or fiber-optic cable. The specification was developed to enable faster communication of LAN-connected computers. Fast Ethernet is also called 100 Base-T. The iFCP protocol overhead for a fast Ethernet link is 6.3%, which provides a bandwidth of 93.70 Mbps (11.71 MBps) for storage traffic.
Implementing SAN Internetworking Solutions 4 Distance Extension Best Practices To implement a successful extended-distance BC/DR solution, follow a set of best practice conventions as follows: 1. Use dedicated bandwidth and rate limiting - If possible, negotiate dedicated bandwidth as part of the SLA with the network service provider. Enable intelligent port rate limiting to ensure the ingress data rate does not exceed the negotiated bandwidth.
Implementing SAN Internetworking Solutions 4 — FastWrite technology - Enable FastWrite technology to reduce protocol overhead for extended-distance write transactions. The technology is very efficient over long distances with large write transactions (such as SDR applications). — Bandwidth management - Enable QoS processing to guarantee bandwidth over a shared link. QoS subdivides port buffers into multiple queues, each with one or more associated drop thresholds.
Implementing SAN Internetworking Solutions 4 7. Zone controller port pairs - When implementing load sharing, create a separate zone for each pair of communicating ports (one initiator and one target per zone). Assign the zones to different intelligent (iFCP) ports on the SAN router. If the IP network correctly distributes the load across two paths, then load sharing is implemented. If the SDR or ADR application performs load balancing across two controllers, then load balancing is implemented.
Implementing SAN Internetworking Solutions 4 14. Explicitly assign unique mSAN_IDs - When iSAN routing is implemented, each Eclipse 1620 SAN router comprises an mSAN (not the case with an Eclipse 2640 SAN router). Each mSAN must be assigned a unique mSAN_ID. The ID ranges from 0 to 255, and is typically the last octet of the management port IP address (although not a requirement). Do not install a SAN router without changing the default mSAN_ID.
Implementing SAN Internetworking Solutions 4 iSCSI Protocol iSCSI is based on SCSI protocol that enables hosts to perform block data I/O operations with a variety of target peripherals. Targets include disk drives, tape devices, optical storage devices, printers, and scanners. A standard host-to-peripheral SCSI connection is based on a parallel transport mechanism with inherent distance and device support limitations.
Implementing SAN Internetworking Solutions 4 iSCSI Server Consolidation Many enterprise-level IT departments have deployed decentralized computing configurations that include low-end, iSCSI-enabled servers directly attached to storage. While server acquisition costs are typically low, licensing and maintenance costs are often very high in terms of dollars and personnel time. The decentralized infrastructure also causes availability and reliability problems.
Implementing SAN Internetworking Solutions 4 As shown in the figure, server consolidation is enabled by installing an Eclipse 2640 SAN Router that provides iSCSI-to-native FCP connectivity. Stranded servers subject to consolidation do not require installation at one physical location. Servers may be located in a data center or remotely; however, consolidation provides logical connectivity and access as though the servers were co-located.
Implementing SAN Internetworking Solutions 4 4-62 McDATA Products in a SAN Environment - Planning Manual
5 Physical Planning Considerations This chapter describes physical planning considerations for incorporating McDATA directors and switches into storage area networks (SANs) and Fibre Channel fabric topologies. The chapter provides planning considerations and recommendations for: • Port connectivity and fiber-optic cabling. • Rack-mount management server, Ethernet local area network (LAN), and remote access support.
Physical Planning Considerations 5 Port Requirements Plan for sufficient shortwave laser, longwave laser, 1.0625 gigabit per second (Gbps), 2.1250 Gbps, and 10.2000 Gbps Fibre Channel ports to meet the needs of the SAN configuration. The number of ports required is equal to the number of device connections (including redundant connections), plus the number of interswitch links (ISLs) between fabric elements, plus the total number of spare port connections.
Physical Planning Considerations 5 — Optical paddles that operate at 1.0625 or 2.1250 Gbps provide eight Fibre Channel port connections. A fully-populated director supports up to 256 connections and can be configured with shortwave or longwave transceivers, or a combination of both. — Optical paddles that operate at 10.2000 Gbps provide two Fibre Channel port connections.
Physical Planning Considerations 5 Consider the following when determining the number and type of transceivers to use: Data Transmission Distance • Distance between a director or fabric switch and the attached Fibre Channel device or between fabric elements communicating through an ISL. • Cost effectiveness. • Device restrictions or requirements with respect to existing fiber-optic (multimode or singlemode) or copper cable.
Physical Planning Considerations 5 Table 5-1 Cable Type and Transmission Rate versus Distance and Link Budget Cable Type and Data Transmission Rate Unrepeated Distance Link Budget 62.5/125 micron multimode at 1.0625 Gbps 250 meters (820 feet) 2.8 dB 62.5/125 micron multimode at 2.1250 Gbps 120 meters (394 feet) 2.2 dB 62.5/125 micron multimode at 10.2000 Gbps 75 meters (246 feet) xxx dB 50/125 micron multimode at 1.0625 Gbps 500 meters (1,640 feet) 3.9 dB 50/125 micron multimode at 2.
Physical Planning Considerations 5 Extended-Distance Ports Through longwave laser transceivers and repeaters or wavelength division multiplexing (WDM) equipment, directors and fabric switches (but not Sphereon 4300 or 4500 Switches) support Fibre Channel data transmission distances of over 100 km. The extended distance feature is enabled on a port-by-port basis by using entries in the RX BB Credit column for a specified port at the Element Manager application’s Configure Ports dialog box.
Physical Planning Considerations 5 Cables LC Connectors Figure 5-1 Fiber-optic jumper cables are required to connect directors, fabric switches, and SAN routers ports to servers, devices, distribution panels, or other elements in a multiswitch fabric or routed SAN. Depending on the attached device and fabric element port, use one of the following types of cable: • Graded-index 62.5/125 micron multimode cable provides a transmission distance of up to 250 meters (1.0625 Gbps), 120 meters (2.
Physical Planning Considerations 5 Routing Fiber-Optic Cables Follow a logical plan for routing fiber-optic cables to avoid confusing connections during installation and operation. Route cables from the access holes at the bottom of the Fabricenter equipment cabinet to fabric element ports. When routing cables to ports be aware: • In a Fibre Channel Protocol (FCP) environment, ports are numbered by physical port number.
Physical Planning Considerations 5 Management Server, LAN, and Remote Access Support Out-of-band (non-Fibre Channel) console access to directors, fabric switches, and SAN routers is provided to perform a variety of operations and management functions. These functions are performed from one or more of the following consoles: Management Server • Through a personal computer (PC) or workstation connected to the management server through a customer LAN segment.
Physical Planning Considerations 5 Management Server Connectivity The management server provides an auto-detecting 10/100 Base-T Ethernet interface that connects to the 24-port hub mounted at the top of the Fabricenter equipment cabinet. Each director CTP card, fabric switch front panel, or SAN router front panel also provides an auto-detecting 10/100 Base-T Ethernet interface that connects to the hub. Factory-installed cables connect the management server, hub, and managed products.
Physical Planning Considerations 5 Remote User Workstations • Interconnecting Fabricenter cabinets - To increase the products managed by one management server, Ethernet hubs in one or more equipment cabinets must be connected. Plan for an Ethernet cable length that meets the distance requirement between cabinets. In addition, plan for an additional 1.5 meters (5 feet) of cable outside the cabinet to provide slack for service clearance, limited cabinet movement, or inadvertent cable pulls.
Physical Planning Considerations 5 • Figure 5-2 Typical Network Configuration (One Ethernet Connection) • 5-12 Part of the dedicated 10/100 megabit per second (Mbps) LAN segment that provides access to managed products. This Ethernet connection is part of the equipment cabinet installation and is required. Connection of remote workstations through the hub is optional. This type of network configuration using one Ethernet connection through the management server is shown in Figure 5-2.
Physical Planning Considerations 5 Figure 5-3 Typical Network Configuration (Two Ethernet Connections) If only one management server connection is used and this connection is provided through the customer intranet, all functions provided by the server are available to users throughout the enterprise. The purpose for dual LAN connections is to provide a dedicated LAN segment that isolates the server and managed products from other users in the enterprise.
Physical Planning Considerations 5 • Up to 12 authorized management workstations can be configured through the director or fabric switch SAN management application to receive unsolicited SNMP trap messages that indicate product operational state changes and failure conditions. • Up to eight authorized management workstations can be configured through a SAN router Element Manager application to send SNMP trap messages that indicate product operational state changes and failure conditions.
Physical Planning Considerations 5 Security Provisions Security provisions are available to restrict unauthorized access to a director, switch, or attached Fibre Channel devices. Access to the director or switch (through the SAN management application, Element Manager application, or SANpilot interface) is restricted by implementing password protection.
Physical Planning Considerations 5 System administrators can use the SAN management application to assign remote workstation access to directors and switches. Remote sessions are allowed for anyone on a customer intranet, disallowed completely, or restricted to specific workstations. Remote users must log into the SAN management application with a user name and password, just as when logging in to the local management server. Passwords are encrypted when sent across the network.
Physical Planning Considerations 5 The fabric element transmits a random value (used only once), an ID value (incremented at each login), and a shared CHAP secret (16-byte random value) to the server. The server concatenates the random value, ID value, and CHAP secret, and calculates a oneway message digest (also called a hash value). The hash value is transmitted to the authenticator (fabric element).
Physical Planning Considerations 5 • RADIUS server support - Remote authentication dial-in user service (RADIUS) is a client-server, UDP-based protocol that supports storage and authentication of passwords and CHAP secrets. Directors, fabric switches, and SAN routers support a RADIUS client (LAN-connected to a primary or secondary RADIUS server) that authenticates CHAP responses and login passwords. The RADIUS server stores: — Management server-to-fabric element (director or fabric switch) CHAP secrets.
Physical Planning Considerations 5 — Authorization errors. — Authentication errors. — Management application user connections. Use of the SANtegrity Authentication feature in conjunction with other security provisions must be carefully planned and coordinated. For additional information, refer to Security Best Practices. Obtain planning assistance from McDATA’s professional services organization before implementing the feature.
Physical Planning Considerations 5 SANtegrity Binding Planning Considerations • Rerouting delay - If a fabric topology changes, directors and fabric switches calculate a new least-cost data transfer path through a fabric, and routing tables immediately implement that path. This may result in Fibre Channel frames being delivered to a destination device out of order, because frames transmitted over the new (shorter) path may arrive ahead of previouslytransmitted frames that traverse the old (longer) path.
Physical Planning Considerations 5 Figure 5-4 Configure Allow/Prohibit Matrix - Active Dialog Box To access the dialog box, ensure the FICON management style is enabled for the director or switch, then select the Allow/Prohibit and Active options from the Element Manager application’s Configure menu. Figure 5-4 shows that port 1 (logical port address 05) is prohibited from communicating with port 6 (logical port address 0A), port 7 (logical port address 0B), and port 8 (logical port address 0C).
Physical Planning Considerations 5 Figure 5-5 PDCM Array - Example Problem A PDCM array configured for Director A prohibits logical port address 05 from communicating with logical port addresses 0A, 0B, and 0C. No PDCM array is configured for Director B. The PDCM array configured for Director A prohibits the source server from transmitting or receiving data across ISL 2. However, internal route tables at both directors indicate a valid server-to-destination device path across ISL 1.
Physical Planning Considerations 5 Preferred Path The preferred path option allows a user to specify and configure one or more ISL data paths between multiple directors or fabric switches in a fabric. At each fabric element, a preferred path consists of a source port on the director or switch being configured, an exit port on the director or switch, and the Domain_ID of the destination director or switch. Each participating director or switch must be configured as part of a desired path.
Physical Planning Considerations 5 A preferred path is configured between a source server and destination device (A or B), traversing Director 1, Director 2, and Director 3. To configure the preferred path through the first director: 1. Select the Preferred Path option from the Element Manager application’s Configure menu. The Configure Preferred Paths dialog box displays. 2. Click Add. The Add Preferred Path dialog box displays (bottom of Figure 5-6). 3.
Physical Planning Considerations 5 Zoning Directors and fabric switches support a user configuration that partitions attached devices into restricted-access groups called zones. Devices in the same zone can recognize and communicate with each other through switched port-to-port connections. Devices in separate zones cannot recognize name server or route table information and therefore cannot communicate with each other.
Physical Planning Considerations 5 Zones are configured through the SAN management application (SANavigator 4.2 or EFCM 8.6) by authorizing or restricting access to name server or route table information (depending on the firmware release level) associated with device N_Ports that attach to director or switch fabric ports (F_Ports).
Physical Planning Considerations 5 ATTENTION ! If zoning is implemented by WWN, removal and replacement of a device HBA or Fibre Channel interface (thereby changing the device WWN) disrupts zone operation and may incorrectly exclude a device from a zone. • The domain identification (Domain_ID) and physical port number of the director or fabric switch port to which the device is attached.
Physical Planning Considerations 5 — The default zone is enabled or disabled separately from the active zone set. — If the default zone is enabled, then all devices not in a specified zone are included in the default zone and can communicate with each other. — If the default zone is disabled and there is no active zone set, then the zoning feature is completely disabled for the fabric and no devices can communicate with each other.
Physical Planning Considerations 5 • Reasons for zone implementation - Determine if zoning is to be implemented for the enterprise. If so, evaluate if the purpose of zoning is to differentiate between operating systems, data sets, user groups, devices, processes, or some combination thereof. Plan the use of zone members, zones, and zone sets accordingly. • Zone members specified by port number or WWN - Determine if zoning is to be implemented by port number or WWN.
Physical Planning Considerations 5 • Each server HBA is explicitly bound to a storage volume or LUN, and access is explicitly authorized (access is blocked by default). • The process is compatible with OSI standards. The following are transparently supported: — Different operating systems and applications. — Different storage volume managers and file systems. — Different fabric devices, including disk drives, tape drives, and tape libraries.
Physical Planning Considerations 5 2. SANtegrity Binding - The SANtegrity Binding feature is recommended for large and complex SANs with fabrics and devices provided by multiple OEMs or that intermix FCP and FICON protocols. The feature is required for FICON-cascaded high-integrity SANs. SANtegrity Binding includes: — Fabric binding (configured and enabled through the SAN management application) that allows only user-specified directors or switches to attach to specified fabrics in a SAN.
Physical Planning Considerations 5 5. Preferred path - A preferred path provides soft control of fabric routing decisions on a switch-by-switch or port-by-port basis. The path instructs a fabric to use a preferred exit port out of a director or fabric switch for a specified receive port and target domain. If a preferred path is prohibited by SANtegrity Binding, PDCM arrays, or hard zoning, the path is not programmed.
Physical Planning Considerations 5 Follow best practices listed here in order of precedence. Logically work in sequence from the most restrictive method to the least restrictive method, ensuring the most restrictive connectivity or routing paths override all other paths. Optional Feature Keys McDATA offers several operating features that are available for the switch as customer-specified options.
Physical Planning Considerations 5 • OpenTrunking - Purchase and enablement of this feature provides dynamic load balancing of Fibre Channel traffic across multiple ISLs. NOTE: SAN routers do not support OpenTrunking. • Full volatility - Purchase and enablement of this feature ensures that no Fibre Channel frames are stored after a director or Fabric switch is powered off or fails, and a memory dump file (that possibly includes classified frames) is not included as part of the data collection procedure.
Physical Planning Considerations 5 After purchasing a feature, obtain the required PFE key through your McDATA marketing representative. A PFE key is encoded to work with the serial number of a unique director or fabric switch and is an alphanumeric string consisting of both uppercase and lowercase characters. The total number of characters may vary. The PFE key is case sensitive and must be entered exactly, including dashes. The following is an example of the format: XxXx-XXxX-xxXX-xX.
Physical Planning Considerations 5 • Connectivity to an IBM S/390 Parallel Enterprise Server (Generation 5 or Generation 6), with one or more FICON channel adapter cards installed, using System Automation for Operating System/390 (SA OS/390) for native FICON, Version 1.3 or later, plus service listed in the appropriate preventive service planning (PSP) bucket. The PSP bucket upgrade is HKYSA30. The minimum OS/390 level for a director or switch without the control unit port (CUP) feature is Version 2.
Physical Planning Considerations 5 SANtegrity Authentication SANtegrity Authentication is a feature that significantly enhances and extends SAN data security by providing password safety; CHAP or DHCHAP verification for fabric elements, management servers, and devices; a PCP user database; CT authentication for the OSMS interface; RADIUS server support; inband and out-of-band access controls lists; encrypted SSH protocol; and security logging.
Physical Planning Considerations 5 Server 1 Storage 1 Trunk TM Server 2 Director A ISL 1 ISL 1 ISL 2 ISL 2 TM Director B Storage 2 Server 3 Figure 5-8 OpenTrunking The figure illustrates two Intrepid 6064 Directors connected by two ISLs. Three servers use the ISLs to communicate with two storage devices. Without trunking, servers 1 through 3 route Fibre Channel traffic from to director B without regard to any data rates.
Physical Planning Considerations 5 When a director or fabric switch (without the full volatility feature installed) powers off or fails, a dump file is written to non-volatile random-access memory (NV-RAM). This dump file retains the last 30 Fibre Channel frames transmitted from the embedded port and the last four frames transmitted to the embedded port. These Fibre Channel frames are then written to diskette and included as part of the data collection procedure.
Physical Planning Considerations 5 CNT WAN Support Fibre Channel-based SANs are typically implemented as discrete islands - isolated networks accessible only from local servers connected through a Fibre Channel fabric. Many companies are striving to interconnect isolated SANs and consolidate computer resources through WAN extension technology. Therefore, edge switches deployed as part of a core-to-edge fabric often require WAN connectivity.
Physical Planning Considerations 5 Figure 5-10 Hardware View (with Element Manager Message) Physical Planning Considerations 5-41
Physical Planning Considerations 5 5-42 McDATA Products in a SAN Environment - Planning Manual
6 Configuration Planning Tasks This chapter describes configuration planning best-practices tasks to be performed before installing one or more McDATA Fibre Channel switching products in a storage area network (SAN) configuration. Table 6-1 summarizes planning tasks described in the chapter.
Configuration Planning Tasks 6 Table 6-1 Configuration Planning Tasks (continued) Task Page Task 13: Complete the Planning Worksheet 6-14 Task 14: Plan AC Power 6-28 Task 15: Plan a Multiswitch Fabric (Optional) 6-29 Task 16: Plan Zone Sets for Multiple Products (Optional) 6-30 Task 17: Plan SAN Routing (Optional) 6-31 Task 18: Complete Planning Checklists 6-34 Task 1: Prepare a Site Plan For each director, fabric switch, SAN router, or FC-512 Fabricenter equipment cabinet installed, design
Configuration Planning Tasks 6 • Power requirements, including an optional uninterruptable power supply (UPS). • Lengths of power cables and location of electrical outlets (for directors, switches, and the management server) having the proper phase, voltage, amperage, and ground connection. DANGER Use the supplied power cords. Ensure the facility power receptacle is the correct type, supplies the required voltage, and is properly grounded.
Configuration Planning Tasks 6 In addition, consider the following when planning cable routing: • The need for additional fiber-optic cables could grow rapidly. Consider installing cable with extra fibers, especially in hard to reach places like underground trenches. Consider locating the equipment cabinet near a fiber-optic patch panel. • Follow proper procedures when moving an installed equipment cabinet to prevent cable or connector damage.
Configuration Planning Tasks 6 NOTE: If the FICON management server feature is enabled, the default operating style is FICON. The open systems management style cannot be enabled. Consider purchasing and enabling the SANtegrity Authentication and SANtegrity Binding features to provide additional data security in a complex and multi-OEM environment. Contact your McDATA representative for information about the features.
Configuration Planning Tasks 6 • Remote user workstations - If remote access to the management server is required, plan to install user workstations with the SAN management and Element Manager applications configured. Administrators can use these remote workstations to configure and monitor directors and fabric switches. Up to 25 sessions can be simultaneously active. Sessions from remote user workstations are disabled if the management server is powered off.
Configuration Planning Tasks 6 • Connect equipment cabinets - Ethernet hubs in multiple equipment cabinets can be connected to provide management server access to up to 48 managed McDATA products. Cabinets can be placed at any distance up to the limit of the 10/100 megabit per second (Mbps) LAN segment.
Configuration Planning Tasks 6 • If installing products and the management server on a public LAN containing other devices, default network addresses may require change to avoid address conflicts with existing devices. For Intrepid-series directors, Sphereon-series fabric switches, and Eclipse-series SAN routers, the IP address, gateway address, and subnet mask are changed through a remote terminal connected to the product’s maintenance port.
Configuration Planning Tasks 6 • Eclipse 1620 SAN Router: — System addresses: • MAC address is unique for each product. • Default IP address is 192.168.111.100. • Subnet mask is 255.255.255.0. • Gateway address is 0.0.0.0. — 10/100 Base-T Ethernet management port addresses: • Default IP address is 192.168.100.100. • Subnet mask is 255.255.255.0. • Gateway address is 0.0.0.0. — Intelligent port (3) addresses: • Default IP address is 0.0.0.0. • Subnet mask is 0.0.0.0. • External IP address is 0.0.0.0.
Configuration Planning Tasks 6 — 10/100 Base-T Ethernet management port addresses: • Default IP address is 192.168.100.100. • Subnet mask is 255.255.255.0. • Gateway address is 0.0.0.0. — Intelligent port (13 through 16) addresses: • Default IP address is 0.0.0.0. • Subnet mask is 0.0.0.0. • External IP address is 0.0.0.0. • Internal IP address is 0.0.0.0.
Configuration Planning Tasks 6 • Determine which (if any) management workstations can have write permission for SNMP variables. • Obtain product-specific trap information from McDATA to load onto SNMP management workstations. Task 8: Plan E-Mail Notification (Optional) As an option, network administrators can configure director and fabric switch e-mail support.
Configuration Planning Tasks 6 • Remote access to products is possible through the maintenance port or an internal modem connection to the management server. These connections are for use by authorized service personnel only and should be carefully monitored. • The number of remote workstations with access to the management server and managed products can and should be restricted. Obtain IP addresses for workstations that should have exclusive access.
Configuration Planning Tasks 6 Task 11: Diagram the Planned Configuration Determine peripheral devices that will connect to each director, switch, or SAN router and where connectivity should be limited (zoning). These devices may include servers, storage control devices, and other fabric elements in a multiswitch fabric. Part of this task may have been performed when the configuration was determined. It might be helpful to draw the configuration diagram. Indicate distances in the diagram if necessary.
Configuration Planning Tasks 6 Rules for Port Names Port names can be up to 24 alphanumeric characters in length. Spaces, hyphens, and underscores are allowed within the name. Each port name must be unique for a product. However, the same port name can be used on separate products. It is recommended that unique port names be used, particularly within a complex multiswitch fabric. Example port names include: Lab server. Test system-2. Printer_001.
Configuration Planning Tasks 6 Product Planning Worksheet (Page 1 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 Configuration Planning Tasks 6-15
Configuration Planning Tasks 6 Product Planning Worksheet (Page 2 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 6-16 McDATA Products in a SAN Environment - Planning Manual Model IP Address Zone
Configuration Planning Tasks 6 Product Planning Worksheet (Page 3 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 Configuration Planning Tasks 6-17
Configuration Planning Tasks 6 Product Planning Worksheet (Page 4 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 6-18 McDATA Products in a SAN Environment - Planning Manual Model IP Address Zone
Configuration Planning Tasks 6 Product Planning Worksheet (Page 5 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Configuration Planning Tasks 6-19
Configuration Planning Tasks 6 Product Planning Worksheet (Page 6 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 6-20 McDATA Products in a SAN Environment - Planning Manual Model IP Address Zone
Configuration Planning Tasks 6 Product Planning Worksheet (Page 7 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 Configuration Planning Tasks 6-21
Configuration Planning Tasks 6 Product Planning Worksheet (Page 8 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 6-22 McDATA Products in a SAN Environment - Planning Manual Model IP Address Zone
Configuration Planning Tasks 6 Product Planning Worksheet (Page 9 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 Configuration Planning Tasks 6-23
Configuration Planning Tasks 6 Product Planning Worksheet (Page 10 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 6-24 McDATA Products in a SAN Environment - Planning Manual Model IP Address Zone
Configuration Planning Tasks 6 Product Planning Worksheet (Page 11 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 Configuration Planning Tasks 6-25
Configuration Planning Tasks 6 Product Planning Worksheet (Page 12 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 6-26 McDATA Products in a SAN Environment - Planning Manual Model IP Address Zone
Configuration Planning Tasks 6 Product Planning Worksheet (Page 13 of 13) Attached Devices Switch Name:____________________________ IP Address:____________________________ Unit Name:__________________________ Port Port Name Location Type Model IP Address Zone 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 Configuration Planning Tasks 6-27
Configuration Planning Tasks 6 Task 14: Plan AC Power Plan for facility power sources for each Fabricenter equipment cabinet, director, fabric switch, or SAN router as follows: • The Fabricenter equipment cabinet operates at 47 to 63 Hertz (Hz), 200 to 240 volts alternating current (VAC), and requires a minimum dedicated 30-ampere service. • The Intrepid 6140 Director operates at 47 to 63 Hz, 200 to 240 VAC, and requires a minimum dedicated 15-ampere service.
Configuration Planning Tasks 6 Task 15: Plan a Multiswitch Fabric (Optional) If a multiswitch fabric topology is to be implemented, carefully plan the physical characteristics and performance objectives of the topology, including the proposed number of fabric elements, characteristics of attached devices, cost, nondisruptive growth requirements, and service requirements. Refer to Fabric Topologies, Planning for Multiswitch Fabric Support, and General Fabric Design Considerations for detailed information.
Configuration Planning Tasks 6 Task 16: Plan Zone Sets for Multiple Products (Optional) If name server zoning is to be implemented, carefully plan the characteristics and security objectives (separation of operating systems, data sets user groups, devices, or processes) of zone members, zones and zone sets. If a fabric topology or routed SAN is implemented, zoning is configured on a fabric-wide or SAN-wide basis. Planning for zoned configurations must be carefully coordinated with planning the topology.
Configuration Planning Tasks 6 Task 17: Plan SAN Routing (Optional) If a routed SAN is to be implemented, carefully plan metropolitan area network (MAN) or wide area network (WAN) connectivity and the integration of SAN routers with standard fibre channel fabric elements. Ensure basic requirements for physical SAN routers are incorporated in the site plan(s) developed while performing Task 1: Prepare a Site Plan.
Configuration Planning Tasks 6 • IP network connections - At each location, cables with appropriate connectors must be routed between the IP transport network and the SAN router. The Eclipse 1620 SAN Router supports Ethernet RJ-45 connectors or SFP optical transceivers with LC duplex connectors. The Eclipse 2640 SAN Router supports SFP optical transceivers with LC duplex connectors.
Configuration Planning Tasks 6 • Negotiate SLA - Network service providers provide IP WAN transport services in accordance with a negotiated service level agreement (SLA). Ensure the SLA specifies the link availability, peak available bandwidth, latency, security level, monitoring level, packet loss, and mean time to repair (MTTR). • Intelligent port addresses - Each intelligent port that supports Internet Fibre Channel protocol (iFCP) requires an IP address and subnet mask.
Configuration Planning Tasks 6 Task 18: Complete Planning Checklists As a guide for planning tasks, complete the planning checklists under this task. Checklists provide detailed planning activities and provide space for a planned completion date for each activity. The customer’s management information system (MIS) project manager should examine the checklists and determine the personnel and resources required for completing planning and installation tasks.
Configuration Planning Tasks 6 Table 6-2 Physical Planning and Hardware Installation Tasks Activity Task Owner Due Date Comments Locate the physical facilities. Connect the facility alternating current (AC) power circuits. If more than one managed product, consider separate power circuits for availability. Obtain an uninterruptable power supply (optional). Recommended. Obtain two outside-access phone lines. One for the modem and the second for a telephone.
Configuration Planning Tasks 6 Table 6-2 Physical Planning and Hardware Installation Tasks (continued) Activity Task Owner Due Date Comments Set up LAN connections to corporate intranet for remote workstation access (optional). Remote workstation access is supported for directors and fabric switches only. Determine peak available bandwidth of the available IP WAN network (optional). If SAN routing is supported, rate limiting must be configured and enabled based on peak available bandwidth.
Configuration Planning Tasks 6 Table 6-3 Operational Setup Tasks (continued) Activity Determine if inband management of the director or switch is to be used, and if so, the type (FICON or open-systems). Task Owner Due Date Comments Management server and Fibre-Channel-attached server peripheral (optional). Determine if the call-home feature is to be used. Determine call-home telephone numbers to be used. Determine if the e-mail notification feature is to be used.
Configuration Planning Tasks 6 Table 6-3 Operational Setup Tasks (continued) Activity Configure extended distance ports. Task Owner Due Date Comments If SAN routing is supported, configure extended distance ports in accordance with IP WAN requirements. Enable and configure optional feature keys. Configure link incident alerts. Configure Ethernet events.
A Product Specifications This appendix lists specifications for McDATA directors, fabric switches, storage area network (SAN) Routers, and the FC-512 Fabricenter equipment cabinet.
Product Specifications Intrepid 10000 Director: Height: 62.2 centimeters (24.5 inches) or 14 rack units. Width: 44.3 centimeters (17.5 inches). Depth: 86.4 centimeters. (34.0 inches). Weight: 152.0 kilograms (335.0 pounds). Sphereon 3232 Switch: Height: 6.5 centimeters (2.6 inches) or 1.5 rack units. Width: 44.5 centimeters (17.5 inches). Depth: 64.1 centimeters (25.2 inches). Weight: 16.8 kilograms (37.0 pounds). Sphereon 4300 Switch: Height: 4.1 centimeters (1.6 inches) or 1 rack unit. Width: 43.
Product Specifications Power Requirements McDATA products have the following nominal power requirements: Intrepid 6064 Director: Input voltage: 100 to 240 VAC. Input current: 2.0 amps at 208 VAC. Input frequency: 47 to 63 Hz. Intrepid 6140 Director: Input voltage: 200 to 240 VAC. Input current: 4.2 amps at 208 VAC. Input frequency: 47 to 63 Hz. Intrepid 10000 Director: Input voltage: 180 to 270 VAC. Input current: 12.0 amps at 208 VAC.
Product Specifications Eclipse 1620 SAN Router: Input voltage: 100 to 240 VAC. Input current: 0.35 amps at 208 VAC. Input frequency: 47 to 63 Hz. Eclipse 2640 SAN Router: Input voltage: 100 to 240 VAC. Input current: 0.95 amps at 208 VAC. Input frequency: 47 to 63 Hz. Heat Dissipation McDATA products have the following maximum heat dissipation characteristics: Intrepid 6064 Director: 490 watts (1,672 BTU/hr). Intrepid 6140 Director: 841 watts (2,873 BTU/hr).
Product Specifications Intrepid 10000 Director: Right and left side: 5.1 centimeters (2.0 inches). Front and rear: 7.6 centimeters (3.0 inches). NOTE: If the Intrepid 10000 Director is installed in a non-McDATA equipment cabinet with a door that does not provide direct airflow over the full height of the unit, 17.8 centimeters (7.0 inches) of front clearance is required. Top and bottom: No clearance required. Sphereon 3232 Switch: Right and left side: No clearance required. Front and rear: 7.
Product Specifications Acoustical Noise and Physical Tolerances This section lists acoustical noise generated, shock and vibration tolerances, and inclination tolerances for McDATA directors, fabric switches, and SAN routers. Acoustical noise generated: Intrepid 6064 Director: 55 dB “A” scale. Sphereon 4300 and 4500 Switches: 64 dB “A” scale. Intrepid 10000 Director: 75 dB “A” scale. All other products: 70 dB “A” scale.
Product Specifications Operating Environment This section specifies environmental requirements for operating McDATA products. Temperature: 400 F to 1040 F (40 C to 400 C). Relative humidity: 8% to 80%. Maximum wet-bulb temperature: 810 F (270 C). Altitude: 10,000 feet (3,048 meters). FC-512 Fabricenter Cabinet Specifications This section lists specifications (dimensions, weight, power requirements, cooling airflow clearances, and service clearances) for the FC-512 Fabricenter equipment cabinet.
Product Specifications Power Requirements The Fabricenter cabinet has the following power requirements: Input voltage: 200 to 240 VAC. Input current: 30.0 amps at 208 VAC. Input frequency: 47 to 63 Hz. Clearances The Fabricenter cabinet has the following cooling airflow and service clearances. Cooling airflow clearances: Right and left side: No clearance required. Front and rear: 15.2 centimeters (6.0 inches). Service clearances: Right and left side: No clearance required. Front and rear: 91.
Product Specifications Figure A-1 Fabricenter Cabinet Footprint Product Specifications A-9
Product Specifications A-10 McDATA Products in a SAN Environment - Planning Manual
B Firmware Summary This appendix summarizes differences and similarities between the Enterprise Operating System (E/OS) for Intrepid 6000-series directors and Sphereon-series fabric switches; Enterprise Operating System, nScale (E/OSn) for the Intrepid 10000 Director; and Enterprise Operating System, internetworking (E/OSi) for Eclipse-series SAN routers. The appendix includes tables that list: • System-related differences. • Fibre Channel protocol-related differences.
Firmware Summary Table B-4 B-2 E/OS versus E/OSn and E/OSi - System-Related Differences (Continued) Feature E/OS 7.0 E/OSn 6.0 Nondisruptive hot code activation (HotCAT) Upgrade nondisruptive to Fibre Channel traffic for single and dual CTP card directors and for fabric switches. Upgrade does not require director CTP card switchover. E/OS allows upgrade or downgrade by greater than one functional release. Director must be set offline to upgrade a single CTP card.
Firmware Summary Table B-4 E/OS versus E/OSn and E/OSi - System-Related Differences (Continued) Feature E/OS 7.0 E/OSn 6.0 E/OSi 4.6 Misaligned word generation Misaligned words not generated. Misaligned word is generated when the port state machine (PSM) transitions from inactive to active state or from active to inactive state. This anomaly has no effect on Fibre Channel device behavior. Misaligned words not generated. Flexible partition (FlexPar) feature support Flexpar feature not supported.
Firmware Summary Table B-5 E/OS versus E/OSn and E/OSi - Fibre Channel Protocol-Related Differences (Continued) Feature B-4 E/OS 7.0 E/OSn 6.0 E/OSi 4.6 Expansion port (E_Port) staging E_Port staging not supported. The Intrepid 10000 Director supports E_Port staging. The director allows only one E_Port connection to a neighbor switch to take part in a fabric build process. Upon process completion, subsequent E_Ports to neighbor switches are brought up (staged).
Firmware Summary Table B-5 E/OS versus E/OSn and E/OSi - Fibre Channel Protocol-Related Differences (Continued) Feature E/OS 7.0 E/OSn 6.0 E/OSi 4.6 Reroute delay behavior With reroute delay enabled, the destination route point is cleared and a delay equal to the error detect time-out value (ED_TOV) is applied before a new route is programmed. During the delay, Class 3 frames are dropped.
Firmware Summary Table B-6 E/OS versus E/OSn and E/OSi - Management-Related Differences (Continued) Feature B-6 E/OS 7.0 E/OSn 6.0 E/OSi 4.6 General SNMP support SNMP interface supports Version 1, Version 2c, and a configuration-related subset of product features. SNMP interface supports Version 1, Version 2c, and Version 3. Read-only use recommended with Versions 1 and 2c. Read-write use recommended with secure Version 3.
Firmware Summary Table B-6 E/OS versus E/OSn and E/OSi - Management-Related Differences (Continued) Feature E/OS 7.0 E/OSn 6.0 E/OSi 4.6 FICON Management Server (FMS) PFE key install and enable behavior FMS is auto-enabled upon PFE installation. Management style automatically changes to FICON. When PFE is installed, the user must explicitly enable FMS. Management style does not automatically change to FICON. FMS PFE key not supported.
Firmware Summary Table B-6 E/OS versus E/OSn and E/OSi - Management-Related Differences (Continued) Feature B-8 E/OS 7.0 E/OSn 6.0 Port group online diagnostics (directors) Diagnostic errors for any single port in a group (port card) cause a group failure indication. Diagnostic errors for any single port in a group (LIM) are indicated individually and do not cause a group failure indication. Port grouping (port cards or LIMs) not supported.
Firmware Summary Table B-6 E/OS versus E/OSn and E/OSi - Management-Related Differences (Continued) Feature E/OS 7.0 E/OSn 6.0 E/OSi 4.6 Special port states and reason codes Port state Inactive with special reason codes Reserved and InvalidOTPConfig not supported. Logical port addresses FE and FF forced offline and placed in special state Inactive with reason code Reserved if FICON CUP is enabled. FICON states are Internal Port (address FE) and Unimplemented (address FF).
Firmware Summary Table B-6 E/OS versus E/OSn and E/OSi - Management-Related Differences (Continued) Feature B-10 E/OS 7.0 E/OSn 6.0 E/OSi 4.6 Diagnostics (port failure state) Port set to Failed state upon failing a user- invoked port diagnostic test. If external port diagnostics are performed without a loopback plug the diagnostic test fails, but port is not set to Failed.
Firmware Summary Table B-6 E/OS versus E/OSn and E/OSi - Management-Related Differences (Continued) Feature E/OS 7.0 E/OSn 6.0 E/OSi 4.6 FICON management style support User-selectable option on a per product basis. Setting is stored on the product. User-selectable option on a per user (different for each login ID) or per product basis. Setting is stored on the management server, not the product. Setting is backed up by the server backup process.
Firmware Summary B-12 McDATA Products in a SAN Environment - Planning Manual
Index A access control list inband 5-18 out-of-band 5-18 role-based access 4-7 any-to-any connectivity 1-26 application I/O profiles 3-31 arbitrated loop switch connectivity features 1-26 default network address 6-8 security features 1-28 serviceability features 1-29 arbitrated loop topology description 3-2 fabric-attached planning considerations 3-11 FL_Port connectivity 3-10 operating characteristics 3-2 private device connectivity 3-7 private loop 3-9 private loop planning considerations 3-10 public dev
Index best practices cabling 3-20 configuration planning 6-1 connectivity 3-20 distance extension 4-55 FCP and FICON intermix 3-47 FICON cascading 3-55 multiswitch fabric topology 3-20 preventing ISL oversubscription 3-33 SAN routing 4-29 security 5-30 binding fabric 5-19 persistent 5-29 SANtegrity 5-19 switch 5-19 business continuance IP versus storage traffic 4-36 operational mode asynchronous data replication 4-38 synchronous data replication 4-38 requirements data priority 4-37 distance 4-37 recovery p
Index D dark fiber distance extension bandwidth 4-46 description 4-39 illustration 4-40 latency 4-46 recovery point objective 4-46 recovery time objective 4-46 data compression algorithm selections 4-26 description 1-21 optimizing WAN use 4-55 set compression level 4-57 data replication asynchronous mode 4-38 dark fiber transport 4-46 IP transport 4-48 SONET/SDH transport 4-47 synchronous mode 4-38 WDM transport 4-47 data transmission distance cable type 5-4 multiswitch fabric requirements 3-21 transceiver
Index Domain_ID assignment director 3-25 fabric switch 3-25 proxy Domain_ID 30 4-13, 4-18 proxy Domain_ID 31 4-13, 4-25 R_Port 4-15 SAN router 4-15 E E/OS description 2-11 management-related properties B-5 operating system differences B-1 operating system similarities B-1 protocol-related properties B-3 system-related properties B-1 E/OSi description 2-12 management-related properties B-5 operating system differences B-1 operating system similarities B-1 protocol-related properties B-3 system-related prop
Index transport technology dark fiber 4-39 IP 4-45 SONET/SDH 4-42 WDM 4-40 transport technology comparison 4-48 F fabric availability nonresilient dual fabric 3-38 nonresilient single fabric 3-38 redundant fabrics 3-38 resilient dual fabric 3-38 resilient single fabric 3-38 fabric binding 5-19 fabric element FCP and FICON intermix environment 3-41 limitations in a fabric 3-19 fabric performance device fan-out ratio 3-35 fabric initialization 3-30 fabric scalability 3-39 I/O requirements 3-31 performance t
Index FICON management server description 5-35 introduction 2-5 plan console support 6-6 firmware application services 2-12 E/OS description 2-11 E/OSi description 2-12 E/OSn description 2-11 fabric services 2-13 Fibre Channel protocol services 2-12 loop services 2-13 network services 2-12 operating system differences B-1 operating system services 2-12 operating system similarities B-1 port services 2-12, 2-13 system management services 2-12 FL_Port connectivity 3-10 FlexPar technology description 4-4 dire
Index I I/O requirements application I/O profiles 3-31 device locality 3-34 ISL oversubscription 3-32 iFCP protocol build fabric events 4-23 comparison to mFCP 4-28 description 4-22 inband product management feature keys 5-35 FMS feature 2-5 OSMS feature 2-5 plan console support 6-6 intelligent port Eclipse 1620 SAN Router 1-23 Eclipse 2640 SAN Router 1-25 implement rate limiting 4-51 set port speed 4-52 inter-FlexPar routing 4-28 interoperability planning 6-4 vendor limitations 3-20 Intrepid 10000 Directo
Index L large fabric fabric initialization 3-30 fabric scalability 3-39 high-bandwidth ISLs 3-30 high-port count directors 3-30 problems 4-3 laser transceiver description 5-3 restrictions 5-5 SFP transceiver 5-7 transmission distance 5-4 latency dark fiber transport 4-46 directors 1-7 IP transport 4-48 SONET/SDH transport 4-47 WDM transport 4-47 LC duplex connector 5-7 LIM assigning BB_Credits 4-50 Intrepid 10000 Director 5-2 load balancing 3-22 local area network comparison to WAN 4-36 latency 4-36 protoc
Index N N_Port DHCHAP authentication 5-17 name conventions, ports 6-14 name server zoning introduction 1-26 planning requirements 6-30 network addresses default settings 6-8 planning 6-7 nickname conventions, ports 6-14 nonresilient fabric dual 3-38 single 3-38 nScale architecture 1-13 O open-system management server description 5-35 introduction 2-5 plan console support 6-6 OpenTrunking feature description 5-37 planning considerations 3-22 support planning 6-29 operating environment A-7 optional feature
Index plan SAN routing 6-31 plan SNMP support 6-10 plan zone sets 6-30 prepare a site plan 6-2 planning worksheet 6-14 port binding 1-28 blocking 1-26 fiber-optic cabling 5-1 logical port addressing 3-42 name conventions 6-14 nickname conventions 6-14 numbering 3-42 port card Intrepid 6064 Director 5-2 Intrepid 6140 Director 5-2 map 3-43 port connections any-to-any connectivity 1-26 Eclipse 1620 SAN Router 1-23 Eclipse 2640 SAN Router 1-24 Intrepid 10000 Director 1-14 Intrepid 6064 Director 1-10 Intrepid 6
Index R S R_Port configuring 4-34 Domain_ID assignment 4-15 operation 4-11 RADIUS server support 5-18 rate limiting description 4-26 implementation 4-51 intelligent port speed selections 4-52 recovery point objective dark fiber transport 4-46 description 4-37 IP transport 4-48 SONET/SDH transport 4-47 WDM transport 4-47 recovery time objective dark fiber transport 4-46 description 4-37 IP transport 4-48 SONET/SDH transport 4-47 WDM transport 4-47 redundant fabrics 3-38 remote fabric feature assigning BB_
Index routing domain (iSAN) 4-25 routing domain (mSAN) 4-18 SAN island consolidation 4-8 Tier 1 (fabrics) 4-8 Tier 2 (mSANs) 4-9 Tier 3 (iSANs) 4-9 zone policy 4-16 SANavigator application description 2-15 GUI description 2-15 main window 2-16 product overview 1-5 SANpilot interface description 2-24 plan console support 6-6 server connectivity 5-14 SANtegrity Authentication feature CHAP authentication 5-16 CT authentication 5-17 description 5-16 DHCHAP authentication 5-17 feature key description 5-37 inban
Index SANavigator application 2-15 SANpilot interface 2-24 SANvergence Manager application 2-21 SONET/SDH distance extension bandwidth 4-47 description 4-42 illustration 4-43 latency 4-47 recovery point objective 4-47 recovery time objective 4-47 specifications director clearances A-4 director dimensions A-1 director heat dissipation A-4 fabric switch clearances A-4 fabric switch dimensions A-1 fabric switch heat dissipation A-4 fabric switch power requirements A-3 Fabricenter cabinet clearances A-8 Fabric
Index I-14 W Z WDM distance extension bandwidth 4-47 description 4-40 illustration 4-41 latency 4-47 recovery point objective 4-47 recovery time objective 4-47 weight directors A-1 fabric switches A-1 Fabricenter cabinet A-7 SAN routers A-1 wide area network comparison to LAN 4-36 dedicated bandwidth 4-55 latency 4-36 optimize use buffering 4-55 data compression 4-55 FastWrite technology 4-56 flow control 4-55 jumbo frames 4-55 protocol stack 4-36 rate limiting 4-51 reliability 4-37 worksheet, planning
