Chapter 1 FIBRE CHANNEL: CONNECTION TO THE FUTURE SECOND EDITION The Fibre Channel Association www.fibrechannel.
Second Edition Copyright © 1998 by the Fibre Channel Association All rights reserved. No part of this book may be reproduced, in any form or by any means whatsoever, without permission in writing from the publisher. While every precaution has been taken in the preparation of this book, the Fibre Channel Association assumes no responsibility for errors or omissions. Neither is any liability assumed for damages resulting from the use of the information contained herein.
FOREWORD The information explosion and the need for high-performance communications for server-to-storage and server-to-server networking have been the focus of much attention during the 90s. Performance improvements in storage, processors, and workstations, along with the move to distributed architectures such as client/server, have spawned increasingly data-intensive and high-speed networking applications.
CONTENTS 1 2 3 WARP SPEED INFORMATION DELIVERY 1 Why Fibre Channel? Fibre Channel Systems 2 4 FIBRE CHANNEL AT WORK 5 Disaster Recovery Multiplatform Storage Enterprise Decision Making Effective Use of IT Resources Explosive Data Growth Management Real-Time Applications Audio-Video Digital Imaging 6 7 8 9 10 11 12 13 14 FIBRE CHANNEL OVERVIEW 15 Interoperability Storage Networks Technology Comparison 16 17 18 21 v
vi Contents 4 PHYSICAL AND SIGNALING LAYER 23 Physical Interface and Media: FC-0 Transmission Protocol: FC-1 Framing and Signaling Protocol: FC-2 24 27 29 UPPER LAYERS 33 Common Services: FC-3 ULP Mapping: FC-4 33 34 CLASSES OF SERVICE AND TOPOLOGIES 35 Classes of Service Topologies Point-to-Point, Dedicated-Bandwidth Topology Loop Shared-Bandwidth Topology Switched Scaled-Bandwidth Topology Summary 38 43 44 45 47 48 FIBRE CHANNEL SERVICES 49 Login Server Fabric/Switch Controller Name Serve
Chapter 1 WARP SPEED INFORMATION DELIVERY Fibre Channel is the solution for Information Technology (IT) professionals who demand reliable, cost-effective information storage and delivery at blazing speeds. Today’s data explosion presents unprecedented enterprise challenges in data warehousing, imaging, integrated audio/video, networked storage, real-time computing, CAD/CAE, archival/backup, and disaster recovery.
Fibre Channel: Connection to the Future 2 l Congestion Free – Fibre Channel’s credit-based flow control delivers data as fast as the destination buffer is able to receive it. l High Efficiency – Real price performance is directly correlated to the efficiency of the technology. Fibre Channel has very little transmission overhead, offering the capability to deliver very high, sustained utilization rates without loss of data.
Chapter 1: Warp Speed Information Delivery l High-performance workgroups l Campus backbones l Digital audio/video networks l Embedded military sensor, processing, and displays l Industrial control systems Supercomputer Mainframe Disk Farm Disk Farm Fibre Channel Switches Front End Processor Midrange Workstation Workstation Supercomputer Center Data Center Enterprise Fibre Channel Switch RAID Workstation Hub Hub Switch JBOD Workstations Design Center Server Department Figure 1.
4 Fibre Channel: Connection to the Future FIBRE CHANNEL SYSTEMS Fibre Channel systems are assembled from adapters, hubs, storage, and switches. Host bus adapters are installed into hosts in the same manner as a SCSI host bus adapter or a network interface card. Hubs link individual elements together to form a shared bandwidth loop. Disk systems integrate a loop into a storage backplane. A port bypass circuit provides the ability to hot swap Fibre Channel disks and links to a hub.
Chapter 2 FIBRE CHANNEL AT WORK Visionary IT managers are delivering bottom-line improvements for their operations with Fibre Channel. They deploy Fibre Channel to take advantage of the unique combination of channel and network features. Fibre Channel networks concurrently provide networked access to storage using storage protocols and server-to-server communications using network protocols. Virtually all data has become mission critical.
6 Fibre Channel: Connection to the Future DISASTER RECOVERY One of today’s most valuable assets is stored information that is moved, utilized, and shared in a timely and cost-effective manner. Banking records, credit card transactions, and inventory status are typical examples of critical pools of stored data. Despite the strongest efforts, no one can avoid disasters that threaten the life of a company, like hurricanes, floods, fires, extended power outages, or sabotage.
Chapter 2: Fibre Channel at Work 7 MULTIPLATFORM STORAGE A typical company has reorganized, combined departments, decentralized, recentralized, bought other companies, outsourced IT operations, and brought IT operations back in-house. The result is an array of products that IT departments are challenged with integrating and utilizing. They must preserve this investment and still meet the information processing, storage, and distribution requirements of the enterprise.
Fibre Channel: Connection to the Future 8 ENTERPRISE DECISION MAKING Information-savvy organizations make data sharing and distribution a key part of their corporate structure. Unless accurate and timely data, the lifeblood of any organization, is kept flowing, decision making suffers.
Chapter 2: Fibre Channel at Work 9 EFFECTIVE USE OF IT RESOURCES Gigabit Fibre Channel networks take advantage of previously untapped power in storage, servers and workstations.
Fibre Channel: Connection to the Future 10 EXPLOSIVE DATA GROWTH Information storage has become a critically important element for enterprises. New applications and image-oriented data have fueled a storage explosion. Storage capacity is doubling every year with no end in sight. IT managers are demanding better ways of maintaining and increasing storage performance while rapidly adding capacity to meet new demands.
Chapter 2: Fibre Channel at Work 11 MANAGEMENT Many companies are returning to the concept of centralized management of data storage, even within distributed IT architectures. The Gartner Group forecasts that over 70 percent of shared storage in networked environments will be centralized by the year 2000. Storage must be viewed as a system, delivering services and protecting data assets.
Fibre Channel: Connection to the Future 12 REAL-TIME APPLICATIONS The speed and reliability of Fibre Channel is being applied to realtime applications. For example, the B-1 bomber is now using Fibre Channel for improved performance in the mission avionics suite.
Chapter 2: Fibre Channel at Work 13 AUDIO-VIDEO Audio and video are rapidly moving to an all-digital format. The Fibre Channel industry is working with the television and movie industry to provide performance-enhancing Fibre Channel solutions. New profiles are under development to map digital audio and video onto Fibre Channel. High-bandwidth, redundant Fibre Channel storage and workstation links deliver scalable, flexible, and reliable systems. Bottlenecks and single points of failure are eliminated.
14 Fibre Channel: Connection to the Future DIGITAL IMAGING Digital imaging applications, like movie post-production, pre-press, and medical imaging, typically move extremely large files and need to move these files in a very short time. Users of these files do not have the time to wait for slow delivery. The speed and reliability of Fibre Channel deliver increased productivity which, in turn, improves the bottom line.
Chapter 3 FIBRE CHANNEL OVERVIEW After a lengthy review of existing equipment and standards, the Fibre Channel standards group realized that channels and networks should be able to share the same fibre. (Note that “fibre” is used throughout this book as a generic term which can indicate either an optical or a copper cable.) IT systems frequently support two or more interfaces, and sharing a port and media makes sense. This reduces hardware costs and the size of the system, since fewer parts are needed.
Fibre Channel: Connection to the Future 16 The ambitious requirements given the standards group were: l Performance from 133 megabits/second to four gigabits/second l Support for distances of up to 10 km l Small connectors l High-bandwidth utilization with distance insensitivity l Greater connectivity than existing multidrop channels l Broad availability (i.e.
Chapter 3: Fibre Channel Overview 17 STORAGE Fibre Channel is the next storage interface. Fibre Channel has been adopted by the major computer systems and storage manufacturers as the next technology for enterprise storage. It eliminates distance, bandwidth, scalability, and reliability limitations of SCSI. Storage Devices and Systems: Fibre Channel is being provided as a standard disk interface. Industry leading RAID manufacturers are shipping Fibre Channel systems.
Fibre Channel: Connection to the Future 18 In a Fibre Channel network, legacy storage systems are interfaced using a Fibre Channel-to-SCSI bridge. IP is used for serverto-server and client/server communications. JBOD Fibre Channel RAID Shared storage Scalable Switch or Hub High performance Servers SCSI Bridge Data integrity SCSI RAID Fast data access and backup Figure 3.
Chapter 3: Fibre Channel Overview 19 l Movie animation and post-production projects to reduce time to market l Quick-response network for imaging applications Fibre Channel was developed by the computer industry for IT applications. Its development focused on removing the performance barriers of legacy LANs.
Fibre Channel: Connection to the Future 20 all data processing into an integrated environment. This requirement opened the door for open, standards-based solutions. Now, companies are connecting their mainframes with enterprise and department servers for distributed client/server architectures. Distributed computing and parallel processing has resulted in a significant increase in process-to-process communications. At the same time, the data storage requirements have exploded.
Chapter 3: Fibre Channel Overview 21 Fibre Channel is attractive because it offers a standards-based solution. With the emphasis on open systems, end users are shying away from proprietary solutions and vertically integrated, singleprovider solutions. Today, these users are integrating the best the industry has to offer into seamless systems. These new systems are being driven by the technology and marketing forces associated with client/server implementations.
22 Fibre Channel: Connection to the Future For the more demanding applications, Class 4 Fibre Channel provides guaranteed delivery and gigabit bandwidth as well as fractional bandwidth Quality of Service. Fibre Channel’s use in both networks and storage provides a price savings due to economies of scale associated with larger volumes. Users can expect their most cost-effective, highestperformance solutions to be built using Fibre Channel. As shown in Table 3.
Chapter 4 PHYSICAL AND SIGNALING LAYER Fibre Channel is structured with independent layers. The five layers of Fibre Channel, shown in Figure 4.1, define the physical media and transmission rates, encoding scheme, framing protocol and flow control, common services, and the upper-layer protocol interfaces. In this chapter, we describe the Fibre Channel Physical (FC-PH) standard, which consists of the three lower layers, FC-0, FC-1, and FC-2. The upper layers are discussed in Chapter 5. Figure 4.
Fibre Channel: Connection to the Future 24 The three FC-PH levels are defined as follows: l FC-0 – Covers the physical characteristics of the interface and media, including the cables, connectors, drivers (ECL, LEDs, shortwave lasers, and long-wave lasers), transmitters, and receivers. l FC-1 – Defines the 8B/10B encoding/decoding and transmission protocol used to integrate the data with the clock information required by serial transmission techniques.
Chapter 4: Physical and Signaling Layer Media Independent Interface TX Serializer RX Deserializer Fibre Channel ASIC GLM Media Independent Interface TX RX Serializer Deserializer Fibre Channel ASIC GBIC or MIA Nomenclature: Speed-Media-Distance-Transmitter Speed: 400 400 MB/s 200 200 MB/s 100- TW - S - EL 100 100 MB/s 100- M5 - I - SL 50 50 MB/s are the two 25 25 MB/s 12.5 12.5 MB/s most common Media: implementations SM Single Mode Fiber M5 50/125 Multimode Fiber M6 62.
Fibre Channel: Connection to the Future 26 Fibre Channel encoding and control logic. GLMs are primarily used to provide factory configurability but may also be field exchanged or upgraded by users. l Gigabit Interface Converters (GBICs) provide a serial interface to the SERDES function. GBICs can be hot inserted or removed from installed devices. These are particularly useful in multi-port devices such as switches and hubs, where single ports can be reconfigured without affecting other ports.
Chapter 4: Physical and Signaling Layer 27 TRANSMISSION PROTOCOL: FC-1 The superior transmission characteristics of a dc-balanced 8B/10B code scheme are used at FC-1 for clock recovery, byte synchronization, and encode/decode. This well-balanced code developed by IBM allows for low-cost component design and provides good transition density for easier clock recovery. A unique special character, called a comma character, ensures proper byte and word alignment.
Fibre Channel: Connection to the Future 28 FRAMING AND SIGNALING PROTOCOL: FC-2 Reliable communications result from Fibre Channel’s FC-2 Framing and Signaling protocol. FC-2 defines a data transport mechanism that is independent of upper layer protocols. FC-2 is self-configuring and supports point-to-point, arbitrated loop, and switched environments (see Figure 4.5). An N_Port is a port on a node, such as a server, workstation, or peripheral. If a port is connected to a loop, it becomes an NL_Port.
Chapter 4: Physical and Signaling Layer 29 l A set of generic functions that are common across multiple upperlayer protocols. l A built-in protocol to aid in managing the operation of the link, control the Fibre Channel configuration, perform error recovery, and recover link and port status information. l Optional headers that may be used for network routing. l Control information in the header to assist hardware routing. l Process to provide segmentation and reassembly of data.
30 Fibre Channel: Connection to the Future 3. Sequence. Fibre Channel places no limits on the size of transfers between applications (in LANs, the software is sensitive to the maximum frame or packet size that can be transmitted). Frame sizes are transparent to software using Fibre Channel because an upper-layer protocol data structure, called a Sequence, is the unit of transfer (see Chapter 5 for a discussion of the upper layers).
Chapter 4: Physical and Signaling Layer 31 Sequences) to be retransmitted. Each Sequence is uniquely identified by the initiator of the Sequence via the Sequence Identifier (SEQ_ID) field within the frame header. Additionally, each frame within the Sequence is uniquely numbered with a Sequence Count (SEQ_CNT). 4. Exchange. An Exchange is composed of one or more nonconcurrent Sequences for a single operation.
32 Fibre Channel: Connection to the Future
Chapter 5 UPPER LAYERS Fibre Channel is a transport service, moving data reliably and fast, with the scalability to meet virtually any application requirements. The Common Services layer, FC-3, and the Upper Layer Protocol (ULP) mapping, FC-4, enhance the functionality and provide common implementations for interoperability. COMMON SERVICES: FC-3 FC-3 defines functions that are available to all N_Ports. Functions that are currently defined include Name Server and Alias Server.
Fibre Channel: Connection to the Future 34 ULP MAPPING: FC-4 FC-4 defines interoperable implementations of Fibre Channel for standard protocols, audio/video, or applications like real-time computing. Each Fibre Channel implementation is specified in a separate FC-4 document. Fibre Channel allows both network and channel protocols to be concurrently transported over the same physical interface.
Chapter 6 CLASSES OF SERVICE AND TOPOLOGIES Fibre Channel provides five different Classes of Service: l Class 1 – Acknowledged Connection Service l Class 2 – Acknowledged Connectionless Service l Class 3 – Unacknowledged Connectionless Service l Class 4 – Fractional Bandwidth Connection-Oriented Service l Class 6 – Uni-Directional Connection Service Fibre Channel connects nodes using three physical topologies that can have variants.
36 Fibre Channel: Connection to the Future Fibre Channel switches relieve each individual port of the responsibility for station management. Each node only has to manage a simple point-to-point connection between itself and the switch. No complex routing algorithms are used. A node simply does the equivalent of dialing a phone number by entering an identification number for the destination node in the header preceding the payload data. If it is an invalid number, the switch rejects it.
Chapter 6: Topologies Fabric Port Node Port Name Name Topology Not Applicable Point-to-Point N_Port 37 Connection Dedicated bandwidth connection between two N_Ports. N_Port N_Port Point-to-Point Loop (Called a Private Loop) Not Applicable NL_Port Shared bandwidth network. Individual NL_Ports use arbitration scheme to gain control of the loop. After control, NL_Port establishes a point-to-point logical connection with another NL_Port.
38 Fibre Channel: Connection to the Future CLASSES OF SERVICE Fibre Channel meets the requirements of a wide range of applications with switches and nodes supporting one or more Classes of Service. No manual set up is required, since Classes of Service that are supported between switches and nodes are determined during login.
Chapter 6: Topologies 39 is connection setup and tear-down, it is an efficient service for large data exchanges. Fibre Channel Class 1 service provides an acknowledgment of receipt for guaranteed delivery. Class 1 also provides full-bandwidth, guaranteed delivery and bandwidth for applications like image transfer and storage backup and recovery. Some applications use the guaranteed delivery feature to move data reliably and quickly without the overhead of a network protocol stack.
40 Fibre Channel: Connection to the Future Class 2: Acknowledged Connectionless Service Class 2 is a connectionless service, independently switching each frame and providing guaranteed delivery with an acknowledgment of receipt. The path between two interconnected devices is not dedicated. The switch multiplexes traffic from N_Ports and NL_Ports without dedicating a path through the switch. Class 2 credit-based flow control eliminates congestion that is found in many connectionless networks.
Chapter 6: Topologies Server 41 Switch Hub JBOD Workstations Figure 6.4 Class 3 service supports storage and networks Class 4: Fractional Bandwidth Acknowledged ConnectionOriented Service Class 4 is a fractional bandwidth, connection-oriented service. Virtual connections are established with bandwidth reservation for a predictable quality of service (QoS).
42 Fibre Channel: Connection to the Future A node may reserve up to 256 concurrent Class 4 connections. Separate functions of Class 4 are setup of the QoS parameters and the connection itself. When a Class 4 connection is active, the switch paces frames from the source node to the destination node. Pacing is the mechanism used by the switch to regulate available bandwidth per VC. This level of control permits congestion management for a switch and guarantees access to the destination node.
Chapter 6: Topologies 43 preemption. Class 6 is ideal for video broadcast applications and realtime systems that move large quantities of data. Intermix Fibre Channel has an optional mode called Intermix. Intermix allows the reservation of full Fibre Channel bandwidth for a dedicated (Class 1) connection. It also allows connectionless traffic within the switch to share the link during idle Class 1 transmissions.
Fibre Channel: Connection to the Future 44 Whether it is a switch, an active hub, or a loop is irrelevant, because station management issues related to topology are not handled by Fibre Channel ports but are the responsibility of the switch. Selecting the topology for an enterprise or application is determined by the system performance requirements, packaging options, and the user’s growth requirements.
Chapter 6: Topologies 45 the two N_Ports have 100% utilization of the bandwidth. The communication is full-duplex; therefore, a one-gigabit transmit and receive link is delivering two Gbps of dedicated bandwidth. LOOP SHARED-BANDWIDTH TOPOLOGY Fibre Channel loop provides a low-cost solution for allowing multiple devices to share gigabit bandwidth. Nodes share access to the loop but have full use of the gigabit bandwidth when making logical pointto-point connections. Up to 127 nodes may be in a loop.
Fibre Channel: Connection to the Future 46 with a second node or switch port. Only one pair may communicate on the loop at one time. Any class of service may be used; however, most loop applications use Class 3. When control is released, another arbitration takes place. Fairness is provided by guaranteeing equal access to all ports. The loop is self-configuring and may operate with or without a switch present.
Chapter 6: Topologies 47 Fibre Channel storage enclosures have a loop built into the backplane. Each backplane node also has a PBC just like those used in hubs. Disks can be hot swapped in and out of the loop. This hotswap feature is the foundation for Fibre Channel’s 7 x 24 on-line storage management capabilities. SWITCHED SCALED-BANDWIDTH TOPOLOGY Fibre Channel switches are extremely easy to install and use because the Fibre Channel protocol provides self-configuration and selfmanagement.
Fibre Channel: Connection to the Future 48 and a frame switch is not required. Some switches 1 1 are dedicated circuit 2 2 switches and some are 3 3 dedicated frame switches. In the frame-switching mode, the bandwidth is Frame Switch dynamically allocated on a link-by-link basis. Based on adaptive routing within the switch, individual frames between switch N N ports are independently switched. During frame Figure 6.
Chapter 7 FIBRE CHANNEL SERVICES Fibre Channel services provide a set of functions, some of which are required by Fibre Channel protocols and some of which provide optional enhancements to Fibre Channel’s basic protocols. Services are typically found in switched topologies, and their functions are outlined in Table 7.1.
50 Fibre Channel: Connection to the Future Login Server The login server is a logical entity within the switch which receives and responds to the switch login requests. The login server also confirms or assigns the N_Port address of the node that initiates the login. The login server can be a proxy agent to provide a name server with the login attributes of the N_Port.
Chapter 7: Fibre Channel Services 51 Management Server Fibre Channel has adopted the industry-accepted Simple Network Management Protocol (SNMP). Management Information Base (MIB) data is associated with each Fibre Channel node, hub, and fabric. SNMP is used to monitor/modify the MIB data. SNMP may use IP over FC-PH as its transport mechanism (the conventional use of SNMP). A generic solution is provided in which SNMP natively uses FC-CT over FC-PH without the requirement of IP.
52 Fibre Channel: Connection to the Future of the group. The fabric assigns the first available N_Port within this hunt group as the destination for the service class. In Class 1, hunt groups facilitate selection of an available N_Port for a dedicated connection. In Class 2 or 3, hunt groups can be deployed to increase bandwidth, reduce latency, or both. Quality of Service Server The quality of service server is used to enter the desired QoS established for Class 4 service on virtual channels.
Chapter 8 SUMMARY Fibre Channel brings to market reliable, high-performance, easy-touse, low-cost communications required by the new breed of dataand communications-intensive applications. It provides new levels of performance for storage and server networks. Fibre Channel’s highspeed links offer more cost-effective solutions than today’s systems. Fibre Channel enables heterogeneous clusters of storage, servers, and workstations.
54 Fibre Channel: Connection to the Future SPECIAL WORKING INTEREST GROUPS (SWIGS) The Fibre Channel Association (FCA) is organized to support the deployment of Fibre Channel applications. It has organized market segment groups responsible for ensuring that Fibre Channel meets the specific needs of these markets. These FCA organizations, called Special Working Interest Groups or SWIGs, publish profiles that map Fibre Channel into interoperable solutions.
Chapter 8: Summary 55 For information on Fibre Channel loop activities contact: Fibre Channel Loop Community P.O. Box 2161 Saratoga, CA 95070 (408) 867-1385 Web site: http://www.fcloop.org For ANSI documentation information, please contact: Global Engineering 15 Inverness Way East Englewood, CO 80112-5704 Phone: (800) 854-7179 or (303) 792-2181 Fax: (303) 792-2192 For the latest information on Fibre Channel standards activities: Web site: http://www.dpt.
56 Fibre Channel: Connection to the Future
GLOSSARY ANSI American National Standards Institute, the coordinating organization for voluntary standards in the United States. Arbitration The process of selecting one respondent from a collection of several candidates that request service concurrently. ATM Asynchronous Transfer Mode. A type of packet switching that transmits fixed-length units of data. Broadcast Sending a transmission to all N_Ports on a fabric.
58 Fibre Channel: Connection to the Future ESCON Enterprise Systems Connection Exchange A term that refers to one of the Fibre Channel “building blocks,” composed of one or more nonconcurrent Sequences. Fabric The facility that connects multiple N_Ports. FC-PH Fibre Channel Physical standard, consisting of the three lower layers, FC-0, FC-1, and FC-2. FC-0 Lowest layer of the FC-PH standard, covering the physical characteristics of the interface and media.
Glossary 59 HiPPI High Performance Parallel Interface, an 800or 1600-Mbit/second interface to supercomputer networks (formerly known as high-speed channel). Developed by ANSI. Hunt Group A set of associated N_Ports attached to a single node and assigned a special identifier that allows any frames containing this identifier to be routed to any available N_Port within the set. Hub A Fibre Channel device that connects nodes into a logical loop by using an apparent star topology.
60 Fibre Channel: Connection to the Future N_Port “Node” port, a Fibre-Channel-defined hardware entity at the end of a link. NL_Port “Node” port connected to an Arbitrated Loop. Network An aggregation of interconnected nodes, including workstations, file servers, and/or peripherals, with its own protocol that supports interaction. Operation A term, defined in FC-2, that refers to one of the Fibre Channel “building blocks” composed of one or more, possibly concurrent, Exchanges.
Glossary 61 SONET Synchronous Optical Network. A standard for optical network elements. Basic level is 51.840 megabit/second (OC-1); higher levels are n times the basic rate (OC-n). Star The physical configuration used with hubs in which each user is connected by communication links radiating out of a central hub that handles all communications. Striping A method for achieving higher bandwidth using multiple N_Ports in parallel to transmit a single information unit across multiple levels.
62 Fibre Channel: Connection to the Future
