- Cisco IP/Multiprotocol Label Switching Specification Sheet

ManualsBrandsCisco Systems ManualsSwitchOL-12518-01

CHAPTER

4-1

Data Center High Availability Clusters Design Guide

OL-12518-01

FCIP over IP/MPLS Core

This chapter discusses the transport of Fibre Channel over IP (FCIP) over IP/Multiprotocol Label

Switching (MPLS) networks and addresses the network requirements from a service provider (SP)

perspective. This chapter also describes service architectures and storage service offerings using FCIP

as a primary storage transport mechanism.

Overview

Storage extension solutions offer connectivity between disparate storage “islands,” and promote

transport solutions that are specifically geared towards carrying storage area network (SAN) protocols

over WAN and MAN networks. This emerging demand is providing a new opportunity for carriers. SPs

can now deliver profitable SAN extension services over their existing optical (Synchronous Optical

Network [SONET]/Synchronous Digital Hierarchy [SDH] and Dense Wavelength Division Multiplexing

[DWDM]) or IP infrastructure. DWDM networks are ideal for high-bandwidth, highly resilient networks

and are typically deployed within metro areas. Transporting storage traffic over the existing

SONET/SDH infrastructure allows SPs to maximize the use of their existing SONET/SDH ring

deployments. Some applications do not mandate stringent requirements offered by optical networks.

These applications can be easily transported over IP networks using FCIP interfaces. The obvious

advantage of transporting storage over IP is the ubiquitous nature of IP.

Disk replication is the primary type of application that runs over an extended SAN network for business

continuance or disaster recovery. The two main types of disk replication are array-based (provided by

EMC

SRDF, Hitachi True Copy, IBM PPRC XD, or HP DRM, and host-based (for example, Veritas

Volume Replicator). Both disk replication types run in synchronous and asynchronous modes. In

synchronous mode, an acknowledgement of a host-disk write is not sent until a copy of the data to the

remote array is completed. In asynchronous mode, host-disk writes are acknowledged before the copy

of the data to the remote array is completed.

Applications that use synchronous replication are highly sensitive to response delays and might not work

with slow-speed or high-latency links. It is important to consider the network requirements carefully

when deploying FCIP in a synchronous implementation. Asynchronous deployments of FCIP are

recommended in networks with latency or congestion issues. With FCIP, Fibre Channel SAN can be

extended anywhere an IP network exists and the required bandwidth is available. FCIP can be extended

over metro, campus, or intercontinental distances using MPLS networks. FCIP may be an ideal choice

for intercontinental and coast-to-coast extension of SAN.

Summary of content (30 pages)

PAGE 1
C H A P T E R 4 FCIP over IP/MPLS Core This chapter discusses the transport of Fibre Channel over IP (FCIP) over IP/Multiprotocol Label Switching (MPLS) networks and addresses the network requirements from a service provider (SP) perspective. This chapter also describes service architectures and storage service offerings using FCIP as a primary storage transport mechanism.
PAGE 2
Chapter 4 FCIP over IP/MPLS Core Typical Customer Requirements Typical Customer Requirements Small-to-medium businesses (SMBs) represent about 90 percent of all companies in the United States. These companies typically employ a few hundred employees and are highly focused on their core services or products. They usually lack IT expertise and manpower to develop, deploy, and maintain LAN, WAN, and SAN infrastructures.
PAGE 3
Chapter 4 FCIP over IP/MPLS Core Typical Customer Requirements The requirements are as follows: • FCIP transport over an optimized IP/MPLS network • Some type of compression mechanism (software or hardware) • Security mechanism (IPSec, encryption, and VPN networks) • End-to-end management of FCIP traffic Compression The primary objective of compression is to reduce the amount of overall traffic on a particular WAN link.
PAGE 4
Chapter 4 FCIP over IP/MPLS Core Typical Customer Requirements Hardware-based compression is available with SAN-OS version 2.0 and with new hardware (MDS 9216i/MLS14/2). Compression is applied per FCIP interface (tunnel) with a variety of modes available. Beginning with SAN-OS 2.0, three compression modes are configurable with additional support for the MPS-14/2 module. Compression Modes and Rate In SAN-OS 1.
PAGE 5
Chapter 4 FCIP over IP/MPLS Core Typical Customer Requirements Figure 4-2 Cisco Compression Solutions 200 MB/sec 150 100 SA-VAM 2001 SA-VAM2 2003 IPS 2003 MPS 2004 132422 50 The following performance data applies to Figure 4-2: • VAM—9.9–12 MB/sec –10.9 MB/sec average • VAM2—19.7–25.4 MB/sec – 19 MB/sec average • IPS—18.6–38.5 MB/sec – 24.6 MB/sec average • MPS—136–192 MB/sec – 186.
PAGE 6
Chapter 4 FCIP over IP/MPLS Core Typical Customer Requirements • SPs providing VPN service to transport FCIP traffic to provide additional security • Using an MPLS extranet for application-specific security Cisco Encryption Solutions For selecting compression solutions for FCIP SAN extension, a user needs to determine the requirements for the encryption solution.
PAGE 7
Chapter 4 FCIP over IP/MPLS Core Using FCIP Tape Acceleration Write Acceleration Write Acceleration is a configurable feature introduced in SAN-OS 1.3 that enhances FCIP SAN extension with the IP Storage Services Module. Write Acceleration is a SCSI protocol spoofing mechanism that improves application performance by reducing the overall service time for SCSI write input/output (I/O) operations and replicated write I/Os over distance.
PAGE 8
Chapter 4 FCIP over IP/MPLS Core Using FCIP Tape Acceleration FCIP Tape Acceleration maintains data integrity in the event of a variety of error conditions. Link errors and resets are handled through Fibre Channel-tape Ethernet LAN services (ELS) recovery mechanisms. Should the remote tape unit signal an error for an I/O that the status has already been returned to “good”, a Deferred Error is signaled to the tape backup application.
PAGE 9
Chapter 4 FCIP over IP/MPLS Core Using FCIP Tape Acceleration • TCP window size • TCP maximum bandwidth • TCP minimum available bandwidth • Round Trip Time (RTT) TCP Window Size TCP uses a sliding window to control the flow of data from end to end. The TCP maximum window size (MWS) is the maximum amount of data the sender allows to be outstanding without acknowledgment at one time. The minimum MWS is 14 KB; the maximum is 32 MB.
PAGE 10
Chapter 4 FCIP over IP/MPLS Core Using FCIP Tape Acceleration The min-available-bw parameter provides the necessary control. Even in the presence of drops, the sender tries aggressively to reach the value configured for this parameter. Even if the congestion window is decreased because of drops, it is increased again on every send so that it is not less than the configured minimum bandwidth. Round Trip Time RTT is a measure of the latency or delay back and forth over the FCIP tunnel.
PAGE 11
Chapter 4 FCIP over IP/MPLS Core Using FCIP Tape Acceleration • Simplified management—Provides a unified management environment independent of whether servers use FCIP to connect to the storage network. • Comprehensive security—Combines the ubiquitous IP security infrastructure with Cisco virtual SANs (VSANs), hardware-based zoning, and hardware-based access control lists (ACLs) to provide robust security.
PAGE 12
Chapter 4 FCIP over IP/MPLS Core Using FCIP Tape Acceleration Multiprotocol Services Module The Cisco MDS 9000 Family 14/2-port Multiprotocol Services Module delivers the intelligence and advanced features required to make multilayer SANs a reality, by integrating in a single module the functions offered by the Cisco 16-Port Fibre Channel Switching Module and the Cisco IP Storage Services Module.
PAGE 13
Chapter 4 FCIP over IP/MPLS Core QoS Requirements in FCIP QoS Requirements in FCIP Currently, most of the FCIP links are dedicated for pure Fibre Channel traffic. But in most cases if QoS is enabled, most of the SAN applications can be transported across the same traffic engineered infrastructure shared with traditional IP traffic. Because there are several QoS models, make sure the right Differentiated Services Code Point (DSCP) is applied to get the assumed results.
PAGE 14
Chapter 4 FCIP over IP/MPLS Core Applications Figure 4-4 Using the GUI to Apply QoS Applications Disaster recovery and business continuance plans drive the need for solutions that protect critical business information and provide continuous access to important data in case of disaster. Disaster recovery applications are intended to replicate data to a remote backup location. The backup site can be located in the same metro area, such as New York and New Jersey, or at transcontinental distances.
PAGE 15
Chapter 4 FCIP over IP/MPLS Core Service Offerings over FCIP Service Offerings over FCIP Figure 4-5 shows a typical service architecture for deploying FCIP over IP/MPLS. Figure 4-5 FCIP over IP/MPLS Architecture Remote Data Center Local Data Center SAN SAN IP/MPLS Gateway FCIP 132439 Gateway FCIP The FCIP gateway is the key component of the overall architecture.
PAGE 16
Chapter 4 FCIP over IP/MPLS Core Service Offerings over FCIP Figure 4-6 FCIP over SP IP/MPLS Core for Disaster Recovery Solutions Remote Sites FC Fabrics Corporate HQ FC IP/MPLS Network FCIP FCIP FC FC Fabrics Cisco 7206 VXR with FCIP PA FC Fabric FC Bandwidth options: DS1 – OC3 TDM Facilities FastE and GigE Facilities FC Fabrics 132427 Cisco MDS 9216 multi-layer edge switch Cisco 7206 VXR with FCIP PA Service Offering Scenario B—Connecting Multiple Sites In certain cases, customers prefer
PAGE 17
Chapter 4 FCIP over IP/MPLS Core Service Offerings over FCIP Figure 4-7 FCIP Connectivity between Second Site and Third Site Third Site FCIP FC Fabric FC SP MPLS Network Optical FC Fabric Secondary Site Cisco MDS 9216 Multi-layer Edge Switch 119870 Cisco MDS 9216 Multi-layer Edge Switch Primary Site Service Offering Scenario C—Host-based Mirroring IP/MPLS networks can be used to implement host-based mirroring based on iSCSI. A typical network setup is shown in Figure 4-8.
PAGE 18
Chapter 4 FCIP over IP/MPLS Core MPLS VPN Core Table 4-1 Possible Service Provider Offerings Storage Service Target Customers Storage Platform Transport Options Protocol CPE • Synchronous • Require no data loss • CLARiiON • Ethernet • DWDM • DWDM • Data replication • High volume/rev. impact • Symmetrix • • SONET • ONS 15530 • (Real-time ext.
PAGE 19
Chapter 4 FCIP over IP/MPLS Core MPLS VPN Core MPLS VPN for Storage Architecture Figure 4-9 SAN 9216 1 14+2 VRF SAN SAN 9216 1 14+2 VRF SAN spse 7609-45 spse 7606-60 1/1 1/2 GE GE 2/1 2/2 1/1 2/4 2/3 GE GE 1/1 1/2 2/3 1/2 GE 2/2 2/1 GSR12410 GSR12410 spse 12408-51 spse 7609-46 2/1 2/4 GE GE 2/2 2/1 2/4 VRF SAN 2/3 SAN 9216 1 14+2 SAN 9216 1 14+2 132426 VRF SAN Multiple storage customers can be supported on the same MPLS network.
PAGE 20
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results When a PE router forwards a packet received from a CE router across the provider network, it labels the packet with the label learned from the destination PE router. When the destination PE router receives the labeled packet, it pops the label and uses it to direct the packet to the correct CE router. Label forwarding across the provider backbone is based on either dynamic label switching or traffic engineered paths.
PAGE 21
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results Figure 4-10 Test Lab Setup and Topology SAN 9216 1 14+2 VRF SAN 1/1 1/2 GE GE 2/1 2/2 Primary path TE 1/1 1/2 GE 2/2 2/1 GSR12410 2/4 spse 7609-46 2/4 GSR12410 2/3 2/4 GE GE 2/2 2/1 2/3 GE GE 1/1 1/2 2/1 spse 7609-45 spse 12408-51 Secondary path TE VRF SAN 2/3 SAN 9216 1 14+2 132429 spse 7606-60 VPN VRF—Specific Configurations MP BGP Configuration—PE1 The following is a sample configuration for PE1 in Figure 4-10.
PAGE 22
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results Gigabit Ethernet Interface Configuration—PE1 The following sample configuration is for the Gigabit Ethernet interface for PE1 in Figure 4-10. interface GigabitEthernet0/0/0 ip vrf forwarding storage ip address 11.11.11.2 255.255.255.
PAGE 23
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results VRF Configuration—PE2 The following are the VRF definitions on the PE2(7500-106) router: ip vrf storage rd 105:106 route-target export 105:106 route-target import 105:106 This test assumes the Cisco MDS 9216/9216i as the CPE. MPLS VPN VRFs allow the SPs to leverage a common backbone to offer shared transport services. To facilitate these services, the provider gets the added security mechanisms of VRFs.
PAGE 24
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results Figure 4-11 Cisco MDS 9216i Connection to GSR MPLS Core GSR 12410 Provider Core GSR 12410 Provider Core GSR 12410 Provider Core GSR 12410 PE 11.11.11.1 132430 / S et/PO thern DS1 bit E Giga rial/DS3/ Se 10.10.10.1 Giga bit E Seria thernet/P l/DS3 O /DS1 S/ GSR 12410 PE Configuring TCP Parameters on CPE (Cisco MDS 9216) A simple ping command from the Cisco MDS 9000 CLI, provides the RTT between the two IP addresses.
PAGE 25
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results Where sustained Gigabit throughput is not required (for example, over an OC-12 or slower WAN link), an MTU of 1500 bytes is adequate. Otherwise, use jumbo frames if possible and set the MTU on the Gigabit Ethernet interface to 2300. Also, set the MTU size on the CPE to 2300 bytes. You also need to consider VPN and MPLS headers when configuring MTU size across the path.
PAGE 26
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results Figure 4-14 shows the average latency with packet size of 1024. Figure 4-14 Average Latency—Packet size 1024 Average Latency (us) Packet size 1024 1000 800 600 400 200 0 10 20 30 40 50 60 70 80 90 100 132433 Latency (us) 1200 Utilization Scenario 3—Cisco MDS 9216i Connection to Cisco 7500 (PE)/GSR (P) In this scenario, the Cisco 7500 is used as a PE and the FCIP traffic is passed over the GSR (P) routers (see Figure 4-15).
PAGE 27
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results The test results reveal that the maximum traffic that can be transported across the Cisco 7500 as PE is around 35 percent, as shown in Figure 4-16. Traffic Load versus Utilization Test Results Traffic Load vs Utilization 40 35 30 25 20 15 10 5 0 10 20 30 40 50 60 70 80 90 100 132435 Utilization (%) Figure 4-16 Traffic Load Scenario 4—Impact of Failover in the Core No convergence testing was done with this FCIP testing.
PAGE 28
Chapter 4 FCIP over IP/MPLS Core Testing Scenarios and Results Scenario 6—Impact of Compression on CPE (Cisco 9216i) Performance You can compress the data stream to reduce WAN link utilization. Some WAN deployments may require compression to obtain adequate throughput. For example, with a 2 to 1 compression ratio, you can obtain 90 Mb/sec of storage data throughput over a 45-Mb/sec DS3 WAN connection. You may also need encryption to protect the confidentiality and integrity of the storage data stream.
PAGE 29
Chapter 4 FCIP over IP/MPLS Core Application Requirements Figure 4-18 Compression Ratio Comparisons 12.0 SA-VAM SA-VAM2 10.0 IPS high-throughput IPS high-compression Compensation Ratio MPS-Mode 1 8.0 MPS-Mode 2 MPS-Mode 3 6.0 4.0 132437 2.0 0.0 ic al l t t .tx .tx ik ul 9 e2 yo as tm .h cp .c ds l fie m am gr y.x ed n n ke t t ls p .ls er .tx .
PAGE 30
Chapter 4 FCIP over IP/MPLS Core Conclusion Table 4-2 Application Requirements Disk mirroring Varies depending on < 1–5 ms for storage array. synchronous. Typically maximum 50 Asynchronous MB per storage array. replication tolerates higher latency (100 ms). File access OS dependent. Synchronous or asynchronous. Synchronous applications are very sensitive to delay. Asynchronous are less sensitive to delay. Depends on the OS and application above it.