An Oracle Deployment Guide September 2011 Deploying a Converged Network Using Oracle CNAs and a Brocade FCoE Switch
Converged Storage Networking Installation Guide Disclaimer Information furnished in this manual is believed to be accurate and reliable. However, Oracle assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. Oracle reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only.
Converged Storage Networking Installation Guide TABLE OF CONTENTS Documentation Conventions................................................................................ 4 Introduction ......................................................................................................... 5 Intended Audience .............................................................................................. 6 Planning ............................................................................................
Converged Storage Networking Installation Guide Brocade 8000 DCB/FCoE Switch .................................................................. 22 Appendix C: Oracle Converged Network Adapter Overview ............................. 23 Kit Contents ................................................................................................... 23 Converged Network Adapter Features and Specifications ............................. 23 Operating System and Technology Requirements .............................
Converged Storage Networking Installation Guide
Documentation Conventions This guide uses the following documentation conventions: CAUTION! indicates the presence of a hazard that has the potential of causing damage to data or equipment. WARNING!! indicates the presence of a hazard that has the potential of causing personal injury. Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this guide, and links to Web sites are shown in underlined blue.
Introduction In the Unified Data Center Fabric Primer: FCoE and Data Center Bridging publication, a converged network is defined as a unified data center fabric: "A unified data center fabric is a networking fabric that combines traditional LAN and storage area network (SAN) traffic on the same physical network with the aim of reducing architecture complexity and enhancing data flow and access.
Intended Audience This guide is intended for system engineers and planners who want to provide converged networking products, solutions, and services to their customers. It is also intended for network planners and administrators who are implementing a converged network for their company. This guide describes how to install an Oracle converged network in preparation for production deployment.
Planning Selecting a Test Architecture When planning to install a converged network, it is important to choose both Fibre Channel and traditional Ethernet-based traffic flows. Combining a test SAN infrastructure and a test LAN infrastructure is often the easiest and most available option for a pilot project. Alternatively, a critical business application test system can closely simulate a production environment.
Converged Network Adapter > FCoE switch > FCoE storage: This implementation is not common because most data centers use Fibre Channel SAN storage. As more storage vendors deliver FCoE storage, more pilot projects will support direct Ethernet connection from the FCoE switch to FCoE-capable storage controllers. In all cases, Ethernet LAN and iSCSI storage connect directly to Ethernet ports on the DCB or FCoE switch. The reference architecture, shown in Figure 2, uses Fibre Channel SAN storage.
Equipment Details Table 1 lists the referenced architecture equipment. Two Sun servers from Oracle were installed with Oracle Solaris 10 9/10. All servers used Oracle’s Sun Storage 10 GbE FCoE PCIe Converged Network Adapter. One Sun ZFS Storage Appliance from Oracle and one Sun Storage 6000 Series Storage Array from Oracle provided the FC SAN storage.
Determine the Configuration Sun Storage 10 GbE FCoE PCIe Converged Network Adapters are supported on multiple hardware platforms and operating systems. Generally, the following specifications apply, but you should always check the Oracle Web site for current information.
For detailed installation instructions, see the Sun Storage 10GbE FCoE PCIe Converged Network Adapter Installation Guide. Install the Adapter Drivers Oracle Solaris, Microsoft Windows and Linux To install the FCoE and Ethernet drivers for Solaris, Windows and Linux: 1. Navigate to http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SunOEM.aspx?oemid=124. 2.
To configure a Brocade 8000 FCoE Switch: 1. Login to switch and enter FCoE command shell. BR8K-21 login: admin Password: ****** BR8K-21:admin> cmsh Use cmsh to enter FCoE command mode 2. Enter the configuration mode. BR8K-21-cmsh# config t Use configure terminal to enter config mode 3. Create and configure the VLAN interface for FCoE.
7. Define Priority Group mapping for groups 0 – 7. BR8K-21-cmsh(conf-ceemap)# priority-table 2 2 2 1 2 2 2 2 NOTE: COS: 0 1 2 3 4 5 6 7 NOTE: COS 3 is the default for FCoE. Don’t try to change it, because the rest of the world is expecting it to be COS 3. 8. Configure LLDP/DCBX for FCoE.
Configure Storage Depending on your storage, you may connect directly to the FCoE switch through FCoE with native FCoE storage, or through other methods (Fibre Channel, iSCSI, NFS, CIFS). Consult your storage array documentation for instructions on how to enable your array and assign disk storage LUNs. The storage installed for this report was two 4 Gb Fibre Channel storage systems (a Sun ZFS Storage Appliance and a Sun Storage 6180 Array).
15
FIGURE 3 ZFS STORAGE APPLIANCE LUN ASSIGNMENT DISPLAY 2. Reboot the servers to discover the assigned LUNs. On the SPARC Enterprise M5000 Server the following command is an intermediate check to ensure that the hardware is seeing the LUNs in a pre-boot environment. The following information is the LUN portion obtained from the SPARC Enterprise M5000 Server NVRAM display. SPARC Enterprise M5000 Server, using Domain console Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
command. Two approaches can be used. Native Oracle Solaris commands can be entered or the SANSurfer CLI can also be used. The Oracle Solaris commands would be performed once on the Sun Fire server and once on the SPARC Enterprise M5000 Server. SCX4450-BRM-03# fcinfo hba-port HBA Port WWN: 210000c0dd1460c9 OS Device Name: /dev/cfg/c2 Manufacturer: QLogic Corp. Model: 375-3681-01 Firmware Version: 05.04.03 FCode/BIOS Version: BIOS: 2.14; fcode: 3.10; EFI: 3.
Alternatively, the following SANsurfer CLI commands can be used. Scanning QLogic FC HBA(s) and device(s), please wait... / SANsurfer FC/CNA HBA CLI v1.7.3 Build 37 Main Menu 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: General Information HBA Information HBA Parameters Target/LUN List Boot Device Utilities Beacon Diagnostics Statistics Virtual FCoE Help Exit Enter Selection: 4 SANsurfer FC/CNA HBA CLI v1.7.
2: 3: Vendor Product ID Product Rev Serial Number Node Name Port Name Port ID : : : : : : : SUN SUN_6180 0760 SQ02400384 20-06-00-80-E5-18-48-B0 20-16-00-80-E5-18-48-B0 01-01-00 Disk (Online) Vendor Product ID Product Rev Serial Number Node Name Port Name Port ID : : : : : : : SUN SUN_6180 0760 SQ94600739 20-06-00-80-E5-18-48-B0 20-17-00-80-E5-18-48-B0 01-02-00 Disk (Online) Vendor Product ID Product Rev Serial Number Node Name Port Name Port ID : : : : : : : SUN Sun Storage 7410 1.
/devices/pci@0,0/pci8086,3605@2/pci8086,3500@0/pci8086,3510@0/pci1077,183@0 ,3/fp@0,0:devctl NOT CONNECTED SCX4450-BRM-03# SCX4450-BRM-03# ./prtpicl | grep 1077 pci1077,184 (obp-device, pci1077,184 (obp-device, pci1077,183 (obp-device, pci1077,183 (obp-device, SCX4450-BRM-03# e6000001a8) e6000001d0) e6000001f8) e600000256) Conclusion This document summarizes how to plan for, manage, and implement an FCoE pilot. This step-by-step guide provides guidance on the implementation of a unified fabric.
Appendix A: FCoE and Enhanced Ethernet Related Materials The following links provide more detailed information, and connect to the IEEE documents that define the Enhanced Ethernet functions: P802.1Qbb (Draft 2.3): Priority-based Flow Control http://www.ieee802.org/1/files/private/bb-drafts/d2/802-1bb-d2-3.pdf P802.1Qaz(Draft 2.5): Enhanced Transmission Selection http://www.ieee802.org/1/files/private/az-drafts/d2/802-1az-d2-5.pdf P802.1Qaz (Draft 2.
Appendix B: Hardware and Software Sun Storage 10 GbE FCoE PCIe Converged Network Adapter The Sun Storage 10 GbE FCoE PCIe Converged Network Adapter is a single-chip, fully-offloaded FCoE initiator, operating in both virtual and non-virtual environments, running over an Enhanced Ethernet fabric. This converged network adapter initiator boosts system performance with 10Gbps speed and full hardware offload for FCoE protocol processing.
Appendix C: Converged Network Adapter Overview This appendix provides a basic overview of the Sun Storage 10GbE FCoE PCIe Converged Network Adapter, which uses QLogic technology. This appendix also describes the various operating systems, storage, and infrastructure configurations that support the Converged Network Adapter, and lists the adapter’s environmental requirements. Detailed information regarding the Sun Storage 10GbE PCIe Converged Network Adapter can be found at: http://www.oracle.
CAUTION!—Do not replace any of the SFP+s connectors. Doing so will void the warranty or serviceability of the Converged Network Adapter. Table 2 lists the features and specifications for this adapter. TABLE 2 CONVERGED NETWORK ADAPTER FEATURES AND SPECIFICATIONS FEATURE DESCRIPTION Form factor Standard low-profile form factor Connector types SFP+ SR optics SFP+ direct-attach twin-axial copper PCIe specification compliance PCI Express Card Version 2.
FEATURE DESCRIPTION Ethernet and NIC support Standard Ethernet and Enhanced Ethernet support for: IEEE 802.1Q VLAN IEEE 802.1p IEEE 802.3x IEEE 802.1Qbb IEEE 802.
Switch Support Software Support Boot Support Host Platform Support The adapter is supported by the platforms listed in Table 4. For up-to-date information, see your Oracle server release notes and Web pages.
Array Support The Sun Storage 10GbE FCoE PCIe Converged Network Adapter supports connecting to, using a supported switch, the following arrays: Sun StorageTek 2540 Sun StorageTek 6140 Sun StorageTek 6180 Sun StorageTek 6540 Sun StorageTek 6580/6780 with 8Gb Fibre Channel host interface cards Disk System Support The Sun Storage 10GbE FCoE PCIe Converged Network Adapter supports connecting to, using a supported switch, the following disk system storage: Sun StorageTek 9980/9985/9985V System Su
Software Support The Con Sun Storage 10GbE FCoE PCIe Converged Network Adapter supports the software utilities and applications listed in Table 5 and Table 6.
SPECIFICATION OPERATING NON-OPERATING Vibration 0.20G in all axes, 5-500 Hz sine 1.0G in all axes, 5–500 Hz sine Shock 5G, 11 ms half-sine 30G 11 ms half-sine Appendix D: Data Center Bridging Technology The following descriptions of Enhanced Ethernet were taken from Ethernet: The Converged Network Ethernet Alliance Demonstration which was presented at the Super Computing 2009 (SC09) conference and published by the Ethernet Alliance in November, 2009.
ETS and DCBX simplify the management of DCB nodes significantly, especially when deployed end-to-end in a converged data center. The DCBX protocol uses Link Layer Discovery Protocol (LLDP) defined by IEEE 802.1AB to exchange and discover DCB capabilities. Priority Flow Control A fundamental requirement for a high performance storage network is guaranteed data delivery. This requirement must be satisfied to transport critical storage data on a converged Ethernet network with minimum latency.
The Internet Small Computer Systems Interface (iSCSI) is a SCSI mass storage transport that operates between the Transport Control Protocol (TCP) and the SCSI Protocol Layers. The iSCSI protocol is defined in RFC 3720 [iSCSI], which was finalized by the Internet Engineering Task Force (IETF) in April 2004. A TCP/IP connection ties the iSCSI initiator and target session components together. Network portals identified by their IP address and TCP port numbers define the endpoints of a connection.
Appendix E: References Unified Data Center Fabric Primer: FCoE and Data Center Bridging, Martin, D. (2010). SearchNetworking.com, retrieved from http://searchnetworking.techtarget.com/tip/0,289483,sid7_gci1378613,00.html Ethernet: The Converged Network Ethernet Alliance Demonstration at SC'09, Ethernet Alliance. (2009), retrieved from http://www.ethernetalliance.org/files/static_page_files/281AD8C4-1D09-3519-AD 7AD835AD525E36/SC09%20white%20paper.
Deploying a Converged Network Using Oracle Copyright © 2011, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the CNAs and a Brocade FCoE Switch contents hereof are subject to change without notice.
