Installing and Administering HyperFabric HP-UX 11i v3 Edition 13 Manufacturing Part Number: B6257-90055 February 2007 Printed in U.S.A. © Copyright 2007 Hewlett-Packard Company.
Legal Notices Copyright 2007 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license required from HP for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents 1. Overview Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Notice of non support for Oracle 10g RAC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HyperFabric Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HyperFabric Adapters . . . . . . . . . . . . . . .
Contents Configuration Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Information You Need . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration Information Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performing the Configuration . . . . . . . . . . . . . . . . . . . . .
Contents Determining Whether a Switch is Faulty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Replacing a HyperFabric Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Replacing a HyperFabric Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Safety Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents 6
Figures Figure 2-1. TCP/UDP/IP Point-To-Point Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Figure 2-2. TCP/UDP/IP Basic Switched Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 2-3. TCP/UDP/IP High Availability Switched Configuration . . . . . . . . . . . . . . . . . . . . . . . . 29 Figure 2-4. HMP Point-To-Point Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 2-5.
Figures 8
Tables Table 2-1. HF2 Speed and Latency w/ TCP/UDP/IP Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Table 2-2. HF2 Speed and Latency w/ HMP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Table 3-1. Important OLAR Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Table 6-1. LED Names (by Adapter) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tables 10
Printing History The manual printing date and part number indicate its current edition. The printing date will change when a new edition is printed. Minor changes may be made at reprint without changing the printing date. The manual part number will change when extensive changes are made. Manual updates may be issued between editions to correct errors or document product changes. To ensure that you receive the updated or new editions, you should subscribe to the appropriate product support service.
1 Overview This chapter contains the following sections that give general information about HyperFabric: Chapter 1 • “Overview” on page 15 • “HyperFabric Products” on page 16 13
Overview • 14 “HyperFabric Concepts” on page 18 Chapter 1
Overview Overview Overview HyperFabric is a HP high-speed, packet-based interconnect for node-to-node communications. HyperFabric provides higher speed, lower network latency and less CPU usage than other industry standard protocols (e.g. Fibre Channel and Gigabit Ethernet). Instead of using a traditional bus based technology, HyperFabric is built around switched fabric architecture, providing the bandwidth necessary for high speed data transfer.
Overview HyperFabric Products HyperFabric Products HyperFabric hardware consists of host-based interface adapter cards, interconnect cables and optional switches. HyperFabric software resides in Application Specific Integrated Circuits (ASICs) and firmware on the adapter cards and includes user space components and HP-UX drivers. Currently, fibre based HyperFabric hardware is available. In addition, a hybrid switch that has 8 fibre ports is available to support HF2 clusters.
Overview HyperFabric Products NOTE In this manual, the terms HyperFabric2 switch or HF2 switch refer to the functional switch (the A6384A switch chassis with one of the switch modules installed). IMPORTANT HF2 adapters and switches are not supported by software versions earlier than those listed in “HyperFabric Adapters” on page 16 and “Switches and Switch Modules” on page 16.
Overview HyperFabric Concepts HyperFabric Concepts Some basic HyperFabric concepts and terms are briefly described below. The fabric is the physical configuration that consists of all of the HyperFabric adapters, the HyperFabric switches (if any are used) and the HyperFabric cables connecting them. The network software controls data transfer over the fabric. A HyperFabric configuration contains two or more HP 9000 systems and optional HyperFabric switches. Each HP 9000 acts as a node in the configuration.
2 Planning the Fabric This chapter contains the following sections offering general guidelines and protocol specific considerations for planning HyperFabric clusters that will run TCP/UDP/IP or HMP applications.
Planning the Fabric 20 • “HyperFabric Functionality for TCP/UDP/IP and HMP Applications” on page 22 • “TCP / UDP / IP” on page 23 • “Hyper Messaging Protocol (HMP)” on page 30 Chapter 2
Planning the Fabric Preliminary Considerations Preliminary Considerations Before beginning to physically assemble a fabric, follow the steps below to be sure all appropriate issues have been considered: Step 1. Read Chapter 1, “Overview,” on page 13 to get a basic understanding of HyperFabric and its components. Step 2. Read this chapter, Planning the Fabric, to gain an understanding of protocol specific configuration guidelines for TCP/UDP/IP and HMP applications. Step 3.
Planning the Fabric HyperFabric Functionality for TCP/UDP/IP and HMP Applications HyperFabric Functionality for TCP/UDP/IP and HMP Applications The following sections in this chapter define HyperFabric features, parameters, and supported configurations for TCP/UDP/IP applications and Hyper Messaging Protocol (HMP) applications. There are distinct differences in supported hardware, available features and performance, depending on which protocol is used by applications running on the HyperFabric.
Planning the Fabric TCP / UDP / IP TCP / UDP / IP TCP/UDP/IP applications are supported on all HF2 (fibre) hardware. Although all HyperFabric adapter cards support HMP applications as well, our focus in this section will be on TCP/UDP/IP HyperFabric applications. Application Availability All applications that use the TCP/UDP/IP stack are supported such as Oracle 9i.
Planning the Fabric TCP / UDP / IP To create a highly available HyperFabric cluster, there cannot be any single point of failure. Once the HP 9000 nodes and the HyperFabric hardware have been configured with no single point of failure, ServiceGuard and EMS can be configured to monitor and fail-over nodes and services using ServiceGuard packages.
Planning the Fabric TCP / UDP / IP For more detailed information on HyperFabric diagnostics see “Running Diagnostics” on page 115 on page 149. Configuration Parameters This section details, in general, the maximum limits for TCP/UDP/IP HyperFabric configurations. There are numerous variables that can impact the performance of any particular HyperFabric configuration. See the “TCP/UDP/IP Supported Configurations” section for guidance on specific HyperFabric configurations for TCP/UDP/IP applications.
Planning the Fabric TCP / UDP / IP • Table 2-1 Speed and Latency: HF2 Speed and Latency w/ TCP/UDP/IP Applications Server Class rp7420 Maximum Speed 2 + 2 Gbps full duplex per link Latency < 42 microsec For a list of HF2 hardware that supports TCP/UDP/IP applications (HP-UXN 11i v3), see HyperFabric Release Notes (B6257-90056). TCP/UDP/IP Supported Configurations Multiple TCP/UDP/IP HyperFabric configurations are supported to match the cost, scaling and performance requirements of each installation.
Planning the Fabric TCP / UDP / IP Figure 2-1 Chapter 2 TCP/UDP/IP Point-To-Point Configurations 27
Planning the Fabric TCP / UDP / IP Switched This configuration offers the same benefits as the point to point configurations illustrated in figure 1, but it has the added advantage of greater connectivity (see Figure 2-2).
Planning the Fabric TCP / UDP / IP High Availability Switched This configuration has no single point of failure. The HyperFabric driver provides end to end HA. If any HyperFabric resource in the cluster fails, traffic will be transparently rerouted through other available resources. This configuration provides high performance and high availability (see Figure 2-3).
Planning the Fabric Hyper Messaging Protocol (HMP) Hyper Messaging Protocol (HMP) Hyper Messaging protocol (HMP) is HP patented, high performance cluster interconnect protocol. HMP provides reliable, high speed, low latency, low CPU overhead, datagram service to applications running on HP-UX platforms. HMP was jointly developed with Oracle Corp. The resulting feature set was tuned to enhance the scalability of the Oracle Cache Fusion clustering technology.
Planning the Fabric Hyper Messaging Protocol (HMP) • Event Monitoring Service (EMS): Supported The HyperFabric EMS monitor allows the system administrator to separately monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem. The monitor can inform the user if the resource being monitored is UP or DOWN. The administrator defines the condition to trigger a notification (usually a change in interface status).
Planning the Fabric Hyper Messaging Protocol (HMP) • Dynamic Resource Utilization (DRU): Partially Supported When a new HyperFabric resource (node, cable or switch) is added to a cluster running an HMP application, the HyperFabric subsystem will dynamically identify the added resource and start using it. The same process takes place when a resource is removed from a cluster.
Planning the Fabric Hyper Messaging Protocol (HMP) A maximum of 8 HyperFabric adapter cards are supported per instance of the HP-UX operating system. The actual number of adapter cards a particular node is able to accommodate also depends on slot availability and system resources. See node specific documentation for details. A maximum of 8 configured IP addresses are supported by the HyperFabric subsystem per instance of the HP-UX operating system.
Planning the Fabric Hyper Messaging Protocol (HMP) There are many variables that can impact HyperFabric performance. If you are considering a configuration that is beyond the scope of the following HP supported configurations, contact your HP representative. Point to Point Large servers like HP’s Superdome can be interconnected to run Oracle RAC 9i and enterprise resource planning applications. These applications are typically consolidated on large servers.
Planning the Fabric Hyper Messaging Protocol (HMP) Enterprise (Database) The HMP enterprise configuration illustrated in Figure 2-5 is very popular for running Oracle RAC 9i. Superdomes or other large servers make up the Database Tier. Database Tier nodes communicate with each other using HMP. Application Tier nodes communicate with each other and to the Database Tier using TCP/UDP/IP.
Planning the Fabric Hyper Messaging Protocol (HMP) Enterprise (Database) - Local Failover Supported Configuration The HMP enterprise configuration is a scalable solution. If higher performance is required, or if eliminating single points of failure is necessary, scaling up to the HMP enterprise configuration with multiple connections between nodes is easily accomplished (see Figure 2-6).
Planning the Fabric Hyper Messaging Protocol (HMP) Technical Computing This configuration is typically used to run technical computing applications. A large number of small nodes are interconnected to achieve high throughput. High availability is not usually a requirement in technical computing environments. HMP provides the high performance, low latency path necessary for these technical computing applications. As many as 56 nodes can be interconnected using HP’s 16 port switches.
Planning the Fabric Hyper Messaging Protocol (HMP) 38 Chapter 2
3 Installing HyperFabric This chapter contains the following sections that describe installing HyperFabric: Chapter 3 • “Checking HyperFabric Installation Prerequisites” on page 41 • “Installing HyperFabric Adapters” on page 42 • “Installing the Software” on page 47 39
Installing HyperFabric • 40 “Installing HyperFabric Switches” on page 52 Chapter 3
Installing HyperFabric Checking HyperFabric Installation Prerequisites Checking HyperFabric Installation Prerequisites Before installing HyperFabric, check to make sure the following hardware and software prerequisites have been met: ✓ Check the HP HyperFabric Release Note for any known problems, required patches, or other information needed for installation. ✓ Confirm the /usr/bin, /usr/sbin, and /sbin directories are in your PATH by logging in as root and using the echo $PATH command.
Installing HyperFabric Installing HyperFabric Adapters Installing HyperFabric Adapters This section contains information about installing HyperFabric adapters in HP 9000 systems. Online Addition and Replacement (OLAR) information is provided in the “Online Addition and Replacement” section on page 62. CAUTION HyperFabric adapters contain electronic components that can easily be damaged by small amounts of electricity.
Installing HyperFabric Installing HyperFabric Adapters IMPORTANT At this time, Superdome systems are not intended for access by users. HP recommends that these systems only be opened by a qualified HP engineer. Failure to observe this requirement can invalidate any support agreement or warranty to which the owner might otherwise be entitled. There are two methods to add or replace OLAR-compatible cards: • NOTE Using the SAM or SMH1 utility.
Installing HyperFabric Installing HyperFabric Adapters Table 3-1 IMPORTANT Important OLAR Terms (Continued) Term Meaning affected card / affected card slot Interface cards and the card slots they reside in, which are in the same power domain as the target slot. In many cases, other interface cards and slots within the system are dependent on the target card.
Installing HyperFabric Installing HyperFabric Adapters Card Compatibility This section explains card compatibility considerations for doing OLAR. Online Addition (OLA) Multiple cards can be added at the same time. When adding a card online, the first issue to resolve is whether the new card is compatible with the system. Each OLAR-capable PCI slot provides a set amount of power. The replacement card cannot require more power than there is available. The card must also operate at the slot’s bus frequency.
Installing HyperFabric Installing HyperFabric Adapters • If the necessary driver is not present and the driver is a dynamically loadable kernel module (DLKM), it can be loaded manually. See the “Dynamically Loadable Kernel Modules” section in “Interface Card OL* Support Guide” for more information. • If the driver is static and not configured in the kernel, then the card cannot be added online. The card could be physically inserted online, but no driver would claim it.
Installing HyperFabric Installing the Software Installing the Software This section describes the HyperFabric file structure and the steps necessary to load the software. The software must be installed on each instance of the HP-UX operating system in the fabric. File Structure The HyperFabric file structure is shown in Figure 3-1 below. Note that the structure is shown for informational purposes only. The user cannot modify any of the files or move them to a different directory.
Installing HyperFabric Installing the Software The global configuration file, which contains the IP addresses for each adapter and each HyperFabric switch (if any) in the fabric. • /sbin/init.d/clic The system boot startup script for the HyperFabric management process. • /var/adm/clic_ip_drv.trc One of the software’s trace files. This file is created when the clic_diag -D TCP_IP command is run. • /var/adm/clic_ip_drv.trc0 One of the HyperFabric software’s trace files.
Installing HyperFabric Installing the Software The 4X HyperFabric PCI 32-bit CRC firmware. This file must not be modified for any reason. • /opt/clic/firmware/clic_fw_hf28c The HyperFabric2 8-bit firmware. This file must not be modified for any reason. • /opt/clic/firmware/clic_fw_hf232c The HyperFabric2 32-bit firmware. This file must not be modified for any reason. • /opt/clic/firmware/clic_fw_db A binary file where adapter-specific configuration information is stored.
Installing HyperFabric Installing the Software Loading the Software Listed below are the steps you must follow to load the HyperFabric software, using the HP-UX swinstall program. Step 1. Log in as root. Step 2. Insert the software media into the appropriate drive. If the software is being loaded from a CD-ROM, go to step 3. Otherwise, go to step 4. Step 3. Mount the CD-ROM drive by using this command: mount device_name where device_name is the name assigned to the CD-ROM drive. Step 4.
Installing HyperFabric Installing the Software Note that these lines are used by the HyperFabric software—and are not comments—so do not remove them from the file. Step 14. Verify that all installed HyperFabric adapters have a software state of “CLAIMED,” by running the ioscan -nf -C clic command. Note: A check is also done to make sure all of the HyperFabric adapters have been claimed when clic_init is activated or when SMH is used to configure HyperFabric. Step 15.
Installing HyperFabric Installing HyperFabric Switches Installing HyperFabric Switches This section contains the information you need to install HyperFabric switches. As stated earlier, in this manual the term HyperFabric2 (HF2) switch refers to the functional switch (the A6384A switch chassis with one of the switch modules installed).
Installing HyperFabric Installing HyperFabric Switches Installing the HF2 Switch This section contains information for installing an HF2 switch. The front of the HF2 switch has a flange—or “wing”—on each side, with two holes for attaching the switch to the rack. Note that the two figures below do not show the flanges. Figure 3-2 below shows the front of the HF2 switch with an A6388A HF2 8-port fibre switch module installed in the switch’s expansion slot.
Installing HyperFabric Installing HyperFabric Switches Figure 3-3 Front of HF2 Switch (A6389A Switch Module Installed) Label showing Ethernet MAC address Integrated Ethernet management LAN card Status Status Status Power A B Ethernet Port Main Port 7 Port 6 Port 11 Port LED colors and meanings legend Integrated 8-port fibre card Ethernet Port Aux Port 5 Port 4 Port 3 Port 10 Port 2 Port 9 Port 1 Port 0 Port 8 A6389A HF2 4-port copper switch module in expansion slot You can install t
Installing HyperFabric Installing HyperFabric Switches With the Rail Kit HP recommends that you install the HF2 switch using the rail kit that is shipped with the switch. The rail kit includes two adjustable rails, screws, nuts, and washers. To install the HF2 switch, you need eight screws and four nuts. Use the square cage nuts if you are installing the HF2 switch in a square-hole rack. Use the u-type clip nuts if you are installing the HF2 switch in a round-hole rack.
Installing HyperFabric Installing HyperFabric Switches One end of each rail has six screw-holes (End-A), the other end has two screw holes (End-B). Figure 3-5 shows both the ends of the rail. Figure 3-5 The Ends of the Rail Kit Step 3. Orient the rails so that End-A faces the back of the rack and aligns with the front end of the switch with flanges. Step 4. Loosen the wing nuts on each rail and adjust the length of each rail to fit the length of the rack.
Installing HyperFabric Installing HyperFabric Switches NOTE HP recommends employing two people to support the weight of the switch because End-A of the rail is not yet secured. Step 8. Secure the switch and End-A of each rail by aligning the two holes in each flange with the two holes in each rack column, and two of the holes in each rail. Secure the entire assembly with two screws in each flange. Step 9.
Installing HyperFabric Installing HyperFabric Switches Without the Rail Kit As mentioned earlier, HP strongly recommends installing the HF2 switch using the rail kit (described in the previous section, “With the Rail Kit” on page 55). When you install the HF2 switch, you will be putting the front of the switch—the end with the flanges (“wings”)—at the back of the rack. The steps for installing the HF2 switch without using the rail kit are as follows: Step 1. Prepare the rack for switch installation. Step 2.
Installing HyperFabric Installing HyperFabric Switches If you want to install another HF2 switch using the rail kit, go to “With the Rail Kit” on page 55. Otherwise, go to Chapter 4, “Configuring HyperFabric,” on page 61.
Installing HyperFabric Installing HyperFabric Switches 60 Chapter 3
4 Configuring HyperFabric This chapter contains the following sections that describe configuring HyperFabric: Chapter 4 • “Configuration Overview” on page 63 • “Information You Need” on page 64 • “Performing the Configuration” on page 69 61
Configuring HyperFabric 62 • “Deconfiguring a HyperFabric Adapter with SMH” on page 74 • “Configuring the HyperFabric EMS Monitor” on page 75 • “Configuring HyperFabric with ServiceGuard” on page 76 Chapter 4
Configuring HyperFabric Configuration Overview Configuration Overview You do not need to configure the HyperFabric switch because the HyperFabric management process performs automatic routing and configuring for the switch. So, configuring HyperFabric consists only of creating the HyperFabric /etc/rc.config.d/clic_global_conf global configuration file on each node in the fabric.
Configuring HyperFabric Information You Need Information You Need When you run the clic_init command or use SMH for configuration, you have to provide certain configuration information. So, before you run clic_init or use SMH, you should get the following information: ❏ For each node in the fabric, determine if that node will need to interoperate with other nodes that are using; any HP-UX 11i v3 system and HyperFabric versions earlier than B.11.31.00.
Configuring HyperFabric Information You Need ❏ For the entire fabric, you need the IP multicast address that all the switches and nodes in the fabric will register to. The address must be a class D address. Note that if you do not have switch management enabled, you do not need this information (clic_init will not prompt you for it). ❏ For each node in the fabric, you need the IP address of the node’s Ethernet LAN interface that is on the same subnet as the switches.
Configuring HyperFabric Information You Need Configuration Information Example For this example, we have added some “dummy” (that is, not valid) addresses to the components in Figure 4-1, Map for Configuration Information Example, below. The “dummy” addresses are used only to show the flow of the information provided as input to the clic_init command and SMH. Do not try to use these addresses in your configuration.
Configuring HyperFabric Information You Need 4. What is the subnet mask of the first adapter? (255.255.255.0) If you do not specify a value for this, a default mask is chosen. You will most likely just accept the default. However, in this example, we are showing a value for the subnet mask just to illustrate the correlation between the “dummy” information in Figure 4-1 and where that information is specified or generated during clic_init and SMH. 5.
Configuring HyperFabric Information You Need 8. If switch management has been enabled, how many switches will be configured? As stated earlier, we recommend that you do not enable switch management. 9. What is the IP address of the first switch (sw_clic0)? (193.0.0.20) 10. What is the Ethernet hardware address of the first switch? (0060b0d00257) 11. What is the IP address of the second switch (sw_clic1)? (193.0.0.21) 12. What is the Ethernet hardware address of the second switch? (0060b0d00256) 13.
Configuring HyperFabric Performing the Configuration Performing the Configuration As explained in “Configuration Overview” on page 63, you must create the global configuration file (/etc/rc.config.d/clic_global_conf) on each node in the fabric. This consists mostly of specifying HyperFabric adapter-related information. (Note that if you are also going to enable switch management—which we do not recommend doing—you need to specify additional configuration information.
Configuring HyperFabric Performing the Configuration Using the clic_init Command Run the clic_init command to create the global configuration file. To use clic_init command to configure the Transparent Local Failover feature on HMP, see “Using SMH” on page 72.
Configuring HyperFabric Performing the Configuration See “Stopping HyperFabric” on page 110 and “Starting HyperFabric” on page 95 for more information. Examples of clic_init Some examples of using the clic_init command are shown below.
Configuring HyperFabric Performing the Configuration Using SMH This section describes how to use SMH1 to configure HyperFabric. For information on how to use SMH to configure and deconfigure local failover feature on HMP, see “Configuring HMP for Transparent Local Failover Support - Using SMH” on page 88.
Configuring HyperFabric Performing the Configuration Step 9. Exit SMH. Once SMH has finished, you do one of the following things: • If you want to configure HyperFabric with MC/ServiceGuard, complete the configuration described in “Configuring HyperFabric with ServiceGuard” on page 76, then run clic_start or use SMH to start HyperFabric.
Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SMH Deconfiguring a HyperFabric Adapter with SMH To use SMH to deconfigure a HyperFabric adapter on an HP 9000 system running HP-UX 11i v3, follow these steps: Step 1. Start SMH. Step 2. Select the “Networking and Communications” area. Step 3. Select the “Network Interfaces” option. Step 4. Select “HyperFabric.” All HyperFabric adapters installed in the system are listed.
Configuring HyperFabric Configuring the HyperFabric EMS Monitor Configuring the HyperFabric EMS Monitor The HyperFabric Event Monitoring Service (EMS) monitor allows system administrators to separately monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem. The monitor can inform the user if the resource being monitored is UP or DOWN. The administrator defines the condition to trigger a notification (usually a change in interface status).
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Configuring HyperFabric with ServiceGuard HyperFabric supports the ServiceGuard HA product. NOTE If you plan to configure HyperFabric with ServiceGuard, please read this section. Otherwise, skip this section and go on to the next chapter, Chapter 5, “Managing HyperFabric,” on page 93. ServiceGuard lets you create HA clusters of HP 9000 server systems.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard NOTE Because the HyperFabric network does not currently support ServiceGuard heartbeat connections, you must use an alternative type of connection for the heartbeat, such as FDDI, Token Ring, 100BaseT, or Ethernet (as shown in Figure 4-2 below).
Configuring HyperFabric Configuring HyperFabric with ServiceGuard How HyperFabric Handles Adapter Failures HyperFabric adapters are handled differently than other types of networking adapters (such as Ethernet, FDDI, and Fibre Channel) in the ServiceGuard environment. In the non-HyperFabric cases, two network links are in a node, and one will be active and one will be idle or in standby.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard available, ServiceGuard has not been notified; HyperFabric handles the local HyperFabric adapter failover. However, the failure of adapter 1 has been logged to /var/adm/clic_log. Figure 4-4 Node with One Failed HyperFabric Adapter node A HyperFabric Resource Active Package A Failed Adapter Package B Active Adapter HF adapter 1 HF adapter 0 Adapter IP addresses: 172.16.10.11 172.16.20.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Figure 4-5 When All HyperFabric Adapters Fail node A node B HyperFabric Resource Failed Packages failover to Node B HyperFabric Resource Active HF adapter 0 HF adapter 1 HF adapter 1 HF adapter 0 HF switch 0 Ethernet Port Package A Package B HF switch 1 Ethernet Port Ethernet Heartbeat LAN 0 Ethernet Heartbeat LAN 1 Configuring HyperFabric with the ServiceGuard Resource Monitor You can configure the HyperFabric Resource Monito
Configuring HyperFabric Configuring HyperFabric with ServiceGuard RESOURCE_UP_VALUE =UP Configuring ServiceGuard with HyperFabric Using SMH You must perform the following steps when using SMH to configure the HyperFabric Resource Monitor with ServiceGuard: smh Clusters High Availability Clusters Cluster Configuration (go through all the steps to create a cluster) Package Configuration Create/Add Package (if creating new packages) Specify Package Name and Nodes Specify Package SUBNET Address Specify Packa
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support HMP supports Local Failover on HP-UX 11i v3. If a HyperFabric resource (adapter, cable, switch or switch port) fails in a cluster, HMP transparently fails over traffic (Local Failover) using another available resource from the card pair. A card pair can be defined as a logical entity comprising of a pair of HF2 adapters on a HP 9000 node.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support • After executing the clic_start command on all node in the cluster, ensure that you run Oracle RAC only after one minute (approximately). • If all the trunks between the switches are down, then execute clic_shutdown followed by clic_start on all the nodes in the cluster, after replacing at least one trunk between the switches. • Maintain at least one trunk between the switches when Oracle RAC is running in the cluster.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support The details of how HyperFabric handles failures of the adapters, links, switch ports, switches, and cables between switches in a cluster are discussed in the following sections.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 1: Adapter, Link or Switch Port Failure (see Figure 4-7) If an adapter or a link or a switch port fails, HMP transparently fails over traffic through the other available link.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 2: Switch Failure (see Figure 4-8) Consider the following illustration where node A is connected to node D with traffic being routed through the HF adapter 1 on both the nodes (A and D), and the HF switch 1 fails. HMP transparently fails over traffic through the other available switch (HF switch 0).
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support If a cable between two switches fails, HMP traffic fails over to the other available cable between those two switches.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support - Using SMH To use SMH to configure HMP for Local Failover Support, complete the following steps: Step 1. Start SMH. Step 2. Select the “Networking and Communications” area. Step 3. Select the “Network Interfaces” area. Step 4. Select “HyperFabric.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Deconfiguring HMP for Local Failover support - Using SMH To use SMH to deconfigure HMP for Local Failover support, complete the following steps: Step 1. Start SMH. Step 2. Select the “Networking and Communications” area. Step 3. Select the “Network Interfaces” area. Step 4. Select “HyperFabric”. All HyperFabric adapter card pairs installed in the system are listed. Step 5.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support - Using the clic_init command You can configure the Transparent Local Failover feature of HMP using clic_init also. Let us consider the following example where we have discussed the configuration in detail. This example uses some “dummy” (that is, not valid) addresses to the components in Figure 4-10.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support You must answer ‘no’ if you want to run applications using HMP (Local Failover or Non-Local Failover) for communication over HyperFabric. In that case, all nodes in the cluster must be running version B.11.00.11 (or) B.11.11.01 (or) later version of HyperFabric software. 3. What is the IP address of the first adapter (clic0)? (192.0.0.1) 4. What is the subnet mask of the first adapter? (255.255.255.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support 8. Select any two of the following clic adapters for CARD_PAIR[0]: clic0 clic1 Enter the first clic adapter from above listed adapters: (clic0) Enter the second clic adapter from above listed adapters: (clic1) 9.
5 Managing HyperFabric This chapter contains the following sections that give information about managing HyperFabric: Chapter 5 • “Starting HyperFabric” on page 95 • “Verifying Communications within the Fabric” on page 97 93
Managing HyperFabric 94 • “Displaying Status and Statistics” on page 101 • “Viewing man Pages” on page 109 • “Stopping HyperFabric” on page 110 Chapter 5
Managing HyperFabric Starting HyperFabric Starting HyperFabric HyperFabric is started in one of these three ways: • As part of the normal local node boot process (HP 9000 system). • By running the HyperFabric clic_start command (described below). • By starting HyperFabric through SMH (described in “Using SMH” on page 96).
Managing HyperFabric Starting HyperFabric The clic_start -? command can be issued to display the online help for clic_start, or look at the clic_start (1m) man page by issuing the man clic_start command. If HyperFabric is already running, you will receive an informational (FYI) message telling you so. Your reaction to this message depends on the situation: • If you have simply forgotten (or did not know) that HyperFabric was already running, you do not have to do anything.
Managing HyperFabric Verifying Communications within the Fabric Verifying Communications within the Fabric You can verify the communications within the fabric by running the clic_probe command, which is described below. You can also use clic_probe to verify the status of specific adapters.
Managing HyperFabric Verifying Communications within the Fabric • -l specifies that you want to do local loopback testing on a particular adapter. Note that if you specify this parameter (-l), you must also specify the -c adapter_ID parameter (described above). • -s specifies that you want to loopback at the switch port attached to a particular adapter. Note that if you specify this parameter (-s), you must also specify the -c adapter_ID parameter (described above).
Managing HyperFabric Verifying Communications within the Fabric clic_probe bently4 CLIC_PROBE: 256 byte packets Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently4.corp7.com:clic3 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. 256 bytes: seq_num = 5. Packet Acknowledged. 256 bytes: seq_num = 6. Packet Acknowledged. 256 bytes: seq_num = 7. Packet Acknowledged.
Managing HyperFabric Verifying Communications within the Fabric --------- bently7.corp4.com CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss. Note that the VRID you specified (194) actually went to the adapter clic1 on bently7. And, as explained earlier, you run the clic_stat -d VRID command to determine the VRID and switch hopcount to specify.
Managing HyperFabric Displaying Status and Statistics Displaying Status and Statistics You can get the status of and statistics associated with many of the HyperFabric components by using the clic_stat command, which is described below. The clic_stat Command The following list contains some of the information that the clic_stat command will provide: • The current fabric map, in textual format. • The status of one or more HyperFabric adapters.
Managing HyperFabric Displaying Status and Statistics Note that turning on performance statistics gathering will affect the performance of HyperFabric, by increasing CPU usage and message latency (the time it takes a message to get from one point to another). It might also decrease the throughput of data. It is recommended that you do not specify this parameter unless you are trying to troubleshoot HyperFabric problems. You must be logged in as root to use this parameter.
Managing HyperFabric Displaying Status and Statistics ============================================================================ • Example 2 If the local node is bently7, and you want to display the management daemon (clic_mgmtd) configuration and statistics data, issue this command: clic_stat -d CFG The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently7.corp2.
Managing HyperFabric Displaying Status and Statistics Instance Number: Adapter Type: Firmware File: Major Num: Mgmt process driver handle: Version: H/W Path: Primary IP address: Uptime time: 1 1X PCI or HSC /opt/clic/firmware/clic_fw_1x32c 238 5 1.0 8/4/0/0 192.0.0.
Managing HyperFabric Displaying Status and Statistics The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:30 2000 Node: bently7.corp2.com ---------------------------------------------------------------------------Adapter ID: clic1 Card Pair: clic0 Instance Number: 1 Adapter Type: 4X PCI or HSC Firmware File: /opt/clic/firmware/clic_fw_hf232c Major Num: 238 Mgmt process driver handle: 5 Version: 1.
Managing HyperFabric Displaying Status and Statistics Mapping message send failures: 0 ============================================================================ • Example 4 If the local node is bently6, and you want to display the VRIDs, IP addresses, switch hopcounts, and physical routes for each HyperFabric adapter in bently6, issue this command: clic_stat -d VRID The generated output if the nodes are connected in a Point-to-Point configuration could look like this: =================================
Managing HyperFabric Displaying Status and Statistics ============================================================================ The generated output if the nodes are connected through a switch could look like this: ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently6.corp4.
Managing HyperFabric Displaying Status and Statistics The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:35 2000 Node: bently7.corp6.
Managing HyperFabric Viewing man Pages Viewing man Pages If you want to be able to view the HyperFabric man pages, you must first add /opt/clic/share/man to your MANPATH environment variable.
Managing HyperFabric Stopping HyperFabric Stopping HyperFabric You can stop HyperFabric only on a local node. Stopping HyperFabric on a node makes all of the HyperFabric adapters in that node unreachable by all other nodes in the fabric. It stops the HyperFabric management process, which stops all interconnect operations on the node. To stop HyperFabric, you use (1) the clic_shutdown command (described below) or (2) SMH (described in “Using SMH” on page 111).
Managing HyperFabric Stopping HyperFabric Using SMH To use SMH to stop HyperFabric on a local HP 9000 system running HP-UX 11i v3, follow these steps: Step 1. Start SMH. Step 2. Select the “Networking and Communications” area. Step 3. Select the “Network Interfaces” area. Step 4. Select “HyperFabric.” Step 5. Pull down the “Actions” menu and select Stop HyperFabric. Note that if HyperFabric is not running on the system, Stop HyperFabric is grayed out and you cannot select it.
Managing HyperFabric Stopping HyperFabric 112 Chapter 5
6 Troubleshooting HyperFabric This chapter contains these sections that describe troubleshooting HyperFabric: Chapter 6 • “Running Diagnostics” on page 115 • “Using Support Tools Manager” on page 120 • “Useful Files” on page 121 113
Troubleshooting HyperFabric 114 • “LED Colors and Their Meanings” on page 123 • “Determining Whether an Adapter or a Cable is Faulty” on page 129 • “Determining Whether a Switch is Faulty” on page 130 • “Replacing a HyperFabric Adapter” on page 132 • “Replacing a HyperFabric Switch” on page 133 Chapter 6
Troubleshooting HyperFabric Running Diagnostics Running Diagnostics Before running HyperFabric diagnostics: 1. Confirm HyperFabric adapters are installed on each node. Use the following command to display a list of HyperFabric adapters installed each node: ioscan -funC clic 2. Check to see if HyperFabric software is installed. Use the following command to display the HyperFabric software version installed on each node: swlist | grep -i hyperfabric 3. Check the patch level.
Troubleshooting HyperFabric Running Diagnostics clic_shutdown clic_init (The answer to the interoperability question must be consistent on all nodes in the fabric.) clic_start Check to make sure all IP addresses have been assigned. Check the subnet. Every HyperFabric adapter in the fabric must be able to communicate with every other HyperFabric adapter. Every HyperFabric adapter in the fabric must be connected, point-to-point or via a switch.
Troubleshooting HyperFabric Running Diagnostics If you are running an Oracle application using HMP and traffic does not appear to be flowing after completing all of the steps listed above, check the ORAHOME/rdbms/log/alert*log files. The display should show Cluster Interconnect IPC version:Oracle using HP-HMP logged in the alert log files. If not, recompile the Oracle application to run using HMP according to the instructions provided in the Oracle Installation and Administration Guide.
Troubleshooting HyperFabric Running Diagnostics packet_data or 0x0004 Trace the packet data. packet_hdr or 0x0008 Trace the protocol headers (TCP/UDP/IP). data_struct or 0x0010 Trace the HyperFabric software CLIC-specific data structures and events. rcv_path or 0x0020 Trace the HyperFabric software receive path. control or 0x0040 Trace the HyperFabric software control messages. all or 0xffff Turn on all possible trace levels. Note that this will have a severe impact on performance.
Troubleshooting HyperFabric Running Diagnostics 256 bytes: seq_num = 0. Packet Acknowledged. 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. --------- CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss. CLIC_PROBE: 256 byte packets Source adapter id: bently6.corp4.com:clic1 Target adapter id: bently8.corp2.com:clic3 256 bytes: seq_num = 0.
Troubleshooting HyperFabric Using Support Tools Manager Using Support Tools Manager Use Support Tools Manager (STM) with HyperFabric to gather information about HyperFabric components and to diagnose hardware problems. Two tools are available in STM for HyperFabric: • The Information Tool provides information about the HyperFabric adapter, without resetting the adapter. • The Diagnostics Tool can be used to run tests on the HyperFabric adapter; the tool reports any failures.
Troubleshooting HyperFabric Useful Files Useful Files When you are troubleshooting HyperFabric-related problems, you might find it useful to look at the contents of the following files: • /etc/rc.config.d/clic_global_conf This is the global configuration file. Check it to confirm that the configuration information is correct. • /var/adm/clic_log This is a global log file that contains a history of significant HyperFabric events.
Troubleshooting HyperFabric Useful Files /var/adm/clic_fw.dumpx This is the default file for a memory dump of an HyperFabric adapter, created when the clic_diag -d command is run without specifying an output file. • /etc/rc.config.d/netconf This file contains IP-related configuration information for all of the networking adapters installed in the HP 9000. IMPORTANT: clic_init and SMH modify this file, adding some HyperFabric-related lines that end with the characters #clic.
Troubleshooting HyperFabric LED Colors and Their Meanings LED Colors and Their Meanings Listed below are the possible colors (and the corresponding meaning) of the LEDs on the HyperFabric adapters and switches. Adapter LEDs Table 6-1 below shows the names of the LEDs on each HyperFabric adapter. Note that the LEDs on the A4920A adapter are labeled, but the labels might be hard to see when a cable is connected to the adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings ✓ If the connection from the adapter to the corresponding switch port (if a switch is used) or the corresponding adapter in the remote node (in a node-to-node configuration) is operational, the “Connected/Traffic,” “Link,” or “Link Connected/Traffic” LED on the adapter shows as solid green.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-2 HyperFabric Adapter LED Colors and Meanings LED “Connected/Traffic” (A4919A and A6092A adapters) Color None Meaning Adapter is not operational. “Link” (A4920A and A6386A adapters) “Link Connected/Traffic” (A4921A adapter) “Error” Chapter 6 Notes • HP 9000 is not operational. • HyperFabric is not running on HP 9000. • Adapter is installed in a slot in HP 9000, but cable is attached incorrectly or no cable is attached at all.
Troubleshooting HyperFabric LED Colors and Their Meanings HF2 Switch LEDs The HF2 switch LED colors and meanings are explained below. ❏ For each “Status” LED on the switch: ✓ If the card/switch module is not operating, the LED is off. — For the integrated Ethernet management LAN adapter card (in the top slot) and the integrated 8-port fibre card (in the middle slot): the card can be safely removed by qualified Hewlett-Packard personnel only.
Troubleshooting HyperFabric LED Colors and Their Meanings ✓ If the connection from the port to the corresponding adapter in the HP 9000 is operational, the LED on the port shows as solid green. ✓ If data is flowing between the port and the corresponding adapter, the LED shows as flashing green. Note that data does not start to flow until HyperFabric initialization has occurred (see “Using the clic_init Command” on page 70).
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-3 HF2 Switch LED Colors and Meanings (Continued) LED “Port x” 128 Color None Meaning Port is not operational. Solid green Connection from port to adapter is operational. Flashing green Data is flowing between the port and the adapter. Notes • A cable is not attached correctly to the port or no cable is attached at all. • The port is connected to a non-operational adapter in an HP 9000.
Troubleshooting HyperFabric Determining Whether an Adapter or a Cable is Faulty Determining Whether an Adapter or a Cable is Faulty If you suspect that either an HyperFabric adapter or a cable attached to it is faulty, follow these steps to determine which component needs replacing: Step 1. Disconnect the cable from the HyperFabric adapter. Step 2.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty Determining Whether a Switch is Faulty If you suspect that a HyperFabric switch is faulty, follow the steps below to determine if the switch needs replacing. The steps for determining if an HF2 switch is faulty are as follows: Step 1. Check the cards/switch module in the switch—the integrated Ethernet management LAN adapter card, integrated 8-port fibre card, and switch module in the expansion slot: a.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty d. If the port is faulty, do not use it. Instead, use a known good port on that card/switch module if one is available. If no good ports are available, replace that card/switch module. Remember, though, that the integrated 8-port fibre card can be removed by qualified Hewlett-Packard personnel only. The A6388A and A6389A switch modules can be removed by you or qualified Hewlett-Packard personnel. Step 5.
Troubleshooting HyperFabric Replacing a HyperFabric Adapter Replacing a HyperFabric Adapter If OLAR is supported for the HP 9000 system and the HyperFabric adapter, and you want to do OLR, see “Online Replacement (OLR)” on page 45. The HP HyperFabric Release Note contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for. If you cannot or do not want to do OLR to replace a HyperFabric adapter, follow these steps: Step 1.
Troubleshooting HyperFabric Replacing a HyperFabric Switch Replacing a HyperFabric Switch If you need to replace a faulty HyperFabric switch, follow these steps: Step 1. If you do not have an HA environment, stop HyperFabric on all nodes that are connected to the faulty switch. (See “Stopping HyperFabric” on page 110.) If you have an HA environment with two switches (where the backup switch will have taken over when the first switch failed), disconnect all cables attached to the faulty switch. Step 2.
Troubleshooting HyperFabric Replacing a HyperFabric Switch 134 Chapter 6
A Safety and Regulatory Information This appendix contains the following sections that contain safety information and regulatory statements for the HyperFabric hardware components: Appendix A • “Safety Symbols” on page 137 • “Regulatory Statements” on page 138 135
Safety and Regulatory Information These components were tested for conformance to various national and international regulations and standards. The scope of this regulatory testing includes electrical and mechanical safety, electromagnetic emissions, immunity, acoustics, and hazardous materials. When required, approvals are obtained from third party test agencies. Approval marks appear on the product label. In addition, various regulatory bodies require some information under the headings noted below.
Safety and Regulatory Information Safety Symbols Safety Symbols The safety-related symbols used in this manual are shown below. WARNING A WARNING denotes a hazard that can cause personal injury. CAUTION A CAUTION denotes a hazard that can damage equipment.
Safety and Regulatory Information Regulatory Statements Regulatory Statements This section contains the regulatory statements for the HyperFabric products.
Safety and Regulatory Information Regulatory Statements Cet appareil numérique de la classe A respecte toutes les exigences du Règlement sur le matériel brouilleur du Canada. Europe RFI Statement This is a Class A product. In a domestic environment, this product may cause radio interference, in which case the user may be required to take adequate measures. Australia and New Zealand EMI Statement This product meets the applicable requirements of the Australia and New Zealand EMC Framework.
Safety and Regulatory Information Regulatory Statements Declarations of Conformity 140 Appendix A
Safety and Regulatory Information Regulatory Statements Appendix A 141
Safety and Regulatory Information Regulatory Statements 142 Appendix A
B Technical Specifications This appendix contains the following sections that contain the technical specifications for the HyperFabric hardware components: Appendix B • “Physical Attributes” on page 144 • “Environmental” on page 145 143
Technical Specifications Physical Attributes Physical Attributes The physical attributes of the HyperFabric adapters and switches are as follows: • HF2 adapter (A6386A PCI [4X]) — 4.2 inch (10.7 cm) height — 1 inch (2.5 cm) width — 7.1 inch (18 cm) length — 4.25 ounces (120 g) weight • HF2 switch chassis (A6384A) — 3.47 inch (8.8 cm) height — 19 inch (48.2 cm) width (at flanges) 17.18 inch (43.6 cm) width (enclosure) — 17.5 inch (44.5 cm) length (depth from back of flanges to back plate) 18.
Technical Specifications Environmental Environmental These environmental specifications are the same for the HyperFabric adapters and switches: • Temperature — -40 to +70 degrees C, non-operating/storage — +5 to +40 degrees C, operating — +20 to +30 degrees C, recommended operating range • Relative Humidity — 15% to 90%, non-operating/storage — 15% to 80% @ 22 degrees C, operating — 15% to 80% @ 22 degrees C, recommended operating range • Altitude — 15,000 feet (4.
Technical Specifications Environmental 146 Appendix B
Index A adapters deconfiguring, 74, 89 faulty, testing for, 129 installing, 42 LED colors and meanings, 123, 125 OLA of, 45 OLR of, 45 regulatory specifications, 138 replacing, 132 C cables, testing for faulty, 129 CLIC, definition of, 18, 47 clic_diag command, 98, 117 clic_init command, 70 after OLA, 45 clic_probe command, 97 to remote node, 117 clic_shutdown command, 110 clic_start command, 95 after OLA, 45 clic_stat command, 101 CLuster InterConnect.
Index planning and preparation for, 44 terminology, 43 overview of configuration, 63 P planning the fabric, 21 prerequisites for installation, 41 R regulatory specifications, 138 relocatable IP addresses, 18 configuring, 81 S SAM using for configuration, 72, 88 using for deconfiguring an adapter, 74, 89 using for starting HyperFabric, 96 using for stopping HyperFabric, 111 software file structure, 47 files, description of, 47 installing, 47 steps for, 50 starting HyperFabric after OLA, 45 using clic_start,
