Intel® NetStructureTM ZT 4901 High Availability Software Technical Product Specification April 2003 Order Number: 273856-002
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.
Contents Contents 1 Document Organization ................................................................................................................ 9 2 Introduction..................................................................................................................................11 2.1 2.2 2.3 3 Host Application Software ..........................................................................................................21 3.1 3.2 3.3 4 Goals of the Host Application .....
Contents 5.2 6 5.1.2.4 Remove Device...................................................................................... 35 5.1.2.5 Driver Synchronization........................................................................... 35 Summary ............................................................................................................................ 36 Redundant Host API....................................................................................................................
Contents 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 8.10 8.11 8.12 9 Slot Control API ...........................................................................................................................85 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 10 imbCloseDriver ...................................................................................................................79 imbDeviceIoControl .....................................................................................................
Contents Figures 1 2 3 4 5 6 High-Availability CPU Architecture ............................................................................................. 11 RSS Processor Board Block Diagram ........................................................................................ 16 RSS Host with Bridge Mezzanine Block Diagram ...................................................................... 17 High-Availability System Backplane Architecture ........................................................
Contents Revision History Date Revision Description April 2003 002 Removed three demonstration utilities from 10.1.2.7 and removed Interhost Communication section.
Contents This page intentionally left blank.
Document Organization 1 This document describes the High Availability Software Development Kit for the Intel® NetStructure™ ZT 4901 I/O Mezzanine Card. Following is a summary of the contents. Chapter 2, “Introduction,” provides an overview of the hardware and software subsystems supported by Intel’s High Availability Software Development Kit. Chapter 3, “Host Application Software,” covers the basic requirements needed for applications to properly leverage Redundant Host architecture.
Document Organization Appendix G, “RH Switchover on OS Crash,” describes how the High-Availability Redundant Host architecture enables the system master board to perform a switchover to the backup host in the event of a system crash under the Linux and VxWorks operating systems. Appendix H, “Data Sheet Reference,” provides links to specifications and user documentation relevant to the High Availability Software Development Kit.
2 Introduction Intel® High Availability (HA) systems feature built-in redundancy for active system components such as power supplies, system master processor boards, and system alarms. Redundant Host (RH) systems are HA systems that feature an architecture allowing the Active Host system master processor board to hand over control of its bus segment to a Standby Host system master processor board.
Introduction Application—Application-specific code, not including application-specific device drivers. Arbitration—Hardware process of a bus master using the hardware REQ# signal to request the PCI bus from the Active Host and then being granted access to the bus with the hardware GNT# signal. Available Host—A Host operating in Owner mode that can own domains and communicate with the rest of the RH system.
Introduction Handover—A type of switchover that is initiated by the Active Host, resulting from a software command or Baseboard Management Controller detected fault wherein the bus segment is quiesced before the transfer of system slot functions. HA SDK—High Availability Software Development Kit High-Availability (HA) system—Constructed from standard components with redundancy to reduce the probability of interruptions. Typically, “five nines” of availability are expected (99.999%).
Introduction Split Mode—Split Mode is a term that refers to a system operating with multiple system master Host boards that each own a single bus segment. Split Mode may refer to either Active/Active or cluster modes. In an Active/Active either of two Hosts can inherit the other Host’s bus segment. In cluster mode each Host’s bus segment is locked to that Host and ownership cannot be transferred to the other Host.
Introduction Intel’s RH software runs on system master processor boards with bridge mezzanine cards in a PICMG 2.13 compliant RSS backplane to provide redundant system master functionality. This allows the failover of control of redundant PCI buses. It provides faster hardware that is PICMG 2.9 and 2.16 compliant. The system makes use of the IPMI infrastructure for fault detection and correction. 2.2.
Introduction Figure 2. RSS Processor Board Block Diagram Base CPU Board CPU/ Chipset Interboard Connector xMC HC Xreq PCI Control/Status CIC Control Iso/Term Clk. Arb. CompactPCI J1/J2 Bus Segment A 2.2.2 Bridge Mezzanine The HASDK driver set works in single and dual bus segment configurations. In order for the dual bus configuration to be supported a bridge mezzanine must be mounted on the processor board.
Introduction Figure 3. RSS Host with Bridge Mezzanine Block Diagram Base CPU Board CPU/ Chipset Interboard Connector Control/ Status Xreq Xreq Control/Status CIC CIC Control Iso/Term 2.2.3 Bridge Mezzanine xMC HC PCI Interboard Connector Clk. Arb. PCI Control Arb. Clk. Iso/Term CompactPCI J1/J2 CompactPCI J1/J2 Bus Segment A Bus Segment B Backplane The RSS system backplane supports two CompactPCI buses accessible by both Redundant Hosts.
Introduction Figure 4. High-Availability System Backplane Architecture 1A 2 3 4 5 6 7 8 13 14 15 16 17 18 19 20 21 1B Host Processor Board Bridge Mezzanine Board 2.3 Bridge Mezzanine Board Redundant Host Processor Board High-Availability Software Approach As shown in the Figure 1, “High-Availability CPU Architecture” on page 11, there are three HighAvailability software components: • Host application • System Management • Backplane Device Drivers 2.3.
Introduction The design of the application should be made as portable as possible. This requires that the design be implemented in a modular approach that isolates the system management requirements from the host application. This division of responsibilities can be achieved through a layered implementation. See Chapter 3, “Host Application Software” for more information. In addition to taking a modular approach, the application designer should recognize the importance of producing a hardened application.
Introduction 2.3.3 Backplane Device Drivers Backplane device drivers are a critical component of High Availability system. The drivers need to be robust in their operations as well as to be dynamic given the “Stated” nature of a Hot Swap architecture. The ability of a driver to remain loaded and initialized even though the Host may not have visibility to the device is critical when Host ownership transfer can occur almost instantaneously.
Host Application Software 3 Through thoughtful design and the use of a layered development approach, an application can be developed that meets the implied robustness of a highly available system and also is a portable entity. In addition to covering the details of developing an application that runs in a High Availability environment, this chapter provides a foundation for understanding the issues that a developer needs to be aware of when deploying in a multi-host architecture. 3.
Host Application Software System Host in the same chassis. In order for a host application to be capable of maintaining the system’s serviceability, these redundant applications should maintain some level of synchronization. The level of synchronization and the level of sophistication of the system’s peripherals determine the failover characteristics of your system. Synchronization issues, in addition to other implementation concerns, are covered in the Section 3.3, “Development Issues” on page 23. 3.
Host Application Software The diagram shows that the host application’s need to understand the particular implementation aspects of the platform’s system management is reduced by placing an intermediary layer that in effect interfaces and translates only the system management information that the host application cares about.
Host Application Software 3.3.2 Graceful Switchover In a Redundant Host environment a graceful switchover is only secondary in importance to data integrity. An effective mechanism is required in order for an application to seamlessly pick up the functionality of a faulted application. The Intel Redundant Host environment has an infrastructure in place to help facilitate such control transitions.
System Management 4 System Management is an all-encompassing term whose definition can vary drastically depending on the type of system that is being developed. System Management can indicate anything from system configuration all the way to active reporting, proactive fault remediation, and comprehensive system security. In a relatively closed system with limited access to external interaction, system management could be limited to chassis management, event logging, and resource management.
System Management The next-generation, high-availability architecture provides this system management infrastructure using IPMI. Through the IPMI API the developer is able to access the status of individual sensors, various management controllers, and to configure the system to initiate switchovers based on events or threshold excursions. See Chapter 8, “IPMI API,” for details. 4.1.
System Management • Upper/Lower critical threshold • Upper/Lower non-recoverable threshold Each range can be set independently for each sensor and the ranges can overlap. This area of configuration is used only to trigger events. These events appear in the System Event Log. Platform Event Filtering (PEF) determines the actions that occur as a result of these events. Only the Upper/Lower non-recoverable threshold is typically configured using the PEF to cause a hardware-initiated takeover to occur. 4.2.
System Management Table 2. RH Channel Alert Destinations Destination # Description 0x00 RESERVED 0x01 RH_CHAN_SET_ALL_MC_FD (Sets CIC Fault Detection Lines) 0x02 RH_CHAN_CLEAR_ALL_MC_FD (Clears CIC Fault Detection Lines) The RH channel acts as a virtual channel that can respond to Alert Actions. This channel supports IPMI commands like Alert Immediate: • In the Alert Policy Table: Create an entry with a unique policy number, channel specified as RH, destination specified as RH_CHAN_SET_ALL_MC_FD.
System Management Request - - Response 1 Completion Code 2 RH BMC Address High Availability Software for the Intel® NetStructureTM ZT 4901 Technical Product Specification 29
System Management This page intentionally left blank.
High Availability CompactPCI Device Drivers 5 This chapter describes the characteristics of highly available software drivers for CompactPCI peripherals in a Redundant Host environment. To fully utilize the High Availability SDK, you must write a peripheral driver that can be started and stopped repeatedly and that can be loaded and initialized even when the device it is servicing is not physically visible to the operating system. 5.
High Availability CompactPCI Device Drivers 5.1.1 Device Driver States There are varying degrees of functionality that are dependent on power modes, operating system Hot Swap implementations, and device characteristics. But for a device driver to function in this High Availability architecture we can generalize the required driver states down to three distinct states. • Initialization • Quiesced • Activation 5.1.1.1 Initialization During initialization, the driver starts up and is loaded.
High Availability CompactPCI Device Drivers Figure 6. Multi-Stated Driver Flowchart AddDevice Driver Not Loaded Find Suitable Driver RemoveDevice / SurpriseRemoval Device Removed Device Not Present 5.1.2 Device Suspended StopDevice RemoveDevice / SurpriseRemoval StartDevice Device Running Adding High-Availability Functionality Operating in a Redundant Host architecture places additional responsibilities on device drivers beyond those issues required to function in a normal Hot Swap environment.
High Availability CompactPCI Device Drivers • • • • 5.1.2.1 Add Device Resume Operations Suspend Operations Remove Device Add Device Add Device is the device driver call made by the Hot Swap Manager either when an asserted ENUM signal is detected or during the kernel load time. The Add Device callback execution indicates to a device driver that an instance of a device that the driver can control has been detected.
High Availability CompactPCI Device Drivers 5.1.2.3 Suspend Operations The counterpart to the Resume Operation callback is the Suspend Operation. The kernel calls the Suspend Operation callback function for each device for which a Host is losing visibility. It cannot be assumed that the driver retains visibility to the backplane device during the Suspend Operation execution. The Suspend Operation should first disconnect the device’s interrupt service routine.
High Availability CompactPCI Device Drivers 5.2 Summary The intent of the HA CompactPCI device driver model is to leverage the native device driver infrastructure to supply a robust Hot Swap capability while limiting the non-proprietary device driver modifications.
Redundant Host API 6.1 Intel-Specific APIs 6.1.
Redundant Host API OUT BOOL*pbReset ); Arguments: Handle – the handle of the current session SourceHost - the number of the source host Domain – the domain number pDestinationHost pointer to the variable receives the number of the host that should own the specified domain if the source host fails and hardware-initiated switchover takes place for it pbReset pointer to the variable receives the state of the flag that indicates whether the specified destination host will perform a reset if the host r
Redundant Host API The following topics specify each of the interface functions. 6.2.1 Definitions and Types The following definitions are provided for terms used in the remaining topics of this chapter. RH (Redundant Host) System. An RH system consists of two or more hosts and one or more domains. An ownership relationship is defined between hosts and domains: hosts own domains. At any given moment of time, no more than one host can own one domain.
Redundant Host API RH Instance ID. A host can be a member of several RH systems simultaneously, similar to multihomed hosts in networking. In that case, the application can use the Redundant Host API from several RH infrastructures. To select a specific RH system, the application uses the RH Instance ID when obtaining the handle to the RH system via RhOpen. RH Instance ID is an implementationdefined character string.
Redundant Host API typedef void (*RH_SLOT_STATE_CALLBACK) ( IN uint32 Domain, IN PHYSICAL_SLOT_ID Slot, IN RH_DOMAIN_SWC_STATE State, IN void *pContext ); typedef void (*RH_SWITCHOVER_CALLBACK) ( IN uint32 Host, IN uint32 Domain, IN void *pContext ); typedef BOOLEAN (*RH_SWITCHOVER_REQUEST_CALLBACK) ( IN uint32 RequestingHost, IN uint32 DestinationHost, IN uint32 Domain, IN void *pContext ); typedef enum { RESET_REQUIRED, RESET_NOT_REQUIRED, UNKNOWN } RH_SLOT_NEEDS_RESET; typedef struct RH_SLOT_DESCRIPTOR_
Redundant Host API typedef void * RH_HANDLE; 6.2.2 Initialization/Termination 6.2.2.
Redundant Host API An RH infrastructure that implements this function shall return the list of RH Instance IDs only for those RH Systems that it services. If multiple RH infrastructures are present on the current host, an intermediate layer of functionality between the application and infrastructures may be defined, that implements this function.
Redundant Host API The current host may be attached to several RH systems. In that case, the parameter Instance ID should be used to specify the RH system that the application wants to work with. Specifying NULL as the value of the parameter InstanceID chooses the default RH system. If the function RhEnumerateInstances is supported, the default RH system shall be the one designated by the first RH Instance ID in the list returned by RhEnumerateInstances.
Redundant Host API Arguments: Handle – the handle of the current session pInstanceID – pointer to the character buffer where the RH Instance ID associated with the given handle is stored as a null-terminated character string InstanceIDLength the size of the buffer; if this size is too small for the output, this function fails.
Redundant Host API Handle – the handle of the current session pCount – pointer to the variable that receives the current number of domains in the system Return Value: HSI_STATUS_SUCCESS returned in the case of success HSI_STATUS_INVALID_PARAMETER invalid session handle Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function returns the number of domains in the RH system that can be owned by the hosts in the system. 6.
Redundant Host API Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function retrieves the list of numbers of known domains that comprise the RH system. Each domain number is an arbitrary uint32 value. Before the call, the caller should allocate a buffer that can accommodate a sufficient number of uint32 values, and pass its address in the pDomainNumbersArray parameter.
Redundant Host API HSI_STATUS RhGetDomainSlotPath ( IN RH_HANDLE Handle, IN uint32 Host, IN uint32 Domain, OUT uint16 *pRootBus, OUT char *pOutSlotPath, IN uint32 SlotPathLength, OUT ULONG *pActualSize ); Arguments: Handle – the handle of the current session Host - the target host number Domain – the domain number pRootBus – pointer to the variable where the infrastructure stores the root bus number of the PCI tree of this domain. This value is 0 for the first or single PCI tree.
Redundant Host API The slot path is stored as a null-terminated sequence of two-character groups. Each group describes one item of the slot path and represents the number (DeviceNumber * 8 + FunctionNumber) for the corresponding PCI-PCI bridge in hexadecimal. The two hexadecimal digits of this number are represented by two characters from the set ‘0’..’9’, ‘A’..’F’. 6.2.3.
Redundant Host API Domain the domain number pSlotNumbersArray pointer to the array where the list of slot numbers for the specified domain is placed ArraySize the size (in items of type PHYSICAL_SLOT_ID) of the buffer initially provided for the array by the caller pActualSize pointer to the variable where the actual number of items in the list is stored (even if the initial size is too small and the function returns the error HSI_STATUS_INSUFFICIENT_BUFFER).
Redundant Host API PhysSlot the physical slot number (represented as combination of Shelf ID and Slot ID) pDomain pointer to the variable where the number of the domain is placed Return Value: HSI_STATUS_SUCCESS returned in the case of success HSI_STATUS_INVALID_PARAMETER invalid session handle Other implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: Get the domain that owns the specified slot.
Redundant Host API HSI_STATUS RhGetHostCount( IN RH_HANDLE Handle, OUT uint32 *pHostCount); Arguments: Handle the handle of the current session pHostCount pointer to the variable where the host count is placed Return Value: HSI_STATUS_SUCCESS returned in the case of success HSI_STATUS_INVALID_PARAMETER invalid session handle Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function gets the number of hosts in the system.
Redundant Host API HSI_STATUS_INSUFFICIENT_BUFFER returned if the buffer provided for the array by the caller is too small; in that case, the array isn’t filled in but the location pointed by pActualSize is set to a correct value to assist the caller in subsequent buffer allocation. Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function retrieves the list of numbers of known hosts that comprise the RH system.
Redundant Host API HSI_STATUS_INSUFFICIENT_BUFFER returned if the buffer OutHostName is too small to store the host name HSI_STATUS_NOT_SUPPORTED returned if this function is not supported by the infrastructure Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function returns the symbolic name for the specified host.
Redundant Host API returns FALSE. This mode can be used for configuration purposes, for example, to update system software on the host. Setting the status to TRUE brings the host back from the isolation mode to the state in which it can own and acquire domains. If the parameter Available is FALSE, the target host must not own any domains when this function is called. 6.2.3.
Redundant Host API 6.2.3.14 RhGetDomainAvailabilityToHost Prototype: HSI_STATUS RhGetDomainAvailabilityToHost( IN RH_HANDLE Handle, IN uint32 Host, IN uint32 Domain, OUT BOOLEAN *pAvailable); Arguments: Handle the handle of the current session Host the host number Domain the domain number pAvailable pointer to the variable that receives a Boolean value: TRUE if the specified host can own the specified domain, FALSE otherwise.
Redundant Host API PhysSlot obtains information for given physical slot number pInfoBuffer pointer to the buffer where the information is placed InfoBufferSize the size (in bytes) of the buffer initially provided for the array by the caller pActualSize pointer to the variable where the required size of the buffer is stored (even if the initial size is too small and the function returns the error HSI_STATUS_INSUFFICIENT_BUFFER).
Redundant Host API SlotPath The slot path from the root bus to the device. The slot path is stored as a null-terminated sequence of two-character groups. Each group describes one item of the slot path and represents the number (DeviceNumber * 8 + FunctionNumber) for the corresponding PCI-PCI bridge in hexadecimal. The two hexadecimal digits of this number are represented by two characters from the set ‘0’..’9’, ‘A’..’F’. BusNumber The bus number for the device.
Redundant Host API OUT RH_SLOT_DESCRIPTOR *pInfoBuffer, IN uint32 InfoBufferSize, OUT uint32 *pActualSize ); Arguments: Handle the handle of the current session PhysSlot the physical slot number below which the devices in question are nested pSlotPath the slot path to the parent bridge for the devices pInfoBuffer pointer to the buffer where the information about devices is placed InfoBufferSize the size (in bytes) of the buffer initially provided for the array by the caller pActualSize pointer t
Redundant Host API OwningHost The number of the host that currently owns the domain this device belongs to RootBusNumber The PCI bus number of the root bus of the PCI hierarchy the device resides in; is 0 for single-root PCI hierarchies. This value is 16 bit to accommodate possible future extensions to PCI that allow more than 256 PCI buses in the system. SlotPath The slot path from the root bus to the nested device. The slot path is stored as a null-terminated sequence of two-character groups.
Redundant Host API This function can be used to enumerate devices nested below physical slots if a PCI-PCI bridge occupies the physical slot. To get information about all devices at the next nesting level, this function should be called with the physical slot number and slot path to the immediate parent bridge. This slot path is taken from the slot information structure for the immediate parent.
Redundant Host API Software connection is the inverse action to software disconnection: it starts the drivers for PCI devices in the domain and resumes normal operation. When initiated for a domain in the DISCONNECTED state, it brings the domain to CONNECTED state through the intermediate CONNECTING state. Software connection can be used to cancel the effect of software disconnection for a domain during switchover preparation.
Redundant Host API Hence, forced switchover is potentially destructive for the owning host and should be used with care. To perform forced switchover, it is sufficient to call the RhPerformSwitchover function. Forced switchover can be initiated either by the owning host (in which case it voluntarily gives up control of this domain), or by the new owner of the domain, or by some third-party host.
Redundant Host API IN uint32 DestinationHost, IN uint32 Timeout, IN BOOLEAN Persist ); Arguments: Handle the handle of the current session pDomains pointer to the array of numbers of the domains to disconnect; all domains must be owned by the same host DomainCount the number of elements in the array of domain numbers DestinationHost the number of the destination host for the intended switchover of the specified domains; value RH_NO_DESTINATION_HOST meaning that no host owns the domains.
Redundant Host API This function just initiates the software disconnection and does not wait for its completion. The function RhGetDomainSwConnectionStatus can be used to track the progress of the pending disconnection.
Redundant Host API Synopsis: This function requests domain software connection. It initiates software connection for the specified domains: • Startup of all devices in the domain • Creation of corresponding software representation for devices (device objects and so forth) If software disconnection is in progress for this domain, this function cancels the software disconnection This function just initiates the software connection–it does not wait for its completion.
Redundant Host API Get domain software connection status. This function returns the current state of the specified domain with respect to software connection/disconnection. There exist two stable (DISCONNECTED, CONNECTED) and two transitional (DISCONNECTING, CONNECTING) states. This function can be used during a cooperative switchover to track progress of a pending software connection or disconnection request. The function can be called on a host that does not own the domain. 6.2.5.
Redundant Host API HSI_STATUS RhPerformSwitchover( IN RH_HANDLE Handle, IN uint32 DestinationHost, IN uint32 *pDomains, IN uint32 DomainCount, IN BOOLEAN Reset, IN BOOLEAN Hostile ); Arguments: Handle – the handle of the current session DestinationHost the number of the host that should own the domains after the switchover; value RH_NO_DESTINATION_HOST means “no host should own the specified domains” pDomains – the array of domain numbers that should be taken over.
Redundant Host API HSI_STATUS RhSetHwDestinationHost( IN RH_HANDLE Handle, IN uint32 SourceHost, IN uint32 *pDomains, IN uint32 DomainCount, IN uint32 DestinationHost, IN BOOLEAN Reset ); Arguments: Handle – the handle of the current session SourceHost – the number of the host for which domain destination is specified pDomains – the array of domain numbers identifying the group of domains that is passed to the specified destination host if the source host fails and hardware-initiated switchover takes
Redundant Host API 6.2.5.
Redundant Host API DomainCallback pointer to the callback function that tracks state of the domain SlotCallback - pointer to the optional callback function that tracks state of separate slots during software connection and disconnection. pContext – an opaque context pointer; passed unchanged to the callback function.
Redundant Host API Callback – pointer to the callback function Context – an opaque context pointer; passed unchanged to the callback function. Systemwide – a Boolean flag; if set to TRUE, notification happens for each switchover even if the current host is neither the source nor the destination of the switchover; if set to FALSE, the host is notified only of those switchovers in which it participates.
Redundant Host API Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function establishes the callback that is called when an attempt is made to take over any domain from the current host. The callback function is called with the requesting host number, new owning host number, and the domain number as parameters.
Redundant Host API The corresponding callback can be used to handle this situation. The callback is called with the bus lock held. Parameters to the callback include a list of entries identifying domain devices in unsafe states. These devices should be reset before the domain can be software connected and the device drivers can be started.
Redundant Host API SlotPath - the slot path from the root bus to the device; represented as a nullterminated character string BusNumber - the bus number for the device.
Redundant Host API HSI_STATUS RhDisableNotification( IN RH_HANDLE Handle IN RH_NOTIFICATION_TYPE NotificationType ); Arguments: Handle – the handle of the current session NotificationType this enumeration specifies the type of notifications to disable Return Value: HSI_STATUS_SUCCESS returned in the case of success HSI_STATUS_INVALID_PARAMETER invalid session handle Other, implementation-defined HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This functio
Hot Swap API 7 See the Intel® NetStructure™ Hot Swap Kit for Linux 2.4 Software Manual for a detailed description of the provided Hot Swap API supported by this software installation. While the Hot Swap Kit manual is specifically tailored for a Linux installation, the Hot Swap API detailed in this manual is identical to the VxWorks* implementation.
Hot Swap API This page intentionally left blank.
8 IPMI API 8.1 imbOpenDriver Prototype: int imbOpenDriver(void) Parameters: None Returns: Int - 0 for Fail and 1 for Success, sets hDevice Description: Establish a link to the IMB driver. 8.2 imbCloseDriver Prototype: void imbCloseDriver() Parameters: None Returns: None Description: Close a link to the IMB driver. 8.
IPMI API LPVOID lpvOutBuffer, DWORD cbOutBuffer, LPDWORD lpcbBytesReturned, LPOVERLAPPEDlpoOverlapped ) Parameters: dummy_hDevice - handle of device dwIoControlCode - control code of operation to perform lpvInBuffer - address of buffer for input data cbInBuffer - size of input buffer lpvOutBuffer - address of output buffer cbOutBuffer - size of output buffer lpcbBytesReturned - address of actual bytes of output lpoOverlapped - address of overlapped struct Returns: BOOL - FALSE for fail and
IPMI API ACCESN_STATUS - ACCESN_OK else error status code Description: Sends a request to an I2C device 8.
IPMI API Returns: ACCESN_STATUS - ACCESN_OK else error status code Description: This function gets the next available async message with a message ID greater than SeqNo. The message looks like an IMB packet and the length and Sequence number are returned 8.
IPMI API ACCESN_STATUS imbUnregisterForAsyncMsgNotification (unsigned int *handleId) Parameters: eventId - EventID handle to unregister Returns: ACCESN_STATUS - ACCESN_OK else error status code Description: This function unregisters the calling application for Asynchronous notification when an SMS message is available with the IMB driver. 8.
IPMI API 8.
Slot Control API 9.
Slot Control API Return Value: HSI_STATUS_SUCCESS if successful HSI_STATUS_INVALID_PARAMETER returned if the handle passed as a parameter is invalid Other HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function is called by a client to terminate a logical session between the client and the Hot Swap Controller driver, established by the call to HsiOpenSlotControl(). Upon return, the handle is no longer valid. 9.
Slot Control API HSI_STATUS HsiGetBoardPresent( IN HSI_SLOT_CONTROL_HANDLE Handle, IN UINT32 Slot, OUT BOOLEAN *pPresent ) Arguments: Handle - The handle of the current session Slot - The physical slot number pPresent - Pointer to the location where the board presence flag is placed: TRUE means a board is present in the slot; FALSE means no board is present in the slot Return value: HSI_STATUS_SUCCESS if successful HSI_STATUS_INVALID_PARAMETER returned if the physical slot number does not correspond
Slot Control API Slot - The physical slot number pHealthy - Pointer to the location where the board health status is placed: TRUE means the board is present and healthy; FALSE means the board is not healthy Return value: HSI_STATUS_SUCCESS if successful HSI_STATUS_INVALID_PARAMETER returned if the physical slot number does not correspond to any actual slot or if the handle is invalid HSI_STATUS_NO_DATA_DETECTED returned if the board health status cannot be currently determined (the slot is not powered)
Slot Control API HSI_STATUS_INVALID_PARAMETER returned if the physical slot number does not correspond to any actual slot or if the handle is invalid Other HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function detects the power status of the specified slot and returns it in the pPower parameter as a logical value: TRUE means the slot power is on; FALSE means the slot power is off. 9.
Slot Control API HSI_STATUS HsiGetSlotReset( IN HSI_SLOT_CONTROL_HANDLE Handle, IN UINT32 Slot, OUT BOOLEAN *pReset ); Arguments: Handle - The handle of the current session Slot - The physical slot number pReset - Pointer to the location where the slot reset status is placed: TRUE means the slot is in the reset state; FALSE means the slot is not in the reset state Return Value: HSI_STATUS_SUCCESS if successful HSI_STATUS_NOT_IMPLEMENTED returned if slot reset functionality is not implemented for the
Slot Control API Reset - The new reset state for the slot: TRUE means the slot is placed in the reset state; FALSE means the slot is taken out of the reset state Return Value: HSI_STATUS_SUCCESS if successful HSI_STATUS_NOT_IMPLEMENTED returned if slot reset functionality is not implemented for the given platform HSI_STATUS_INVALID_PARAMETER returned if the physical slot number does not correspond to any actual slot or if the handle is invalid Other HSI_STATUS values returned if other errors occurred dur
Slot Control API HSI_STATUS_INVALID_PARAMETER returned if the physical slot number does not correspond to any actual slot or if the handle is invalid Other HSI_STATUS values returned if other errors occurred during execution of this function Synopsis: This function detects the state of the M66EN signal line for the specified slot (reflecting whether 66 MHz operation is enabled for the specified slot) and returns it in the pM66Enable parameter as a logical value: TRUE means that the signal is asserted (66 M
Slot Control API This function controls the state of the M66EN signal line for the specified slot (reflecting whether or not 66 MHz operation is enabled for the specified slot), depending on the value of the parameter M66Enable. M66Enable = TRUE means that the signal line is not driven by the Hot Swap Controller (potentially enabling 66 MHz operation for the slot); M66Enable = FALSE means that the signal is driven low by the Hot Swap Controller (disabling 66 MHz operation for the slot). 9.
Slot Control API The callback function has the following prototype: VOID (*HSI_SLOT_EVENT_CALLBACK)( IN void *pContext, IN BOOLEAN HscError, IN HSI_SLOT_EVENT_INFO *pSlotInfo ); The arguments have the following semantics: pContext - Opaque context pointer. This is the same value that was originally passed to HsiSetSlotEventCallback(). HscError - The value TRUE indicates that a hardware error has been detected in the Hot Swap Controller, and FALSE indicates a state change in one of the slots.
Demonstration Utilities 10 The purpose of the demonstration utility is to demonstrate and expose the main functionality and features of the HSSDK driver set, the Application Programming Interface (API) and the Redundant Host (RH) capabilities of the ZT 5524 System Master CPU board. It also serves as a test tool for exercising the APIs while acting as a programming tutorial. The functional interfaces are listed below: • • • • • • 10.
Demonstration Utilities • Cluster A Standby Host is a host that does not control a bus domain. A Standby Host is referred to as being in Standby mode. An Active Host is a host that owns at least one bus segment. Functionality such as software initiated handovers, hardware initiated failovers, switchovers, event reporting and alarms are exercised. 10.1.2.
Demonstration Utilities 10.1.2.4 Switchover Functions The RHDemo exposes the following functionality: • • • • • • • 10.1.2.5 Prepare for Switchover Cancel Prepare for Switchover Get Domain Software Connection Status Get Slot Software Connection Status Perform Switchover Set Hardware Concession Host Get Hardware Concession Host Host Domain Enumeration and Association The RHDemo enumerates hosts and domains, reports host-domain associations and returns useful data on the domains, hosts and slots.
Demonstration Utilities • Enable Switchover Request Notification • Enable Unsafe Switchover Notification • Disable Notification 10.1.3 IPMI Interface The IPMI interface is an important element of the RSS system architecture.
Demonstration Utilities 10.1.4 Hot Swap Interface The basic purpose of the CompactPCI hot swap functionality is to allow orderly insertion or extraction of CompactPCI boards without affecting operation of the system involved. The hot swap interface in this demo operates under Linux* and VxWorks*. The current demo version does not support hot swap functionality. However, this new HS module does demonstrate manipulation of the Hot Swap API (HS API). 10.1.4.
Demonstration Utilities Table 3. PCI Tree Information Retrieval Flags Flags Meanings PRES Device is present. CONN Device is software connected. CONF Device’s software connection failed. The following information is displayed: • • • • • • • • • • 10.1.4.1.
Demonstration Utilities • Slot state flags If the slot is not empty, the following fields are also present: • • • • • • • • 10.1.4.3 Vendor ID Device ID Subsystem vendor ID Subsystem ID Revision ID Class, subclass, programming interface Header type HS-CSR, if any Slot State When the slot information structure is filled in as a result of a call, the HsStateFlags field contains a set of flags representing the current state of the slot. The following flags are defined: Table 5.
Demonstration Utilities 102 High Availability Software for the Intel® NetStructureTM ZT 4901 Technical Product Specification
Software Installation Software Installation A.1 A Linux The Redundant Host software package in Linux is broken out into two RPM packages. To achieve full Hot Swap Redundant Host capability, both packages must be installed. The packages can be installed individually if only specific functionality is required. In order for the Redundant Host functionality to be enabled properly the Hot Swap Kit for Linux must first be installed. See the Intel® NetStructure™ Hot Swap Kit for Linux 2.
Software Installation 2. Patch and rebuild the kernel with Redundant Host Support, then copy this kernel image to the / usr/src/redhat/BUILD/CompactPCI-RH-1.0/kernel_patches directory as “linux-” (for example, “linux-2.4.18”). The binary RPM spec includes this kernel image. 3. Make any appropriate edits to the RHSK configuration files 4. Reboot A.3 Installing RH Source RPM A.3.1 Source Installation Make sure you have administrator login privileges: bash# rpm –iv CompactPCI-RH-1.0-1.src.
Software Installation CONFIG_RH options also must be enabled. The Redundant Host Software is dependant on both the Hot Swap support and IPMI drivers to be enabled. bash# bash# bash# bash# A.3.4 make make make make dep install modules modules.dep Making RH Configuration Changes Make any required changes to the following files located under the /usr/src/redhat directory (see Section A.4, “Configuring the Redundant Host Infrastructure” on page 105 for more information): BUILD/CompactPCI-RH-1.
Software Installation A.4.3 /CompactPCI-RH-1.0/app/lib After building all the projects in the application subdirectory, this directory contains the following shared object modules and library: • • • • libIpmiApi.so libRhApi.so libSlotCntrlApi.so libBrandsHatch.a The shared object modules provide dynamically linkable access to the exposed IPMI, Redundant Host, and Slot Control APIs, while the library provides a statically linkable entity for the Redundant Host applications.
Redundant Host Function Return Values B HSI_STATUS_SUCCESS The specified operation completed successfully. HSI_STATUS_BUS_NOT_FOUND The operation failed because the required bus was not found. HSI_STATUS_BUS_RESET The operation failed because the required bus was in reset. HSI_STATUS_BUS_SEG_NOT_CONTOLLED The operation failed because the required bus segment was not controlled by the local Host. HSI_STATUS_BUSY The operation failed because the device was busy with some other operation.
Redundant Host Function Return Values 108 HSI_STATUS_DEVICE_SEARCH_FAILED The search for this device failed to be resolved. This does not mean that the device does not exist, but simply that the Universal PCI table located in the querying Host did not resolve this search. HSI_STATUS_FAILED_DEVICE_EXTRACT_SEND The PCI Configuration Module was unable to successfully send a device extraction message from the detecting Host to the Host that has no visibility of the backplane device.
Redundant Host Function Return Values HSI_STATUS_INVALID_PARAMETER The specified operation could not be completed because one or more input parameters were not valid. Examples: • NULL pointer • PCI bus number greater than 255 • Slot number out of range • Malformed Subsystem ID mask for the Alternate HS_CSR Interface HSI_STATUS_NO_DATA_DETECTED The specified operation could not be completed because no meaningful data could be returned to the caller as the result of the operation.
Redundant Host Function Return Values HSI_STATUS_NOT_AVAILIBLE The specified operation could not be completed because necessary functionality was not available at the time of the call. HSI_STATUS_NOT_READY The specified operation failed because the device was not ready to handle the operation HSI_STATUS_NOT_SUPPORTED The specified operation is not supported HSI_STATUS_OBJECT_DOES_NOT_EXIST The specified operation could not be completed because an object specified in input parameters did not exist.
Redundant Host Function Return Values HSI_STATUS_UNABLE_TO_SEND_PACKET The Redundant Host was unsuccessful in sending an interHost message between the redundant system masters. HSI_STATUS_UNSUCCESSFUL_TRANSLATION The Redundant Host driver failed to translate the Slot-Path information for a P2P Bridge into a bus-devicefunction descriptor.
Redundant Host Function Return Values This page intentionally left blank.
HSK Device Driver Interface for VxWorks* 5.4 C The knowledge required to recompile the VxWorks kernel, as well as understand how the HSK device driver integrates with VxWorks, requires a high degree of competency with this operating system in order to gain the most benefit from an RH based system. Whether you are modifying an existing driver or designing the device driver from scratch, adding RH-aware functionality is a straightforward process for an experienced VxWorks programmer.
HSK Device Driver Interface for VxWorks* 5.4 struct _RH_HSK_DRV_OBJ { CB_RH_ADD_DEVICE AddDevice; CB_RH_PNP StartDevice; CB_RH_PNP StopDevice; CB_RH_PNP RemoveDevice; CB_RH_PNP SurpriseRemoval; PRH_DEVICE_INFO DeviceInfo; PRH_DRIVER_EXT DriverObjectExt; } RH_HSK_DRV_OBJ, *PRH_HSK_DRV_OBJ; The DriverExtension contains a pointer to a structure defined by the driver writer, and is context specific to the registered device driver.
HSK Device Driver Interface for VxWorks* 5.4 UINT8 BaseClass; UINT16 SubVendorID; UINT16 SubSystemID; } RH_COMPAT_DEVICE, *PRH_COMPAT_DEVICE; C.3 HSK Driver Instantiation Code Segment The following code segment populates the HA Driver object. Return status validation has been omitted, but an HA device driver should respond appropriately to failed return values.
HSK Device Driver Interface for VxWorks* 5.4 A driver can have itself removed from the HSK Manager’s registry by calling the rhHskUnregisterDriver routine. Use the RH driver object as an input parameter for this routine. If a driver requests to unregister itself and any of the driver’s devices are not in the uninitialized state, the rhHskUnregisterDriver function returns with an unsuccessful status. The following code segment illustrates an AddDevice function.
HSK Device Driver Interface for VxWorks* 5.4 C.4.2 HSI_STATUS StartDevice StartDevice is called for a device driver to commence or resume activity with its associated device. Before this callback is invoked, the device should be fully initialized, and the device driver should be ready to begin hardware interaction.
HSK Device Driver Interface for VxWorks* 5.4 Pointer to a device object. This data is a device context allowing the device driver to identify the specific device whose state is changing to stopped. Device-specific data is located in this object.
HSK Device Driver Interface for VxWorks* 5.4 Syntax ( PRH_DEVICE_OBJ deviceobject ) Parameters deviceobject Pointer to a device object. This data is a device context allowing the device driver to identify the specific device that experienced a surprise removal by an operator. Device-specific data is located in this object.
HSK Device Driver Interface for VxWorks* 5.4 Parameters pci A PCI location structure. This structure contains the PCI bus, device, and function location of the device being associated with the message callback routine. callback This parameter is a callback pointer, registered with the HSK Manager, that is the receiver function for messages being sent to a particular device’s registering device driver. The message callback format is specified in Section C.6.1, “RH_HSK_RH_PROCESS_PACKET” on page 121.
HSK Device Driver Interface for VxWorks* 5.4 C.6.1 RH_HSK_RH_PROCESS_PACKET RH_HSK_RH_PROCESS_PACKET is called when a message packet is being redirected to a device driver for synchronization purposes. The RH driver validates the data packet header in addition to performing a CRC-16 check of the data payload. The payload part of the packet is driver dependent and defined by the driver developer. The RH driver confirms the validity of the CRC-16 value within the packet without validating packet contents.
HSK Device Driver Interface for VxWorks* 5.4 pPackett Pointer to a data packet being sent to the specified device driver. It is the responsibility of the device driver to validate the packet contents. Upon returning from this callback, the RH driver deallocates the data packet. The data packet must be smaller than 1KB.
RH Device Driver Interface for Linux* 2.4 D The High-Availability RH architecture leverages both the capabilities of the native hot-pluggable Linux driver model and the Hot-Swap Kit drivers to offer ultra-quick takeovers while maintaining maximum device serviceability. The Linux hot-pluggable driver model not only provides hot extraction of backplane devices, but also dynamic device insertion.
RH Device Driver Interface for Linux* 2.4 This page intentionally left blank.
Design Guideline for Peripheral VendorsE The following topics present guidelines for designing a device driver for use in the Intel NetStructure Redundant Host environment. E.1 Non Bus Mastering Peripheral Peripheral devices that are not masters p resent no complications for a Redundant Host environment. These devices do not perform data writes into System Master memory. They only request the System Master to read data from the device.
Design Guideline for Peripheral Vendors This page intentionally left blank.
Porting ZT 5550 HA Applications to PICMG 2.12 F The PICMG 2.12 base API (described in Chapter 6) and IPMI replace the functionality of the Host Controller API used with the ZT 5550 system master board. This appendix provides information for porting applications that were written for the ZT 5550 to a PICMG 2.12 based system. The following table summarizes the changes in the functionality.
Porting ZT 5550 HA Applications to PICMG 2.12 RhGetHwDestinationHost, RhPrepareForSwitchover, RhCancelPrepareForSwitchov er, RhGetDomainSwConnectionSt atus, RhGetSlotSwConnectionStatus Event Notification 128 HAEnableNotification RhEnableDomainStateNotificat ion, RhEnableSwitchoverNotificatio n, RhEnableSwitchoverRequestN otification, RhEnableUnsafeSwitchoverNo tification HADisableNotification RhDisableNotification Host Control HAHostControl This functionality has been moved to IPMI.
RH Switchover on OS Crash G The High-Availability RH architecture enables the system master board to perform a switchover to the backup host in the event of a system crash. Under the Linux* operating system the RH Software patches the Linux kernel to perform a switchover whenever a kernel panic occurs. In addition, the host board can be forced to reboot under these circumstances by simply adding the string “panic=1” in the append statement found in the lilo.conf configuration file.
RH Switchover on OS Crash This page intentionally left blank.
Data Sheet Reference H.1 H CompactPCI Information about CompactPCI specifications is available from PICMG* (PCI Industrial Computers Manufacturers Group): https://www.picmg.org/compactpci.stm H.2 User Documentation The latest Intel NetStructure product information and manuals are available on the Intel® NetStructure™ Website. BIOS and driver updates are also available from this site. http:// developer.intel.com/design/network/products/cbp/linecard.htm. H.
Data Sheet Reference This page intentionally left blank.
Index A activation 32 API hot swap 26, 77 IMPI 25 redundant host 25, 37 slot control 26, 85 switchover 61 architecture high availability CPU 11 B backplane 17, 20 baseboard management 26 bridge mezzanine 16 C channel alert destinations 28 channel definitions 27 chassis management 19, 25 code modularity 24 CompactPCI 31, 131 configuration 25 D demonstration utilities 95 device add 34 driver synchronization 35 remove 35 resume operations 34 suspend operations 35 documentation 131 driver 31, 123 design 31 stat
imbOpenDriver 79 imbRegisterForAsyncMsgNotification 82 imbSendIpmiRequest 81 imbSendTimedI2cRequest 80 imbSetLocalBmcAddr 83 imbUnregisterForAsyncMsgNotification 82 Initialization 32 int rhHskUnregisterMsgCallback 120 interface 95 IPMI API 25 API IPMI 79 L Linux 103, 123 M mode active/active 96 active/standby 96 cluster 96 modularity code 24 multiple mode 96 P peripheral vendors 125 PICMG 127 portability 21 PRH_DEVICE_OBJ AddDevice 116 process packet 120 Q quiesced 32 R redundancy 21, 23 redundant host 19,
slot control API 26 software 21 division of labor 22 portability 21 redundancy 21 serviceability 21 switchover 24, 97, 129 forced 62 fully cooperative 61 hardware initiated 63 hostile 63 partially cooperative 62 switchover API 61 system management 19, 25 T Terminology 11 threshold 26 U user interface 95 utilities 95 V VxWorks 106, 131 HSK device driver 113 Intel® NetStructureTM ZT 4901 High Availability Software Technical Product Specification 135
This page intentionally left blank.
