Best Practices for SGeRAC and Oracle RAC on HP-UX 11i, March 2009

ManualsBrandsHP ManualsSoftwareHP Serviceguard Extension for RAC (SGeRAC)

Introduction ...................................................................................................................................2

Audience...................................................................................................................................2

Terms and definitions...................................................................................................................2

Basic configuration......................................................................................................................3

Configurations for Oracle RAC.........................................................................................................4

Public network............................................................................................................................4

VIP high availability.................................................................................................................5

Network for cluster communication.................................................................................................6

General principles...................................................................................................................6

Common configuration.............................................................................................................7

Alternate configuration – multiple RAC databases .........................................................................8

Alternate configuration – fast reconfiguration with low heartbeat timeout.........................................10

Guidelines for changing cluster parameters................................................................................11

Storage...................................................................................................................................12

CFS.....................................................................................................................................13

CVM...................................................................................................................................15

SLVM ..................................................................................................................................15

ASM over raw devices ...........................................................................................................15

ASM over SLVM....................................................................................................................16

Serviceguard packages..............................................................................................................16

Serviceguard Extension for RAC Toolkit.....................................................................................17

Standard packages................................................................................................................18

Extended Distance Cluster for RAC...............................................................................................18

OCR and Voting Disk.............................................................................................................18

Binaries ...............................................................................................................................18

Database data files................................................................................................................18

Database recovery data..........................................................................................................18

Conclusion..................................................................................................................................18

Document revision history...............................................................................................................19

References...................................................................................................................................19

HP documentation.....................................................................................................................19

Oracle documentation ...............................................................................................................19

For more information.....................................................................................................................21

Best Practices for SGeRAC and Oracle RAC

on HP-UX 11i

March 2009

Summary of content (21 pages)

PAGE 1
Best Practices for SGeRAC and Oracle RAC on HP-UX 11i March 2009 Introduction ................................................................................................................................... 2 Audience................................................................................................................................... 2 Terms and definitions...................................................................................................................
PAGE 2
Introduction This paper describes common best practices for configuring clusters with Serviceguard (SG), Serviceguard Extension for RAC (SGeRAC), Oracle® Clusterware, and Oracle® Real Application Cluster (RAC).
PAGE 3
• RIP – Serviceguard Relocatable IP Address user for client application access and failovers with package • SG-HB – Serviceguard Heartbeat traffic. • SGeRAC – Serviceguard Extension for RAC extends Serviceguard to support Oracle RAC. • SLVM – Shared Logical Volume Manger allows multi-system shared access to LVM volumes for RAC. • VIP – Virtual IP address is used by OC to configure access to Oracle clients and for remote failover to reject client connections.
PAGE 4
Figure 1.
PAGE 5
Client failover is needed when failures affect existing and new client sessions. These failures include node failures (e.g. power failures) and network failures (e.g. all redundant network interface or links failed). Protection is available at three levels: 1. Oracle Fast Application Notification (FAN) 2. Remote virtual IP address (VIP) fail over 3. Client connect timeout Clients that are FAN integrated or using the FAN API may interrupt existing sessions and failover.
PAGE 6
Remote failover In the case of a catastrophic failure such as node failure or network failure, OC fails over the VIP address to a surviving node. Network for cluster communication The network for cluster communication is used for private internal cluster communications. Although one network adapter per node is sufficient for the private network, HP recommends that a minimum of two network adapters for each network to be used for higher availability.
PAGE 7
• RAC GCS (cache fusion) traffic may be very high, so a separate network for RAC-DB-IC may be needed.3 One RAC-DB-IC may interfere with another RAC-DB-IC on the same cluster. RAC-DB-IC may also interfere with heartbeat traffic. • Some networks are not supported by CFS/CVM, so the RAC-DB-IC traffic may be on a separate network. • Certain kinds of fast re-configurations require a minimum of two Serviceguard heartbeat networks, whereas CSS-HB and RAC-DB-IC do not support multiple network for HA purposes.
PAGE 8
When both the primary and standby interfaces fail, Serviceguard resolves the interconnect failure by performing a cluster reconfiguration. After Serviceguard completes its reconfiguration, SGeRAC notifies CSS and CSS updates RAC. Serviceguard allows addition and deletion of subnets. When subnets are changed, be sure to keep the SG-HB on the same network as CSS-HB and RAC-DB-IC to maintain the monitoring behavior.
PAGE 9
Figure 3. Alternative configuration – multiple RAC databases SG-HB CSS-HB RAC-DB1-IC LAN 1 Private (standby) LAN 2 LAN 3 Private (primary) RAC-DB2-IC Private (primary) LAN 4 Node A LAN 1 LAN 2 LAN 3 LAN 4 Private (standby) Node B Timeouts The SG-HB timeout (MEMBER_TIMEOUT) should be set based on service availability requirements.
PAGE 10
Alternate configuration – fast reconfiguration with low heartbeat timeout Figure 4. Alternative configuration – low Serviceguard member timeout SG-HB1 CSS-HB RAC-DB1-IC LAN 1 Private (standby) LAN 2 LAN 3 Private (primary) SG-HB2 Private (primary) LAN 4 Node A LAN 1 LAN 2 LAN 3 LAN 4 Private (standby) Node B When RAC-DB-IC traffic is very high and SG-HB timeout is low, there is a probability of RAC-DB-IC traffic interfering with SG-HB traffic and causing unnecessary timeouts.
PAGE 11
limitation where if all interconnect fails (e.g. primary and standby switch failed at the same time), all nodes are halted. If there is a concern with simultaneous failure of both switches, Serviceguard supports multiple standby and additional standby switches may be added. Without subnet monitoring, double networks failures on the CSS-HB network will be discovered by CSS-HB timeout and CSS determines which node to evict. Starting with A.11.
PAGE 12
Note 2: SLVM timeout would be documented in the LVM Whitepaper entitled LVM Link and Node Failure Recovery Time found on the HP Technical Documentation site at docs.hp.com.
PAGE 13
For Oracle Clusterware data files, starting with 10.2.0.2, Oracle introduced a CSS disktimeout to capture hang I/O hangs; however, for configurations with SGeRAC, the disktimeout is automatically set to three seconds lower than CSS MISSCOUNT and is only indirectly configurable based on CSS MISSCOUNT. Setting I/O timeouts for SLVM or CVM does not affect disktimeout. If CSS receives I/O error, CSS attempts to reopen Oracle Clusterware data files until disktimeout is reached.
PAGE 14
Note that if you use CFS for Oracle data files, the Oracle Disk Manager (ODM) library from Symantec must be enabled to get near raw I/O performance. When Oracle Clusterware home and/or Oracle database home is on CFS, a CFS file system full problem may negatively impact Oracle Clusterware and/or Oracle database home on all nodes, e.g. unable to log or dump.
PAGE 15
For online changes, CFS/CVM supports online reconfiguration of disk groups, volumes, file system, and nodes. For shared storage access failure, the CVM I/O timeout default value is 30 seconds for each link.
PAGE 16
The preferred storage configuration, with a minimum of two ASM disk groups per database, is as follows: • VG/local FS for Oracle Clusterware home • VG/local FS for Oracle home • Raw devices for OCR and Voting Disk (can also use SLVM VG) • For each RAC database o Two ASM disk groups o One for database data and the other for flash recovery ASM over SLVM SGeRAC supports ASM over SLVM. The ASM disk groups members must be raw logical volumes (LV) managed by SLVM.
PAGE 17
Storage required by OC must be available before OC starts, and similarly, storage required by RAC database must be available before the RAC instance starts. There is a need to coordinate to ensure the combined stack starts up and shuts down in the proper sequence, and a need to automate startup and shutdown sequences, if desired. HP recommends that you use Serviceguard packages to coordinate between SG/SGeRAC and OC/RAC.
PAGE 18
configured with a dependency on the CFS MNP. The Oracle Clusterware MNP ensures the required storage is available before signaling the start of Oracle Clusterware. Each RAC database MNP ensures the storage for the database is started before starting the database instance. The init process monitors Oracle Clusterware. Oracle Clusterware monitors the database instance. The MNP monitoring is to the extent that the MNP status reflects the status of what it has started. The MNP does not restart the application.
PAGE 19
Document revision history Date Description March 2009 Updated for SG/SGeRAC A.11.19 March 2008 Updated document for general Oracle Clusterware and RAC usage June 2007 Updated recommendation for Oracle Clusterware home and Oracle database home when using CFS March 2007 Updated document with Cluster interconnect configurations, Cluster interconnect subnet monitoring, SGeRAC Toolkits, and Extended Distance Cluster Added multiple RAC database configurations.
PAGE 20
• Oracle Real Application Clusters Installation Guide 11g Release 1 (11.1) for Linux and UNIX, B28264-03, November 2007 http://download.oracle.com/docs/cd/B28359_01/install.111/b28264.pdf • Oracle Clusterware Administration and Deployment Guide 11g Release 1 (11.1), B28255-03, September 2007 http://download.oracle.com/docs/cd/B28359_01/rac.111/b28255.pdf • Oracle Real Application Clusters Administration and Deployment Guide 11g Release 1 (11.1), B28254-04, November 2007 http://download.oracle.
PAGE 21
For more information To learn more about HP Serviceguard Solutions for HP-UX 11i, please visit: www.hp.com/go/serviceguardsolutions For technical documentation: www.docs.hp.com/hpux/ha © Copyright 2006-2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services.