Architecture considerations and best practices for architecting an Oracle RAC solution with Serviceguard and SGeRAC

ManualsBrandsHP ManualsSoftwareHP Serviceguard Extension for RAC (SGeRAC)

Architecture considerations and best

practices for architecting an Oracle RAC

solution with Serviceguard and SGeRAC

Technical white paper

Table of contents

Introduction ......................................................................................................................................... 2

Scope ............................................................................................................................................. 2

Technologies overview ......................................................................................................................... 2

Serviceguard and Serviceguard Extension for RAC (SGeRAC) ............................................................... 2

Serviceguard ............................................................................................................................... 2

Serviceguard Extension for RAC ..................................................................................................... 3

SGeRAC Toolkit ........................................................................................................................... 4

Oracle RAC .................................................................................................................................... 4

Cluster protocol ............................................................................................................................ 5

Networking ................................................................................................................................. 5

Storage ....................................................................................................................................... 6

Architecting an Oracle RAC solution with SGeRAC.................................................................................. 7

Cluster management—who controls what and when? ............................................................................... 8

Network HA configuration choices ................................................................................................... 10

Network high-availability ............................................................................................................ 11

Cluster Interconnect Subnet monitoring .......................................................................................... 12

Public network configuration ........................................................................................................ 13

Private network configuration ....................................................................................................... 14

Configuration Option 1—recommended configuration ....................................................................... 14

Configuration Option 1a ................................................................................................................ 15

Configuration Option 2 .................................................................................................................. 16

Storage management choices .......................................................................................................... 18

Serviceguard CFS ....................................................................................................................... 18

Oracle Disk Manager ..................................................................................................................... 20

Storage Checkpoints ...................................................................................................................... 20

Storage Mapping ........................................................................................................................... 21

Best practices when using CFS for RAC ............................................................................................ 21

SLVM and CVM ......................................................................................................................... 22

Best practices when using SLVM for Oracle RAC ............................................................................... 22

Cluster-wide Device Special Files ..................................................................................................... 23

ASM over SLVM ......................................................................................................................... 23

Criteria for choosing the right storage management for RAC ........................................................... 23

Oracle RAC with SGeRAC Toolkit ........................................................................................................ 24

Why Use the SGeRAC Toolkit? ........................................................................................................ 24

Conclusion ........................................................................................................................................ 26

References ........................................................................................................................................ 27

Summary of content (27 pages)

PAGE 1
Architecture considerations and best practices for architecting an Oracle RAC solution with Serviceguard and SGeRAC Technical white paper Table of contents Introduction ......................................................................................................................................... 2 Scope ............................................................................................................................................. 2 Technologies overview ..............................
PAGE 2
Introduction Beginning with its 10g release, Oracle Real Application Clusters (RAC) requires pre-integrated clusterware from Oracle called Cluster Ready Services or CRS (also referred to as Oracle Clusterware). HP strongly recommends a combination of Oracle Clusterware and Serviceguard/SGeRAC for Oracle RAC solutions on HP-UX 11i.
PAGE 3
Figure 1: Serviceguard Cluster Application packages Node A Application packages Node B Storage links • All systems are physically connected to each disk • Maximum cluster size is 16 nodes • Each application runs on only one host at a time • Hosts can run multiple applications • Failover is possible to any node that is physically connected to the data Application packages Node C Storage links 2-node cluster SAN QS Node 1 QS Node 2 (switches not shown) Serviceguard Extension for RAC Serviceguard E
PAGE 4
Figure 2 shows an example of SGeRAC cluster Figure 2: SGeRAC Cluster Cache Fusion (Ethernet or IPoIB) RAC Node A RAC Node B Storage links NOTE: Cluster heartbeat and Cache Fusion can share the same network RAC Node C Storage links SAN SGeRAC Toolkit The SGeRAC Toolkit provides documentation and scripts to simplify the integration of SGeRAC and the Oracle RAC stack, including facilitating easy configuration of cluster interconnect monitoring.
PAGE 5
CSS manages the Oracle cluster membership and provides its own group membership service to RAC instances. CSS manages the cluster by controlling which nodes are members of the cluster and by notifying other members when a node joins or leaves the cluster. If third-party cluster software such as SGeRAC is used in the environment, CSS utilizes the group membership service provided by the third-party cluster software. CRS manages high availability operation within a cluster.
PAGE 6
The Oracle VIP must be configured on the public network for each RAC instance. Clients use VIP to connect to the RAC database. If a node fails, CRS fails the VIP over to another cluster node to provide an immediate node-down response to the clients’ connection request. This increases the availability of other RAC instances to the clients because the clients no longer have to wait for a network timeout before the connection request fails over to another instance in the cluster.
PAGE 7
Since ASM supports only database data files and log files, the following storage management methods are required for non-database files if ASM is used on HP-UX: • A local file system or Cluster File System (CFS) for Oracle Clusterware binaries and configuration files and RAC binaries and configuration files. • Shared storage for Oracle Clusterware data: voting disk and OCR files. Prior to RAC 11gR2, this storage has to be configured either as shared raw devices, shared raw volumes, or files in the CFS.
PAGE 8
Figure 3 shows the combined stack. Figure 3: Serviceguard and Oracle RAC integrated stack • SGeRAC cluster membership providing RAC membership • HP -UX supported volume managers and file systems to store Oracle and application data – e.g .
PAGE 9
RAC also has its own group membership. When you configure RAC, you choose all nodes or a subset of nodes in the cluster to run RAC instances for a specific RAC database. By default, RAC Instance Membership Recovery (IMR) is set to 17 minutes. This means that when a critical RAC resource fails, RAC will wait for 17 minutes before performing its own recovery.
PAGE 10
RAC: • RAC interconnect network If all RAC interconnect networks fail for a node, RAC evicts from the group the instance running on the impacted node. The time to eviction of the instance is dependent on the IMR value; the default value is 17 minutes. In this case, only the RAC instance is down and not the node; therefore, the cluster is not impacted.
PAGE 11
Since Oracle Clusterware doesn’t support redundant networks and doesn’t monitor NICs, either APA or Serviceguard primary/standby LANs should be used to provide network high availability. Figure 4 shows an example of Serviceguard primary/standby LANs that are used to provide network HA for public and private networks in an Oracle RAC configuration.
PAGE 12
• It allows one NIC to be a standby for multiple NICs which can reduce the total IO slots required relative to the APA. • It supports the failover to different NIC types in the same family, like failover from 100B-T Ethernet to 1 Gb Ethernet. • It monitors all links and can detect the loss of all links and take failover action. HP-UX APA supports two configuration modes: Failover Group and Link Aggregate.
PAGE 13
Public network configuration Oracle only supports one active public network for client connections. Use APA or Serviceguard Local LAN failover to configure network high availability for the public network. This is shown in Figure 5.
PAGE 14
Private network configuration There are a number of choices available when configuring the private network for the combined Oracle RAC and SGeRAC stack. This section discusses several different options and the supported HA configuration for each option. Configuration Option 1—recommended configuration In this configuration, Serviceguard heartbeat, CFS interconnect (if CFS is used), Oracle Clusterware CSS heartbeat, and RAC interconnect are configured on the same network.
PAGE 15
For faster cluster recovery: • Use a common, dedicated HA network for Serviceguard heartbeat, CFS interconnect (if CFS is used), CSS heartbeat, and RAC interconnect. • Use Serviceguard Local LAN failover for network HA. • Configure the Serviceguard heartbeat on the same network as the CSS heartbeat. CSS only supports one active network and Serviceguard heartbeat will fail and cause the node to be evicted if the network used by CSS fails.
PAGE 16
Figure 7 shows the supported HA configuration for the additional Serviceguard heartbeat network.
PAGE 17
Figure 8 shows the supported HA configurations for the two private networks. Either APA Active/Standby or Serviceguard local LAN failover can be configured to provide network HA. In addition, if the RAC interconnect traffic requires higher bandwidth, APA trunking can be configured to combine the bandwidth of multiple NICs in primary/standby configuration along with Serviceguard local LAN failover for HA.
PAGE 18
Storage management choices Beginning with Serviceguard A.11.17, Serviceguard/SGeRAC supports the following storage management options for Oracle RAC: • Serviceguard Cluster File System (CFS) • Shared Logical Volume Manager (SLVM) • VERITAS Cluster Volume Manager (CVM) • ASM with SLVM • ASM with raw disks (on HP-UX 11i v3 or later only) Both SLVM and CVM have been available for quite some time. The Serviceguard CFS storage option and ASM with SLVM option are available starting with Serviceguard A.11.17.
PAGE 19
The CFS also simplifies backup and recovery in an RAC environment. Without CFS, the archive logs of each instance are stored in a local file system on the node where the instance is running. • Performing a backup of the database requires access to archive logs of all instances. The common method would be to use Network File System (NFS). The node running a backup job would use NFS to mount all instances’ archive log file systems.
PAGE 20
Oracle Disk Manager One of the major key features provided with Serviceguard SMS CFS for RAC is the database accelerator, ODM. ODM is a standard API specified by Oracle for database I/O. The VERITAS implementation of ODM improves performance by providing the database direct access to the underlying storage without passing through the actual file system interface. The resulting performance is equivalent to performance when using raw devices.
PAGE 21
Storage Mapping The Storage Mapping feature enables the mapping of data files to physical devices. Access to mapping information allows for a detailed understanding of the storage hierarchy in which files reside. This information is critical to effectively evaluating I/O performance. The VERITAS Storage Mapping option supports the set of storage APIs called Oracle Mapping (ORAMAP) that lets Oracle determine the mapping information for files and devices.
PAGE 22
Oracle RAC supports the use of CFS for Oracle Clusterware software and Oracle RAC software. When the software is installed on CFS, node-specific directories are created to hold log and trace files that are generated from Oracle Clusterware processes and the RAC instance running on each node. CAUTION: All the log and trace files from all nodes in the cluster are stored in a shared CFS mount point; therefore, the file system can fill up quickly.
PAGE 23
Cluster-wide Device Special Files Using Cluster-wide Device Special Files (cDSF) provides additional clarity and simplicity when planning and implementing an LVM storage configuration on HP-UX. Cluster-wide Device Files, available starting with Serviceguard A.11.20, ensure that each storage device used by a cluster has a unique device file name across all nodes. For more information, see Managing Serviceguard eighteenth edition. ASM over SLVM Starting with SGeRAC A.11.
PAGE 24
Table 2 shows a high-level comparison between the solutions. Table 2.
PAGE 25
Beginning with Oracle 10.1.0.4, Oracle made improvements in coordination between Oracle Clusterware and third-party clusterware, enabling the use of SGeRAC packages to manage the storage needs of Oracle Clusterware and RAC database instances: • Support for the on-demand startup and shutdown of Oracle Clusterware 23 and RAC database instances 24 The startup and shutdown of Oracle Clusterware—as HP-UX is taken up to init level 3 and taken down to a lower level—can be performed automatically or on demand.
PAGE 26
Conclusion The following are highlights of HA features provided by Serviceguard and SGeRAC for Oracle RAC: • Online node addition and replacement. • Advanced I/O fencing mechanism provides data integrity protection among nodes both inside and outside of the cluster. • I/O fencing for a single instance DB or for the application itself (Oracle Clusterware does not provide this). • Ability to secure multiple paths to the data storage array (Oracle relies on the O/S vendor for this).
PAGE 27
References HP Documentation: http://www.hp.com/go/hpux-serviceguard-docs • Managing Serviceguard, 18th Edition, September 2010 • Using Serviceguard Extension for RAC, 9th Edition, September 2010 • Oracle Database 11g Release 2 Real Application Clusters with SLVM/RAW on HP-UX Installation Cookbook • Sample Configurations with SGeRAC and Oracle RAC 10gR2, March 2009 • VERITAS Storage Foundation™ 5.