Sample Configuration with HP Serviceguard Extension for RAG and Oracle Real Application Clusters 11g release 2 using Cluster File System

ManualsBrandsHP ManualsSoftwareHP Serviceguard Extension for RAC (SGeRAC)

Sample configuration with

HP Serviceguard Extension for RAC and

Oracle Real Application Clusters 11g

release 2 using Cluster File System

Technical white paper

Table of contents

Introduction ......................................................................................................................................... 3

Audience ........................................................................................................................................ 3

Terms and definitions ........................................................................................................................ 3

Reference materials .......................................................................................................................... 4

Configuration planning considerations ................................................................................................... 4

Capacity planning ........................................................................................................................... 5

Failover time requirements ................................................................................................................. 5

Planning for high availability ................................................................................................................. 6

Public network HA............................................................................................................................ 6

Virtual IP address (VIP) HA ................................................................................................................ 7

Network HA for cluster communication ............................................................................................... 7

Storage high availability ................................................................................................................. 11

Multiple nodes ............................................................................................................................... 11

Power considerations ...................................................................................................................... 11

Storage requirements ......................................................................................................................... 11

OC requirements: OCR and voting disk ............................................................................................ 12

RAC database files ......................................................................................................................... 12

Oracle binaries (Grid Infrastructure and RAC database) ..................................................................... 12

Recovery files ................................................................................................................................ 12

Sample configuration prerequisites ...................................................................................................... 13

Software ....................................................................................................................................... 13

Server hardware requirement .......................................................................................................... 13

Network requirements .................................................................................................................... 13

Storage requirements ...................................................................................................................... 13

Cluster for CFS (ccia1 and ccia2) .................................................................................................... 14

Sample configuration ......................................................................................................................... 14

Assumptions for this sample configuration ......................................................................................... 14

Create a SGeRAC cluster with CFS for Oracle 11g R2 ....................................................................... 15

Prerequisites for Oracle 11g R2 (sample installation) .......................................................................... 18

Install and configure Oracle 11g R2 Grid Infrastructure on a local file system ....................................... 21

Install Oracle RAC 11g R2 Database binary on CFS .......................................................................... 21

Configure Oracle Disk Manager (ODM) ........................................................................................... 22

Create RAC demo database on CFS ................................................................................................ 23

Configure SG packages with SGeRAC ............................................................................................. 24

Start and stop cluster ...................................................................................................................... 27

Summary of content (31 pages)

PAGE 1
Sample configuration with HP Serviceguard Extension for RAC and Oracle Real Application Clusters 11g release 2 using Cluster File System Technical white paper Table of contents Introduction ......................................................................................................................................... 3 Audience ........................................................................................................................................ 3 Terms and definitions.............
PAGE 2
Appendix ......................................................................................................................................... 29 Sample configuration for CFS cluster ................................................................................................ 29 Package configuration for CFS (OC MNP) ........................................................................................ 29 Package configuration for CFS (Oracle RAC MNP) ...............................................
PAGE 3
Introduction This document discusses the various aspects of planning and implementing an Oracle Real Application Clusters 11g release 2 (RAC 11g R2) solution on HP-UX 11i v3 with HP Serviceguard Extension for RAC version A.11.19 (SGeRAC A.11.19). This document also presents a sample configuration for RAC 11g R2 with SGeRAC A.11.19 and Cluster File System (CFS) 5.0.1 with HP Serviceguard Storage Management Suite A.03.00 (SG SMS A.03.00).
PAGE 4
Term Definition MNP Multi-node package—a Serviceguard package that runs on multiple nodes at the same time, and can be independently started and halted on individual nodes.
PAGE 5
Capacity planning Proper capacity planning helps ensure that sufficient resources are available to meet expected service levels. Processor capacity In a cluster environment, cluster members maintain heartbeat traffic between nodes. Insufficient CPU processing capacity affects application performance. A longer timeout means it takes longer to reconfigure a cluster, and more time is needed before a node failure or network failure can be detected.
PAGE 6
Service (RAC-DB-IC) failure is discovered, the speed of recovery actions by impacted components— for example, SG, Group Membership Service (GMS), Cluster Synchronization Service (CSS), and/or RAC)—and database recovery time. – For a complete SG cluster interconnect failure, SG sees the failure within the MEMBER_TIMEOUT timeframe.
PAGE 7
Virtual IP address (VIP) HA This section describes high availability for VIP. SG local LAN failover mechanism—preferred choice For client public network HA in SGeRAC configurations, the preferred network HA method is to use SG primary and standby. SG monitors the redundant network, so additional auto-port aggregation (APA) software is not required. This mechanism is fastest to detect and recover from network failure. It can use a single standby LAN as a standby for multiple primary LANs.
PAGE 8
The network traffic between cluster nodes can be categorized as follows: • SG-HB–SG-HB and communications traffic—supported over single or multiple subnets • CSS-HB–CSS-HB traffic and communications traffic for OC—use a single logical connection over a single subnet network • RAC-DB-IC–RAC instance peer-to-peer traffic and communications for Global Cache Service (GCS) and Global Enqueue Service (GES), formerly Cache Fusion (CF) and Distributed Lock Manager (DLM)—network HA is provided by the HP-UX 11i platf
PAGE 9
Figure 1. Preferred: single subnet with Ethernet primary and standby, including GAB/LLT Figure 1 shows a common configuration where SG-HB, CSS-HB, RAC-DB-IC, and GAB/LLT share the same network for cluster communications. This configuration is recommended and suitable for most installations where the RAC traffic does not saturate the network and negatively impact other components (SG-HB, CSS-HB, and GAB/LLT traffic). There is one network with sufficient bandwidth.
PAGE 10
Figure 2 shows a configuration that addresses the possibility that heavy RAC-DB-IC traffic may affect SG-HB, illustrating one way to overcome the heavy RAC-DB-IC traffic affecting the SG-HB. This configuration uses two SG-HB networks. With multiple SG-HB networks, SG MEMBER_TIMEOUT can be set as short as the systems allow, enabling faster cluster reformation. The subnet for the CSS-HB and RAC-DB-IC has redundancy with the primary and standby adapters and switches that are configured.
PAGE 11
Figure 3 is a variation of figure 2 that shows the CSS-HB and RAC-DB1-IC residing on the same subnet as the SG-HB. The RAC-DB2-IC is for a second database and is on a separate network. It does not affect the HB and RAC-DB1-IC traffic. If the primary (LAN1) fails, SG performs a local LAN failover. If both the primary (LAN1) and standby (LAN2) fail, the RAC IMR reforms and evicts the affected database instance.
PAGE 12
OC requirements: OCR and voting disk OCR Each OCR requires at least 300 MB of disk space. The OCR must be shared and accessible by all cluster nodes. OC uses the OCR to keep Oracle cluster information and cluster database configuration information. OCR is also used to keep track of processes OC controls. Voting disk Each voting disk requires at least 300 MB of disk space. As with the OCR, the voting disk must be shared and accessible by all cluster nodes.
PAGE 13
Sample configuration prerequisites In the sample configuration, the following prerequisites apply: Software • HP-UX 11i v3 1003 High Availability Operating Environment (HAOE) or later (Oracle RAC 11g R2 is not supported on HP-UX 11i v2) • Serviceguard version A.11.19 (SG A.11.19) • Serviceguard Extension for RAC version A.11.19 (SGeRAC A.11.19) • HP Serviceguard Management Suites (SG SMS) bundle A.03.
PAGE 14
– /dev/dsk/c4t1d0 /dev/rdsk/c4t1d0 – /dev/dsk/c4t2d0 /dev/rdsk/c4t2d0 – /dev/dsk/c4t3d0 /dev/rdsk/c4t3d0 • Redundant link: – /dev/dsk/c5t0d0 /dev/rdsk/c5t0d0 – /dev/dsk/c5t1d0 /dev/rdsk/c5t1d0 – /dev/dsk/c5t2d0 /dev/rdsk/c5t2d0 – /dev/dsk/c5t3d0 /dev/rdsk/c5t3d0 Cluster for CFS (ccia1 and ccia2) Figure 4. Cluster for CFS Sample configuration The following sections describe the process for configuring Oracle RAC 11g R2 on CFS. Assumptions for this sample configuration 1.
PAGE 15
6. There are two private disks (one disk per node for the local file system that is used for local Oracle storage). Create a SGeRAC cluster with CFS for Oracle 11g R2 In this sample, both the Oracle RAC software and data files reside on CFS. The Oracle Clusterware binaries home is on a local file system.
PAGE 16
This value might be adjusted to a lower value for faster cluster reformation after careful consideration of the guidelines in the Serviceguard user guide: http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02437444/c02437444.pdf Create cluster (sample) # cmapplyconf –C cluster.
PAGE 17
Create disk group multi-node package Add the disk group to the cluster.
PAGE 18
/dev/vx/dsk/cfsdg1/vol1 10485760 /dev/vx/dsk/cfsdg1/vol2 10485760 /dev/vx/dsk/cfsdg1/vol3 1048576 36455 9796231 0% /cfs/mnt1 36455 9796231 0% /cfs/mnt2 34142 951039 3% /cfs/mnt3 View configuration # cmviewcl CLUSTER cluster_ccia STATUS up NODE ccia1 ccia2 STATUS up up STATE running running MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-MP-2 SG-CFS-MP-3 STATUS up up up up up STATE running running running running running AUTO_RUN enabled enabled enabled enabled enabled
PAGE 19
Create the Oracle user home and modify the Oracle user to the new home directory on each node. # mkdir /home/oracle # # chown oracle:oinstall /home/oracle usermod –d /home/oracle oracle Change password on each node # passwd oracle Set up ssh without a password for the Oracle user on all nodes—alternatively, this can be done during Oracle 11g R2 Clusterware installation using Oracle installer.
PAGE 20
Create Oracle Cluster Software home directory For installing Oracle Cluster Software on a local file system, create the directories on each node. # # # mkdir chown chmod –p –R -R /u01/app/11.2.0/crs oracle:oinstall /u01/app/11.2.0/crs 775 /u01/app/11.2.0 The Oracle Cluster Software home directory is as follows: /u01/app/11.2.0/crs/ When installing Oracle Cluster Software, you should set the ORACLE_HOME environment to specify this directory. Please note, at installation, and before running the root.
PAGE 21
Change directory permission on each node (if needed). # chmod 775 /cfs Install and configure Oracle 11g R2 Grid Infrastructure on a local file system Oracle 11g R2 Grid Infrastructure contains OC. Log in as “oracle” user. $ export DISPLAY=:0.0 $ cd <11gR2 Cluster Software disk directory> $ ./runInstaller Notes: 1. Specify inventory directory as “/u01/app/oraInventory” and operating system group name as “oinstall”. 2. Specify Cluster Ready Services (CRS) HOME as /u01/app/11.2.0/crs.
PAGE 22
3. When prompted, run root.sh on each node. Configure Oracle Disk Manager (ODM) ODM is required when Oracle RAC is used with SGeRAC and CFS. For this configuration example, the ODM libraries are included with the HP SG SMS bundle for RAC. Check ODM availability 1. Check the VRTSdbed license. # /opt/VRTS/bin/vxlictest -n “VERITAS Storage Foundation for Oracle” - f “ODM” ODM feature is licensed 2. Check that the VRTSodm package is installed. # swlist VRTSodm VRTSodm VRTSodm.ODM-KRN VRTSodm.
PAGE 23
Create RAC demo database on CFS Export environment variables for “oracle” user export ORACLE_BASE=/cfs/mnt1/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1 export ORA_CRS_HOME=/u01/app/11.2.
PAGE 24
12. Configure Database Storage as needed. 13. Create Database. Configure SG packages with SGeRAC Starting and stopping OC in a SG package is recommended; doing so will help ensure that OC will start after SGeRAC is started and stop before SGeRAC is halted. SG packages should also be used to synchronize storage activation and deactivation with Oracle Cluster Software and RAC instances.
PAGE 25
3. Edit the pkgConfigFile package configuration file. Because the OCR and voting disk reside on CFS, the OC package is configured to depend on the CFS package. package_name crsp package_type node_name node_name multi_node ccia1 ccia2 TKIT_DIR TKIT_SCRIPT_DIR ORA_CRS_HOME /etc/cmcluster/crsp /opt/cmcluster/SGeRAC/toolkit/crsp /u01/app/11.2.
PAGE 26
ORACLE_HOME ORACLE_USER /cfs/mnt1/oracle/product/11.2.0/db_1 oracle dependency_name SG-CFS-MP-1-DEP dependency_condition dependency_location SG-CFS-MP-1=up same_node dependency_name SG-CFS-MP-2-DEP dependency_condition dependency_location SG-CFS-MP-2=up same_node dependency_name crsp-DEP dependency_condition dependency_location crsp=up same_node service_name service_cmd service_restart racp_monitor "$SGCONF/scripts/sgerac/erac_tk_rac.
PAGE 27
Running package racp on node ccia1 Successfully started package racp on node ccia1 Running package racp on node ccia2 Successfully started package racp on node ccia2 cmrunpkg: All specified packages are running # cmviewcl CLUSTER cluster_ccia NODE ccia1 ccia2 STATUS up STATUS up up STATE running running MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-MP-2 SG-CFS-MP-3 crsp racp STATUS up up up up up up up STATE running running running running running running running AUTO_RUN ena
PAGE 28
Start and stop Oracle RAC 11g R2 instance Oracle RAC packages can be started and stopped in order to start RAC DB instances. When halting the SG cluster (cmhaltcl –f), the package dependencies ensure that the Oracle RAC package is halted before the CFSs are unmounted and the OC package gets halted.
PAGE 29
Appendix Sample configuration for CFS cluster The following sections describe sample configurations with CFS: Cluster configuration file for cluster (ccia1 and ccia2) CLUSTER_NAME QS_HOST cluster_ccia fender QS_POLLING_INTERVAL QS_TIMEOUT_EXTENSION 120000000 0 NODE_NAME NETWORK_INTERFACE STATIONARY_IP NETWORK_INTERFACE NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE NODE_NAME NETWORK_INTERFACE STATIONARY_IP NETWORK_INTERFACE NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE AUTO_START_TIMEOUT NETWORK_PO
PAGE 30
Package configuration for CFS (Oracle RAC MNP) package_name package_type node_name node_name racp multi_node ccia1 ccia2 TKIT_DIR TKIT_SCRIPT_DIR /etc/cmcluster/racp /opt/cmcluster/SGeRAC/toolkit/racp ORACLE_DBNAME ORACLE_HOME ORACLE_USER orcl /cfs/mnt1/oracle/product/11.2.
PAGE 31
For more information www.hp.com/go/serviceguardsolutions www.hp.com/go/sgerac References HP documentation All of the following materials can be found on the HP Business Support Center. • HP Serviceguard Storage Management Suite Version Release Notes http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?lang=en&cc=us&taskId=101&pr odClassId=10008&contentType=SupportManual&docIndexId=64255&prodTypeId=18964&prodSeriesId=41 62092 • Serviceguard Version A.11.19 Release Notes http://bizsupport1.