Using Serviceguard Extension for RAC Version A.11.
Legal Notices © Copyright 2013 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Advantages of using SGeRAC.........................................................................9 Preface......................................................................................................10 Where to find Documentation on the Web......................................................12 1 Introduction to Serviceguard Extension for RAC............................................13 What is a Serviceguard Extension for RAC Cluster? ..................................................
Network Monitoring...........................................................................................................29 SGeRAC Heartbeat Network..........................................................................................29 CSS Heartbeat Network.................................................................................................29 RAC Cluster Interconnect................................................................................................29 Public Client Access...
Mirror Detachment Policies with CVM..........................................................................56 Using CVM 5.x.................................................................................................................56 Preparing the Cluster for Use with CVM 5.x......................................................................56 Starting the Cluster and Identifying the Master Node....................................................57 Converting Disks from LVM to CVM.................
SGeRAC Toolkit Package Cleanup ......................................................................................95 Additional Information........................................................................................................95 3 Support of Oracle RAC ASM with SGeRAC.................................................96 Introduction............................................................................................................................
Quorum Server Status..................................................................................................121 CVM Package Status...................................................................................................121 Status After Moving the Package to Another Node...........................................................122 Status After Package Switching is Enabled......................................................................123 Status After Halting a Node.................
Keeping Kernels Consistent...........................................................................................145 Example of Rolling Upgrade .............................................................................................145 Limitations of Rolling Upgrades .........................................................................................149 Non-Rolling Software Upgrades.............................................................................................
Advantages of using SGeRAC HP Serviceguard Extension for RAC (SGeRAC) amplifies the availability and simplifies the management of Oracle Real Application Cluster (RAC). SGeRAC allows you to integrate Oracle RAC into a Serviceguard cluster while also easily managing the dependency between Oracle Clusterware and Oracle RAC with a full range of storage management options.
Preface This guide covers how to use Serviceguard Extension for Oracle Real Application Cluster (RAC) to configure Serviceguard clusters for use with RAC software, on HP High Availability clusters running the HP-UX operating system. The contents are as follows: • Chapter 1— Introduction to Serviceguard Extension for RAC Describes a Serviceguard cluster and provides a roadmap for using this guide. This chapter should be used as a supplement to Chapters 1–3 of the Managing Serviceguard user’s guide.
• VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Configuration Guide Extracts. • VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Administration Guide Extracts. If you will be using Veritas Cluster Volume Manager (CVM) and Veritas Cluster File System (CFS) from Symantec with Serviceguard refer to the HP Serviceguard Storage Management Suite Version A.03.01 for HP-UX 11i v3 Release Notes.
Where to find Documentation on the Web • SGeRAC Documentation Go to www.hp.com/go/hpux-serviceguard-docs, and then click HP Serviceguard Extension for RAC. • Related Documentation Go to www.hp.com/go/hpux-serviceguard-docs, www.hp.com/go/ hpux-core-docs, and www.hp.com/go/hpux-ha-monitoring-docs. For more information see the latest versions of the following documents: 12 ◦ Clusters for High Availability: a Primer of HP Solutions.
1 Introduction to Serviceguard Extension for RAC Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real Application Cluster (RAC), formerly known as Oracle Parallel Server RDBMS, to run on HP high availability clusters under the HP-UX operating system. This chapter introduces Serviceguard Extension for RAC and shows where to find different kinds of information in this book.
When properly configured, Serviceguard Extension for RAC provides a highly available database that continues to operate even if one hardware component fails. Group Membership Group membership allows multiple instances of RAC to run on each node. Related processes are configured into groups. Groups allow processes in different instances to choose which other processes to interact with. This allows the support of multiple databases within one RAC cluster.
are supported are those specified by Hewlett-Packard, and you can create your own multi-node packages. For example, the packages HP supplies for use with the Veritas Cluster Volume Manager (CVM) and the Veritas Cluster File (CFS) System (on HP-UX releases that support Veritas CFS and CVM. Also, see “About Veritas CFS and CVM from Symantec” (page 16)). • Multi-node package. A system multi-node package must run on all nodes that are active in the cluster. If it fails on one active node, that node halts.
For information on configuring CFS and CVM with Serviceguard, refer to the latest edition of the Managing Serviceguard user’s guide at www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard. Package Dependencies When CFS is used as shared storage, the application and software using the CFS storage should be configured to start and stop using Serviceguard packages.
NOTE: Beginning with HP-UX 11i v3 1109 HA-OE/DC-OE, SGeRAC is included as a licensed bundle at no additional cost. To install SGeRAC A.11.20 on your system during 1109 HA-OE/DC-OE installation, you must select T1907BA (SGeRAC) in the Software tab.
Oracle 10gR2/11gR1/11gR2 RAC uses the following two subnets for cluster communication purposes: • CSS Heartbeat Network (CSS-HB)—Oracle Clusterware running on the various nodes of the cluster communicate among themselves using this network. • RAC Cluster Interconnect Network (RAC-IC)—Database instances of a database use this network to communicate among themselves. NOTE: In this document, the generic terms CRS and Oracle Clusterware will subsequently be referred to as Oracle Cluster Software.
is handled by ensuring at least one instance of the multi-node package remains running in the cluster.
Configuring Packages for Oracle Listeners Oracle listeners can be configured as packages within the cluster (called listener packages). Each node with a RAC instance can be configured with a listener package. Listener packages are configured to automatically fail over from the original node to an adoptive node. When the original node is restored, the listener package automatically fails back to the original node. In the listener package ASCII configuration file, the FAILBACK_POLICY is set to AUTOMATIC.
and Package 2 disk. Oracle instance 2 now handles all database access, since instance 1 has gone down. Figure 4 After Node Failure In the above figure, pkg1 and pkg2 are not instance packages. They are shown to illustrate the movement of the packages. Larger Clusters Serviceguard Extension for RAC supports clusters of up to 16 nodes. The actual cluster size is limited by the type of storage and the type of volume manager used.
Figure 5 Four-Node RAC Cluster In this type of configuration, each node runs a separate instance of RAC and may run one or more high availability packages as well. The figure shows a dual Ethernet configuration with all four nodes connected to a disk array (the details of the connections depend on the type of disk array). In addition, each node has a mirrored root disk (R and R).
Figure 6 Eight-Node Cluster with EVA, XP or EMC Disk Array FibreChannel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP 9000 Servers Configuration Guide, available through your HP representative.
4. 5. Restart the Serviceguard cluster. Restart Oracle Clusterware (for Oracle 10gR2, 11gR1, and 11gR2) and Oracle RAC database instance on all nodes. Use the following steps to disable the GMS authorization: 1. 2. 3. 4. 5. If Oracle RAC database instance and Oracle Clusterware (for Oracle 10gR2, 11gR1, and 11gR2) are running, shut them down on all nodes. Halt the Serviceguard cluster. Edit /etc/opt/nmapi/nmutils.conf and comment the GMS_USER[] settings on all nodes. Restart the Serviceguard cluster.
Configuring Clusters with Serviceguard Manager You can configure clusters and packages in Serviceguard Manager. You must have root (UID=0) access to the cluster nodes.
2 Serviceguard Configuration for Oracle 10gR2, 11gR1, or 11gR2 RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 10gR2/11gR1/11gR2 with Serviceguard.
CSS Timeout When SGeRAC is on the same cluster as Oracle Cluster Software, the CSS timeout is set to a default value of 600 seconds (10 minutes) at Oracle software installation. This timeout is configurable with Oracle tools and should not be changed without ensuring that the CSS timeout allows enough time for Serviceguard Extension for RAC reconfiguration and to allow multipath (if configured) reconfiguration to complete.
The file /var/opt/oracle/oravg.conf must not be present so Oracle Cluster Software will not activate or deactivate any shared storage. Multipathing Multipathing is automatically configured in HP-UX 11i v3 (this is often called native multipathing). Multipathing is supported through either SLVM pvlinks or CVM Dynamic Multipath (DMP). In some configurations, SLVM or CVM does not need to be configured for multipath as the multipath is provided by the storage array.
Manual Startup and Shutdown Manual listener startup and shutdown is supported through the following commands: srvctl and lsnrctl. Network Monitoring SGeRAC cluster provides network monitoring. For networks that are redundant and monitored by Serviceguard cluster, Serviceguard cluster provides local failover capability between local network interfaces (LAN) that is transparent to applications utilizing User Datagram Protocol (UDP) and Transport Control Protocol (TCP).
Manual Startup and Shutdown Manual RAC instance startup and shutdown is supported through the following commands: srvctl or sqlplus. Shared Storage It is expected the shared storage is available when the RAC instance is started. Since the RAC instance expects the shared storage to be available, ensure the shared storage is activated. For SLVM, the shared volume groups must be activated and for CVM, the disk group must be activated. For CFS, the cluster file system must be mounted.
(DLPI) and supported over Serviceguard heartbeat subnet networks, including primary and standby links. • Highly available virtual IP () (only when using Oracle Grid Infrastructure 11.2.0.2) — IP addresses, which Oracle Database and Oracle ASM instances use to ensure highly available and load balanced across the provided set of cluster interconnect interfaces.
Volume Planning with SLVM Storage capacity for the Oracle database must be provided in the form of logical volumes located in shared volume groups. The Oracle software requires at least two log files for each Oracle instance, several Oracle control files and data files for the database itself. For all these files, Serviceguard Extension for RAC uses HP-UX raw logical volumes located in volume groups that are shared between the nodes in the cluster.
Fill out the Veritas Volume worksheet to provide volume names for volumes that you will create using the Veritas utilities. The Oracle DBA and the HP-UX system administrator should prepare this worksheet together. Create entries for shared volumes only. For each volume, enter the full pathname of the raw volume device file. Be sure to include the desired size in MB. Following are sample worksheets filled out. Refer to Appendix B: “Blank Planning Worksheets”, for samples of blank worksheets.
Installing Serviceguard Extension for RAC Installing Serviceguard Extension for RAC includes updating the software and rebuilding the kernel to support high availability cluster operation for Oracle Real Application Clusters.
nomenclature. You are not required to migrate to agile addressing when you upgrade to 11i v3, though you should seriously consider its advantages. It is possible, though not a best practice, to have legacy DSFs on some nodes and agile addressing on others—this allows you to migrate the names on different nodes at different times, if necessary. NOTE: The examples in this document use legacy naming conventions.
Limitations of cDSFs • cDSFs are supported only within a single cluster; you cannot define a cDSF group that crosses cluster boundaries. • A node can belong to only one cDSF group. • cDSFs are not supported by CVM, CFS, or any other application that assumes DSFs reside only in /dev/disk and /dev/rdisk. • cDSFs do not support disk partitions. Such partitions can be addressed by a device file using the agile addressing scheme, but not by a cDSF.
any network configured for Oracle cluster interconnect must also be configured as SGeRAC A.11.20 heartbeat network. NOTE: Do not configure Serviceguard heartbeat and Oracle cluster interconnect in mutually exclusive networks. 2. Configure one network with standby interfaces for both Oracle cluster interconnect and Serviceguard heartbeat and register the standby interface by Oracle oifcfg command. For Oracle Grid Infrastructure 11.2.0.
Figure 7 Single Network for Cluster Communications Alternate Configuration—Fast Reconfiguration with Low Node Member Timeout High RAC-IC traffic may interfere with SG-HB traffic and cause unnecessary member timeout if Serviceguard cluster configuration parameter MEMBER_TIMEOUT is low. If MEMBER_TIMEOUT cannot be increased, use of an additional network dedicated for SG-HB alone avoids unnecessary member timeouts when RAC-IC traffic is high.
with a CSS cluster reconfiguration. It will wait for the CSS misscount time interval before handling the CSS-HB subnet failure (by bringing down the node on which the CSS-HB subnet has failed). The default value of CSS misscount in SGeRAC configurations is 600 seconds. As shown in Figure 8, CLUSTER_INTERCONNECT_SUBNET can be used in conjunction with the NODE_FAIL_FAST_ENABLED package configuration parameter to monitor the CSS-HB network.
Figure 9 RAC/RAC-IC Traffic Separation—Multiple Database Configuration As shown in Figure 9, each primary and standby pair protects against a single failure. If the subnet with SG-HB (lan1/lan2) fails, Serviceguard will resolve the subnet failure with a Serviceguard cluster reconfiguration. If the 192.168.2.0 subnet (lan3 and lan4) fails, Oracle instance membership recovery (IMR) will resolve the interconnect failure subnet, unless Serviceguard subnet monitoring is used.
When Cluster Interconnect Subnet Monitoring is used The Cluster Interconnect Subnet Monitoring feature is used to monitor the CSS-HB network. If the Serviceguard cluster configuration parameter MEMBER_TIMEOUT is changed, then it is necessary to also change the Oracle Clusterware parameter CSS miscount.
Cluster Interconnect Monitoring Restrictions In addition to the above limitations, the Cluster Interconnect Monitoring feature has the following restriction: • Cluster Lock device/Quorum Server/Lock Lun must be configured in the cluster. Creating a Storage Infrastructure with LVM In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes.
Selecting Disks for the Volume Group Obtain a list of the disks on both nodes and identify which device files are used for the same disk on both. Use the following command on each node to list available disks as they are known to each system: # lssf /dev/dsk/* In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which are the device names for the same disks on both ftsys9 and ftsys10.
# lvcreate -m 1 -M n -c y -s g -n redo1.log -L 408 /dev/vg_rac • -m 1—Specifies single mirroring. • -M n—Ensures that mirror write cache recovery is set off. • -c y—Mirror consistency recovery is enabled. • -s g—Mirroring is PVG-strict. It occurs between different physical volume groups. • -n redo1.log—Specify the name of the logical volume. • -L 28—allocates 28 megabytes. NOTE: Use the -c y options for both redo logs and control files.
NOTE: Contact Oracle to determine if your version of Oracle RAC allows “resilvering” and to appropriately configure the mirror consistency recovery policy for your logical volumes. In the case Oracle “resilvering” is available, create logical volumes for use as Oracle data files by using the same options as in the following example: # lvcreate -m 1 -M n -c n -s g -n system.dbf -L 408 /dev/vg_rac • -m 1—Specifies single mirroring. • -M n—Ensures that mirror write cache recovery is set to OFF.
# mkdir /dev/vg_rac 2. Create a control file named group in the directory /dev/vg_rac, as follows: # mknod /dev/vg_rac/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the form 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. Use the following command to display a list of existing volume groups: # ls -l /dev/*/group 3.
Table 1 Required Oracle File Names for Demo Database (continued) Logical Volume Name LV Size (MB) Raw Logical Volume Path Name Oracle File Size (MB)* ops1log2.log 128 /dev/vg_rac/rops1log2.log 120 ops1log3.log 128 /dev/vg_rac/rops1log3.log 120 ops2log1.log 128 /dev/vg_rac/rops2log1.log 120 ops2log2.log 128 /dev/vg_rac/rops2log2.log 120 ops2log3.log 128 /dev/vg_rac/rops2log3.log 120 opssystem.dbf 408 /dev/vg_rac/ropssystem.dbf 400 opssysaux.dbf 808 /dev/vg_rac/ropssysaux.
all disk and file system (LVM) related configurations. For CVM/CFS related configurations, the fsweb tool launches the Veritas Enterprise Administrator (VEA) tool. NOTE: Serviceguard Manager is the graphical user interface for Serviceguard. It is available as a “plug-in” to the System Management Homepage (SMH). SMH is a web-based graphical user interface (GUI) that replaces SAM as the system administration GUI as of HP-UX 11i v3 (but, you can run the SAM terminal interface).
either the pathnames of the raw logical volumes instead of using the defaults. If you do not wish to install the demo database, select install software only. Creating a Storage Infrastructure with CFS In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM) or Veritas Cluster Volume Manager (CVM). You can also use a mixture of volume types, depending on your needs.
NOTE: 2. CVM 5.x or later does not require rootdg. Create the cluster ASCII file: # cd /etc/cmcluster # cmquerycl -C clm.asc -n ever3a -n ever3b Edit the cluster file. 3. Create the cluster: # cmapplyconf -C clm.asc 4. Start the cluster: # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 5. STATUS up STATUS up up STATE running running Configure the Cluster Volume Manager (CVM).
# /etc/vx/bin/vxdisksetup -i c4t4d0 8. Create the disk group for RAC. Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: # vxdg -s init cfsdg1 c4t4d0 9. Create the disk group multi-node package.
The following output will be displayed: Package name “SG-CFS-MP-3” was generated to control the resource. Mount point “/cfs/mnt3” that was associated with the cluster. NOTE: The diskgroup and mount point multi-node packages (SG-CFS-DG_ID# and SG-CFS-MP_ID#) do not monitor the health of the disk group and mount point. They check that the application packages that depend on them have access to the disk groups and mount points.
1. Unmount CFS mount points. # cfsumount /cfs/mnt1 # cfsumount /cfs/mnt2 # cfsumount /cfs/mnt3 2. Delete mount point multi-node package.
NOTE: The Oracle 11gR2 OUI allows only ASM over SLVM, ASM over raw device files, Cluster File System for Clusterware files, and Database files. Initializing the Veritas Volume Manager If you are about to create disk groups for the first time, you need to initialize the Volume Manager. This is done by creating a disk group known as rootdg that contains at least one disk.
3. Configure the Cluster Volume Manager (CVM). To configure and start the CVM stack, you need to configure the system multi-node package, SG-CFS-pkg. The SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and supports multiple subnets. Use the cmapplyconf command: # cmapplyconf -P /etc/cmcluster/cfs/SG-CFS-pkg.conf # cmrunpkg SG-CFS-pkg When CVM starts, it selects a master node. From this node, you must issue the disk group configuration commands.
PACKAGE SG-CFS-pkg STATUS up STATE running AUTO_RUN enabled SYSTEM yes IMPORTANT: After creating these files, use the vxedit command to change the ownership of the raw volume files to oracle and the group membership to dba, and to change the permissions to 660.
WARNING! The above file should never be edited. After the above command completes, start the cluster and create disk groups for shared use as described in the following sections. Starting the Cluster and Identifying the Master Node Run the cluster to activate the special CVM package: # cmruncl After the cluster is started, it will run with a special system multi-node package named VxVM-CVM-pkg that is on all nodes.
NAME rootdg ops_dg STATE ID enabled enabled,shared 971995699.1025.node1 972078742.1084.node2 Creating Volumes Use the vxassist command to create logical volumes. The following is an example: # vxassist -g ops_dg make log_files 1024m This command creates a 1024MB volume named log_files in a disk group named ops_dg. The volume can be referenced with the block device file /dev/vx/dsk/ops_dg/log_files or the raw (character) device file /dev/vx/rdsk/ops_dg/log_files.
Table 2 Required Oracle File Names for Demo Database (continued) Volume Name Size (MB) Raw Device File Name Oracle File Size (MB) ops2log1.log 128 /dev/vx/rdsk/ops_dg/ops2log1.log 120 ops2log2.log 128 /dev/vx/rdsk/ops_dg/ops2log2.log 120 ops2log3.log 128 /dev/vx/rdsk/ops_dg/ops2log3.log 120 opssystem.dbf 508 /dev/vx/rdsk/ops_dg/opssystem.dbf 500 opssysaux.dbf 808 /dev/vx/rdsk/ops_dg/opssysaux.dbf 800 opstemp.dbf 258 /dev/vx/rdsk/ops_dg/opstemp.dbf 250 opsusers.
Prerequisites for Oracle 10gR2, 11gR1, or 11gR2 (Sample Installation) The following sample steps prepare an SGeRAC cluster for Oracle 10gR2, 11gR1, or 11gR2. Refer to the Oracle documentation for Oracle installation details. 1. Create inventory groups on each node. Create the Oracle inventory group if one does not exist, create the OSDBA group, and create the Operator Group (optional). # groupadd oinstall # groupadd dba # groupadd oper 2. Create Oracle user on each node.
# chown -R oracle:oinstall /mnt/app/crs/oracle/product/10.2.0/crs # chmod -R 775 /mnt/app/crs/oracle/product/10.2.0/crs 8. Create Oracle base directory (for RAC binaries on local file system). If installing RAC binaries on local file system, create the oracle base directory on each node. # mkdir -p /mnt/app/oracle # chown -R oracle:oinstall /mnt/app/oracle # chmod -R 775 /mnt/app/oracle # usermod -d /mnt/app/oracle oracle 9. Create Oracle base directory (for RAC binaries on cluster file system).
The following is a sample of the mapping file for DBCA: system=/dev/vg_rac/ropssystem.dbf sysaux=/dev/vg_rac/ropssysaux.dbf undotbs1=/dev/vg_rac/ropsundotbs01.dbf undotbs2=/dev/vg_rac/ropsundotbs02.dbf example=/dev/vg_rac/ropsexample1.dbf users=/dev/vg_rac/ropsusers.dbf redo1_1=/dev/vg_rac/rops1log1.log redo1_2=/dev/vg_rac/rops1log2.log redo2_1=/dev/vg_rac/rops2log1.log redo2_2=/dev/vg_rac/rops2log2.log control1=/dev/vg_rac/ropsctl1.ctl control2=/dev/vg_rac/ropsctl2.ctl control3=/dev/vg_rac/ropsctl3.
NOTE: The volume groups are supported with Serviceguard. The steps shown in the following section are for configuring the volume groups in Serviceguard clusters LVM version 1.0. For more information on using and configuring LVM version 2.x, see the HP-UX System Administrator's Guide: Logical Volume Management located at www.hp.com/go/ hpux-core-docs —> HP-UX 11i v3. Installing Oracle 10gR2, 11gR1, or 11gR2 Cluster Software The following sample steps for an SGeRAC cluster for Oracle 10gR2, 11gR1, or 11gR2.
Installing RAC Binaries on Cluster File System Logon as a “oracle” user: $ export ORACLE_BASE=/cfs/mnt1/oracle $ export DISPLAY={display}:0.0 $ cd <10gR2/11g RAC installation disk directory> $ ./runInstaller Use following guidelines when installing on a local file system: 1. In this example, the path to ORACLE_HOME is located on a CFS directory /cfs/mnt1/ oracle/product//db_1. 2. Select installation for database software only. 3. When prompted, run root.sh on each node.
Creating a RAC Demo Database on CFS Export environment variables for “oracle” user: export ORACLE_BASE=/cfs/mnt1/oracle export ORACLE_HOME=$ORACLE_BASE/product//db_1 export ORA_CRS_HOME=/mnt/app/crs/oracle/product//crs LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32 export LD_LIBRARY_PATH SHLIB_PATH export \ PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin: CLASSPATH=$ORACLE_HOME/jre:$ORA
IMPORTANT: Beginning with HP Serviceguard Storage Management Suite (SMS) Version CVM/CFS 5.1 SP1, /opt/VRTSodm/lib/libodm.sl is changed to /opt/VRTSodm/ lib/libodm.so on itanium(IA) architecture. If you are using itanium architecture and HP Serviceguard SMS Version CVM/CFS 5.1 SP1 or later, then you must use /opt/VRTSodm/lib/libodm.so file instead of /opt/VRTSodm/ lib/libodm.sl file. #ll -L /opt/VRTSodm/lib/libodm.sl output: -r-xr-xr-x 1 root sys 94872 Aug 25 2009 /opt/VRTSodm/lib/libodm.
io req: io calls: comp req: comp calls: io mor cmp: io zro cmp: cl receive: cl ident: cl reserve: cl delete: cl resize: cl same op: cl opt idn: cl opt rsv: **********: 3. 9102431 6911030 73480659 5439560 461063 2330 66145 18 8 1 0 0 0 332 17 Verify that the Oracle disk manager is loaded: # kcmodule -P state odm Output: state loaded 4. In the alert log, verify the Oracle instance is running. The log should contain output similar to the following: For CFS 5.0.
Using Serviceguard Packages to Synchronize with Oracle 10gR2/11gR1/11gR2 RAC It is recommended to start and stop Oracle Cluster Software in a Serviceguard package—the Oracle Cluster Software will start after SGeRAC is started, and will stop before SGeRAC is halted. Serviceguard packages should also be used to synchronize storage activation and deactivation with Oracle Cluster Software and RAC instances.
RAC MNP packages are required to define a 'same node up' package dependency on the OC MNP, since Oracle Clusterware must be running before any RAC database instances can be started. When using ASM for storage management, the ASM DG MNP must have a package dependency on the OC MNP and the corresponding RAC MNP packages must have a "same node up" dependency on the ASM DG MNP.
Figure 12 Dependency structure in the case of ASM over SLVM and ASM over HP-UX raw disks In case of ASM over SLVM: • The SLVM Volume groups used for Oracle Clusterware storage are configured in the OC-MNP package. • The SLVM Volume groups used for RAC database storage are configured in the ASMDG MNP package. In case of ASM over HP-UX raw disks • You must not specify any HP-UX raw disks information either in OC-MNP package or in ASMDG MNP package.
File Description erac_tk_asmdg.1 Toolkit ASMDG MNP module ADF file. erac_tk_asmdg A symbolic link to the latest version of ASMDG MNP module file (for example, erac_tk_asmdg.1). The files under /etc/cmcluster/scripts/sgerac are the module script files: Files Description erac_tk_oc.sh Toolkit OC MNP module script file. erac_tk_rac.sh Toolkit RAC MNP module script file. erac_tk_asmdg.sh Toolkit ASMDG MNP module script file. oc_gen.
Files Description rac_dbi.check Toolkit monitor script that checks if the RAC instance is running. rac_dbi.sh Toolkit script to start, stop, and check the RAC instance. The files under /opt/cmcluster/SGeRAC/toolkit/asmp are for the ASMDG MNP: Files Description toolkit_asmdg.sh The entry point to the Toolkit entity. It is an interface between the ASMDG MNP package control script and the asm_dg.* files listed below. asm_dg.
You must follow the storage configuration procedures required for SLVM, CVM, CFS, ASM, or ASM over SLVM respectively to configure the storage for Oracle Clusterware (OCR and voting disk) and RAC database (control files, data files, redo-logs, and so on). Certain package parameters and some specific parameter values are needed to configure the Toolkit MNPs. In addition, some Toolkit specific configurations are needed in the Toolkit configuration file.
Attributes Description KILL_ASM_FOREGROUNDS This parameter specifies the requirement to kill if any ASM foreground processes that have file descriptors open on the dismounted disk group volumes. This parameter can be set to either Yes (default) or No. ASM_DISK GROUP ASM Disk groups used by the database instance ASM_VOLUME_GROUP: Volume groups used in the ASM disk groups for this database instance.
Attributes Description down on the local node, the RAC MNP on that node is not halted.To continue monitoring and come back from the maintenance mode on one node, you must remove the "rac.debug" file on that node. You must ensure that the Oracle RAC instance is properly running after the maintenance phase. OC_TKIT_DIR This parameter contains a directory path where the Oracle Clusterware toolkit configuration file (oc.conf) is generated. The toolkit maintenance mode flag file (oc.
Toolkit Multi-Node package Storage Management HP-UX version Oracle Clusterware CVM raw(version 5.0.1) 11i v3 Oracle Clusterware CFS(version 5.1 SP1) 11i v3 RAC Instance CFS(version 5.1 SP1) 11i v3 RAC Instance ASM over HP-UX raw 11i v3 disks 11i v3 SGeRAC Toolkit Administration Serviceguard package commands administer the Toolkit OC MNP and RAC MNP and ASMDG MNP .
Attributes Description local_lan_failover_allowed Set by default to yes so that the cluster can switch LANs locally in the event of a failure. node_fail_fast_enabled Set by default to no. run_script_timeout, halt_script_timeout Default value is 600 seconds for a 4 node cluster. This value is suggested as an initial value. You might need to tune it according to your environment. script_log_file Set by default to $SGRUN/log/$SG_PACKAGE.log TKIT_DIR Set to the OC MNP working directory.
Attributes Description For the package using CFS DEPENDENCY_NAME MP-MNP-name DEPENDENCY_CONDITION MP-MNP-PKG=UP DEPENDENCY_LOCATION SAME_NODE NOTE: For modular style CFS DG-MP package, as a dependency modular style CFS DG-MP MNP must be mentioned in OC MNP configuration file. For the ASMDG MNP: NOTE: If ASM over SLVM is being used for the RAC database, it is recommended to use the new ASMDG package to manage the ASM disk group.
Attributes Description vgchange_cmd, vg • Set vgchange_cmd to vgchange -a s. When you use ASM over HP-UX raw disks, ignore this step. • Specify the name(s) of the Shared Volume Groups in vg[0], vg[1]... When using ASM over HP-UX raw disks, ignore this step. run_script_timeout, halt_script_timeout Default value is 600 seconds for a 4 node cluster. This value is suggested as an initial value.
Attributes Description ORA_CRS_HOME, OC_TKIT_DIR It is not required to set these values, the cmapplyconf command automatically sets them during package configuration based on the setting in the OC MNP. Set by default to . Refer to “Toolkit Configuration File Parameters” (page 73) for the descriptions. DB_SERVICE By default, this parameter is commented. To start and stop the Oracle RAC Database services as part of RAC-MNP, uncomment the parameter and specify service names to start.
Attributes Description For the package using CFS: DEPENDENCY_NAME OC-MNP-name DEPENDENCY_CONDITION OC-MNP-PKG=UP DEPENDENCY_LOCATION SAME_NODE DEPENDENCY_NAME MP-MNP-name DEPENDENCY_CONDITION MP-MNP-PKG=UP DEPENDENCY_LOCATION SAME_NODE NOTE: For modular style CFS DG-MP package, as a dependency OC MNP and modular style CFS DG-MP MNP must be mentioned in RAC MNP configuration file.
Edit the package template files based on the description in “Modular Packages Configuration Parameters” (page 76). 4. Apply the package configuration file: # cmapplyconf -P pkgConfigFile ASMDG creation procedures NOTE: This procedure is required only if you are using ASM over SLVM for RAC database and if you are planning to use the ASMDG package to manage your ASM disk group. 1. On one node of the cluster, create an ASMDG MNP working directory under /etc/cmcluster. # mkdir /etc/cmcluster/ASMDGMNP-Dir 2.
# cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_rac -t rac_dbi.conf pkgConfigFile Otherwise, create the package configuration file and set the Oracle Clusterware parameters in this file directly: # cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_rac pkgConfigFile Edit the package template files based on the description in “Modular Packages Configuration Parameters” (page 76). 4.
Attributes Description SERVICE_NAME Specify a single SERVICE_NAME, corresponding to the service definition in the control script. This service invokes Toolkit script "toolkit_oc.sh check". SERVICE_HALT_TIMEOUT Default value is 300 seconds for a 4 node cluster. This value is suggested as an initial value. It may need to be tuned for your environment. For the ASMDG MNP: NOTE: If ASM over SLVM is being used for RAC database, HP recommends that you use the new ASM DG package to manage the ASM disk group.
Attributes Description NODE_FAIL_FAST_ENABLED Set by default to NO. RUN_SCRIPT, HALT_SCRIPT Set to the package control script. RUN_SCRIPT_TIMEOUT, HALT_SCRIPT_TIMEOUT Default value is 600 seconds for a 4 node cluster. This value is suggested as an initial value. It may need to be tuned for your environment. STORAGE_GROUP If the database is stored in a CVM disk group, specify it using this parameter. DEPENDENCY_NAME, DEPENDENCY_CONDITION, DEPENDENCY_LOCATION Configure a dependency on the OC MNP.
When SLVM or ASM over SLVM is used for Oracle Clusterwaer storage, specify the corresponding SLVM Volume Group names and set activation to shared mode. • set VGCHANGE to "vgchange -a s" • specify the names of the Shared Volume Groups in vg[0], vg[1].... NOTE: You must not specify SLVM Volume groups used by RAC databases here. When using ASM over SLVM for RAC database storage, specify the SLVM Volume Group of member disks in the ASM DG MNP package configuration file.
In the function customer_defined_run_cmds: • start the RAC instance using the command: /toolkit_dbi.sh start In the function customer_defined_halt_cmds: • stop the RAC instance using the command: /toolkit_dbi.sh stop For the ASMDG MNP: • set VGCHANGE to "vgchange -a s" . When you use ASM over HP-UX raw disks, ignore this step. • specify the names of the ASM Shared Volume Groups in VG[0], VG[1]...
ASMDG MNP creation procedures NOTE: This procedure is required only if you are using ASM over SLVM for RAC database and if you are planning to use the ASMDG package to manage your ASM disk group. 1. On one node of the cluster, create the ASMDG MNP working directory under /etc/cmcluster and copy over the files in the Toolkit directory /opt/cmcluster/SGeRAC/toolkit/asmp. # mkdir /etc/cmcluster/YourOwn-ASMDGMNP-Dir # cd /etc/cmcluster/YourOwn-ASMDGMNP-Dir . # cp /opt/cmcluster/SGeRAC/toolkit/asmp/* . 2. 3.
SGeRAC Toolkit configuration for Modular and Legacy Packages OC MNP startup procedures 1. On each node of the cluster, halt the Oracle Clusterware if it is running. : $ORA_CRS_HOME/bin/crsctl stop crs 2. On one node of the cluster, start the OC MNP via cmrunpkg. : # cmrunpkg Use cmviewcl to check the package status. There must be only one OC MNP configured in the cluster. 3.
Oracle RAC MNP startup procedures 1. On one node of the cluster, start the RAC MNP via cmrunpkg. # cmrunpkg Your-RACMNP-Name Use cmviewcl to check the package status. 2. After the package is up and running, verify that the RAC instance is running. On one node of the cluster, login as the Oracle administrator and enter: : $ORACLE_HOME/bin/srvctl status instance -d $databaseName -i $instanceName Messages like the following should be seen: Instance is running on node 3.
NOTE: The step assumes that asm_dg.conf file created in legacy package creation exists in ASMDGMNP working directory on all the cluster nodes 4. Edit the new ASMDG MNP configuration file and set the Toolkit parameter TKIT_DIR to the directory created in Step 1. NOTE: If you specified multiple values for ASM_VOLUME_GROUP and ASM_DISKGROUP fields in the legacy ASMDG toolkit configuration file (asm_dg.
Migration of Legacy CFS Disk group and Mount point Packages to Modular CFS Disk group and Mount point Packages (CFS DG-MP) Beginning with the SG A.11.20 patch PHSS_41628 and SG CFS A.11.20 patch PHSS_41674, new modular CFS Disk group and Mount point feature has been introduced. It allows to consolidate all disk group and mount point packages for an application into a single modular package.
Attributes Description DEPENDENCY_NAME DG-MNP-name DEPENDENCY_CONDITION DG-MNP-PKG=UP DEPENDENCY_LOCATION SAME_NODE OC MNP ascii file with modular style CFS DG-MP package : 7. 8. Attributes Description DEPENDENCY_NAME OC-DGMP-name DEPENDENCY_CONDITION OC-DGMP-PKG=UP DEPENDENCY_LOCATION SAME_NODE Backup all RAC MNP configuration file.
13. Delete all legacy style Disk group MNPs and Mount Point MNPs from the cluster. # cmdeleteconf -p # cmdeleteconf -p 14. Apply and run both modular CFS DG-MP packages for Oracle Clusterware and RAC database storage created in step number [1] and [3] # cmapplyconf -P # cmapplyconf -P # cmrunpkg 15. Apply the updated OC MNP configuration file which was modified in step number [6].
NOTE: 5. 4. 5. If Database services are already running then you must not perform the steps 4 and If Database services are not running then halt the RAC-MNP using cmhaltpkg . Re-start the RAC-MNP. cmrunpkg SGeRAC Toolkit Package Cleanup 1. 2. 3. Shutdown the RAC MNP. # cmhaltpkg Delete the RAC MNP configuration. # cmdeleteconf -p Remove the RAC MNP working directory from all nodes.
3 Support of Oracle RAC ASM with SGeRAC Introduction This chapter discusses the use of the Oracle 10g Release 2 (10g R2) and11g Release 1 (11g R1) database server feature called Automatic Storage Management (ASM) in configurations of HP Serviceguard Extension for Real Application Clusters (SGeRAC). We begin with a brief review of ASM—functionality, pros, cons, and method of operation. Then, we look in detail at how we configure ASM with SGeRAC (version A.11.17 or later is required).
ASM over SLVM SGeRAC support for ASM-over-SLVM continues in HPUX 11i v3. SGeRAC 11.17.01 or later is required for ASM-over-SLVM configurations on HP-UX 11i v3.
Figure 13 1-1 mapping between SLVM logical and physical volumes for ASM configuration The SLVM volume groups are marked as shared volume groups and exported across the SGeRAC cluster using standard SGeRAC procedures. Please note that, for the case in which the SLVM PVs being used by ASM are disk array LUs, the requirements in this section do not place any constraints on the configuration of the LUs.
• Extend each LV to the maximum size possible on that PV (the number of extents available in a PV can be determined via vgdisplay -v ) • Configure LV timeouts, based on the PV timeout and number of physical paths, as described in the previous section. If a PV timeout has been explicitly set, its value can be displayed via pvdisplay -v. If not, pvdisplay will show a value of default, indicating that the timeout is determined by the underlying disk driver.
or later) to support ASM on raw disks/disk array LUs. In HP-UX 11i v3, new DSF is introduced. SGeRAC will support the DSF format that ASM support with the restriction that native multipathing feature is enabled. The advantages for “ASM-over-raw” are as follows: • There is a small performance improvement from one less layer of volume management. • Online disk management (adding disks, deleting disks) is supported with ASM-over-raw.
Additional Documentation on the Web and Scripts • Oracle Clusterware Installation Guide 11g Release 1 (11.1) for HP-UX at www.oracle.com/ pls/db111/portal.portal_db?selected=11&frame= → HP-UX Installation Guides → Clusterware Installation Guide for HP-UX • ASM related sections in Oracle Manuals ◦ Oracle® Database Administrator's Guide 10g R2 (10.2) at www.oracle.com/pls/db102/ portal.
4 SGeRAC Toolkit for Oracle RAC 10gR2 or later Introduction This chapter discusses how Serviceguard Extension for RAC Toolkit enables a new framework for the integration of Oracle 10g Release 2 (10.2.0.1) or later version of Real Application Clusters (Oracle RAC1) with HP Serviceguard Extension for Real Application Clusters A.11.17 or later (SGeRAC2). SGeRAC Toolkit leverages the multi-node package and simple package dependency features introduced by HP Serviceguard (SG) A.11.
(on HP-UX 11i v3). Oracle 11gR2 is supported only on HP-UX 11i v3 (11.31) with SGeRAC A.11.19 or later. • The RAC database files can be configured as shared raw logical volumes managed by SGeRAC using SLVM or CVM. Beginning with SGeRAC A.11.17, the RAC database files may be configured as shared files managed by SGeRAC using CFS. Also, beginning with Oracle 10g R2 and SGeRAC A.11.17, the RAC database files may also be configured as files in Oracle ASM (Automatic Storage Management) Disk Groups.
NOTE: The above mentioned steps are the mandatory prerequisite steps to be performed before you configure SGeRAC toolkit for CRS, ASMDG (if storage is ASM/SLVM), and RAC MNP’s. Support for invocation of Oracle Clusterware commands from customer-developed scripts This includes invocation of such commands from SGeRAC package control scripts or module scripts; therefore, SGeRAC packages can invoke commands to start up and shutdown Oracle Clusterware and/or RAC database instances.
Simple package dependencies have the following features/restrictions: • cmrunpkg will fail if the user attempts to start a package that has a dependency on another package that is not running. The package manager will not attempt to start a package if its dependencies are not met. If multiple packages are specified to cmrunpkg, they will be started in dependency order. If the AUTO_RUN attribute is set to YES, the package manager will start the packages automatically in dependency order.
An example of a bottleneck created if we only have a package for Oracle Clusterware is this: if we concentrate all storage management in the Oracle Clusterware package, then any time there is a change in the storage configuration for one database (for example, an SLVM volume group is added), we would have to modify the Oracle Clusterware package. These are the main arguments in favor of having separate packages for Oracle Clusterware and each RAC database.
Figure 14 Resources managed by SGeRAC and Oracle Clusterware and their dependencies Startup and shutdown of the combined Oracle RAC-SGeRAC stack The combined stack is brought up in proper order by cmrunnode or cmruncl as follows. 1. SGeRAC starts up. 2. The SGeRAC package manager starts up Oracle Clusterware via the Oracle Clusterware MNP, ensuring that the storage needed is made available first.
group MNPs or ASMDG MNPs that are dependent on the particular RAC MNP in the case of the storage needed by the RAC database being managed by CFS or ASM disk groups respectively. Next, SGeRAC package manager shuts down Oracle Clusterware via the Oracle Clusterware MNP, followed by the storage needed by Oracle Clusterware (this requires subsequent shutdown of mount point and disk group MNPs in the case of the storage needed by Oracle Clusterware being managed by CFS).
status of the ASM diskgroups mentioned in that ASMDG MNP. If one or more ASM diskgroup is in a dismounted state, the check function will report failure—the ASM diskgroup is dismounted without using cmhaltpkg. The service that invokes the function fails at this point and the SGeRAC package manager fails the corresponding ASMDG MNP and the RAC MNP that is dependent on ASMDG MNP.
In this case, Oracle Clusterware quorum and voting disk and RAC database files are stored in raw logical volumes managed by SLVM or CVM. The management of SLVM or CVM storage for Oracle Clusterware and database is specified in the package configuration of the respective MNPs. Use Case 2: Oracle Clusterware storage and database storage in CFS Figure 16 Use Case 2 Setup In this case, Oracle Clusterware quorum and registry device data is stored in files in a CFS.
dependent on Oracle Clusterware MNP. Disk groups that are exclusively used by a RAC DB should be managed in separate ASM DG MNP. If different RAC Database uses different ASM Disk groups then those, ASM DGs should not be configured in a single ASMDG MNP. As RAC DB Instance MNP 3 and RAC DB Instance MNP 4 use completely different ASM diskgroups, they are made dependent on their respective ASMDG MNP(ASMDG MNP 2, ASMDG MNP 3).
3. 4. maintenance mode. The maintenance mode message will appear in ASMDG MNP package log files, for example, “ASM DG MNP pausing ASM DG instance checking and entering maintenance mode.” The user can maintain the Oracle ASM disk groups on that node while Oracle ASMDG MNP package is still running. After the maintenance work is completed, the user can remove the created asm_dg.debug in step 2 to bring the Oracle ASMDG MNP package out of maintenance mode to resume normal monitoring by Serviceguard.
Figure 18 Internal structure of SGeRAC for Oracle Clusterware Figure 19 Internal structure of SGeRAC for ASMDG MNP Serviceguard Extension for RAC Toolkit internal file structure 113
Figure 20 Internal structure of SGeRAC for RAC DB instance 114 SGeRAC Toolkit for Oracle RAC 10gR2 or later
5 Maintenance This chapter includes information about carrying out routine maintenance on a Real Application Cluster configuration. Starting with version SGeRAC A.11.17, all log messages from cmgmsd log to /var/adm/syslog/syslog.log by default. As presented here, these tasks differ in some details from the similar tasks described in the Managing Serviceguard documentation.
CLUSTER cluster_mo NODE minie STATUS up STATUS up STATE running Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up NODE mo PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 STATUS up NAME lan0 lan1 lan3 STATE running Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 NAME lan0 lan1 lan3 MULTI_NODE_PACKAGES PACKAGE SG-CF
NODE_NAME mo STATUS up Dependency_Parameters: DEPENDENCY_NAME SG-CFS-pkg PACKAGE SG-CFS-MP-1 NODE_NAME minie STATUS up STATUS up Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 PACKAGE SG-CFS-MP-2 NODE_NAME minie STATUS up STATUS up Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 PACKAGE SG-CFS-MP-3 NODE_NAME minie STATUS up STATUS up Dependenc
Cluster Status The status of a cluster may be one of the following: • Up. At least one node has a running cluster daemon, and reconfiguration is not taking place. • Down. No cluster daemons are running on any cluster node. • Starting. The cluster is in the process of determining its active membership. At least one cluster daemon is running. • Unknown. The node on which the cmviewcl command is issued cannot communicate with other nodes in the cluster.
Package Switching Attributes Packages also have the following switching attributes: • Package Switching. Enabled—the package can switch to another node in the event of failure. • Switching Enabled for a Node. Enabled—the package can switch to the referenced node. Disabled—the package cannot switch to the specified node until the node is enabled for the package using the cmmodpkg command. Every package is marked Enabled or Disabled for each node that is either a primary or adoptive node for the package.
Network Status The network interfaces have only status, as follows: • Up. • Down. • Unknown—Whether the interface is up or down cannot be determined. This can happen when the cluster is down. A standby interface has this status. NOTE: Serial Line Status has been de-supported as of Serviceguard A.11.18.
ftsys10 up running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 28.1 32.
PACKAGE VxVM-CVM-pkg NODE ftsys8 STATUS up STATE running STATUS down NODE STATUS ftsys9 up Script_Parameters: ITEM STATUS Service up STATE halted STATE running MAX_RESTARTS 0 RESTARTS 0 NAME VxVM-CVM-pkg.
Alternate NODE ftsys10 up STATUS up enabled ftsys9 (current) STATE running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 28.1 32.1 NAME lan0 lan1 Now pkg2 is running on node ftsys9. Note that it is still disabled from switching.
Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover min_package_node Failback automatic Script_Parameters: ITEM STATUS Resource up Subnet up Resource up Subnet up Resource up Subnet up Resource up Subnet up NODE_NAME manx manx burmese burmese tabby tabby persian persian NAME /resource/random 192.8.15.0 /resource/random 192.8.15.0 /resource/random 192.8.15.0 /resource/random 192.8.15.
NOTE: • All of the checks below are performed when you run cmcheckconf without any arguments (or with only -v, with or without -k or -K). cmcheckconf validates the current cluster and package configuration, including external scripts and pre-scripts for modular packages, and runs cmcompare to check file consistency across nodes. (This new version of the command also performs all of the checks that were done in previous releases.) See “Checking Cluster Components” (page 125) for details.
Table 4 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Comments • Same physical volumes on each node • Physical volumes connected on each node Volume groups (package) cmcheckconf (1m), cmapplyconf (1m) Checked only on nodes configured to run the package. LVM logical volumes (package) cmcheckconf (1m), cmapplyconf (1m) Checked for modular packages only, as part of package validation (cmcheckconf -P).
Table 4 Verifying Cluster Components (continued) Component (Context) Tool or Command; More Information Comments exist and are executable. Service commands whose paths are nested within an unmounted shared filesystem are not checked. IP addresses (cluster) cmcheckconf (1m), cmapplyconf (1m) Commands check that all IP addresses configured into the cluster are in each node's /etc/ hosts.
Limitations Serviceguard does not check the following conditions: • Access Control Policies properly configured • File systems configured to mount automatically on boot (that is, Serviceguard does not check /etc/fstab) • Shared volume groups configured to activate on boot • Volume group major and minor numbers unique • Redundant storage paths functioning properly • Kernel parameters and driver configurations consistent across nodes • Mount point overlaps (such that one file system is obscured w
1) Create a directory for RAC toolkit configuration file. This directory should be same as the one which is created on the existing cluster nodes. For example; mkdir /etc/cmcluster// 2) Copy the rac_dbi.conf from RAC multi node package directory from other nodes to the RAC multi node package directory of new node. Step b1 and b2 must be repeated for all the RAC MNP packages in the cluster. c.
# vgchange -S y -c y /dev/vg_rac This command is issued from the configuration node only, and the cluster must be running on all nodes for the command to succeed. Note that both the -S and the -c options are specified. The -S y option makes the volume group shareable, and the -c y option causes the cluster ID to be written out to all the disks in the volume group. This command specifies the cluster that a node must be a part of to obtain shared access to the volume group.
1. 2. Ensure that the Oracle RAC database is not active on either node. From node 2, use the vgchange command to deactivate the volume group: # vgchange -a n /dev/vg_rac 3. From node 2, use the vgexport command to export the volume group: # vgexport -m /tmp/vg_rac.map.old /dev/vg_rac This dissociates the volume group from node 2. 4. From node 1, use the vgchange command to deactivate the volume group: # vgchange -a n /dev/vg_rac 5.
Adding Additional Shared LVM Volume Groups To add capacity or to organize your disk resources for ease of management, you may wish to create additional shared volume groups for your Oracle RAC databases. If you decide to use additional shared volume groups, they must conform to the following rules: • Volume groups should include different PV links to each logical unit on the disk array. • Volume group names must be the same on all nodes in the cluster.
Monitoring Hardware Good standard practice in handling a high-availability system includes careful fault monitoring so as to prevent failures if possible, or at least to react to them swiftly when they occur. The following should be monitored for errors or warnings of all kinds.
13. Start up the Oracle RAC instances on all nodes. 14. Activate automatic cluster startup. NOTE: As you add new disks to the system, update the planning worksheets (described in Appendix B: “Blank Planning Worksheets”), so as to record the exact configuration you are using. Replacing Disks The procedure for replacing a faulty disk mechanism depends on the type of disk configuration you are using and on the type of Volume Manager software.
6. Issue the following command to extend the logical volume to the newly inserted disk: # lvextend -m 1 /dev/vg_sg01 /dev/dsk/c2t3d0 7. Finally, use the lvsync command for each logical volume that has extents on the failed physical volume. This synchronizes the extents of the new disk with the extents of the other mirror.
NOTE: After executing one of the commands above, any I/O queued for the device will restart. If the device replaced in step #2 was a mirror copy, then it will begin the resynchronization process that may take a significant amount of time to complete. The progress of the resynchronization process can be observed using the vgdisplay(1M), lvdisplay(1M) or pvdisplay(1M) commands.
the bus without harm.) When using inline terminators and Y cables, ensure that all orange-socketed termination packs are removed from the controller cards. NOTE: You cannot use inline terminators with internal FW/SCSI buses on D and K series systems, and you cannot use the inline terminator with single-ended SCSI buses. You must not use an inline terminator to connect a node to a Y cable. Figure 21 shows a three-node cluster with two F/W SCSI buses.
Replacement of I/O Cards After an I/O card failure, you can replace the card using the following steps. It is not necessary to bring the cluster down to do this if you are using SCSI inline terminators or Y cables at each node. 1. Halt the node by using Serviceguard Manager or the cmhaltnode command. Packages should fail over normally to other nodes. 2. Remove the I/O cable from the card. With SCSI inline terminators, this can be done without affecting the disks or other nodes on the bus. 3.
1. Use the cmgetconf command to obtain a fresh ASCII configuration file, as follows: # cmgetconf config.ascii 2. Use the cmapplyconf command to apply the configuration and copy the new binary file to all cluster nodes: # cmapplyconf -C config.ascii This procedure updates the binary file with the new MAC address and thus avoids data inconsistency between the outputs of the cmviewconcl and lanscan commands. Monitoring RAC Instances The DB Provider provides the capability to monitor RAC databases.
6 Troubleshooting Go to www.hp.com/go/hpux-serviceguard-docs, and then click HP Serviceguard . In the User Guide section, click on the latest Managing Serviceguard manual and see the “Troubleshooting your Cluster” chapter. NOTE: 140 Troubleshooting All messages from cmgmsd log to /var/adm/syslog/syslog.log by default.
A Software Upgrades Serviceguard Extension for RAC (SGeRAC) software upgrades can be done in the two following ways: • rolling upgrade • non-rolling upgrade Instead of an upgrade, moving to a new version can be done with: • migration with cold install Rolling upgrade is a feature of SGeRAC that allows you to perform a software upgrade on a given node without bringing down the entire cluster. SGeRAC supports rolling upgrades on version A.11.
Rolling Software Upgrades SGeRAC version A.11.15 and later allow you to roll forward to any higher revision provided all of the following conditions are met. • The upgrade must be done on systems of the same architecture (HP 9000 or Integrity Servers). • All nodes in the cluster must be running on the same version of HP-UX. • Each node must be running a version of HP-UX that supports the new SGeRAC version. • Each node must be running a version of Serviceguard that supports the new SGeRAC version.
1. 2. Install Serviceguard A.11.19 patch PHSS_42216 on all the cluster nodes before upgrading from Serviceguard A.11.19 to HP-UX 11i v3 1109 HA-OE/DC-OE, Serviceguard and SGeRAC A.11.20. Otherwise, upgraded Serviceguard and SGeRAC A.11.20 node will not be able to join the existing 11.19 cluster. Select the licensed SGeRAC A.11.20 bundle. To upgrade the 1109 HA-OE/DC-OE: On the first node: a. Halt the node using the following command: cmhaltnode b. 3.
NOTE: Please make sure that Serviceguard Extension for RAC is installed on all the nodes in the cluster and all nodes are up and running before attempting to deploy Oracle RAC in this cluster. 5. Repeat steps 3 and 4 for all the nodes in the cluster to complete the upgrade. Upgrading from Serviceguard A.11.18 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along with SGeRAC To upgrade from Serviceguard A.11.18 to HP-UX 11i v3 1109 HA-OE/DC-OE: 1. Perform a rolling upgrade to Serviceguard A.11.19. 2.
1. 2. 3. Halt Oracle (RAC, Clusterware) software on the local node (if running). Halt Serviceguard/SGeRAC on the local node by issuing the Serviceguard cmhaltnode command. Edit the /etc/rc.config.d/cmcluster file to include the following line: AUTOSTART_CMCLD = 0 4. 5. Upgrade the HP-UX OS (if required), Serviceguard, and SGeRAC to the new release (SGeRAC requires the compatible version of Serviceguard and OS).
Figure 22 Running Cluster Before Rolling Upgrade 1. 2. Halt Oracle (RAC, Clusterware) software on node 1. Halt node 1. This will cause the node’s packages to start up on an adoptive node. You can do this in Serviceguard Manager, or from the command line issue the following: # cmhaltnode -f node1 This will cause the failover package to be halted cleanly and moved to node 2. The Serviceguard daemon on node 1 is halted, and the result is shown in Figure 23.
Figure 24 Node 1 Upgraded to SG/SGeRAC 11.16 1. If you prefer, restart the cluster on the upgraded node (node 1). You can do this in Serviceguard Manager, or from the command line issue the following: # cmrunnode node1 2. 3. At this point, different versions of the Serviceguard daemon (cmcld) are running on the two nodes, as shown in Figure 25. Start Oracle (Clusterware, RAC) software on node 1. Figure 25 Node 1 Rejoining the Cluster 1. 2. Halt Oracle (RAC, Clusterware) software on node 2.
4. When upgrading is finished, enter the following command on node 2 to restart the cluster on node 2: # cmrunnode node2 5. Start Oracle (Clusterware, RAC) software on node 2. Figure 26 Running Cluster with Packages Moved to Node 1 Move PKG2 back to its original node. Use the following commands: # cmhaltpkg pkg2 # cmrunpkg -n node2 pkg2 # cmmodpkg -e pkg2 The cmmodpkg command re-enables switching of the package that is disabled by the cmhaltpkg command. The final running cluster is shown in Figure 27.
Figure 27 Running Cluster After Upgrades Limitations of Rolling Upgrades The following limitations apply to rolling upgrades: • During a rolling upgrade, you should issue Serviceguard/SGeRAC commands (other than cmrunnode and cmhaltnode) only on a node containing the latest revision of the software. Performing tasks on a node containing an earlier revision of the software will not work or will cause inconsistent results.
For more information on support, compatibility, and features for SGeRAC, refer to the Serviceguard and Serviceguard Extension for RAC Compatibility and Feature Matrix, located at www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard Extension for RAC . • You cannot delete Serviceguard/SGeRAC software (via swremove) from a node while the cluster is in the process of a rolling upgrade.
refer to the Serviceguard Compatibility and Feature Matrix, located at www.hp.com/go/ hpux-serviceguard-docs —> HP Serviceguard Extension for RAC. 6. Recreate any user accounts needed for the cluster applications. 7. Recreate the network and storage configurations (Set up stationary IP addresses and create LVM volume groups and/or CVM disk groups required for the cluster). 8. Recreate the SGeRAC cluster. 9. Restart the cluster. 10. Reinstall the cluster applications, such as RAC. 11. Restore the data.
to A.11.19 in preparation for a rolling upgrade to A.11.20, continue with the following subsection that provides information on upgrading to A.11.19.
B Blank Planning Worksheets This appendix reprints blank planning worksheets used in preparing the RAC cluster. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Instance 1 Redo Log: _____________________________________________________ Instance 2 Redo Log 1: _____________________________________________________ Instance 2 Redo Log 2: _____________________________________________________ Instance 2 Redo Log 3: _____________________________________________________ Instance 2 Redo Log: _____________________________________________________ Instance 2 Redo Log: _____________________________________________________ Data: System ______________________________
Index A activation of volume groups in shared mode, 130 administration cluster and package states, 115 array replacing a faulty mechanism, 134, 135, 136 B building a cluster CVM infrastructure, 53 building an RAC cluster displaying the logical volume infrastructure, 47 logical volume infrastructure, 42 building logical volumes for RAC, 46 C CFS, 49, 52 cluster state, 120 status options, 118 Cluster Communication Network Monitoring, 36 cluster volume group creating physical volumes, 43 creating a storage i
maintenance adding disk hardware, 133 making changes to shared volume groups, 130 Migration, 90 Legacy to Modular packages, 90 monitoring hardware, 133 N network status, 120 node halting status, 123 in an RAC cluster, 13 status and state, 118 non-rolling upgrade DRD, 151 O online hardware maintenance by means of in-line SCSI terminators, 136 Online node addition and deletion, 128 Online reconfiguration, 128 opsctl.ctl Oracle demo database files, 46, 58 opslog.
used with CVM, 56 system.dbf Oracle demo database files, 47, 59 T temp.
