Using Serviceguard Extension for RAC HP Part Number: T1859-91001 Published: September 2010
Legal Notices © Copyright 2010 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents Printing History ................................................................................................................11 Preface..............................................................................................................................13 1 Introduction to Serviceguard Extension for RAC.......................................................15 What is a Serviceguard Extension for RAC Cluster? ......................................................................
Manual Startup and Shutdown..................................................................................................29 Network Monitoring.......................................................................................................................30 SGeRAC Heartbeat Network.....................................................................................................30 CSS Heartbeat Network...............................................................................................
Initializing the Veritas Volume Manager.........................................................................................55 Using CVM 5.x or later....................................................................................................................55 Preparing the Cluster and the System Multi-node Package for use with CVM 5.x or later......55 Mirror Detachment Policies with CVM................................................................................57 Using CVM 5.x...............
Creating Mirrored Logical Volumes for RAC Redo Logs and Control Files.............................78 Creating Mirrored Logical Volumes for RAC Data Files...........................................................78 Creating RAC Volume Groups on Disk Arrays .............................................................................79 Creating Logical Volumes for RAC on Disk Arrays.......................................................................80 Oracle Demo Database Files ...........................
Types of Cluster and Package States..............................................................................................109 Examples of Cluster and Package States..................................................................................109 Types of Cluster and Package States........................................................................................111 Cluster Status .................................................................................................................
Example of Rolling Upgrade ........................................................................................................132 Step 1. .......................................................................................................................................133 Step 2. .......................................................................................................................................134 Step 3. ...........................................................................
List of Figures 1-1 1-2 1-3 1-4 1-5 1-6 2-1 2-2 2-3 4-1 A-1 A-2 A-3 A-4 A-5 A-6 Overview of Oracle RAC Configuration on HP-UX ....................................................................15 Group Membership Services.........................................................................................................16 Before Node Failure.......................................................................................................................22 After Node Failure......................
List of Tables 1 2-1 2-2 3-1 3-2 3-3 10 Document Edition and Printing Date............................................................................................11 Required Oracle File Names for Demo Database .........................................................................47 Required Oracle File Names for Demo Database .........................................................................59 RAC Software, Archive, Datafiles, SRVM.............................................................
Printing History Table 1 Document Edition and Printing Date Printing Date Part Number Edition June 2003 T1859-90006 First Edition June 2004 T1859-90017 Second Edition February 2005 T1859-90017 Second Edition February 2005 Update October 2005 T1859-90033 Third Edition December 2005 T1859-90033 Third Edition, First Reprint March 2006 T1859-90038 Third Edition, Second Reprint May 2006 T1859-90038 Third Edition May 2006 Update February 2007 T1859-90043 Fourth Edition February 2007 Jun
Preface This user guide covers how to use the Serviceguard Extension for RAC (Oracle Real Application Cluster) to configure Serviceguard clusters for use with Oracle Real Application Cluster software on HP High Availability clusters running the HP-UX operating system. The contents are as follows: • Chapter 1, “Introduction to Serviceguard Extension for RAC,” describes a Serviceguard cluster and provides a roadmap for using this guide.
• • VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Configuration Guide Extracts. VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Administration Guide Extracts. If you will be using Veritas Cluster Volume Manager (CVM) and Veritas Cluster File System (CFS) from Symantec with Serviceguard refer to the HP Serviceguard Storage Management Suite Version A.03.00 for HP-UX 11i v3 Release Notes.
1 Introduction to Serviceguard Extension for RAC Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real Application Cluster (RAC), formerly known as Oracle Parallel Server RDBMS, to run on HP high availability clusters under the HP-UX operating system. This chapter introduces Serviceguard Extension for RAC and shows where to find different kinds of information in this book.
databases. Further, when properly configured, Serviceguard Extension for RAC provides a highly available database that continues to operate even if one hardware component should fail. Group Membership Oracle RAC systems implement the concept of group membership, which allows multiple instances of RAC to run on each node. Related processes are configured into groups. Groups allow processes in different instances to choose which other processes to interact with.
There are also packages that run on several cluster nodes at once, and do not fail over. These are called system multi-node packages and multi-node packages. As of Serviceguard Extension for RAC A.11.18, the non-failover packages that are supported are those specified by Hewlett-Packard, and you can create your own multi-node packages.
Package Dependencies When CFS is used as shared storage, the application and software using the CFS storage should be configured to start and stop using Serviceguard packages. These application packages should be configured with a package dependency on the underlying multi-node packages, which manages the CFS and CVM storage reserves. Configuring the application to be start/stop through Serviceguard package is to ensure the synchronization of storage activation/deactivation and application startup/shutdown.
CRS manages Oracle's cluster resources based on configuration—including start, stop, and monitor, and failover of the resources. EVM publishes events generated by CRS and may run scripts when certain events occur. When installed on a SGeRAC cluster, both the Oracle Cluster Software and RAC can continue to rely on the shared storage capability, networking monitoring, as well as other capabilities provided through Serviceguard and SGeRAC.
How Cluster Interconnect Subnet Works The CLUSTER_INTERCONNECT_SUBNET parameter works similar to the existing SUBNET package configuration parameter. The most notable difference is in the failure handling of the subnets monitored using these individual parameters.
How Serviceguard Works with Oracle 9i RAC Serviceguard provides the cluster framework for Oracle, a relational database product in which multiple database instances run on different cluster nodes. A central component of Real Application Clusters is the distributed lock manager (DLM), which provides parallel cache management for database instances.
Figure 1-3 Before Node Failure Figure 1-4 shows the condition where node 1 has failed and Package 1 has been transferred to node 2. Oracle instance 1 is no longer operating, but it does not fail over to node 2. The IP address for package 1 was transferred to node 2 along with the package. Package 1 continues to be available and is now running on node 2. Also, node 2 can now access both the Package 1 disk and Package 2 disk. Oracle instance 2 now handles all database access, since instance 1 has gone down.
Figure 1-4 After Node Failure In the above figure, pkg1 and pkg2 are not instance packages. They are shown to illustrate the movement of the packages. Larger Clusters Serviceguard Extension for RAC supports clusters of up to 16 nodes. The actual cluster size is limited by the type of storage and the type of volume manager used. Up to Four Nodes with SCSI Storage You can configure up to four nodes using a shared F/W SCSI bus; for more than four nodes, FibreChannel must be used.
Figure 1-5 Four-Node RAC Cluster In this type of configuration, each node runs a separate instance of RAC and may run one or more high availability packages as well. The figure shows a dual Ethernet configuration with all four nodes connected to a disk array (the details of the connections depend on the type of disk array). In addition, each node has a mirrored root disk (R and R).
Figure 1-6 Eight-Node Cluster with EVA, XP or EMC Disk Array FibreChannel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP 9000 Servers Configuration Guide, available through your HP representative.
4. 5. Restart the Serviceguard cluster. Restart Oracle Clusterware (for Oracle 10g, 11gR1, and 11gR2) and Oracle RAC database instance on all nodes. Use the following steps to disable the GMS authorization: 1. 2. 3. 4. 5. 26 If Oracle RAC database instance and Oracle Clusterware (for Oracle 10g, 11gR1, and 11gR2) are running, shut them down on all nodes. Halt the Serviceguard cluster. Edit /etc/opt/nmapi/nmutils.conf and comment the GMS_USER[] settings on all nodes. Restart the Serviceguard cluster.
2 Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 10g/11gR1/11gR2 with Serviceguard.
CSS Timeout When SGeRAC is on the same cluster as Oracle Cluster Software, the CSS timeout is set to a default value of 600 seconds (10 minutes) at Oracle software installation. This timeout is configurable with Oracle tools and should not be changed without ensuring that the CSS timeout allows enough time for Serviceguard Extension for RAC (SGeRAC) reconfiguration and to allow multipath (if configured) reconfiguration to complete.
Multipathing Multipathing is automatically configured in HP-UX 11i v3 (this is often called native multipathing). Multipathing is supported through either SLVM pvlinks or CVM Dynamic Multipath (DMP). In some configurations, SLVM or CVM does not need to be configured for multipath as the multipath is provided by the storage array.
Network Monitoring SGeRAC cluster provides network monitoring. For networks that are redundant and monitored by Serviceguard cluster, Serviceguard cluster provides local failover capability between local network interfaces (LAN) that is transparent to applications utilizing User Datagram Protocol (UDP) and Transport Control Protocol (TCP).
NOTE: srvctl and sqlplus are Oracle commands. Manual Startup and Shutdown Manual RAC instance startup and shutdown is supported through the following commands: srvctl or sqlplus. Shared Storage It is expected the shared storage is available when the RAC instance is started. Since the RAC instance expects the shared storage to be available, ensure the shared storage is activated. For SLVM, the shared volume groups must be activated and for CVM, the disk group must be activated.
• • • • RAC GCS (cache fusion) traffic may be very high, so an additional dedicated heartbeat network for Serviceguard needs to be configured. Some networks, such as Infiniband, are not supported by CFS/CVM, so the CSS-HB/RAC-IC traffic may need to be on a separate network that is different from SG-HB network. Certain configurations for fast re-configurations requires a dual Serviceguard heartbeat network, and CSS-HB/RAC-IC does not support multiple networks for HA purposes.
raw logical volume device file. Be sure to include the desired size in MB. Following is a sample worksheet filled out. However, this sample is only representative. For different versions of the Oracle database, the size of files are different. Refer to Appendix B: “Blank Planning Worksheets”, for samples of blank worksheets. Make as many copies as you need. Fill out the worksheet and keep it for future reference.
Oracle Control File 2: ___/dev/vg_ops/ropsctl2.ctl______110______ Oracle Control File 3: ___/dev/vg_ops/ropsctl3.ctl______110______ Instance 1 Redo Log 1: ___/dev/vg_ops/rops1log1.log_____120______ Instance 1 Redo Log 2: ___/dev/vg_ops/rops1log2.log_____120_______ Instance 1 Redo Log 3: ___/dev/vg_ops/rops1log3.
NOTE: All nodes in the cluster must be either SGeRAC nodes or Serviceguard nodes. For the up-to-date version compatibility for Serviceguard and HP-UX, see the SGeRAC release notes for your version. 1. 2. 3. 4. Mount the distribution media in the tape drive, CD, or DVD reader. Run Software Distributor, using the swinstall command. Specify the correct input device. Choose the following bundle from the displayed list: Serviceguard Extension for RAC 5.
NOTE: 4.1 and later versions of Veritas Volume Manager (VxVM) and Dynamic Multipathing (DMP) from Symantec are supported on HP-UX 11i v3, but do not provide multipathing and load balancing. Instead, DMP acts as a pass-through driver, allowing multipathing and load balancing to be controlled by the HP-UX I/O subsystem.
NOTE: For more detailed information, see the SGeRAC Toolkit README file located in the /opt/cmcluster/SGeRAC/toolkit directory. SGeRAC Toolkit The SGeRAC Toolkit supports both legacy and modular package formats, the following new files are created to support the modular package format: /etc/cmcluster/modules/sgerac/erac_tk_oc /etc/cmcluster/modules/sgerac/erac_tk_oc.1 /etc/cmcluster/modules/sgerac/erac_tk_asmdg /etc/cmcluster/modules/sgerac/erac_tk_asmdg.
CLUSTER_INTERCONNECT_SUBNET can be used to recover from Oracle cluster communications network failures in certain configurations. Single Network for Cluster Communications The single network configuration is the most common configuration for network installations. In this configuration, there is sufficient bandwidth for all cluster communications traffic to go through one network. If there are multiple databases, all database traffic goes through the same network.
NOTE: Starting with Serviceguard A.11.19, the faster failover capability is in core Serviceguard. This configuration can be used for faster failover. Figure 2-2 SG-HB/RAC-IC Traffic Separation Each primary and standby pair protects against a single failure. With the SG-HB on more than one subnet, a single subnet failure will not trigger a Serviceguard reconfiguration.
Oracle Clusterware Package: PACKAGE_NAME PACKAGE_TYPE LOCAL_LAN_FAILOVER_ALLOWED NODE_FAIL_FAST_ENABLED DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION CRS_PACKAGE MULTI_NODE YES NO CI-PACKAGE CI-PACKAGE=UP SAME_NODE Oracle Cluster Interconnect Subnet Package: Package to monitor the CSS-HB subnet PACKAGE_NAME CI-PACKAGE PACKAGE_TYPE MULTI_NODE LOCAL_LAN_FAILOVER_ALLOWED YES NODE_FAIL_FAST_ENABLED YES CLUSTER_INTERCONNECT_SUBNET 192.168.1.
The database instances that use 192.168.2.0 must have cluster_interconnects defined in their SPFILE or PFILE as follows: orcl1.cluster_interconnects=’192.168.2.1’ orcl2.cluster_interconnects=’192.168.2.2’ NOTE: Do not configure CLUSTER_INTERCONNECT_SUBNET in the RAC instance package if the RAC-IC network is the same as CSS-HB network.
NOTE: 1. The “F” represents the Serviceguard failover time as given by the max_reformation_duration field of cmviewcl –v –f line output. 2. SLVM timeout is documented in the whitepaper, LVM link and Node Failure Recovery Time. Limitations of Cluster Communication Network Monitor The Cluster Interconnect Monitoring feature does not coordinate with any feature handling subnet failures (including self).
NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. The steps shown in the following section are for configuring the volume groups in Serviceguard clusters LVM version 1.0. For more information on using and configuring LVM version 2.x, see the HP-UX 11i Version 3: HP-UX System Administrator's Guide: Logical Volume Management located at: www.hp.com/go/hpux-core-docs —> HP-UX 11i v3. For LVM version 2.
1. First, set up the group directory for vgops: # mkdir /dev/vg_ops 2. Next, create a control file named group in the directory /dev/vg_ops, as follows: # mknod /dev/vg_ops/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the form 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured.
NOTE: With LVM 2.1 and above, mirror write cache (MWC) recovery can be set to ON for RAC Redo Logs and Control Files volumes. Example: # lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vg_ops NOTE: The character device file name (also called the raw logical volume name) is used by the Oracle DBA in building the RAC database. Creating Mirrored Logical Volumes for RAC Data Files Following a system crash, the mirrored logical volumes need to be resynchronized, which is known as “resilvering.
NOTE: The character device file name (also called the raw logical volume name) is used by the Oracle DBA in building the OPS database. Creating RAC Volume Groups on Disk Arrays The procedure described in this section assumes that you are using RAID-protected disk arrays and LVM’s physical volume links (PV links) to define redundant data paths from each node in the cluster to every logical unit on the array.
LVM will now recognize the I/O channel represented by/dev/dsk/c0t15d0 as the primary link to the disk. If the primary link fails, LVM will automatically switch to the alternate I/O channel represented by /dev/dsk/c1t3d0. Use the vgextend command to add additional disks to the volume group, specifying the appropriate physical volume name for each PV link. Repeat the entire procedure for each distinct volume group you wish to create.
Table 2-1 Required Oracle File Names for Demo Database (continued) Logical Volume Name LV Size (MB) Raw Logical Volume Path Name Oracle File Size (MB)* pwdfile.ora 5 /dev/vg_ops/rpwdfile.ora 5 opsundotbs1.dbf 508 /dev/vg_ops/ropsundotbs1.log 500 opsundotbs2.dbf 508 /dev/vg_ops/ropsundotbs2.log 500 example1.dbf 168 /dev/vg_ops/ropsexample1.
3. On ftsys10 (and other nodes, as necessary), create the volume group directory and the control file named group. # mkdir /dev/vg_ops # mknod /dev/vg_ops/group c 64 0xhh0000 For the group file, the major number is always 64, and the hexadecimal minor number has the form 0xhh0000 where hh must be unique to the volume group you are creating. If possible, use the same number as on ftsys9. Use the following command to display a list of existing volume groups: # ls -l /dev/*/group 4.
For more information, refer to your version of the Serviceguard Extension for RAC Release Notes and HP Serviceguard Storage Management Suite Release Notes located at www.hp.com/go/hpux-serviceguard-docs. CAUTION: Once you create the disk group and mount point packages, you must administer the cluster with CFS commands, including cfsdgadm, cfsmntadm, cfsmount, and cfsumount. You must not use the HP-UX mount or umount command to provide or remove access to a shared file system in a CFS environment.
4. Start the cluster: # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 5. STATUS up STATUS up up STATE running running Configure the Cluster Volume Manager (CVM). Configure the system multi-node package, SG-CFS-pkg, to configure and start the CVM/CFS stack. Unlike VxVM-CVM-pkg, the SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and supports multiple subnets.
The following output will be displayed: Package name “SG-CFS-DG-1” was generated to control the resource shared disk group “cfsdg1” is associated with the cluster. 10. Activate the disk group. # cfsdgadm activate cfsdg1 11. Creating volumes and adding a cluster filesystem.
NOTE: The diskgroup and mount point multi-node packages (SG-CFS-DG_ID# and SG-CFS-MP_ID#) do not monitor the health of the disk group and mount point. They check that the application packages that depend on them have access to the disk groups and mount points. If the dependent application package loses access and cannot read and write to the disk, it will fail. However, the DG or MP multi-node package will not fail. 13. Mount cluster filesystem.
# cfsumount /cfs/mnt2 # cfsumount /cfs/mnt3 2. Delete mount point multi-node package.
NOTE: The Oracle 11gR2 OUI allows only ASM over SLVM, ASM over raw device files, Cluster File System for Clusterware files, and Database files. Initializing the Veritas Volume Manager If you are about to create disk groups for the first time, you need to initialize the Volume Manager. This is done by creating a disk group known as rootdg that contains at least one disk.
When CVM starts, it selects a master node. From this node, you must issue the disk group configuration commands. To determine the master node, issue the following command from each node in the cluster: # vxdctl -c mode The following output will be displayed: mode: enabled: cluster active SLAVEmaster: ever3b or mode: enabled: cluster active MASTERmaster: ever3b • Converting disks from LVM to CVM. Use the vxvmconvert utility to convert LVM volume groups into CVM disk groups.
IMPORTANT: After creating these files, use the vxedit command to change the ownership of the raw volume files to oracle and the group membership to dba, and to change the permissions to 660. Example: # cd /dev/vx/rdsk/ops_dg # vxedit -g ops_dg set user=oracle * # vxedit -g ops_dg set group=dba * # vxedit -g ops_dg set mode=660 * The logical volumes are now available on the primary node, and the raw logical volume names can now be used by the Oracle DBA.
WARNING! The above file should never be edited. After the above command completes, start the cluster and create disk groups for shared use as described in the following sections. Starting the Cluster and Identifying the Master Node Run the cluster to activate the special CVM package: # cmruncl After the cluster is started, it will run with a special system multi-node package named VxVM-CVM-pkg that is on all nodes.
NAME rootdg ops_dg STATE ID enabled enabled,shared 971995699.1025.node1 972078742.1084.node2 Creating Volumes Use the vxassist command to create logical volumes. The following is an example: # vxassist -g ops_dg make log_files 1024m This command creates a 1024MB volume named log_files in a disk group named ops_dg. The volume can be referenced with the block device file /dev/vx/dsk/ops_dg/log_files or the raw (character) device file /dev/vx/rdsk/ops_dg/log_files.
Table 2-2 Required Oracle File Names for Demo Database (continued) Volume Name Size (MB) Raw Device File Name Oracle File Size (MB) ops1log3.log 128 /dev/vx/rdsk/ops_dg/ops1log3.log 120 ops2log1.log 128 /dev/vx/rdsk/ops_dg/ops2log1.log 120 ops2log2.log 128 /dev/vx/rdsk/ops_dg/ops2log2.log 120 ops2log3.log 128 /dev/vx/rdsk/ops_dg/ops2log3.log 120 opssystem.dbf 508 /dev/vx/rdsk/ops_dg/opssystem.dbf 500 opssysaux.dbf 808 /dev/vx/rdsk/ops_dg/opssysaux.dbf 800 opstemp.
the package control script (on HP-UX releases that support them). See “About Veritas CFS and CVM from Symantec” (page 18). For more detailed information on the package configuration process, refer to the Managing Serviceguard Sixteenth Edition user’s guide. Prerequisites for Oracle 10g, 11gR1, or 11gR2 (Sample Installation) The following sample steps prepare a SGeRAC cluster for Oracle 10g, 11gR1, or 11gR2. Refer to the Oracle documentation for Oracle installation details. 1.
# chmod 775 /mnt/app 7. Create Oracle cluster software home directory. For installing Oracle cluster software on local file system, create the directories on each node. # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # chown -R oracle:oinstall /mnt/app/crs/oracle/product/10.2.0/crs # chmod -R 775 /mnt/app/crs/oracle/product/10.2.0/crs 8. Create Oracle base directory (for RAC binaries on local file system). If installing RAC binaries on local file system, create the oracle base directory on each node.
The following is a sample of the mapping file for DBCA: system=/dev/vg_ops/ropssystem.dbf sysaux=/dev/vg_ops/ropssysaux.dbf undotbs1=/dev/vg_ops/ropsundotbs01.dbf undotbs2=/dev/vg_ops/ropsundotbs02.dbf example=/dev/vg_ops/ropsexample1.dbf users=/dev/vg_ops/ropsusers.dbf redo1_1=/dev/vg_ops/rops1log1.log redo1_2=/dev/vg_ops/rops1log2.log redo2_1=/dev/vg_ops/rops2log1.log redo2_2=/dev/vg_ops/rops2log2.log control1=/dev/vg_ops/ropsctl1.ctl control2=/dev/vg_ops/ropsctl2.ctl control3=/dev/vg_ops/ropsctl3.
NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. The steps shown in the following section are for configuring the volume groups in Serviceguard clusters LVM version 1.0. For more information on using and configuring LVM version 2.x, see the HP-UX System Administrator's Guide: Logical Volume Management located at: www.hp.com/go/hpux-core-docs —> HP-UX 11i v3.
1. 2. 3. In this example, the path to ORACLE_HOME is on a local file system /mnt/app/oracle/ product//db_1. Select installation for database software only. When prompted, run root.sh on each node. Installing RAC Binaries on Cluster File System Logon as a “oracle” user: $ export ORACLE BASE=/cfs/mnt1/oracle $ export DISPLAY={display}:0.0 $ cd <10g/11g RAC installation disk directory> $ ./runInstaller Use following guidelines when installing on a local file system: 1.
Creating a RAC Demo Database on CFS Export environment variables for “oracle” user: export ORACLE_BASE=/cfs/mnt1/oracle export ORACLE_HOME=$ORACLE_BASE/product//db_1 export ORA_CRS_HOME=/mnt/app/crs/oracle/product//crs LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32 export LD_LIBRARY_PATH SHLIB_PATH export \ PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin: CLASSPATH=$ORACLE_HOME/jre:$ORA
Configuring Oracle to Use Oracle Disk Manager Library NOTE: 1. 2. 3. The following steps are specific to CFS 4.1 or later. Login as a Oracle user. Shutdown database. Link the Oracle Disk Manager library into Oracle home. For Oracle 10g on HP 9000 Systems: $ rm ${ORACLE_HOME}/lib/libodm10.sl $ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm10.sl For Oracle 10g on Integrity Systems: $ rm ${ORACLE_HOME}/lib/libodm10.so $ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm10.
cl opt idn: cl opt rsv: **********: 3. 0 332 17 Verify that the Oracle disk manager is loaded: # kcmodule -P state odm Output: state loaded 4. In the alert log, verify the Oracle instance is running. The log should contain output similar to the following: For CFS 4.1: Oracle instance running with ODM: VERITAS 4.1 ODM Library, Version 1.1 For CFS 5.0: Oracle instance running with ODM: VERITAS 5.0 ODM Library, Version 1.0 Configuring Oracle to Stop Using Oracle Disk Manager Library NOTE: 1. 2. 3.
Preparing Oracle Cluster Software for Serviceguard Packages CRS starts all RAC instances by default. You must disable CRS from starting up RAC instances when you configure your RAC instances in SGeRAC packages. • On each node of the cluster, disable the automatic startup of the Oracle Clusterware at boot time. Login as root and enter: : $ORA_CRS_HOME/bin/crsctl disable crs Wait until Oracle Cluster Software completely stops. (Check CRS logs or check for Oracle processes, ps -ef | grep ocssd.
• DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION mp1 SG-CFS-MP-1=UP SAME_NODE DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION mp2 SG-CFS-MP-2=UP SAME_NODE DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION mp3 SG-CFS-MP-3=UP SAME_NODE Start and stop Oracle Cluster Software. In the Serviceguard package control script, configure the Oracle Cluster Software start in the customer_defined_run_cmds function. For 10g 10.1.0.4 or later: /sbin/init.d/init.crs start For 10g 10.2.0.
3 Serviceguard Configuration for Oracle 9i RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 9i with Serviceguard.
Data: User data ___/dev/vg_ops/ropsdata1.dbf_200__________ Data: User data ___/dev/vg_ops/ropsdata2.dbf__200__________ Data: User data ___/dev/vg_ops/ropsdata3.dbf__200__________ Data: Rollback ___/dev/vg_ops/ropsrollback.dbf__300_________ Parameter: spfile1 /dev/vg_ops/ropsspfile1.ora __5_____ Instance 1 undotbs1: /dev/vg_ops/ropsundotbs1.dbf___312___ Instance 2 undotbs2: /dev/vg_ops/ropsundotbs2.dbf___312___ Data: example1__/dev/vg_ops/ropsexample1.
Considerations on using CFS for RAC datafiles and Server Management Storage (SRVM) Use the following list when considering to use CFS for database storage: • Single file system view. — Simpler setup for archive recovery since archive area is visible by all nodes. • • • • • Oracle create database files. Online changes (OMF – Oracle Managed Files) within CFS. Better manageability. Manual intervention when modifying volumes, DGs, disks. Requires the SGeRAC and CFS software.
Data: Temp ___/dev/vx/rdsk/ops_dg/temp.dbf______100_______ Data: Users ___/dev/vx/rdsk/ops_dg/users.dbf_____120_________ Data: Tools ___/dev/vx/rdsk/ops_dg/tools.dbf____15___________ Data: User data ___/dev/vx/rdsk/ops_dg/data1.dbf_200__________ Data: User data ___/dev/vx/rdsk/ops_dg/data2.dbf__200__________ Data: User data ___/dev/vx/rdsk/ops_dg/data3.dbf__200__________ Data: Rollback ___/dev/vx/rdsk/ops_dg/rollback.dbf__300_________ Parameter: spfile1 /dev/vx/rdsk/ops_dg/spfile1.
access between multiple nodes by applications that can manage read/write access contention, such as Oracle Real Application Cluster (RAC). NOTE: CVM (and CFS - Cluster File System) are supported on some, but not all, current releases of HP-UX. See the latest release notes for your version of Serviceguard at www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard. Veritas Storage Management Products Veritas Volume Manager (VxVM) 3.5 is not supported on HP-UX 11i v3. If you are running VxVM 3.
NOTE: Starting with Oracle 10g and later, Oracle Clusterware processes register with cmgmsd while Oracle server processes register with Oracle Clusterware. The maximum number of Oracle server processes supported is determined by Oracle Clusterware. The maximum number of processes registered by Oracle Clusterware should not exceed the maximum supported by cmgmsd. Oracle foreground server processes are needed to handle the requests of the DB client connected to the DB instance.
NOTE: For more information, see the Serviceguard Version A.11.20 Release Notes at www.hp.com/go/hpux-serviceguard-docs —> High Availability. Creating Volume Groups and Logical Volumes If your volume groups have not been set up, use the procedure in the next sections. If you have already done LVM configuration, skip ahead to the section ““Installing Oracle Real Application Clusters”.
NOTE: For more information on using LVM, refer to the HP-UX Managing Systems and Workgroups manual. Building Mirrored Logical Volumes for RAC with LVM Commands After you create volume groups and define physical volumes for use in them, you define mirrored logical volumes for data, logs, and control files. It is recommended that you use a shell script to issue the commands described in the next sections.
that mirroring is PVG-strict—it occurs between different physical volume groups; the -n system.dbf option lets you specify the name of the logical volume; and the -L 408 option allocates 408 megabytes. Example for “MWC”: # lvcreate -m 1 -M y -s g -n system.dbf -L 408 /dev/vg_ops The -M y option ensures that mirror write cache recovery is set ON.
/dev/dsk/c0t15d0 /dev/dsk/c1t3d0 Use the following procedure to configure a volume group for this logical disk: 1. First, set up the group directory for vg_ops: # mkdir /dev/vg_ops 2. Next, create a control file named group in the directory /dev/vg_ops, as follows: # mknod /dev/vg_ops/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the format: 0xhh0000 where hh must be unique to the volume group you are creating.
Table 3-2 Required Oracle File Names for Demo Database Logical Volume Name LV Size (MB) Raw Logical Volume Path Name Oracle File Size (MB)* opsctl1.ctl 108 /dev/vg_ops/ropsctl1.ctl 100 opsctl2.ctl 108 /dev/vg_ops/ropsctl2.ctl 100 opsctl3.ctl 108 /dev/vg_ops/ropsctl3.ctl 100 ops1log1.log 28 /dev/vg_ops/rops1log1.log 20 ops1log2.log 28 /dev/vg_ops/rops1log2.log 20 ops1log3.log 28 /dev/vg_ops/rops1log3.log 20 ops2log1.log 28 /dev/vg_ops/rops2log1.log 20 ops2log2.
Displaying the Logical Volume Infrastructure To display the volume group, use the vgdisplay command: # vgdisplay -v /dev/vg_ops Exporting the Logical Volume Infrastructure Before the Oracle volume groups can be shared, their configuration data must be exported to other nodes in the cluster. This is done either in Serviceguard Manager or by using HP-UX commands, as shown in the following sections. NOTE: Serviceguard Manager is the graphical user interface for Serviceguard.
and to build the correct Oracle runtime executables. When the executables are installed to a cluster file system, the Oracle installer has an option to install the executables once. When executables are installed to a local file system on each node, the Oracle installer copies the executables to the other nodes in the cluster. For details on Oracle installation, refer to the Oracle installation documentation.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # configured for the quorum server. When one subnet fails, Serviceguard uses the next available subnet to communicate with the quorum server. QS_HOST is used to specify the quorum server and QS_ADDR can be used to specify additional IP addresses for the quorum server. The QS_HOST entry must be specified (only once) before any other QS parameters. Only one QS_ADDR entry is used to specify the additional IP address.
# # # # # # # # # # # # # package weights; node capacity is checked against the corresponding package weight to determine if the package can run on that node. CAPACITY_NAME specifies a name for the capacity. The capacity name can be any string that starts and ends with an alphanumeric character, and otherwise contains only alphanumeric characters, dot (.), dash (-), or underscore (_). Maximum string length is 39 characters. Duplicate capacity names are not allowed.
NETWORK_INTERFACE lan2 STATIONARY_IP 10.142.2.4 STATIONARY_IP fec0::8e02:0:0:a8e:4 STATIONARY_IP 3ffe:1000:0:8e02::a8e:4 # CLUSTER_LOCK_LUN # # # # # # Route information route id 1: 16.89.114.145 route id 2: 10.142.1.4 route id 3: 10.142.2.4 CAPACITY_NAME CAPACITY_VALUE # Warning: There are no standby network interfaces for lan0. # Possible standby Network Interfaces for lan1: lan3. # Warning: There are no standby network interfaces for lan2. # Cluster Timing Parameters (microseconds).
HOSTNAME_ADDRESS_FAMILY IPV4 # IP Monitor Configuration Parameters. # The following set of three parameters can be repeated as necessary. # SUBNET is the subnet to be configured whether or not to be monitored # at IP layer. # IP_MONITOR is set to ON if the subnet is to be monitored at IP layer. # IP_MONITOR is set to OFF if the subnet is not to be monitored at IP layer. # POLLING_TARGET is the IP address to which polling messages are sent # from each network interface in the subnet.
# # # # # # # # # # # # # # # # # # # # # weight, repeat this process for each weight. Note: for the reserved weight "package_limit", the default weight is always one. This default cannot be changed in the cluster configuration file, but it can be overriden in the package configuration file. # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # Access Control Policy Parameters. # # # # # List of cluster aware LVM Volume Groups.
# VOLUME_GROUP # # # # # # # # /dev/vg02 List of OPS Volume Groups. Formerly known as DLM Volume Groups, these volume groups will be used by OPS or RAC cluster applications via the vgchange -a s command. (Note: the name DLM_VOLUME_GROUP is also still supported for compatibility with earlier versions.
IMPORTANT: Creating a rootdg disk group is only necessary the first time you use the Volume Manager. CVM 5.x or later does not require a rootdg. 2. Create the cluster file. # cd /etc/cmcluster # cmquerycl -C clm.asc -n ever3a -n ever3b Edit cluster file. 3. Create the cluster. # cmapplyconf -C clm.asc 4. Start the cluster. # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 5.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: # /etc/vx/bin/vxdisksetup -i c4t4d0 8. Create the disk group for RAC. Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: # vxdg -s init cfsdg1 c4t4d0 9. Create the disk Group Multi-Node package.
Package name “SG-CFS-MP-2” was generated to control the resource. Mount point “/cfs/mnt2” was associated with the cluster. # cfsmntadm add cfsdg1 volsrvm /cfs/cfssrvm all=rw The following output will be displayed: Package name “SG-CFS-MP-3” was generated to control the resource. Mount point “/cfs/cfssrvm” that was associated with the cluster. 13. Mount cluster filesystem. # cfsmount /cfs/mnt1 # cfsmount /cfs/mnt2 # cfsmount /cfs/cfssrvm 14. Check CFS mount points.
# cfsmntadm delete /cfs/cfssrvm The following output will be generated: Mount point “/cfs/cfssrvm” was disassociated from the cluster Cleaning up resource controlling shared disk group “cfsdg1” Shared disk group “cfsdg1” was disassociated from the cluster. NOTE: 3. The disk group is deleted if there is no dependency. Delete DG MNP. # cfsdgadm delete cfsdg1 The following output will be generated: Shared disk group “cfsdg1” was disassociated from the cluster.
IMPORTANT: Creating a rootdg disk group is only necessary the first time you use the Volume Manager. CVM 5.0 or later does not require a rootdg. Using CVM 5.x or later This section has information on how to prepare the cluster and the system multi-node package with CVM 5.x or later only, without the CFS filesystem (on HP-UX releases that support them). See “About Veritas CFS and CVM from Symantec” (page 18). For more detailed information on how to configure CVM 5.
mode: enabled: cluster active - SLAVEmaster: ever3b • Converting disks from LVM to CVM. Use the vxvmconvert utility to convert LVM volume groups into CVM disk groups. Before you can do this, the volume group must be deactivated—any package that uses the volume group must be halted. This procedure is described in the Managing Serviceguard user guide Appendix G. • Initializing disks for CVM. You need to initialize the physical disks that will be employed in CVM disk groups.
IMPORTANT: After creating these files, use the vxedit command to change the ownership of the raw volume files to oracle and the group membership to dba, and to change the permissions to 660. Example: # cd /dev/vx/rdsk/ops_dg # vxedit -g ops_dg set user=oracle * # vxedit -g ops_dg set group=dba * # vxedit -g ops_dg set mode=660 * The logical volumes are now available on the primary node, and the raw logical volume names can now be used by the Oracle DBA.
CLUSTER bowls STATUS up NODE spare split strike STATUS up up up STATE running running running SYSTEM_MULTI_NODE_PACKAGES: PACKAGE STATUS VxVM-CVM-pkg up STATE running When CVM starts up, it selects a master node. From this node, you must issue the disk group configuration commands. To determine the master node, issue the following command from each node in the cluster: # vxdctl -c mode One node will identify itself as the master. Create disk groups from this node.
# vxdg list IMPORTANT: After creating these files, use the vxedit command to change the ownership of the raw volume files to oracle and the group membership to dba, and to change the permissions to 660. Example: # cd /dev/vx/rdsk/ops_dg # vxedit -g ops_dg set user=oracle * # vxedit -g ops_dg set group=dba * # vxedit -g ops_dg set mode=660 * The logical volumes are available on the primary node, and the Oracle DBA can use the raw logical volume names.
Table 3-3 Required Oracle File Names for Demo Database (continued) Volume Name Size (MB) Raw Device File Name Oracle File Size (MB) opsdata2.dbf 208 /dev/vx/rdsk/ops_dg/opsdata2.dbf 200 opsdata3.dbf 208 /dev/vx/rdsk/ops_dg/opsdata3.dbf 200 opsrollback.dbf 308 /dev/vx/rdsk/ops_dg/opsroolback.dbf 300 opsspfile1.ora 5 /dev/vx/rdsk/ops_dg/opsspfile1.ora 5 opsundotbs1.dbf 320 /dev/vx/rdsk/ops_dg/opsundotbs1.dbf 312 opsundotbs2.dbf 320 /dev/vx/rdsk/ops_dg/opsundotbs2.
Install Oracle Software into CFS Home Oracle RAC software is installed using the Oracle Universal Installer. This section describes installation of Oracle RAC software onto a CFS home. 1. Oracle pre-installation steps. a. Create user accounts. Create user and group for Oracle accounts on all nodes using the following commands: # groupadd -g 99 dba # useradd -g dba -u 999 -d /cfs/mnt1/oracle oracle Create Oracle home directory on CFS.
1. Set up environment variables. Use the following as an example: export ORACLE_BASE=/cfs/mnt1/oracle export ORACLE_HOME=$ORACLE_BASE/product/9.2.0.2 export PATH=$ORACLE_HOME/bin:$PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32 export LD_LIBRARY_PATH SHLIB_PATH export CLASSPATH=/opt/java1.
3. Link the Oracle Disk Manager library into Oracle home using the following commands: For HP 9000 systems: $ rm ${ORACLE_HOME}/lib/libodm9.sl $ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm9.sl For Integrity systems: $ rm ${ORACLE_HOME}/lib/libodm9.so $ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm9.so 4. Start the Oracle database. Verify Oracle Disk Manager is Running NOTE: 1. 2. The following steps are specific to CFS 4.1 or later.
For CFS 5.0: Oracle instance running with ODM: VERITAS 5.0 ODM Library, Version 1.0 Configuring Oracle to Stop using Oracle Disk Manager Library NOTE: 1. 2. 3. The following steps are specific to CFS 4.1 or later. Login as Oracle user. Shutdown the database. Change directories: $ cd ${ORACE_HOME}/lib 4. Remove the file linked to the ODM library: For HP 9000 systems: $ rm libodm9.sl $ ln -s ${ORACLE_HOME}/lib/libodmd9.sl ${ORACLE_HOME}/lib/libodm9.sl For Integrity systems: $ rm libodm9.
1. 2. 3. 4. Shut down the Oracle applications, if any. Shut down Oracle. Deactivate the database volume groups or disk groups. Shut down the cluster (cmhaltnode or cmhaltcl). If the shutdown sequence described above is not followed, cmhaltcl or cmhaltnode may fail with a message that GMS clients (RAC 9i) are active or that shared volume groups are active.
1. 2. In the ASCII package configuration file, set the AUTO_RUN parameter to NO, or if you are using Serviceguard Manager to configure packages, set Automatic Switching to Disabled. This keeps the package from starting up immediately when the node joins the cluster, and before RAC is running. You can then manually start the package using the cmmodpkg -e packagename command after RAC is started.
• • • • • • • • • CVM_ACTIVATION_CMD. Do not use the VG[] or VXVM_DG[] parameters for CVM disk groups. If you are using VxVM disk groups without CVM, enter the names of VxVM disk groups that will be imported using the VXVM_DG[] array parameters. Enter one disk group per array element. Do not use theCVM_DG[] or VG[] parameters for VxVM disk groups without CVM. Also, do not specify an activation command. Add the names of logical volumes and file systems that will be mounted on them.
Enter the names of the LVM volume groups you wish to activate in shared mode in the VG[] array. Use a different array element for each RAC volume group. (Remember that RAC volume groups must also be coded in the cluster configuration file using OPS_VOLUME_GROUP parameters.
5. Copy the Oracle shell script templates from the ECMT default source directory to the package directory. # cd /etc/cmcluster/pkg/${SID_NAME} # cp -p /opt/cmcluster/toolkit/oracle/* Example: # cd /etc/cmcluster/pkg/ORACLE_TEST0 # cp -p /opt/cmcluster/toolkit/oracle/* Edit haoracle.conf as per README. 6. Gather the package service name for monitoring Oracle instance processes. In Serviceguard Manager, this information can be found under the “Services” tab.
4 Maintenance and Troubleshooting This chapter includes information about carrying out routine maintenance on a Real Application Cluster configuration. Starting with version SGeRAC A.11.17, all log messages from cmgmsd log to /var/adm/syslog/syslog.log by default. As presented here, these tasks differ in some details from the similar tasks described in the Managing Serviceguard documentation.
minie up running Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up NODE mo PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 STATUS up NAME lan0 lan1 lan3 STATE running Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 NAME lan0 lan1 lan3 MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg NODE_NAME minie STATUS up STATUS up Scr
Dependency_Parameters: DEPENDENCY_NAME SG-CFS-pkg PACKAGE SG-CFS-MP-1 NODE_NAME minie STATUS up STATE running STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up NODE_NAME minie STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up PACKAGE SG-CFS-MP-3 NODE_NAME minie STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up AUTO_RUN enabled SYSTEM no SWITCHING
Cluster Status The status of a cluster may be one of the following: • • • • Up. At least one node has a running cluster daemon, and reconfiguration is not taking place. Down. No cluster daemons are running on any cluster node. Starting. The cluster is in the process of determining its active membership. At least one cluster daemon is running. Unknown. The node on which the cmviewcl command is issued cannot communicate with other nodes in the cluster.
For multi-node packages, node switching Disabled means the package cannot start on that node. Status of Group Membership The state of the cluster for Oracle RAC is one of the following: • • Up. Services are active and being monitored. The membership appears in the output of cmviewcl -l group. Down. The cluster is halted and GMS services have been stopped. The membership does not appear in the output of the cmviewcl -l group.
Packages can also be configured with one of two values for the FAILBACK_POLICY parameter: • • AUTOMATIC—with this setting, a package following a failover returns to its primary node when the primary node becomes available again. MANUAL—with this setting, a package following a failover must be moved back to its original node by a system administrator. Failover and failback policies are displayed in the output of the cmviewcl -v command.
Quorum Server Status If the cluster is using a quorum server for tie-breaking services, the display shows the server name, state and status following the entry for each node, as in the following excerpt from the output of cmviewcl -v: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Quorum Server Status: NAME STATUS lp-qs up ...
CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 56/36.
CLUSTER example NODE ftsys9 STATUS up STATUS up PACKAGE pkg1 pkg2 NODE ftsys10 STATE running STATUS up up STATUS up STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE running Both packages are now running on ftsys9 and pkg2 is enabled for switching. Ftsys10 is running the daemon and no packages are running on ftsys10.
Primary Alternate Alternate Alternate up up up up enabled enabled enabled enabled manx burmese tabby persian Online Reconfiguration The online reconfiguration feature provides a method to make configuration changes online to a Serviceguard Extension for RAC (SGeRAC) cluster. Specifically, this provides the ability to add and/or delete nodes from a running SGeRAC Cluster, and to reconfigure SLVM Volume Group (VG) while it is being accessed by only one node.
Manager that occurs when the configuration is applied. However, in some cases you may want to manually make a volume group sharable. For example, if you wish to add a new shared volume group without shutting down the cluster, you can use the manual method to do it online. When convenient, it's a good practice to bring down the cluster and reconfigure it to include the new volume group. 1. Use the vgchange command on each node to ensure that the volume group to be shared is currently inactive on all nodes.
NOTE: If you wish to change the capacity of a volume group at a later time, you must deactivate and unshare the volume group first. If you add disks, you must specify the appropriate physical volume group name and make sure the /etc/lvmpvg file is correctly updated on both nodes. Making Offline Changes to Shared Volume Groups You may need to change the volume group configuration of RAC shared logical volumes to add capacity to the data files or to add log files.
where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system after the volume groups that are already configured. 13. Use the vgimport command, specifying the map file you copied from the configuration node. In the following example, the vgimport command is issued on the second node for the same volume group that was modified on the first node: # vgimport -v -m /tmp/vg_ops.map /dev/vg_ops /dev/dsk/c0t2d0/dev/dsk/c1t2d0 14.
NOTE: For CVM without CFS, if you are adding a disk group to the cluster configuration, make sure you also modify any package or create the package control script that imports and deports this disk group. If you are adding a CVM disk group, be sure to add the STORAGE_GROUP entry for the disk group to the package ASCII file. For CVM with CFS, if you are adding a disk group to the cluster configuration, make sure you also create the corresponding multi-node package.
of further action in case of a problem. For example, you could configure a disk monitor to report when a mirror was lost from a mirrored volume group being used in a non-RAC package. For additional information, refer to www.hp.com/go/hpux-ha-monitoring-docs —> HP Event Monitoring Service. Using EMS Hardware Monitors A set of hardware monitors is available for monitoring and reporting on memory, CPU, and many other system values.
NOTE: If your LVM installation requires online replacement of disk mechanisms, the use of disk arrays may be required, because software mirroring of JBODs with MirrorDisk/UX does not permit hot swapping for disks that are activated in shared mode. Replacing a Mechanism in an HA Enclosure Configured with Exclusive LVM Non-Oracle data that is used by packages may be configured in volume groups that use exclusive (one-node-at-a-time) activation.
# pvchange -a N [pv path] Alternatively, use the pvchange -a N [pv path] command to detach a disk (all paths to the disk) and close it. Use this to allow diagnostics or replace a multi-ported disk. NOTE: If the volume group is mirrored, applications can continue accessing data on mirror copies after the commands above. If the volume is not mirrored, then any access attempts to the device may hang indefinitely or time out. This depends upon the LV timeout value configured for the logical volume. 2.
5. Activate the volume group on one node in exclusive mode, then deactivate the volume group: # vgchange -a e vg_ops This will synchronize the stale logical volume mirrors. This step can be time-consuming, depending on hardware characteristics and the amount of data. 6. Deactivate the volume group: # vgchange -a n vg_ops 7.
Figure 4-1 F/W SCSI Buses with Inline Terminators The use of inline SCSI terminators allows you to do hardware maintenance on a given node by temporarily moving its packages to another node and then halting the original node while its hardware is serviced. Following the replacement, the packages can be moved back to the original node. Use the following procedure to disconnect a node that is attached to the bus with an inline SCSI terminator or with a Y cable: 1. 2. 3. 4. 5. 6. 7. 8.
4. 5. 6. 7. Remove the defective I/O card. Install the new card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed. In SAM, select the option to attach the new I/O card. Add the node back into the cluster by using Serviceguard Manager or the cmrunnode command.
Monitoring RAC Instances The DB Provider provides the capability to monitor RAC databases. RBA (Role Based Access) enables a non-root user to have the capability to monitor RAC instances using Serviceguard Manager.
A Software Upgrades Serviceguard Extension for RAC (SGeRAC) software upgrades can be done in the two following ways: • rolling upgrade • non-rolling upgrade Instead of an upgrade, moving to a new version can be done with: • migration with cold install Rolling upgrade is a feature of SGeRAC that allows you to perform a software upgrade on a given node without bringing down the entire cluster. SGeRAC supports rolling upgrades on version A.11.
For more information on support, compatibility, and features for SGeRAC, refer to the Serviceguard Compatibility and Feature Matrix, located at www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard Extension for RAC. Steps for Rolling Upgrades Use the following steps when performing a rolling SGeRAC software upgrade: 1. Halt Oracle (RAC, Clusterware) software on the local node (if running). 2. Halt Serviceguard/SGeRAC on the local node by issuing the Serviceguard cmhaltnode command. 3. Edit the /etc/rc.
NOTE: While you are performing a rolling upgrade, warning messages may appear while the node is determining what version of software is running. This is a normal occurrence and not a cause for concern. Figure A-1 Running Cluster Before Rolling Upgrade Step 1. 1. 2. Halt Oracle (RAC, Clusterware) software on node 1. Halt node 1. This will cause the node’s packages to start up on an adoptive node.
Figure A-2 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 and install the new version of Serviceguard and SGeRAC (A.11.16), as shown in Figure A-3. NOTE: If you install Serviceguard and SGeRAC separately, Serviceguard must be installed before installing SGeRAC. Figure A-3 Node 1 Upgraded to SG/SGeRAC 11.16 Step 3. 1. If you prefer, restart the cluster on the upgraded node (node 1).
Figure A-4 Node 1 Rejoining the Cluster Step 4. 1. 2. Halt Oracle (RAC, Clusterware) software on node 2. Halt node 2. You can do this in Serviceguard Manager, or from the command line issue the following: # cmhaltnode -f node2 This causes both packages to move to node 1. See Figure A-5. 3. 4. Upgrade node 2 to Serviceguard and SGeRAC (A.11.16) as shown in Figure A-5. When upgrading is finished, enter the following command on node 2 to restart the cluster on node 2: # cmrunnode node2 5.
Figure A-5 Running Cluster with Packages Moved to Node 1 Step 5. Move PKG2 back to its original node. Use the following commands: # cmhaltpkg pkg2 # cmrunpkg -n node2 pkg2 # cmmodpkg -e pkg2 The cmmodpkg command re-enables switching of the package that is disabled by the cmhaltpkg command. The final running cluster is shown in Figure A-6.
Figure A-6 Running Cluster After Upgrades Limitations of Rolling Upgrades The following limitations apply to rolling upgrades: • During a rolling upgrade, you should issue Serviceguard/SGeRAC commands (other than cmrunnode and cmhaltnode) only on a node containing the latest revision of the software. Performing tasks on a node containing an earlier revision of the software will not work or will cause inconsistent results.
Non-Rolling Software Upgrades A non-rolling upgrade allows you to perform a software upgrade from any previous revision to any higher revision or between operating system versions. For example, you may do a non-rolling upgrade from SGeRAC A.11.14 on HP-UX 11i v1 to A.11.16 on HP-UX 11i v2, given both are running the same architecture. The cluster cannot be running during a non-rolling upgrade, therefore it is necessary to halt the entire cluster in order to perform the upgrade.
8. 9. 10. 11. Recreate the SGeRAC cluster. Restart the cluster. Reinstall the cluster applications, such as RAC. Restore the data. Upgrade Using DRD DRD stands for Dynamic Root Disk. Using a Dynamic Root Disk on HP-UX 11i v3 allows you to perform the update on a clone of the root disk, then halt the node and reboot it from the updated clone root disk. You can obtain the DRD software free from www.software.hp.com—search for “DynRootDisk”.
B Blank Planning Worksheets This appendix reprints blank planning worksheets used in preparing the RAC cluster. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Disk Group Name: __________________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name:__________________________________
Index A activation of volume groups in shared mode, 119 adding packages on a running cluster, 105 administration cluster and package states, 109 array replacing a faulty mechanism, 123, 124, 125 AUTO_RUN parameter, 104 AUTO_START_TIMEOUT in sample configuration file, 83 B building a cluster CVM infrastructure, 54, 93 building an RAC cluster displaying the logical volume infrastructure, 48, 82 logical volume infrastructure, 42, 76 building logical volumes for RAC, 47, 80 C CFS, 49, 53, 89 deleting from the
HEARTBEAT_IP in sample configuration file, 83 high availability cluster defined, 15 I in-line terminator permitting online hardware maintenance, 126 installing Oracle RAC, 49, 82 installing software Serviceguard Extension for RAC, 34, 74 IP in sample package control script, 106 IP address switching, 22 L lock disk replacing a faulty mechanism, 126 logical volumes blank planning worksheet, 142 creating, 47, 80 creating for a cluster, 44, 58, 59, 78, 97 creating the infrastructure, 42, 76 disk arrays, 46, 7
optimizing packages for large numbers of storage units, 106 physical volumes creating for clusters, 43, 77 filled in planning worksheet, 141 planning worksheets for logical volume planning, 32, 33, 71, 73 worksheets for physical volume planning, 141 planning worksheets blanks, 141 point to point connections to storage devices, 24 PVG-strict mirroring creating volume groups with, 43, 77 R RAC group membership services, 21 overview of configuration, 15 status, 113 RAC cluster defined, 15 removing packages on
VXVM_DG in package control script, 106 W worksheet logical volume planning, 32, 33, 71, 73 worksheets physical volume planning, 141 worksheets for planning blanks, 141 146 Index
