Using Serviceguard Extension for RAC HP Part Number: T1859-90059 Published: September 2008
Legal Notices © Copyright 2003-2008 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents Printing History ................................................................................................................11 Preface..............................................................................................................................13 1 Introduction to Serviceguard Extension for RAC.......................................................15 What is a Serviceguard Extension for RAC Cluster? ......................................................................
Network Monitoring.......................................................................................................................29 SGeRAC Heartbeat Network.....................................................................................................30 CSS Heartbeat Network.............................................................................................................30 RAC Cluster Interconnect.......................................................................................
Using CVM 4.x or later....................................................................................................................57 Preparing the Cluster and the System Multi-node Package for use with CVM 4.x or later......58 Mirror Detachment Policies with CVM................................................................................59 Using CVM 3.x.................................................................................................................................
Creating Mirrored Logical Volumes for RAC Data Files...........................................................80 Creating RAC Volume Groups on Disk Arrays .............................................................................81 Creating Logical Volumes for RAC on Disk Arrays.......................................................................82 Oracle Demo Database Files ...........................................................................................................
Examples of Cluster and Package States..................................................................................107 Types of Cluster and Package States........................................................................................109 Cluster Status ...........................................................................................................................110 Node Status and State .................................................................................................
Example of Rolling Upgrade ........................................................................................................130 Step 1. .......................................................................................................................................131 Step 2. .......................................................................................................................................132 Step 3. ...........................................................................
List of Figures 1-1 1-2 1-3 1-4 1-5 1-6 2-1 2-2 2-3 2-4 4-1 A-1 A-2 A-3 A-4 A-5 A-6 Overview of Oracle RAC Configuration on HP-UX ....................................................................15 Group Membership Services.........................................................................................................16 Before Node Failure.......................................................................................................................22 After Node Failure..................
List of Tables 1 2-1 2-2 3-1 3-2 3-3 10 Document Edition and Printing Date............................................................................................11 Required Oracle File Names for Demo Database .........................................................................48 Required Oracle File Names for Demo Database .........................................................................62 RAC Software, Archive, Datafiles, SRVM.............................................................
Printing History Table 1 Document Edition and Printing Date Printing Date Part Number Edition June 2003 T1859-90006 First Edition Print, CD-ROM (Instant Information), and Web (http:// www.docs.hp.com/) June 2004 T1859-90017 Second Edition Print, CD-ROM (Instant Information), and Web (http:// www.docs.hp.com/) February 2005 T1859-90017 Second Edition February 2005 Update Web (http://www.docs.hp.
Preface The Seventh edition of the manual includes information for Logical Volume Manager (See “Creating a Storage Infrastructure with LVM” (page 43)), Serviceguard Extension for RAC (Oracle Real Application Cluster) Version A.11.18 on HP-UX 11i v2 and 11i v3, Veritas Cluster File System (CFS)/Cluster Volume Manager (CVM) from Symantec version 5.0, Cluster Interconnect Subnet Monitoring feature, SGeRAC Toolkit.
• • • • • • • VERITAS Volume Manager Administrator’s Guide. This contains a glossary of VERITAS terminology. VERITAS Volume Manager Storage Administrator Administrator’s Guide VERITAS Volume Manager Reference Guide VERITAS Volume Manager Migration Guide VERITAS Volume Manager for HP-UX Release Notes VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Configuration Guide Extracts. VERITAS Storage Foundation for Oracle RAC.
1 Introduction to Serviceguard Extension for RAC Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real Application Cluster (RAC), formerly known as Oracle Parallel Server RDBMS, to run on HP high availability clusters under the HP-UX operating system. This chapter introduces Serviceguard Extension for RAC and shows where to find different kinds of information in this book.
databases. Further, when properly configured, Serviceguard Extension for RAC provides a highly available database that continues to operate even if one hardware component should fail. Group Membership Oracle RAC systems implement the concept of group membership, which allows multiple instances of RAC to run on each node. Related processes are configured into groups. Groups allow processes in different instances to choose which other processes to interact with.
There are also packages that run on several cluster nodes at once, and do not fail over. These are called system multi-node packages and multi-node packages. As of Serviceguard Extension for RAC A.11.
Package Dependencies When CFS is used as shared storage, the application and software using the CFS storage should be configured to start and stop using Serviceguard packages. These application packages should be configured with a package dependency on the underlying multi-node packages, which manages the CFS and CVM storage reserves. Configuring the application to be start/stop through Serviceguard package is to ensure the synchronization of storage activation/deactivation and application startup/shutdown.
• • CSS Heartbeat Network (CSS-HB): Oracle Clusterware running on the various nodes of the cluster communicate among themselves using this network. RAC Cluster Interconnect Network (RAC-IC): Database instances of a database use this network to communicate among themselves. NOTE: In this document, the generic terms “CRS” and “Oracle Clusterware” will subsequently be referred to as “Oracle Cluster Software”.
For example, when a multi-node package (pkgA) is configured to run on all nodes of the cluster, and configured to monitor a subnet (SubnetA) using the CLUSTER_INTERCONNECT_SUBNET parameter: • • • If more than one instance of pkgA is running in the cluster and SubnetA fails on one of the nodes where the instance of pkgA is running, the failure is handled by halting the instance of pkgA on the node where the subnet has failed.
Group Membership The group membership service (GMS) is the means by which Oracle instances communicate with the Serviceguard cluster software. GMS runs as a separate daemon process that communicates with the cluster manager. This daemon is an HP component known as cmgmsd. The cluster manager starts up, monitors, and shuts down the cmgmsd.
Figure 1-3 Before Node Failure Figure 1-4 shows the condition where node 1 has failed and Package 1 has been transferred to node 2. Oracle instance 1 is no longer operating, but it does not fail over to node 2. The IP address for package 1 was transferred to node 2 along with the package. Package 1 continues to be available and is now running on node 2. Also, node 2 can now access both Package 1's disk and Package 2’s disk. Oracle instance 2 now handles all database access, since instance 1 has gone down.
Figure 1-4 After Node Failure In the above figure, pkg1 and pkg2 are not instance packages. They are shown to illustrate the movement of the packages. Larger Clusters Serviceguard Extension for RAC supports clusters of up to 16 nodes. The actual cluster size is limited by the type of storage and the type of volume manager used. Up to Four Nodes with SCSI Storage You can configure up to four nodes using a shared F/W SCSI bus; for more than 4 nodes, FibreChannel must be used.
Figure 1-5 Four-Node RAC Cluster In this type of configuration, each node runs a separate instance of RAC and may run one or more high availability packages as well. The figure shows a dual Ethernet configuration with all four nodes connected to a disk array (the details of the connections depend on the type of disk array). In addition, each node has a mirrored root disk (R and R').
Figure 1-6 Eight-Node Cluster with XP or EMC Disk Array FibreChannel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP 9000 Servers Configuration Guide, available through your HP representative.
2 Serviceguard Configuration for Oracle 10g RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 10g with Serviceguard.
This timeout is configurable with Oracle tools and should not be changed without ensuring that the CSS timeout allows enough time for Serviceguard Extension for RAC (SGeRAC) reconfiguration and to allow multipath (if configured) reconfiguration to complete. On a single point of failure, for example, node failure, Serviceguard reconfigures first and SGeRAC delivers the new group membership to CSS via NMAPI2. If there is a change in group membership, SGeRAC updates the members of the new membership.
Multipath Multipath is supported through either SLVM pvlinks or CVM Dynamic Multipath (DMP). In some configurations, SLVM or CVM does not need to be configured for multipath as the multipath is provided by the storage array. Since Oracle Cluster Software checks availability of the shared device for the vote disk through periodic monitoring, the multipath detection and failover time must be less than CRS's timeout specified by the Cluster Synchronization Service (CSS) MISSCOUNT.
network interfaces (LAN) that is transparent to applications utilizing User Datagram Protocol (UDP) and Transport Control Protocol (TCP). For virtual IP addresses (floating or package IP address) in Serviceguard, which also provides remote failover capability of network connection endpoints between cluster nodes and transparent local failover capability of network connection endpoints between redundant local network interfaces.
Shared Storage It is expected the shared storage is available when the RAC instance is started. Since the RAC instance expects the shared storage to be available, ensure the shared storage is activated. For SLVM, the shared volume groups must be activated and for CVM, the disk group must be activated. For CFS, the cluster file system must be mounted. Oracle Cluster Software requires shared storage for the Oracle Cluster Registry (OCR) and a vote device.
A failure of CSS-HB/RAC-IC network in such configuration does not force Serviceguard to reform the cluster. If Serviceguard is not configured to monitor the network, Oracle will take at least CSS misscount time interval to resolve the network failure. The default value of CSS misscount in SGeRAC configurations is 600 seconds. To avoid longer recovery times, manage Oracle Clusterware and RAC-DB instances using Serviceguard multi-node packages.
NOTE: For specific CFS Serviceguard Storage Management Suite product information refer to your version of the HP Serviceguard Storage Management Suite Release Notes.
Instance 2 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops2log1.log____120________ Instance 2 Redo Log 2: ___/dev/vx/rdsk/ops_dg/ops2log2.log____120________ Instance 2 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops2log3.log____120________ Instance 2 Redo Log: _________________________________________________ Instance 2 Redo Log: __________________________________________________ Data: System ___/dev/vx/rdsk/ops_dg/opssystem.dbf___500__________ Data: Sysaux ___/dev/vx/rdsk/ops_dg/opssysaux.
multipathing (DMP) for clusters. CVM supports concurrent storage read/write access between multiple nodes by applications which can manage read/write access contention, such as Oracle Real Application Cluster (RAC). NOTE: CVM (and CFS - Cluster File System) are supported on some, but not all, current releases of HP-UX. See the latest release notes for your version of Serviceguard at http://www.docs.hp.com -> High Availability - > Serviceguard.
The SGeRAC Toolkit uses Serviceguard/SGeRAC version A.11.17 (or later) multi-node packages and package dependency features to provide a uniform, intuitive, and easy-to-manage method to co-ordinate between SGeRAC and Oracle Clusterware. Additionally, the SGeRAC Toolkit helps to manage all the storage options supported by SGeRAC; CFS, SLVM, CVM, ASM (over SLVM), and SGeRAC support ASM over the raw device files on HP-UX 11i v3 only. For more information, refer to the Serviceguard Extension for RAC Version A.
the network and will perform a local LAN failover (transparent to CSS and RAC) if the primary interface fails. NOTE: A package with the CLUSTER_INTERCONNECT_SUBNET parameter is available for both Modular and Legacy packages. A package with this parameter can be configured only when all nodes of the cluster are running SGeRAC version A.11.18 or higher. For more information, see the latest edition of the Managing Serviceguard Fifteenth Edition user’s guide at http://docs.hp.com. -> High Availability.
Figure 2-2 SG-HB/RAC-IC Traffic Separation Each primary and standby pair protects against a single failure. With the SG-HB on more than one subnet, a single subnet failure will not trigger a Serviceguard reconfiguration. If the subnet with CSS-HB fails, unless subnet monitoring is used, CSS will resolve the interconnect subnet failure with a CSS cluster reconfiguration.
DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION CI-PACKAGE CI-PACKAGE=UP SAME_NODE Oracle Cluster Interconnect Subnet Package: Package to monitor the CSS-HB subnet PACKAGE_NAME CI-PACKAGE PACKAGE_TYPE MULTI_NODE LOCAL_LAN_FAILOVER_ALLOWED YES NODE_FAIL_FAST_ENABLED YES CLUSTER_INTERCONNECT_SUBNET 192.168.1.
NOTE: Do not configure CLUSTER_INTERCONNECT_SUBNET in the RAC instance package if the RAC-IC network is the same as CSS-HB network. The following is an example of the relevant package configuration parameters: Oracle RAC Instance Package PACKAGE_NAME PACKAGE_TYPE LOCAL_LAN_FAILOVER_ALLOWED NODE_FAIL_FAST_ENABLED CLUSTER_INTERCONNECT_SUBNET RAC_PACKAGE MULTI_NODE YES NO 192.168.2.
CLUSTER_INTERCONNECT_SUBNET can be used in conjunction with the NODE_FAIL_FAST_ENABLED option to monitor the CSS-HB network. A failure of CSS-HB subnet on a node should be handled by bringing down that node. Thus, set NODE_FAIL_FAST_ENABLED to YES for the package monitoring the CSS-HB subnet.
• 195 seconds or • 40 times Serviceguard NODE_TIMEOUT plus 25 times Serviceguard HEARTBEAT_INTERVAL + 120 seconds When Cluster Interconnect Subnet Monitoring is not used The Cluster Interconnect Monitoring is not used to monitor the CSS HB subnet and if the default values of any of the following parameters has to be changed: • Oracle Clusterware parameter CSS misscount or • Serviceguard cluster configuration parameter NODE_TIMEOUT Then the CSS misscount parameter should be the greater of either: • 19
• • • Cluster Lock device/Quorum Server/Lock Lun must be configured in the cluster. CLUSTER_INTERCONNECT_SUBNET can be used to monitor only IPv4 subnets.
NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. See the Managing Systems and Workgroups: A Guide for HP-UX System Administrators and for LVM version 2.x see the HP-UX 11i Version 3: HP-UX System Administrator's Guide: Logical Volume Management located at: http://docs.hp.com -> Core HP-UX 11iv3 -> LVM Volume Manager Creating Volume Groups and Logical Volumes If your volume groups have not been set up, use the procedure in the next sections.
Building Mirrored Logical Volumes for RAC with LVM Commands After you create volume groups and define physical volumes for use in them, you define mirrored logical volumes for data, logs, and control files. It is recommended that you use a shell script to issue the commands described in the next sections. The commands you use for creating logical volumes vary slightly depending on whether you are creating logical volumes for RAC redo log files or for use with Oracle data.
The -m 1 option specifies single mirroring; the -M n option ensures that mirror write cache recovery is set off; the -c y means that mirror consistency recovery is enabled; the -s g means that mirroring is PVG-strict, that is, it occurs between different physical volume groups; the -n system.dbf option lets you specify the name of the logical volume; and the -L 408 option allocates 408 megabytes.
Assume that the disk array has been configured, and that both the following device files appear for the same LUN (logical disk) when you run the ioscan command: /dev/dsk/c0t15d0 /dev/dsk/c1t3d0 Use the following procedure to configure a volume group for this logical disk: 1. First, set up the group directory for vg_ops: # mkdir /dev/vg_ops 2.
Table 2-1 Required Oracle File Names for Demo Database Logical Volume Name LV Size (MB) Raw Logical Volume Path Name Oracle File Size (MB)* opsctl1.ctl 118 /dev/vg_ops/ropsctl1.ctl 110 opsctl2.ctl 118 /dev/vg_ops/ropsctl2.ctl 110 opsctl3.ctl 118 /dev/vg_ops/ropsctl3.ctl 110 ops1log1.log 128 /dev/vg_ops/rops1log1.log 120 ops1log2.log 128 /dev/vg_ops/rops1log2.log 120 ops1log3.log 128 /dev/vg_ops/rops1log3.log 120 ops2log1.log 128 /dev/vg_ops/rops2log1.log 120 ops2log2.
Exporting the Logical Volume Infrastructure Before the Oracle volume groups can be shared, their configuration data must be exported to other nodes in the cluster. This is done either in Serviceguard Manager or by using HP-UX commands, as shown in the following sections. NOTE: Serviceguard Manager is the graphical user interface for Serviceguard. It is available as a “plug-in” to the System Management Homepage (SMH).
For details on Oracle installation, refer to the Oracle installation documentation. As part of this installation, the Oracle installer installs the executables and optionally, the Oracle installer can build an Oracle demo database on the primary node. The demo database files can be the character (raw) device files names for the logical volumes create earlier. For a demo database on SLVM or CVM, create logical volumes as shown in Table 2-1: “Required Oracle File Names for Demo Database ”.
# # # # # # # # For example, to configure a quorum server running on node “qshost” with 120 seconds for the QS_POLLING_INTERVAL and to add 2 seconds to the system assigned value for the quorum server timeout, enter: # # # # # # # # # # # # Definition of nodes in the cluster. Repeat node definitions as necessary for additional nodes. NODE_NAME is the specified nodename in the cluster. It must match the hostname and both cannot contain full domain name.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # First line must be USER_NAME, second USER_HOST, and third USER_ROLE. Enter a value after each. # # # # # # List of cluster aware LVM Volume Groups. These volume groups will be used by package applications via the vgchange -a e command. Neither CVM or VxVM Disk Groups should be used here. For example: VOLUME_GROUP /dev/vgdatabase VOLUME_GROUP /dev/vg02 # # # # # # # # List of OPS Volume Groups.
For more information refer to your version of the Serviceguard Extension for RAC Release Notes and HP Serviceguard Storage Management Suite Release Notes located at http://docs.hp.com -> High Availability. Creating a SGeRAC Cluster with CFS for Oracle 10g The following software needs to be pre-installed in order to use this configuration: NOTE: If you are installing CFS 4.1 or 5.
5. Configure the Cluster Volume Manager (CVM) Configure the system multi-node package, SG-CFS-pkg, to configure and start the CVM/CFS stack. Unlike VxVM-CVM-pkg, the SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and supports multiple subnets. # cfscluster config -s The following output will be displayed: CVM is now configured Starting CVM...
# vxassist -g cfsdg1 make vol3 300m # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol1 The following output will be displayed: version 6 layout 10485760 sectors, 10485760 blocks of size 1024, log size 16384 blocks largefiles supported # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2 The following output will be displayed: version 6 layout 10485760 sectors, 10485760 blocks of size 1024, log size 16384 blocks largefiles supported # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3 The following output will be displayed: version 6 layout
10485760 /dev/vx/dsk/cfsdg1/vol3 307200 19651 9811985 1802 286318 0% /cfs/mnt2 1% /cfs/mnt3 15.
NOTE: 3. The disk group package is deleted if there is no dependency. Delete Disk Group Multi-node Package # cfsdgadm delete cfsdg1 The following output will be generated: Shared disk group “cfsdg1” was disassociated from the cluster. NOTE: “cfsmntadm delete” also deletes the disk group if there is no dependent package. To ensure the disk group deletion is complete, use the above command to delete the disk group package. 4.
Preparing the Cluster and the System Multi-node Package for use with CVM 4.x or later The following steps describe how to prepare the cluster and the system multi-node package with CVM 4.x or later only. 1. Create the Cluster file: # cd /etc/cmcluster # cmquerycl -C clm.asc -n ever3a -n ever3b Edit Cluster file 2. Create the Cluster # cmapplyconf -C clm.asc • Start the Cluster # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 3.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: # /etc/vx/bin/vxdisksetup -i c4t4d0 • Create the Disk Group for RAC Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: # vxdg -s init ops_dg c4t4d0 4.
NOTE: The specific commands for creating mirrored and multi-path storage using CVM are described in the HP-UX documentation for the Veritas Volume Manager. Using CVM 3.x This section has information on how to prepare the cluster and the system multi-node package with CVM 3.x; (on HP-UX releases that support them; see “About Veritas CFS and CVM from Symantec” (page 18)). Preparing the Cluster for Use with CVM 3.x In order to use the Veritas Cluster Volume Manager (CVM) version 3.
that uses the volume group must be halted. This procedure is described in the Managing Serviceguard Fifteenth Edition user’s guide Appendix G. Initializing Disks for CVM You need to initialize the physical disks that will be employed in CVM disk groups. If a physical disk has been previously used with LVM, you should use the pvremove command to delete the LVM header data from all the disks in the volume group (this is not necessary if you have not previously used the disk with LVM).
This policy can be re-set on a disk group basis by using the vxedit command, as follows: # vxedit set diskdetpolicy=global NOTE: The specific commands for creating mirrored and multi-path storage using CVM are described in the HP-UX documentation for the Veritas Volume Manager. Oracle Demo Database Files The following set of volumes is required for the Oracle demo database which you can create during the installation process.
Example: 1. Create an ASCII file, and define the path for each database object. control1=/u01/ORACLE/db001/ctrl01_1.ctl 2. Set the following environment variable where filename is the name of the ASCII file created. # export DBCA_RAW_CONFIG=/filename Adding Disk Groups to the Cluster Configuration For CVM 4.x or later, if the multi-node package was configured for disk group activation, the application package should be configured with package dependency to ensure the CVM disk group is active.
5. 6. Enable Remote Access (ssh and remsh) for Oracle User on all Nodes Create File System for Oracle Directories In the following samples, /mnt/app is a mounted file system for Oracle software. Assume there is a private disk c4t5d0 at 18 GB size on all nodes. Create the local file system on each node.
# chmod 755 /dev/vg_ops b. Change Permission and Ownership of Oracle Cluster Software Vote Device and Database Files # chown oracle:oinstall /dev/vg_ops/r* # chmod 660 /dev/vg_ops/r* c. Change Permission of OCR Device # chown root:oinstall /dev/vg_ops/rora_ocr # chmod 640 /dev/vg_ops/rora_ocr d.
# chown -R oracle:oinstall /cfs/mnt3 b. Create Directory for Oracle Demo Database on CFS Create the CFS directory to store Oracle database files. Run commands only on one node. # chmod 775 /cfs # chmod 775 /cfs/mnt2 # cd /cfs/mnt2 # mkdir oradata # chown oracle:oinstall oradata # chmod 775 oradata NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. The steps shown in the following section are for configuring the volume groups in Serviceguard clusters LVM version 1.0.
Installing Oracle 10g RAC Binaries The following sample steps for a SGeRAC cluster for Oracle 10g. Refer to the Oracle documentation for Oracle installation details. Installing RAC Binaries on a Local File System Logon as a “oracle” user: $ export ORACLE BASE=/mnt/app/oracle $ export DISPLAY={display}:0.0 $ cd <10g RAC installation disk directory> $ ./runInstaller Use following guidelines when installing on a local file system: 1.
1. Setting up Listeners with Oracle Network Configuration Assistant Use the Oracle network configuration assistant to configure the listeners with the following command: $ netca 2. Create Database with Database Configuration Assistant Use the Oracle database configuration assistant to create the database with the following command: $ dbca Use following guidelines when installing on a local file system: a. In this sample, the database name and SID prefix are ver10. b.
output for CFS 4.1: VRTSodm 4.1m VERITAS Oracle Disk Manager VRTSodm.ODM-KRN 4.1m VERITAS ODM kernel files VRTSodm.ODM-MAN 4.1m VERITAS ODM manual pages VRTSodm.ODM-RUN 4.1m VERITAS ODM commands output for CFS 5.0: VRTSodm 5.0.01.01 Veritas Oracle Disk Manager VRTSodm.ODM-KRN 5.0.01.01 Veritas ODM kernel files VRTSodm.ODM-MAN 5.0.01.01 Veritas ODM manual pages VRTSodm.ODM-RUN 5.0.01.01 Veritas ODM commands 3. Check that libodm.sl is present #ll -L /opt/VRTSodm/lib/libodm.
io zro cmp: cl receive: cl ident: cl reserve: cl delete: cl resize: cl same op: cl opt idn: cl opt rsv: **********: 3. 2330 66145 18 8 1 0 0 0 332 17 Verify that the Oracle Disk Manager is loaded: # kcmodule -P state odm Output: state loaded 4. In the alert log, verify the Oracle instance is running. The log should contain output similar to the following: For CFS 4.1: Oracle instance running with ODM: VERITAS 4.1 ODM Library, Version 1.1 For CFS 5.0: Oracle instance running with ODM: VERITAS 5.
Preparing Oracle Cluster Software for Serviceguard Packages • Stopping the Oracle Cluster Software on each Node For 10g 10.1.0.04 or later: # /sbin/init.d/init.crs stop For 10g 10.2.0.01 or later: # /bin/crsctl stop crs Wait until Oracle Cluster Software completely stops. (Check CRS logs or check for Oracle processes, ps -ef | grep ocssd.bin) • Change Oracle Cluster Software from Starting at Boot Time on each Node For 10g 10.1.0.04 or later: # /sbin/init.d/init.crs disable For 10g 10.2.0.
• DEPENDENCY_CONDITION DEPENDENCY_LOCATION SG-CFS-MP-2=UP SAME_NODE DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION mp3 SG-CFS-MP-3=UP SAME_NODE Starting and Stopping Oracle Cluster Software In the Serviceguard package control script, configure the Oracle Cluster Software start in the customer_defined_run_cmds function For 10g 10.1.0.04 or later: /sbin/init.d/init.crs start For 10g 10.2.0.
3 Serviceguard Configuration for Oracle 9i RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 9i with Serviceguard.
Data: Users ___/dev/vg_ops/ropsusers.dbf_____120_________ Data: Tools ___/dev/vg_ops/ropstools.dbf____15___________ Data: User data ___/dev/vg_ops/ropsdata1.dbf_200__________ Data: User data ___/dev/vg_ops/ropsdata2.dbf__200__________ Data: User data ___/dev/vg_ops/ropsdata3.dbf__200__________ Data: Rollback ___/dev/vg_ops/ropsrollback.dbf__300_________ Parameter: spfile1 /dev/vg_ops/ropsspfile1.ora __5_____ Instance 1 undotbs1: /dev/vg_ops/ropsundotbs1.
Considerations on using CFS for RAC datafiles and Server Management Storage (SRVM) Use the following list when considering to use CFS for database storage: • Single file system view — Simpler setup for archive recovery since archive area is visible by all nodes • • • • • Oracle create database files Online changes (OMF - Oracle Managed Files) within CFS Better manageability Manual intervention when modifying volumes, DGs, disks Requires the SGeRAC and CFS software CFS and SGeRAC is available in selected
Instance 2 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops2log1.log____20________ Instance 2 Redo Log 2: ___/dev/vx/rdsk/ops_dg/ops2log2.log____20________ Instance 2 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops2log3.log____20_________ Instance 2 Redo Log: _________________________________________________ Instance 2 Redo Log: __________________________________________________ Data: System ___/dev/vx/rdsk/ops_dg/system.dbf___400__________ Data: Temp ___/dev/vx/rdsk/ops_dg/temp.
The Veritas CVM package for version 3.5 is named VxVM-CVM-pkg; the package for CVM version 4.1 and later is named SG-CFS-pkg. CVM allows you to activate storage on one node at a time, or you can perform write activation on one node and read activation on another node at the same time (for example, allowing backups). CVM provides full mirroring and dynamic multipathing (DMP) for clusters.
NOTE: Starting with Oracle 10g and later, Oracle Clusterware processes register with cmgmsd while Oracle server processes register with Oracle Clusterware. The maximum number of Oracle server processes supported is determined by Oracle Clusterware. The maximum number of processes registered by Oracle Clusterware should not exceed the maximum supported by cmgmsd. Oracle foreground server processes are needed to handle the requests of the DB client connected to the DB instance.
NOTE: For more information, see the Serviceguard Version A.11.18 Release Notes at http://docs.hp.com -> High Availability -> Serviceguard Creating Volume Groups and Logical Volumes If your volume groups have not been set up, use the procedure in the next sections. If you have already done LVM configuration, skip ahead to the section ““Installing Oracle Real Application Clusters”.
NOTE: For more information on using LVM, refer to the HP-UX Managing Systems and Workgroups manual. Building Mirrored Logical Volumes for RAC with LVM Commands After you create volume groups and define physical volumes for use in them, you define mirrored logical volumes for data, logs, and control files. It is recommended that you use a shell script to issue the commands described in the next sections.
consistency recovery. With a mirror consistency policy of “NONE”, SLVM does not perform the resynchronization. NOTE: Contact Oracle to determine if your version of Oracle RAC allows “resilvering” and to appropriately configure the mirror consistency recovery policy for your logical volumes. Create logical volumes for use as Oracle data files by using the same options as in the following example: # lvcreate -m 1 -M n -c n -s g -n system.
2. Next, create a control file named group in the directory /dev/vg_ops, as follows: # mknod /dev/vg_ops/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the format: 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. Use the following command to display a list of existing volume groups: # ls -l /dev/*/group 3.
Table 3-2 Required Oracle File Names for Demo Database (continued) Logical Volume Name LV Size (MB) Raw Logical Volume Path Name Oracle File Size (MB)* ops1log3.log 28 /dev/vg_ops/rops1log3.log 20 ops2log1.log 28 /dev/vg_ops/rops2log1.log 20 ops2log2.log 28 /dev/vg_ops/rops2log2.log 20 ops2log3.log 28 /dev/vg_ops/rops2log3.log 20 opssystem.dbf 408 /dev/vg_ops/ropssystem.dbf 400 opstemp.dbf 108 /dev/vg_ops/ropstemp.dbf 100 opsusers.dbf 128 /dev/vg_ops/ropsusers.
NOTE: Serviceguard Manager is the graphical user interface for Serviceguard. It is available as a “plug-in” to the System Management Homepage (SMH). SMH is a web-based graphical user interface (GUI) that replaces SAM as the system administration GUI as of HP-UX 11i v3 (but you can still run the SAM terminal interface; see “Using SAM” on page 32 of the Managing Serviceguard Fifteenth Edition user’s guide).
For a demo database on SLVM or CVM, create logical volumes as shown in Table 3-2: “Required Oracle File Names for Demo Database ”, earlier in this chapter. As the installer prompts for the database file names, use the pathnames of the raw logical volumes instead of using the defaults. NOTE: If you do not wish to install the demo database, select install software only.
# # QS_HOST qshost # QS_POLLING_INTERVAL 120000000 # QS_TIMEOUT_EXTENSION 2000000 # # # # # # # # # # # # Definition of nodes in the cluster. Repeat node definitions as necessary for additional nodes. NODE_NAME is the specified nodename in the cluster. It must match the hostname and both cannot contain full domain name. Each NETWORK_INTERFACE, if configured with IPv4 address, must have ONLY one IPv4 address entry with it which could be either HEARTBEAT_IP or STATIONARY_IP.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 8 login names from the /etc/passwd file on user host. 2. USER_HOST is where the user can issue Serviceguard commands. If using Serviceguard Manager, it is the COM server. Choose one of these three values: ANY_SERVICEGUARD_NODE, or (any) CLUSTER_MEMBER_NODE, or a specific node. For node, use the official hostname from domain name server, and not an IP addresses or fully qualified name. 3.
If you are installing CFS 4.1 or 5.0 with either the Storage Management Suite (SMS) bundle or Mission Critical Operating Environment (MCOE), use the appropriate product number as described below: • • SGeRAC and CFS 4.1 are included with the HP Serviceguard SMS bundle or MCOE. SGeRAC and CFS 5.0 are included with the HP Serviceguard Storage Management Suite bundle or MCOE.
CVM is now configured Starting CVM...It might take a few minutes to complete When CVM starts up, it selects a master node, which is the node from which you must issue the disk group configuration commands. To determine the master node, issue the following command from each node in the cluster: # vxdctl -c mode The following output will be displayed: mode: enabled: cluster active - SLAVEmaster: ever3b or mode: enabled: cluster active - MASTERslave: ever3b 6.
largefiles supported # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2 The following output will be displayed: version 6 layout 10485760 sectors, 10485760 blocks of size 1024, log size 16384 blocks largefiles supported # newfs -F vxfs /dev/vx/rdsk/cfsdg1/volsrvm The following output will be displayed: version 6 layout 307200 sectors, 307200 blocks of size 1024, log size 1024 blocks largefiles supported 12.
PACKAGE SG-CFS-pkg SG-CFS-DG-1 SG-CFS-MP-1 SG-CFS-MP-2 SG-CFS-MP-3 STATUS up up up up up STATE running running running running running AUTO_RUN enabled enabled enabled enabled enabled SYSTEM yes no no no no Deleting CFS from the Cluster Use the following steps to halt the applications that are using CFS file systems: 1. Unmount CFS Mount Points # cfsumount /cfs/mnt1 # cfsumount /cfs/mnt2 # cfsumount /cfs/cfssrvm 2.
Creating a Storage Infrastructure with CVM In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM), Veritas Volume Manager (VxVM), or Veritas Cluster Volume Manager (CVM).
NOTE: To prepare the cluster for CVM configuration, you need to be sure MAX_CONFIGURED_PACKAGES to minimum of 3 (the default value for MAX_CONFIGURED_PACKAGES for Serviceguard A.11.17 is 150) cluster configuration file. In the sample set the value to 10. 2. Create the Cluster # cmapplyconf -C clm.asc • Start the Cluster # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 3.
# vxdg -s init ops_dg c4t4d0 4. Creating Volumes and Adding a Cluster Filesystem # vxassist -g ops_dg make vol1 10240m #vxassist -g ops_dg make vol2 10240m # vxassist -g ops_dg make volsrvm 300m 5.
NOTE: Cluster configuration is described in the previous section. To prepare the cluster for CVM disk group configuration, you need to ensure that only one heartbeat subnet is configured. Then use the following command, which creates the special package that communicates cluster information to CVM: # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf WARNING! The above file should never be edited.
Creating Disk Groups for RAC Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: # vxdg -s init ops_dg c0t3d2 Verify the configuration with the following command: # vxdg list NAME rootdg ops_dg STATE enabled enabled,shared ID 971995699.1025.node1 972078742.1084.node2 Creating Volumes Use the vxassist command to create logical volumes.
Table 3-3 Required Oracle File Names for Demo Database Volume Name Size (MB) Raw Device File Name Oracle File Size (MB) opsctl1.ctl 108 /dev/vx/rdsk/ops_dg/opsctl1.ctl 100 opsctl2.ctl 108 /dev/vx/rdsk/ops_dg/opsctl2.ctl 100 opsctl3.ctl 108 /dev/vx/rdsk/ops_dg/opsctl3.ctl 100 ops1log1.log 28 /dev/vx/rdsk/ops_dg/oops1log1.log 20 ops1log2.log 28 /dev/vx/rdsk/ops_dg/ops1log2.log 20 ops1log3.log 28 /dev/vx/rdsk/ops_dg/ops1log3.log 20 ops2log1.log 28 /dev/vx/rdsk/ops_dg/ops2log1.
# export DBCA_RAW_CONFIG=/filename Adding Disk Groups to the Cluster Configuration For CVM 4.x or later, if the multi-node package was configured for disk group activation, the application package should be configured with package dependency to ensure the CVM disk group is active. For CVM 3.5 and CVM 4.x or later (without using multi-node package) after creating units of CVM storage with VxVM commands, you need to specify the disk groups in each package configuration ASCII file.
# chown oracle:dba /cfs/cfssrvm/ora_srvm 2. Install Oracle RAC Software a. Install Oracle (software only) with Oracle Universal Installer as oracle user # su - oracle When using CFS for SRVM, set SRVM_SHARED_CONFIG $ export SRVM_SHARED_CONFIG=/cfs/cfssrvm/ora_srvm b. Set DISPLAY $ export DISPLAY=${display}:0.0 c. Run Oracle Universal Installer and follow installation steps $ cd $ .
5.0.01.01 Veritas ODM manual pages VRTSodm.ODM-RUN 5.0.01.01 Veritas ODM commands 3. Check that libodm.sl is present: # ll -L /opt/VRTSodm/lib/libodm.sl The following output will be displayed: -rw-r--r-- 1 root sys 14336 Apr 25 18:42 /opt/VRTSodm/lib/libodm.sl Configure Oracle to use Oracle Disk Manager Library NOTE: 1. 2. 3. The following steps are specific to CFS 4.1 or later.
cl delete: cl resize: cl same op: cl opt idn: cl opt rsv: **********: 3. 1 0 0 0 332 17 Verify that Oracle Disk Manager is loaded with the following command: # kcmodule -P state odm The following output will be displayed: state loaded 4. In the alert log, verify the Oracle instance is running. The log should contain output similar to the following: For CFS 4.1: Oracle instance running with ODM: VERITAS 4.1 ODM Library, Version 1.1 For CFS 5.0: Oracle instance running with ODM: VERITAS 5.
the cluster will reform, and the package control scripts will bring up the database instances and application programs. When Oracle has been started, you can use the SAM process management area or the ps -ef command on both nodes to verify that all RAC daemons and Oracle processes are running. Starting Up and Shutting Down Manually To start up and shut down RAC instances without using packages, you can perform the following steps. • Starting up involves the following sequence: 1.
DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION mp1 SG-CFS-MP-1=UP SAME_NODE mp2 SG-CFS-MP-2=UP SAME_NODE mp3 SG-CFS-MP-3=UP SAME_NODE Configuring Packages that Access the Oracle RAC Database You can also use packages to start up applications that access the RAC instances.
Use the following procedure to create a control scripts for the sample package pkg1. First, generate a control script template: # cmmakepkg -s /etc/cmcluster/pkg1/control.sh You may customize the script, as described in the section, ““Customizing the Package Control Script”.” Customizing the Package Control Script Check the definitions and declarations at the beginning of the control script using the information in the Package Configuration worksheet.
Optimizing for Large Numbers of Storage Units A set of four variables is provided to allow performance improvement when employing a large number of filesystems or storage groups. For more detail, see the comments in the control script template. They are: • • • CONCURRENT_VGCHANGE_OPERATIONS—defines a number of parallel LVM volume group activations during package startup as well and deactivations during package shutdown.
is the main difference between an Oracle RAC Instance Package and a single instance Oracle package. Information for Creating the Oracle RAC Instance Package on a SGeRAC Node Use the following steps to set up the pre-package configuration on a SGeRAC node: 1. Gather the RAC Instance SID_NAME. If you are using Serviceguard Manager, this is in the cluster Properties. Example: SID_NAME=ORACLE_TEST0 For an ORACLE RAC Instance for a two-node cluster, each node would have an SID_NAME. 2.
4 Maintenance and Troubleshooting This chapter includes information about carrying out routine maintenance on an Real Application Cluster configuration. As presented here, these tasks differ in some details from the similar tasks described in the Managing Serviceguard documentation.
Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up NODE mo PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 STATUS up NAME lan0 lan1 lan3 STATE running Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 NAME lan0 lan1 lan3 MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg NODE_NAME minie STATUS up STATUS up Script_Parameters: ITEM
DEPENDENCY_NAME SG-CFS-pkg PACKAGE SG-CFS-MP-1 NODE_NAME minie SATISFIED yes STATUS up STATE running STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up PACKAGE SG-CFS-MP-2 NODE_NAME minie STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up PACKAGE SG-CFS-MP-3 NODE_NAME minie STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up SYSTEM no SWITCHING enable
Cluster Status The status of a cluster may be one of the following: • • • • Up. At least one node has a running cluster daemon, and reconfiguration is not taking place. Down. No cluster daemons are running on any cluster node. Starting. The cluster is in the process of determining its active membership. At least one cluster daemon is running. Unknown. The node on which the cmviewcl command is issued cannot communicate with other nodes in the cluster.
For multi-node packages, node switching Disabled means the package cannot start on that node. Status of Group Membership The state of the cluster for Oracle RAC is one of the following: • • Up. Services are active and being monitored. The membership appears in the output of cmviewcl -l group. Down. The cluster is halted and GMS services have been stopped. The membership does not appear in the output of the cmviewcl -l group.
• • Recovering. A corrupt message was received on the serial line, and the line is in the process of resynchronizing. Unknown. We cannot determine whether the serial line is up or down. This can happen when the remote node is down. Failover and Failback Policies Packages can be configured with one of two values for the FAILOVER_POLICY parameter: • • CONFIGURED_NODE. The package fails over to the next node in the node list in the package configuration file. MIN_PACKAGE_NODE.
PACKAGE ops_pkg2 STATUS up STATE running AUTO_RUN disabled NODE ftsys10 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Start configured_node Failback manual Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NAME ftsys10 ftsys9 (current) Quorum Server Status If the cluster is using a quorum server for tie-breaking services, the display shows the server name, state and status following the entry for each node, as in the following excerpt from the output of
NODE STATUS ftsys9 up Script_Parameters: ITEM STATUS Service up STATE running MAX_RESTARTS 0 RESTARTS 0 NAME VxVM-CVM-pkg.srv Status After Moving the Package to Another Node After issuing the following command: # cmrunpkg -n ftsys9 pkg2 the output of the cmviewcl -v command is as follows: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 56/36.
Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 28.1 32.1 NAME lan0 lan1 Now pkg2 is running on node ftsys9. Note that it is still disabled from switching.
Script_Parameters: ITEM STATUS Resource up Subnet up Resource up Subnet up Resource up Subnet up Resource up Subnet up NODE_NAME manx manx burmese burmese tabby tabby persian persian NAME /resource/random 192.8.15.0 /resource/random 192.8.15.0 /resource/random 192.8.15.0 /resource/random 192.8.15.
NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. For more information on using and configuring LVM version 2.x, see the HP-UX 11i Version 3: HP-UX System Administrator's Guide: Logical Volume Management located at: http://docs.hp.com -> Core HP-UX 11iv3 -> LVM Volume Manager For LVM version 2 compatibility requirements see the Serviceguard/SGeRAC/SMS/Serviceguard Mgr Plug-in Compatibility and Feature Matrix at http://docs.hp.
Activated volume group in shared mode. This node is the Server. When the same command is entered on the second node, the following message is displayed: Activated volume group in shared mode. This node is a Client. NOTE: Do not share volume groups that are not part of the RAC configuration unless shared access is controlled.
# vgchange -S y -c y /dev/vg_ops 11. On node 2, issue the following command: # mkdir /dev/vg_ops 12. Create a control file named group in the directory /dev/vg_ops, as in the following: # mknod /dev/vg_ops/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the format: 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. 13.
NOTE: For CVM without CFS, if you are adding a disk group to the cluster configuration, make sure you also modify any package or create the package control script that imports and deports this disk group. If you are adding a CVM disk group, be sure to add the STORAGE_GROUP entry for the disk group to the package ASCII file. For CVM with CFS, if you are adding a disk group to the cluster configuration, make sure you also create the corresponding multi-node package.
when a mirror was lost from a mirrored volume group being used in a non-RAC package. Refer to the manual Using the Event Monitoring Service (B7609-90022) for additional information. Using EMS Hardware Monitors A set of hardware monitors is available for monitoring and reporting on memory, CPU, and many other system values. Refer to the EMS Hardware Monitors User’s Guide (B6191-90015) for additional information. Adding Disk Hardware As your system expands, you may need to add disk hardware.
NOTE: If your LVM installation requires online replacement of disk mechanisms, the use of disk arrays may be required, because software mirroring of JBODs with MirrorDisk/UX does not permit hot swapping for disks that are activated in shared mode. Replacing a Mechanism in an HA Enclosure Configured with Exclusive LVM Non-Oracle data that is used by packages may be configured in volume groups that use exclusive (one-node-at-a-time) activation.
# pvchange -a N [pv path] Alternatively, use the pvchange -a N [pv path] command to detach a disk (all paths to the disk) and close it. Use this to allow diagnostics or replace a multi-ported disk. NOTE: If the volume group is mirrored, applications can continue accessing data on mirror copies after the commands above. If the volume is not mirrored, then any access attempts to the device may hang indefinitely or time out. This depends upon the LV timeout value configured for the logical volume. 2.
5. Activate the volume group on one node in exclusive mode then deactivate the volume group: # vgchange -a e vg_ops This will synchronize the stale logical volume mirrors. This step can be time-consuming, depending on hardware characteristics and the amount of data. 6. Deactivate the volume group: # vgchange -a n vg_ops 7.
Figure 4-1 F/W SCSI Buses with In-line Terminators The use of in-line SCSI terminators allows you to do hardware maintenance on a given node by temporarily moving its packages to another node and then halting the original node while its hardware is serviced. Following the replacement, the packages can be moved back to the original node. Use the following procedure to disconnect a node that is attached to the bus with an in-line SCSI terminator or with a Y cable: 1. 2. 3. 4. 5. 6. 7. 8.
4. 5. 6. 7. Remove the defective I/O card. Install the new card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed. In SAM, select the option to attach the new I/O card. Add the node back into the cluster by using Serviceguard Manager or the cmrunnode command.
Monitoring RAC Instances The DB Provider provides the capability to monitor RAC databases. RBA (Role Based Access) enables a non-root user to have the capability to monitor RAC instances using Serviceguard Manager.
A Software Upgrades Serviceguard Extension for RAC (SGeRAC) software upgrades can be done in the two following ways: • rolling upgrade • non-rolling upgrade Instead of an upgrade, moving to a new version can be done with: • migration with cold install Rolling upgrade is a feature of SGeRAC that allows you to perform a software upgrade on a given node without bringing down the entire cluster. SGeRAC supports rolling upgrades on version A.11.
For more information on support, compatibility, and features for SGeRAC, refer to the Serviceguard Compatibility and Feature Matrix, located at http://docs.hp.com -> High Availability -> Serviceguard Extension for RAC. Steps for Rolling Upgrades Use the following steps when performing a rolling SGeRAC software upgrade: 1. Halt Oracle (RAC, CRS, Clusterware, OPS) software on the local node (if running). 2. Halt Serviceguard/SGeRAC on the local node by issuing the Serviceguard cmhaltnodecommand. 3.
NOTE: While you are performing a rolling upgrade, warning messages may appear while the node is determining what version of software is running. This is a normal occurrence and not a cause for concern. Figure A-1 Running Cluster Before Rolling Upgrade Step 1. 1. 2. Halt Oracle (RAC, CRS, Clusterware, OPS) software on node 1. Halt node 1. This will cause the node’s packages to start up on an adoptive node.
Figure A-2 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 and install the new version of Serviceguard and SGeRAC (A.11.16), as shown in Figure A-3. NOTE: If you install Serviceguard and SGeRAC separately, Serviceguard must be installed before installing SGeRAC. Figure A-3 Node 1 Upgraded to SG/SGeRAC 11.16 Step 3. 1. Restart the cluster on the upgraded node (node 1) (if desired).
Figure A-4 Node 1 Rejoining the Cluster Step 4. 1. 2. Halt Oracle (RAC, CRS, Clusterware, OPS) software on node 2. Halt node 2. You can do this in Serviceguard Manager, or from the command line issue the following: # cmhaltnode -f node2 This causes both packages to move to node 1; see Figure A-5. 3. 4. Upgrade node 2 to Serviceguard and SGeRAC (A.11.16) as shown in Figure A-5. When upgrading is finished, enter the following command on node 2 to restart the cluster on node 2: # cmrunnode node2 5.
Figure A-5 Running Cluster with Packages Moved to Node 1 Step 5. Move PKG2 back to its original node. Use the following commands: # cmhaltpkg pkg2 # cmrunpkg -n node2 pkg2 # cmmodpkg -e pkg2 The cmmodpkg command re-enables switching of the package, which is disabled by the cmhaltpkg command. The final running cluster is shown in Figure A-6.
Figure A-6 Running Cluster After Upgrades Limitations of Rolling Upgrades The following limitations apply to rolling upgrades: • During a rolling upgrade, you should issue Serviceguard/SGeRAC commands (other than cmrunnode and cmhaltnode) only on a node containing the latest revision of the software. Performing tasks on a node containing an earlier revision of the software will not work or will cause inconsistent results.
Non-Rolling Software Upgrades A non-rolling upgrade allows you to perform a software upgrade from any previous revision to any higher revision or between operating system versions. For example, you may do a non-rolling upgrade from SGeRAC A.11.14 on HP-UX 11i v1 to A.11.16 on HP-UX 11i v2, given both are running the same architecture. The cluster cannot be running during a non-rolling upgrade, therefore it is necessary to halt the entire cluster in order to perform the upgrade.
7. Recreate the network and storage configurations (that is, set up stationary IP addresses and create LVM volume groups and/or CVM disk groups required for the cluster). 8. Recreate the SGeRAC cluster. 9. Restart the cluster. 10. Reinstall the cluster applications, such as RAC. 11. Restore the data.
B Blank Planning Worksheets This appendix reprints blank planning worksheets used in preparing the RAC cluster. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Disk Group Name: __________________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name:__________________________________
Index A activation of volume groups in shared mode, 117 adding packages on a running cluster, 103 administration cluster and package states, 107 array replacing a faulty mechanism, 121, 122, 123 AUTO_RUN parameter, 102 AUTO_START_TIMEOUT in sample configuration file, 50, 85 B building a cluster CVM infrastructure, 57, 92 building an RAC cluster displaying the logical volume infrastructure, 48, 83 logical volume infrastructure, 43, 78 building logical volumes for RAC, 47, 82 C CFS, 52, 56, 87 deleting from
HEARTBEAT_IP in sample configuration file, 50, 85 high availability cluster defined, 15 O lock disk replacing a faulty mechanism, 124 logical volumes blank planning worksheet, 140 creating, 47, 82 creating for a cluster, 45, 61, 80, 96 creating the infrastructure, 43, 78 disk arrays, 46, 81 filled in planning worksheet, 32, 33, 73, 75 lssf using to obtain a list of disks, 44, 79 LV in sample package control script, 104 LVM creating on disk arrays, 46, 81 LVM commands exporting files, 49, 84 online hardwa
physical volumes creating for clusters, 44, 79 filled in planning worksheet, 139 planning worksheets for logical volume planning, 32, 33, 73, 75 worksheets for physical volume planning, 139 planning worksheets blanks, 139 point to point connections to storage devices, 24 PVG-strict mirroring creating volume groups with, 44, 79 R RAC group membership services, 20 overview of configuration, 15 status, 111 RAC cluster defined, 15 removing packages on a running cluster, 103 removing Serviceguard Extension for
worksheets physical volume planning, 139 worksheets for planning blanks, 139 144 Index
