Using Serviceguard Extension for RAC HP Part Number: T1859-90060 Published: March 2009
Legal Notices © Copyright 2009 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents Printing History ................................................................................................................11 Preface..............................................................................................................................13 1 Introduction to Serviceguard Extension for RAC.......................................................15 What is a Serviceguard Extension for RAC Cluster? ......................................................................
Manual Startup and Shutdown..................................................................................................29 Network Monitoring.......................................................................................................................29 SGeRAC Heartbeat Network.....................................................................................................30 CSS Heartbeat Network...............................................................................................
Creating a Storage Infrastructure with CVM.......................................................................................59 Initializing the Veritas Volume Manager.........................................................................................59 Using CVM 4.x or later....................................................................................................................59 Preparing the Cluster and the System Multi-node Package for use with CVM 4.x or later......
Building Mirrored Logical Volumes for RAC with LVM Commands............................................82 Creating Mirrored Logical Volumes for RAC Redo Logs and Control Files.............................82 Creating Mirrored Logical Volumes for RAC Data Files...........................................................82 Creating RAC Volume Groups on Disk Arrays .............................................................................83 Creating Logical Volumes for RAC on Disk Arrays......................
4 Maintenance and Troubleshooting..........................................................................113 Reviewing Cluster and Package States with the cmviewcl Command............................................113 Types of Cluster and Package States..............................................................................................113 Examples of Cluster and Package States..................................................................................
Steps for Rolling Upgrades ...........................................................................................................136 Keeping Kernels Consistent.....................................................................................................136 Example of Rolling Upgrade ........................................................................................................136 Step 1. ....................................................................................................
List of Figures 1-1 1-2 1-3 1-4 1-5 1-6 2-1 2-2 2-3 4-1 A-1 A-2 A-3 A-4 A-5 A-6 Overview of Oracle RAC Configuration on HP-UX ....................................................................15 Group Membership Services.........................................................................................................16 Before Node Failure.......................................................................................................................22 After Node Failure......................
List of Tables 1 2-1 2-2 3-1 3-2 3-3 10 Document Edition and Printing Date............................................................................................11 Required Oracle File Names for Demo Database .........................................................................46 Required Oracle File Names for Demo Database .........................................................................64 RAC Software, Archive, Datafiles, SRVM.............................................................
Printing History Table 1 Document Edition and Printing Date Printing Date Part Number Edition June 2003 T1859-90006 First Edition June 2004 T1859-90017 Second Edition February 2005 T1859-90017 Second Edition February 2005 Update October 2005 T1859-90033 Third Edition December 2005 T1859-90033 Third Edition, First Reprint March 2006 T1859-90038 Third Edition, Second Reprint May 2006 T1859-90038 Third Edition May 2006 Update February 2007 T1859-90043 Fourth Edition February 2007 Jun
Preface This user's guide covers how to use the Serviceguard Extension for RAC (Oracle Real Application Cluster) to configure Serviceguard clusters for use with Oracle Real Application Cluster software on HP High Availability clusters running the HP-UX operating system. The contents are as follows: • Chapter 1, “Introduction,” describes a Serviceguard cluster and provides a roadmap for using this guide. This chapter should be used as a supplement to Chapters 1–3 of the Managing Serviceguard user’s guide.
• • VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Configuration Guide Extracts. VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite Administration Guide Extracts. If you will be using Veritas Cluster Volume Manager (CVM) and Veritas Cluster File System (CFS) from Symantec with Serviceguard refer to the HP Serviceguard Storage Management Suite Version A.02.01 for HP-UX 11i v3 Release Notes.
1 Introduction to Serviceguard Extension for RAC Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real Application Cluster (RAC), formerly known as Oracle Parallel Server RDBMS, to run on HP high availability clusters under the HP-UX operating system. This chapter introduces Serviceguard Extension for RAC and shows where to find different kinds of information in this book.
databases. Further, when properly configured, Serviceguard Extension for RAC provides a highly available database that continues to operate even if one hardware component should fail. Group Membership Oracle RAC systems implement the concept of group membership, which allows multiple instances of RAC to run on each node. Related processes are configured into groups. Groups allow processes in different instances to choose which other processes to interact with.
There are also packages that run on several cluster nodes at once, and do not fail over. These are called system multi-node packages and multi-node packages. As of Serviceguard Extension for RAC A.11.
Package Dependencies When CFS is used as shared storage, the application and software using the CFS storage should be configured to start and stop using Serviceguard packages. These application packages should be configured with a package dependency on the underlying multi-node packages, which manages the CFS and CVM storage reserves. Configuring the application to be start/stop through Serviceguard package is to ensure the synchronization of storage activation/deactivation and application startup/shutdown.
NOTE: In this document, the generic terms “CRS” and “Oracle Clusterware” will subsequently be referred to as “Oracle Cluster Software”. The use of the term CRS will still be used when referring to a sub-component of Oracle Cluster Software. For more detail information on Oracle 10g/11gR1 RAC refer to Chapter 2: “Serviceguard Configuration for Oracle 10g or 11gR1 RAC”.
• • • If more than one instance of pkgA is running in the cluster and SubnetA fails on one of the nodes where the instance of pkgA is running, the failure is handled by halting the instance of pkgA on the node where the subnet has failed. If pkgA is running on only one node of the cluster and SubnetA fails on that node, pkgA will continue to run on that node after the failure.
The cluster manager starts up, monitors, and shuts down the cmgmsd. When an Oracle instance starts, the instance registers itself with cmgmsd; thereafter, if an Oracle instance fails, cmgmsd notifies other members of the same group to perform recovery. If cmgmsd dies unexpectedly, Serviceguard will fail the node with a TOC (Transfer of Control). Configuring Packages for Oracle RAC Instances Oracle instances can be configured as packages with a single node in their node list.
Figure 1-3 Before Node Failure Figure 1-4 shows the condition where node 1 has failed and Package 1 has been transferred to node 2. Oracle instance 1 is no longer operating, but it does not fail over to node 2. The IP address for package 1 was transferred to node 2 along with the package. Package 1 continues to be available and is now running on node 2. Also, node 2 can now access both Package 1's disk and Package 2’s disk. Oracle instance 2 now handles all database access, since instance 1 has gone down.
Figure 1-4 After Node Failure In the above figure, pkg1 and pkg2 are not instance packages. They are shown to illustrate the movement of the packages. Larger Clusters Serviceguard Extension for RAC supports clusters of up to 16 nodes. The actual cluster size is limited by the type of storage and the type of volume manager used. Up to Four Nodes with SCSI Storage You can configure up to four nodes using a shared F/W SCSI bus; for more than 4 nodes, FibreChannel must be used.
Figure 1-5 Four-Node RAC Cluster In this type of configuration, each node runs a separate instance of RAC and may run one or more high availability packages as well. The figure shows a dual Ethernet configuration with all four nodes connected to a disk array (the details of the connections depend on the type of disk array). In addition, each node has a mirrored root disk (R and R').
Figure 1-6 Eight-Node Cluster with EVA, XP or EMC Disk Array FibreChannel switched configurations also are supported using either an arbitrated loop or fabric login topology. For additional information about supported cluster configurations, refer to the HP 9000 Servers Configuration Guide, available through your HP representative.
4. 5. Restart the Serviceguard cluster Restart Oracle Clusterware (for Oracle 10g and 11gR1) and Oracle RAC database instance on all nodes. Use the following steps to disable the GMS authorization: Disable GMS authorization 1. 2. 3. 4. 5. 26 If Oracle RAC database instance and Oracle Clusterware (for Oracle 10g and 11gR1) are running shut them down on all nodes. Halt the Serviceguard cluster. Edit /etc/opt/nmapi/nmutils.conf and comment the GMS_USER[] settings on all nodes.
2 Serviceguard Configuration for Oracle 10g or 11gR1 RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 10g/11gR1 with Serviceguard.
CSS Timeout When SGeRAC is on the same cluster as Oracle Cluster Software, the CSS timeout is set to a default value of 600 seconds (10 minutes) at Oracle software installation. This timeout is configurable with Oracle tools and should not be changed without ensuring that the CSS timeout allows enough time for Serviceguard Extension for RAC (SGeRAC) reconfiguration and to allow multipath (if configured) reconfiguration to complete.
Multipathing Multipathing is supported through either SLVM pvlinks or CVM Dynamic Multipath (DMP). In some configurations, SLVM or CVM does not need to be configured for multipath as the multipath is provided by the storage array. Since Oracle Cluster Software checks availability of the shared device for the vote disk through periodic monitoring, the multipath detection and failover time must be less than CRS's timeout specified by the Cluster Synchronization Service (CSS) MISSCOUNT.
network interfaces (LAN) that is transparent to applications utilizing User Datagram Protocol (UDP) and Transport Control Protocol (TCP). For virtual IP addresses (floating or package IP address) in Serviceguard, which also provides remote failover capability of network connection endpoints between cluster nodes and transparent local failover capability of network connection endpoints between redundant local network interfaces.
Shared Storage It is expected the shared storage is available when the RAC instance is started. Since the RAC instance expects the shared storage to be available, ensure the shared storage is activated. For SLVM, the shared volume groups must be activated and for CVM, the disk group must be activated. For CFS, the cluster file system must be mounted. Oracle Cluster Software requires shared storage for the Oracle Cluster Registry (OCR) and a vote device.
In the above cases, you will see a longer time to recover some network failures beyond those protected by primary and standby, unless Serviceguard is configured to monitor the network. A failure of CSS-HB/RAC-IC network in such configuration does not force Serviceguard to reform the cluster. If Serviceguard is not configured to monitor the network, Oracle will take at least CSS misscount time interval to resolve the network failure. The default value of CSS misscount in SGeRAC configurations is 600 seconds.
NOTE: For specific CFS Serviceguard Storage Management Suite product information refer to your version of the HP Serviceguard Storage Management Suite Release Notes. Volume Planning with CVM Storage capacity for the Oracle database must be provided in the form of volumes located in shared disk groups. The Oracle software requires at least two log files for each Oracle instance, several Oracle control files and data files for the database itself.
Instance 2 Redo Log 2: ___/dev/vx/rdsk/ops_dg/ops2log2.log____120________ Instance 2 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops2log3.log____120________ Instance 2 Redo Log: _________________________________________________ Instance 2 Redo Log: __________________________________________________ Data: System ___/dev/vx/rdsk/ops_dg/opssystem.dbf___500__________ Data: Sysaux ___/dev/vx/rdsk/ops_dg/opssysaux.dbf___800__________ Data: Temp ___/dev/vx/rdsk/ops_dg/opstemp.
multiple nodes by applications which can manage read/write access contention, such as Oracle Real Application Cluster (RAC). NOTE: CVM (and CFS - Cluster File System) are supported on some, but not all, current releases of HP-UX. See the latest release notes for your version of Serviceguard at http://www.docs.hp.com -> High Availability - > Serviceguard. Veritas Storage Management Products Veritas Volume Manager (VxVM) 3.5 is not supported on HP-UX 11i v3. If you are running VxVM 3.
Additionally, the SGeRAC Toolkit helps to manage all the storage options supported by SGeRAC; CFS, SLVM, CVM, ASM (over SLVM), ASM over the raw device files on HP-UX 11i v3 only. For more information, refer to the Serviceguard Extension for RAC Version A.11.19 Release Notes located at http://docs.hp.com -> High Availability -> Serviceguard Extension for Real Application Clusters -> Release Notes.
NOTE: CVM 4.x or later with CFS does not use the STORAGE_GROUP parameter because the disk group activation is performed by the multi-node package. CVM 3.x or 4.x and later without CFS uses the STORAGE_GROUP parameter in the ASCII package configuration file in order to activate the disk group (on HP-UX releases that support Veritas CFS and CVM; see “About Veritas CFS and CVM from Symantec” (page 18)). Do not enter the names of LVM volume groups or VxVM disk groups in the package ASCII configuration file.
Alternate Configuration - Fast Reconfiguration with Low Node Member Timeout A high RAC-IC traffic may interfere with SG-HB traffic and cause unnecessary member timeout if Serviceguard cluster configuration parameter MEMBER_TIMEOUT is low. If MEMBER_TIMEOUT cannot be increased, use of an additional network dedicated for SG-HB alone avoids unnecessary member timeouts when RAC-IC traffic is high.
NODE_FAIL_FAST_ENABLED parameter is set to NO for the Oracle Clusterware package, and is set to YES for the package monitoring CSS-HB subnet (Oracle Cluster Interconnect Subnet Package as shown in the package configuration parameters examples below). NOTE: Do not configure CLUSTER_INTERCONNECT_SUBNET in the RAC Instance package due to the RAC-IC network being the same as CSS-HB network.
failure. In SGeRAC configurations, default value of IMR time interval may be as high as seventeen minutes. CLUSTER_INTERCONNECT_SUBNET can be configured for RAC instance MNP to monitor the RAC-IC subnet that is different from CSS-HB subnet. The parameter file (SPFILE or PFILE) for RAC instances must have cluster_interconnects parameter defined, to hold IP address in the appropriate subnet if the RAC Instances use a RAC-IC network different from CSS-HB network.
• • • For SLVM: F + SLVM timeout + 15 seconds For CVM/CFS: 3 times F + 15 seconds When both SLVM and CVM/CFS are used, then take the max of the above two calculations. NOTE: 1. The “F” represents the Serviceguard failover time as given by the max_reformation_duration field of cmviewcl –v –f line output. 2. SLVM timeout is documented in the whitepaper, LVM link and Node Failure Recovery Time.
NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. The steps shown in the following section are for configuring the volume groups in Serviceguard clusters LVM version 1.0. For more information on using and configuring LVM version 2.x, see the HP-UX 11i Version 3: HP-UX System Administrator's Guide: Logical Volume Management located at: http://docs.hp.com -> Core HP-UX 11iv3 -> LVM Volume Manager For LVM version 2.
The major number is always 64, and the hexadecimal minor number has the form 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. Use the following command to display a list of existing volume groups: # ls -l /dev/*/group 3.
mirror resynchronization (NOMWC) or fast resynchronization (MWC) for control and redo files and no mirror resynchronization (NONE) for the datafiles since Oracle would perform resilvering on the datafiles based on the redo log. If Oracle does not perform “resilvering” of RAC data files that are mirrored logical volumes, choose a mirror consistency policy of “NOMWC”. This is done by disabling mirror write caching and enabling mirror consistency recovery. With “NOMWC”, SLVM performs the resynchronization.
disks you have defined on the array. If you are using SAM, choose the type of disk array you wish to configure, and follow the menus to define alternate links. If you are using LVM commands, specify the links on the command line. The following example shows how to configure alternate links using LVM commands. The following disk configuration is assumed: 8/0.15.0 8/0.15.1 8/0.15.2 8/0.15.3 8/0.15.4 8/0.15.
NOTE: The default maximum number of volume groups in HP-UX version 2.0 is 10. If you intend to create enough new volume groups that the total exceeds ten, you must increase the maxvgs system parameter and then re-boot the system. Use the kctune utility to change kernel parameter area, then choose Configurable Parameters, maxvgs appears on the list.
The size of the logical volume is larger than the Oracle file size because Oracle needs extra space to allocate a header in addition to the file's actual data capacity. Create these files if you wish to build the demo database. The three logical volumes at the bottom of the table are included as additional data files, which you can create as needed, supplying the appropriate sizes.
# ls -l /dev/*/group 4. Import the volume group data using the map file from node ftsys9. On node ftsys10 (and other nodes, as necessary), enter: # vgimport -s -m /tmp/vg_ops.map /dev/vg_ops Installing Oracle Real Application Clusters NOTE: Some versions of Oracle RAC requires installation of additional software. Refer to your version of Oracle for specific requirements. Before installing the Oracle Real Application Cluster software, make sure the storage cluster is running.
# used by any other cluster as a cluster lock device. # # # # # # # # LUN lock disk parameters. Use the CLUSTER_LOCK_LUN parameter to define the device on a per node basis. The device may only be used for this purpose and by only a single cluster. # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # Quorum Server Parameters. Use the QS_HOST, QS_ADDR, QS_POLLING_INTERVAL, and QS_TIMEOUT_EXTENSION parameters to define a quorum server.
# delete the NETWORK_INTERFACE definition containing IPv6 # addresses. # # # # # # # # # # # # # # # # # # # # # # # # # Note: This configuration contains IPv4 STATIONARY_IP or HEARTBEAT_IP addresses. To obtain an IPv6-only cluster on supported platforms, comment out any IPv4 STATIONARY_IPs or HEARTBEAT_IPs. If this leaves any NETWORK_INTERFACE without any STATIONARY_IP or HEARTBEAT_IP, comment out the NETWORK_INTERFACE as well.
# # CAPACITY_NAME CAPACITY_VALUE # Warning: There are no standby network interfaces for lan0. # Possible standby Network Interfaces for lan1: lan3. # Warning: There are no standby network interfaces for lan2. NODE_NAME everd NETWORK_INTERFACE lan0 STATIONARY_IP 16.89.114.145 NETWORK_INTERFACE lan1 HEARTBEAT_IP 10.142.1.4 STATIONARY_IP fec0::8e01:0:0:a8e:4 STATIONARY_IP 3ffe:1000:0:8e01::a8e:4 NETWORK_INTERFACE lan3 NETWORK_INTERFACE lan2 STATIONARY_IP 10.142.2.
# the standby LAN interface will continue to be used until cmmodnet -e lanX # is issued for the primary LAN interface. NETWORK_AUTO_FAILBACK # # # # # # # # # YES The HOSTNAME_ADDRESS_FAMILY parameter specifies the Internet Protocol address family to which Serviceguard will attempt to resolve cluster node names and quorum server host names. If the parameter is set to IPV4, Serviceguard will attempt to resolve the names to IPv4 addresses only. This is the default value.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # For example, if CAPACITY_NAME "memory" is not defined for node1, then node1's "memory" capacity is assumed to be infinite. Now even if pkgA, pkgB, and pkgC each specify the maximum weight of 1000000 for WEIGHT_NAME "memory", all three packages are eligible to run at the same time on node1, assuming all other requirements are met.
# # # # # # # # the configuration, and stop the process. The maximum number of access policies that can be configured in the cluster is 200. # # # # # # List of cluster aware LVM Volume Groups. These volume groups will be used by package applications via the vgchange -a e command. Neither CVM or VxVM Disk Groups should be used here. For example: VOLUME_GROUP /dev/vgdatabase VOLUME_GROUP /dev/vg02 # # # # # # # # List of OPS Volume Groups.
/cfs/mnt1 – for Oracle Base and Home /cfs/mnt2 – for Oracle datafiles /cfs/mnt3 - for OCR and Vote device Initializing the Veritas Volume Manager Use the following steps to create a two node SGeRAC cluster with CFS and Oracle: 1. Initialize the Veritas Volume Manager If not already done, install the VxVM license key on all nodes. Use the following command: # vxinstall (For each question raised when running the vxinstall command to install the VxVM licence key, respond with “n”.) NOTE: 2. CVM 4.
6. Converting Disks from LVM to CVM You can use the vxvmconvert utility to convert LVM volume groups into CVM disk groups. Before you can do this, the volume group must be deactivated, which means that any package that uses the volume group must be halted. This procedure is described in the Managing Serviceguard Sixteenth Edition user’s guide Appendix G. 7. Initializing Disks for CVM/CFS You need to initialize the physical disks that will be employed in CVM disk groups.
307200 sectors, 307200 blocks of size 1024, log size 1024 blocks largefiles supported 12. Configure Mount Point # cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw The following output will be displayed: Package name “SG-CFS-MP-1” was generated to control the resource. Mount point “/cfs/mnt1” was associated with the cluster. #cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw The following output will be displayed: Package name “SG-CFS-MP-2” was generated to control the resource.
SG-CFS-MP-2 SG-CFS-MP-3 up up running running enabled enabled no no CAUTION: Once you create the disk group and mount point packages, it is critical that you administer the cluster with the cfs commands, including cfsdgadm, cfsmntadm, cfsmount, and cfsumount. If you use the general commands such as mount and umount, it could cause serious problems, such as writing to the local file system instead of the cluster file system.
Stopping CVM...CVM is stopped # cfscluster unconfig The following output will be displayed: CVM is now unconfigured Creating a Storage Infrastructure with CVM In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM), Veritas Volume Manager (VxVM), or Veritas Cluster Volume Manager (CVM).
Edit Cluster file 2. Create the Cluster # cmapplyconf -C clm.asc • Start the Cluster # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 3. STATUS up STATUS up up STATE running running Configure the Cluster Volume Manager (CVM) Configure the system multi-node package, SG-CFS-pkg, to configure and start the CVM stack. Unlike VxVM-CVM-pkg, the SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and supports multiple subnets.
4. Creating Volumes and Adding a Cluster Filesystem # vxassist -g ops_dg make vol1 10240m #vxassist -g ops_dg make vol2 10240m # vxassist -g ops_dg make vol3 300m 5.
NOTE: Cluster configuration is described in the previous section. To prepare the cluster for CVM disk group configuration, you need to ensure that only one heartbeat subnet is configured. Then use the following command, which creates the special package that communicates cluster information to CVM: # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf WARNING! The above file should never be edited.
Creating Disk Groups for RAC Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: # vxdg -s init ops_dg c0t3d2 Verify the configuration with the following command: # vxdg list NAME rootdg ops_dg STATE enabled enabled,shared ID 971995699.1025.node1 972078742.1084.node2 Creating Volumes Use the vxassist command to create logical volumes.
Table 2-2 Required Oracle File Names for Demo Database Volume Name Size (MB) Raw Device File Name Oracle File Size (MB) opsctl1.ctl 118 /dev/vx/rdsk/ops_dg/opsctl1.ctl 110 opsctl2.ctl 118 /dev/vx/rdsk/ops_dg/opsctl2.ctl 110 opsctl3.ctl 118 /dev/vx/rdsk/ops_dg/opsctl3.ctl 110 ops1log1.log 128 /dev/vx/rdsk/ops_dg/ops1log1.log 120 ops1log2.log 128 /dev/vx/rdsk/ops_dg/ops1log2.log 120 ops1log3.log 128 /dev/vx/rdsk/ops_dg/ops1log3.log 120 ops2log1.
Adding Disk Groups to the Cluster Configuration For CVM 4.x or later, if the multi-node package was configured for disk group activation, the application package should be configured with package dependency to ensure the CVM disk group is active. For CVM 3.5 and CVM 4.x or later (without using multi-node package) after creating units of CVM storage with VxVM commands, you need to specify the disk groups in each package configuration ASCII file.
# mkdir /dev/vg01 # mknod /dev/vg01/group c 64 0x010000 # vgcreate /dev/vg01 /dev/dsk/c4t5d0 # lvcreate -L 16000 /dev/vg01 # newfs -F vxfs /dev/vg01/rlvol1 # mkdir -p /mnt/app # mount /dev/vg01/lvol1 /mnt/app # chmod 775 /mnt/app 7. Create Oracle Cluster Software Home Directory For installing Oracle Cluster Software on local file system, create the directories on each node. # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # chown -R oracle:oinstall /mnt/app/crs/oracle/product/10.2.
c. Change Permission of OCR Device # chown root:oinstall /dev/vg_ops/rora_ocr # chmod 640 /dev/vg_ops/rora_ocr d. Create Raw Device Mapping File for Oracle Database Configuration Assistant In this example, the database name is “ver10” # ORACLE_BASE=/mnt/app/oracle; export ORACLE_BASE # mkdir -p $ORACLE_BASE/oradata/ver10 # chown -R oracle:oinstall $ORACLE_BASE/oradata # chmod -R 755 $ORACLE_BASE/oradata The following is a sample of the mapping file for DBCA: system=/dev/vg_ops/ropssystem.
# cd /cfs/mnt2 # mkdir oradata # chown oracle:oinstall oradata # chmod 775 oradata NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. The steps shown in the following section are for configuring the volume groups in Serviceguard clusters LVM version 1.0. For more information on using and configuring LVM version 2.x, see the HP-UX 11i Version 3: HP-UX System Administrator's Guide: Logical Volume Management located at: http://docs.hp.
1. 2. 3. In this example, the path to ORACLE_HOME is on a local file system /mnt/app/oracle/ product//db_1. Select installation for database software only. When prompted, run root.sh on each node. Installing RAC Binaries on Cluster File System Logon as a “oracle” user: $ export ORACLE BASE=/cfs/mnt1/oracle $ export DISPLAY={display}:0.0 $ cd <10g/11g RAC installation disk directory> $ ./runInstaller Use following guidelines when installing on a local file system: 1.
export ORACLE_BASE=/cfs/mnt1/oracle export ORACLE_HOME=$ORACLE_BASE/product//db_1 export ORA_CRS_HOME=/mnt/app/crs/oracle/product//crs LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32 export LD_LIBRARY_PATH SHLIB_PATH export \ PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin: CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib export CLASSPATH exp
Configuring Oracle to Use Oracle Disk Manager Library NOTE: 1. 2. 3. The following steps are specific to CFS 4.1 or later. Login as Oracle user Shutdown database Link the Oracle Disk Manager library into Oracle home For Oracle 10g on HP 9000 Systems: $ rm ${ORACLE_HOME}/lib/libodm10.sl $ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm10.sl For Oracle 10g on Integrity Systems: $ rm ${ORACLE_HOME}/lib/libodm10.so $ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm10.
cl opt idn: cl opt rsv: **********: 3. 0 332 17 Verify that the Oracle Disk Manager is loaded: # kcmodule -P state odm Output: state loaded 4. In the alert log, verify the Oracle instance is running. The log should contain output similar to the following: For CFS 4.1: Oracle instance running with ODM: VERITAS 4.1 ODM Library, Version 1.1 For CFS 5.0: Oracle instance running with ODM: VERITAS 5.0 ODM Library, Version 1.0 Configuring Oracle to Stop Using Oracle Disk Manager Library NOTE: 1. 2. 3.
Preparing Oracle Cluster Software for Serviceguard Packages • Stopping the Oracle Cluster Software on each Node For 10g 10.1.0.4 or later: # /sbin/init.d/init.crs stop For 10g 10.2.0.1 or later: # /bin/crsctl stop crs Wait until Oracle Cluster Software completely stops. (Check CRS logs or check for Oracle processes, ps -ef | grep ocssd.bin) • Change Oracle Cluster Software from Starting at Boot Time on each Node For 10g 10.1.0.4 or later: # /sbin/init.d/init.crs disable For 10g 10.2.0.
• DEPENDENCY_CONDITION DEPENDENCY_LOCATION SG-CFS-MP-2=UP SAME_NODE DEPENDENCY_NAME DEPENDENCY_CONDITION DEPENDENCY_LOCATION mp3 SG-CFS-MP-3=UP SAME_NODE Starting and Stopping Oracle Cluster Software In the Serviceguard package control script, configure the Oracle Cluster Software start in the customer_defined_run_cmds function For 10g 10.1.0.4 or later: /sbin/init.d/init.crs start For 10g 10.2.0.
3 Serviceguard Configuration for Oracle 9i RAC This chapter shows the additional planning and configuration that is needed to use Oracle Real Application Clusters 9i with Serviceguard.
Data: User data ___/dev/vg_ops/ropsdata1.dbf_200__________ Data: User data ___/dev/vg_ops/ropsdata2.dbf__200__________ Data: User data ___/dev/vg_ops/ropsdata3.dbf__200__________ Data: Rollback ___/dev/vg_ops/ropsrollback.dbf__300_________ Parameter: spfile1 /dev/vg_ops/ropsspfile1.ora __5_____ Instance 1 undotbs1: /dev/vg_ops/ropsundotbs1.dbf___312___ Instance 2 undotbs2: /dev/vg_ops/ropsundotbs2.dbf___312___ Data: example1__/dev/vg_ops/ropsexample1.
Considerations on using CFS for RAC datafiles and Server Management Storage (SRVM) Use the following list when considering to use CFS for database storage: • Single file system view — Simpler setup for archive recovery since archive area is visible by all nodes • • • • • Oracle create database files Online changes (OMF - Oracle Managed Files) within CFS Better manageability Manual intervention when modifying volumes, DGs, disks Requires the SGeRAC and CFS software CFS and SGeRAC is available in selected
Instance 2 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops2log1.log____20________ Instance 2 Redo Log 2: ___/dev/vx/rdsk/ops_dg/ops2log2.log____20________ Instance 2 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops2log3.log____20_________ Instance 2 Redo Log: _________________________________________________ Instance 2 Redo Log: __________________________________________________ Data: System ___/dev/vx/rdsk/ops_dg/system.dbf___400__________ Data: Temp ___/dev/vx/rdsk/ops_dg/temp.
version 4.1 and later is named SG-CFS-pkg. CVM allows you to activate storage on one node at a time, or you can perform write activation on one node and read activation on another node at the same time (for example, allowing backups). CVM provides full mirroring and dynamic multipathing (DMP) for clusters. CVM supports concurrent storage read/write access between multiple nodes by applications which can manage read/write access contention, such as Oracle Real Application Cluster (RAC).
NOTE: Starting with Oracle 10g and later, Oracle Clusterware processes register with cmgmsd while Oracle server processes register with Oracle Clusterware. The maximum number of Oracle server processes supported is determined by Oracle Clusterware. The maximum number of processes registered by Oracle Clusterware should not exceed the maximum supported by cmgmsd. Oracle foreground server processes are needed to handle the requests of the DB client connected to the DB instance.
NOTE: For more information, see the Serviceguard Version A.11.19 Release Notes at http://docs.hp.com -> High Availability -> Serviceguard Creating Volume Groups and Logical Volumes If your volume groups have not been set up, use the procedure in the next sections. If you have already done LVM configuration, skip ahead to the section ““Installing Oracle Real Application Clusters”.
NOTE: For more information on using LVM, refer to the HP-UX Managing Systems and Workgroups manual. Building Mirrored Logical Volumes for RAC with LVM Commands After you create volume groups and define physical volumes for use in them, you define mirrored logical volumes for data, logs, and control files. It is recommended that you use a shell script to issue the commands described in the next sections.
consistency recovery. With a mirror consistency policy of “NONE”, SLVM does not perform the resynchronization. NOTE: Contact Oracle to determine if your version of Oracle RAC allows “resilvering” and to appropriately configure the mirror consistency recovery policy for your logical volumes. Create logical volumes for use as Oracle data files by using the same options as in the following example: # lvcreate -m 1 -M n -c n -s g -n system.
2. Next, create a control file named group in the directory /dev/vg_ops, as follows: # mknod /dev/vg_ops/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the format: 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. Use the following command to display a list of existing volume groups: # ls -l /dev/*/group 3.
Table 3-2 Required Oracle File Names for Demo Database (continued) Logical Volume Name LV Size (MB) Raw Logical Volume Path Name Oracle File Size (MB)* ops1log3.log 28 /dev/vg_ops/rops1log3.log 20 ops2log1.log 28 /dev/vg_ops/rops2log1.log 20 ops2log2.log 28 /dev/vg_ops/rops2log2.log 20 ops2log3.log 28 /dev/vg_ops/rops2log3.log 20 opssystem.dbf 408 /dev/vg_ops/ropssystem.dbf 400 opstemp.dbf 108 /dev/vg_ops/ropstemp.dbf 100 opsusers.dbf 128 /dev/vg_ops/ropsusers.
NOTE: Serviceguard Manager is the graphical user interface for Serviceguard. It is available as a “plug-in” to the System Management Homepage (SMH). SMH is a web-based graphical user interface (GUI) that replaces SAM as the system administration GUI as of HP-UX 11i v3 (but you can still run the SAM terminal interface; see “Using SAM” on page 32 of the Managing Serviceguard Sixteenth Edition user’s guide).
For a demo database on SLVM or CVM, create logical volumes as shown in Table 3-2: “Required Oracle File Names for Demo Database ”, earlier in this chapter. As the installer prompts for the database file names, use the pathnames of the raw logical volumes instead of using the defaults. NOTE: If you do not wish to install the demo database, select install software only. Cluster Configuration ASCII File The following is an example of an A.11.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # For cluster of up to 4 nodes it is 0.2*MEMBER_TIMEOUT. It increases as number of nodes increases and reaches to 0.5*MEMBER_TIMEOUT for 16 nodes If quorum server is configured on busy network or if quorum server polling is experiencing timeouts (syslog messages) or if quorum server is used for large number of clusters, such default time (as mentioned above) might not be sufficient.
# Node capacity definition is optional, but if CAPACITY_NAME is specified, # CAPACITY_VALUE must also be specified; CAPACITY_NAME must come first. # To specify more than one capacity, repeat this process for each capacity. # # # # # # # # # NOTE: If a given capacity is not defined for a node, Serviceguard assumes that capacity is infinite on that node.
# Warning: There are no standby network interfaces for lan0. # Possible standby Network Interfaces for lan1: lan3. # Warning: There are no standby network interfaces for lan2. # Cluster Timing Parameters (microseconds). # # # # # # # # # # # # The MEMBER_TIMEOUT parameter defaults to 14000000 (14 seconds). If a heartbeat is not received from a node within this time, it is declared dead and the cluster reforms without that node. A value of 10 to 25 seconds is appropriate for most installations.
# ICMP Echo Request messages will be included by default with IP_MONITOR # set to ON, and with its gateway listed as a POLLING_TARGET. SUBNET 16.89.112.0 IP_MONITOR ON POLLING_TARGET 16.89.112.1 SUBNET 10.142.1.0 IP_MONITOR OFF SUBNET 10.142.2.0 IP_MONITOR OFF SUBNET fec0:0:0:8e01:: IP_MONITOR OFF SUBNET 3ffe:1000:0:8e01:: IP_MONITOR OFF SUBNET fec0:0:0:8e02:: IP_MONITOR OFF SUBNET 3ffe:1000:0:8e02:: IP_MONITOR OFF # Package Configuration Parameters.
# of 0.1 for the package: # # WEIGHT_NAME processor # WEIGHT_DEFAULT 0.1 # # WEIGHT_NAME # WEIGHT_DEFAULT # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # Access Control Policy Parameters. # # # # # # List of cluster aware LVM Volume Groups. These volume groups will be used by package applications via the vgchange -a e command. Neither CVM or VxVM Disk Groups should be used here.
OPS_VOLUME_GROUP OPS_VOLUME_GROUP /dev/vg_dd0 /dev/vg_dd1 Creating a Storage Infrastructure with CFS In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM), Veritas Volume Manager (VxVM), or Veritas Cluster Volume Manager (CVM). You can also use a mixture of volume types, depending on your needs.
3. Create the Cluster # cmapplyconf -C clm.asc 4. Start the Cluster # cmruncl # cmviewcl The following output will be displayed: CLUSTER ever3_cluster NODE ever3a ever3b 5. STATUS up STATUS up up STATE running running Configure the Cluster Volume Manager (CVM) Configure the system multi-node package, SG-CFS-pkg, to configure and start the CVM/CFS stack. Unlike VxVM-CVM-pkg, the SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and supports multiple subnets.
9. Create the Disk Group Multi-Node package. Use the following command to add the disk group to the cluster: # cfsdgadm add cfsdg1 all=sw The following output will be displayed: Package name “SG-CFS-DG-1” was generated to control the resource shared disk group “cfsdg1” is associated with the cluster. 10. Activate the Disk Group # cfsdgadm activate cfsdg1 11.
# cfsmount /cfs/cfssrvm 14. Check CFS Mount Points # bdf | grep cfs /dev/vx/dsk/cfsdg1/vol1 10485760 /dev/vx/dsk/cfsdg1/vol2 10485760 /dev/vx/dsk/cfsdg1/volsrvm 307200 36455 9796224 19651 9796224 3246 284957 0% /cfs/mnt1 0% /cfs/mnt2 1% /cfs/cfssrvm 15.
The following output will be generated: Shared disk group “cfsdg1” was disassociated from the cluster. NOTE: “cfsmntadm delete” also deletes the disk group if there is no dependent package. To ensure the disk group deletion is complete, use the above command to delete the disk group package. 4. De-configure CVM # cfscluster stop The following output will be generated: Stopping CVM...
Preparing the Cluster and the System Multi-node Package for use with CVM 4.x or later 1. Create the Cluster file: # cd /etc/cmcluster # cmquerycl -C clm.asc -n ever3a -n ever3b Edit Cluster file NOTE: To prepare the cluster for CVM configuration, you need to be sure MAX_CONFIGURED_PACKAGES to minimum of 3 (the default value for MAX_CONFIGURED_PACKAGES for Serviceguard A.11.19 is 300) cluster configuration file. In the sample set the value to 10. 2. Create the Cluster # cmapplyconf -C clm.
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example: # /etc/vx/bin/vxdisksetup -i c4t4d0 • Create the Disk Group for RAC Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example: # vxdg -s init ops_dg c4t4d0 4.
NOTE: The specific commands for creating mirrored and multi-path storage using CVM are described in the HP-UX documentation for the Veritas Volume Manager. Using CVM 3.x This section has information on how to prepare the cluster with CVM 3.x (on HP-UX releases that support Veritas CFS and CVM; see “About Veritas CFS and CVM from Symantec” (page 18)). Preparing the Cluster for Use with CVM 3.x In order to use the Veritas Cluster Volume Manager (CVM) version 3.
the volume group must be halted. This procedure is described in the Managing Serviceguard Sixteenth Edition user’s guide Appendix G. Initializing Disks for CVM Initialize the physical disks that will be employed in CVM disk groups. If a physical disk has been previously used with LVM, you should use the pvremove command to delete the LVM header data from all the disks in the volume group (this is not necessary if you have not previously used the disk with LVM).
This policy can be re-set on a disk group basis by using the vxedit command, as follows: # vxedit set diskdetpolicy=global NOTE: The specific commands for creating mirrored and multi-path storage using CVM are described in the HP-UX documentation for the Veritas Volume Manager. Oracle Demo Database Files The following set of volumes is required for the Oracle demo database which you can create during the installation process.
Example: # ln -s /dev/vx/rdsk/ops_dg/opsctl1.ctl \ /u01/ORACLE/db001/ctrl01_1.ctl Example, Oracle9: 1. Create an ASCII file, and define the path for each database object. control1=/dev/vx/rdsk/ops_dg/opsctl1.ctl or control1=/u01/ORACLE/db001/ctrl01_1.ctl 2. Set the following environment variable where filename is the name of the ASCII file created. # export DBCA_RAW_CONFIG=/filename Adding Disk Groups to the Cluster Configuration For CVM 4.
c. Set up CFS directory for Oracle datafiles. # cd /cfs/mnt2 # mkdir oradata # chown oracle:dba oradata # chmod 755 oradata # ll total 0drwxr-xr-x 2 root root dba 96 Jun 3 13:45 oradat d. 96 Jun 3 11:43 lost+founddrwxr-xr-x 2 oracle Set up CFS directory for Server Management. Preallocate space for srvm (200MB) # prealloc /cfs/cfssrvm/ora_srvm 209715200 # chown oracle:dba /cfs/cfssrvm/ora_srvm 2. Install Oracle RAC Software a.
Verify that Oracle Disk Manager is Configured NOTE: 1. The following steps are specific to CFS 4.1 or later. Check the license for CFS 4.1 or later. #/opt/VRTS/bin/vxlictest -n “VERITAS Storage Foundation for Oracle” -f “ODM” output: ODM feature is licensed 2. Check that the VRTSodm package is installed: # swlist VRTSodm output for CFS 4.1: VRTSodm 4.1m VERITAS Oracle Disk Manager VRTSodm.ODM-KRN 4.1m VERITAS ODM kernel files VRTSodm.ODM-MAN 4.1m VERITAS ODM manual pages VRTSodm.ODM-RUN 4.
commit: create: delete: identify: io: reidentify: resize: unidentify: mname: vxctl: vxvers: io req: io calls: comp req: comp calls: io mor cmp: io zro cmp: cl receive: cl ident: cl reserve: cl delete: cl resize: cl same op: cl opt idn: cl opt rsv: **********: 3.
$ ln -s ${ORACLE_HOME}/lib/libodmd9.so ${ORACLE_HOME}/lib/libodm9.so 5. Restart the database Using Packages to Configure Startup and Shutdown of RAC Instances To automate the startup and shutdown of RAC instances on the nodes of the cluster, you can create packages which activate the appropriate volume groups and then run RAC. Refer to the section “Creating Packages to Launch Oracle RAC Instances” NOTE: The maximum number of RAC instances for Oracle 9i is 127 per cluster.
To create the package with Serviceguard Manager select the cluster. Go to the actions menu and choose configure package. To modify a package, select the package. For an instance package, create one package for each instance. On each node, supply the SID name for the package name. To create a package on the command line, use the cmmakepkg command to get an editable configuration file. Set the AUTO_RUN parameter to YES,if you want the instance to start up as soon as the node joins the cluster.
Using Serviceguard Manager to Write the Package Control Script As you complete the tabs for the configuration, the control script can be generated automatically. When asked to supply the pathname of the package run and halt scripts, use the filenames from the ECM toolkit. For more information, use the Help key. When you create a package control script this way, you do not need to do any further editing, but you may customize the script if you wish.
NOTE: Use care in defining service run commands. Each run command is executed by the control script in the following way: • The cmrunserv command executes each run command and then monitors the process id of the process created by the run command. • When the command started by cmrunserv exits, Serviceguard determines that a failure has occurred and takes appropriate action, which may include transferring the package to an adoptive node.
Do not define the RAC instance as a package service. Instead, include the commands that start up an RAC instance in the customer_defined_run_commands section of the package control script. Similarly, you should include the commands that halt an RAC instance in the customer_defined_halt_commands section of the package control script. Define the Oracle monitoring command as a service command, or else use the special Oracle script provided with the ECM Toolkit.
7. Gather how to start the database using an ECMT script. In Serviceguard Manager, enter this filename for the control script start command. /etc/cmcluster/pkg/${SID_NAME}/toolkit.sh start Example:/etc/cmcluster/pkg/ORACLE_TEST0/toolkit.sh start 8. Gather how to stop the database using an ECMT script. In Serviceguard Manager, enter this filename for the control script stop command. /etc/cmcluster/pkg/${SID_NAME}/toolkit.sh stop Example: /etc/cmcluster/pkg/ORACLE_TEST0/toolkit.
4 Maintenance and Troubleshooting This chapter includes information about carrying out routine maintenance on an Real Application Cluster configuration. As presented here, these tasks differ in some details from the similar tasks described in the Managing Serviceguard documentation.
Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up NODE mo PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 STATUS up NAME lan0 lan1 lan3 STATE running Quorum_Server_Status: NAME STATUS white up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up STANDBY up PATH 0/0/0/0 0/8/0/0/4/0 0/8/0/0/6/0 NAME lan0 lan1 lan3 MULTI_NODE_PACKAGES PACKAGE SG-CFS-pkg NODE_NAME minie STATUS up STATUS up Script_Parameters: ITEM
DEPENDENCY_NAME SG-CFS-pkg PACKAGE SG-CFS-MP-1 NODE_NAME minie SATISFIED yes STATUS up STATE running STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up PACKAGE SG-CFS-MP-2 NODE_NAME minie STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up PACKAGE SG-CFS-MP-3 NODE_NAME minie STATUS up STATE running Dependency_Parameters: DEPENDENCY_NAME SG-CFS-DG-1 NODE_NAME mo STATUS up SYSTEM no SWITCHING enable
Cluster Status The status of a cluster may be one of the following: • • • • Up. At least one node has a running cluster daemon, and reconfiguration is not taking place. Down. No cluster daemons are running on any cluster node. Starting. The cluster is in the process of determining its active membership. At least one cluster daemon is running. Unknown. The node on which the cmviewcl command is issued cannot communicate with other nodes in the cluster.
For multi-node packages, node switching Disabled means the package cannot start on that node. Status of Group Membership The state of the cluster for Oracle RAC is one of the following: • • Up. Services are active and being monitored. The membership appears in the output of cmviewcl -l group. Down. The cluster is halted and GMS services have been stopped. The membership does not appear in the output of the cmviewcl -l group.
Packages can also be configured with one of two values for the FAILBACK_POLICY parameter: • • AUTOMATIC. With this setting, a package, following a failover, returns to its primary node when the primary node becomes available again. MANUAL. With this setting, a package, following a failover, must be moved back to its original node by a system administrator. Failover and failback policies are displayed in the output of the cmviewcl -v command.
Quorum Server Status If the cluster is using a quorum server for tie-breaking services, the display shows the server name, state and status following the entry for each node, as in the following excerpt from the output of cmviewcl -v: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Quorum Server Status: NAME STATUS lp-qs up ...
CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up STANDBY up PATH 56/36.
CLUSTER example NODE ftsys9 STATUS up STATUS up PACKAGE pkg1 pkg2 NODE ftsys10 STATE running STATUS up up STATUS up STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE running Both packages are now running on ftsys9 and pkg2 is enabled for switching. Ftsys10 is running the daemon and no packages are running on ftsys10.
Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled Alternate up enabled Alternate up enabled NAME manx burmese tabby persian Online Reconfiguration The online reconfiguration feature provides a method to make configuration changes online to a Serviceguard Extension for RAC (SGeRAC) cluster.
NOTE: For more information, see the Serviceguard Version A.11.19 Release Notes at http://docs.hp.com -> High Availability -> Serviceguard Making LVM Volume Groups Shareable Normally, volume groups are marked to be activated in “shared” mode when they are listed with the OPS_VOLUME_GROUP parameter in the cluster configuration file or in Serviceguard Manager. which occurs when the configuration is applied. However, in some cases you may want to manually make a volume group sharable.
NOTE: Do not share volume groups that are not part of the RAC configuration unless shared access is controlled. Deactivating a Shared Volume Group Issue the following command from each node to deactivate the shared volume group: # vgchange -a n /dev/vg_ops Remember that volume groups remain shareable even when nodes enter and leave the cluster. NOTE: If you wish to change the capacity of a volume group at a later time, you must deactivate and unshare the volume group first.
# mknod /dev/vg_ops/group c 64 0xhh0000 The major number is always 64, and the hexadecimal minor number has the format: 0xhh0000 where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. 13. Use the vgimport command, specifying the map file you copied from the configuration node.
NOTE: For CVM without CFS, if you are adding a disk group to the cluster configuration, make sure you also modify any package or create the package control script that imports and deports this disk group. If you are adding a CVM disk group, be sure to add the STORAGE_GROUP entry for the disk group to the package ASCII file. For CVM with CFS, if you are adding a disk group to the cluster configuration, make sure you also create the corresponding multi-node package.
of further action in case of a problem. For example, you could configure a disk monitor to report when a mirror was lost from a mirrored volume group being used in a non-RAC package. For additional information, refer to http://docs.hp.com -> High Availability -> Event Monitoring Service and HA Monitors Using EMS Hardware Monitors A set of hardware monitors is available for monitoring and reporting on memory, CPU, and many other system values.
NOTE: If your LVM installation requires online replacement of disk mechanisms, the use of disk arrays may be required, because software mirroring of JBODs with MirrorDisk/UX does not permit hot swapping for disks that are activated in shared mode. Replacing a Mechanism in an HA Enclosure Configured with Exclusive LVM Non-Oracle data that is used by packages may be configured in volume groups that use exclusive (one-node-at-a-time) activation.
# pvchange -a N [pv path] Alternatively, use the pvchange -a N [pv path] command to detach a disk (all paths to the disk) and close it. Use this to allow diagnostics or replace a multi-ported disk. NOTE: If the volume group is mirrored, applications can continue accessing data on mirror copies after the commands above. If the volume is not mirrored, then any access attempts to the device may hang indefinitely or time out. This depends upon the LV timeout value configured for the logical volume. 2.
5. Activate the volume group on one node in exclusive mode then deactivate the volume group: # vgchange -a e vg_ops This will synchronize the stale logical volume mirrors. This step can be time-consuming, depending on hardware characteristics and the amount of data. 6. Deactivate the volume group: # vgchange -a n vg_ops 7.
Figure 4-1 F/W SCSI Buses with In-line Terminators The use of in-line SCSI terminators allows you to do hardware maintenance on a given node by temporarily moving its packages to another node and then halting the original node while its hardware is serviced. Following the replacement, the packages can be moved back to the original node. Use the following procedure to disconnect a node that is attached to the bus with an in-line SCSI terminator or with a Y cable: 1. 2. 3. 4. 5. 6. 7. 8.
4. 5. 6. 7. Remove the defective I/O card. Install the new card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed. In SAM, select the option to attach the new I/O card. Add the node back into the cluster by using Serviceguard Manager or the cmrunnode command.
Monitoring RAC Instances The DB Provider provides the capability to monitor RAC databases. RBA (Role Based Access) enables a non-root user to have the capability to monitor RAC instances using Serviceguard Manager.
A Software Upgrades Serviceguard Extension for RAC (SGeRAC) software upgrades can be done in the two following ways: • rolling upgrade • non-rolling upgrade Instead of an upgrade, moving to a new version can be done with: • migration with cold install Rolling upgrade is a feature of SGeRAC that allows you to perform a software upgrade on a given node without bringing down the entire cluster. SGeRAC supports rolling upgrades on version A.11.
For more information on support, compatibility, and features for SGeRAC, refer to the Serviceguard Compatibility and Feature Matrix, located at http://docs.hp.com -> High Availability -> Serviceguard Extension for RAC. Steps for Rolling Upgrades Use the following steps when performing a rolling SGeRAC software upgrade: 1. Halt Oracle (RAC, CRS, Clusterware, OPS) software on the local node (if running). 2. Halt Serviceguard/SGeRAC on the local node by issuing the Serviceguard cmhaltnodecommand. 3.
NOTE: While you are performing a rolling upgrade, warning messages may appear while the node is determining what version of software is running. This is a normal occurrence and not a cause for concern. Figure A-1 Running Cluster Before Rolling Upgrade Step 1. 1. 2. Halt Oracle (RAC, CRS, Clusterware, OPS) software on node 1. Halt node 1. This will cause the node’s packages to start up on an adoptive node.
Figure A-2 Running Cluster with Packages Moved to Node 2 Step 2. Upgrade node 1 and install the new version of Serviceguard and SGeRAC (A.11.16), as shown in Figure A-3. NOTE: If you install Serviceguard and SGeRAC separately, Serviceguard must be installed before installing SGeRAC. Figure A-3 Node 1 Upgraded to SG/SGeRAC 11.16 Step 3. 1. Restart the cluster on the upgraded node (node 1) (if desired).
Figure A-4 Node 1 Rejoining the Cluster Step 4. 1. 2. Halt Oracle (RAC, CRS, Clusterware, OPS) software on node 2. Halt node 2. You can do this in Serviceguard Manager, or from the command line issue the following: # cmhaltnode -f node2 This causes both packages to move to node 1; see Figure A-5. 3. 4. Upgrade node 2 to Serviceguard and SGeRAC (A.11.16) as shown in Figure A-5. When upgrading is finished, enter the following command on node 2 to restart the cluster on node 2: # cmrunnode node2 5.
Figure A-5 Running Cluster with Packages Moved to Node 1 Step 5. Move PKG2 back to its original node. Use the following commands: # cmhaltpkg pkg2 # cmrunpkg -n node2 pkg2 # cmmodpkg -e pkg2 The cmmodpkg command re-enables switching of the package, which is disabled by the cmhaltpkg command. The final running cluster is shown in Figure A-6.
Figure A-6 Running Cluster After Upgrades Limitations of Rolling Upgrades The following limitations apply to rolling upgrades: • During a rolling upgrade, you should issue Serviceguard/SGeRAC commands (other than cmrunnode and cmhaltnode) only on a node containing the latest revision of the software. Performing tasks on a node containing an earlier revision of the software will not work or will cause inconsistent results.
Non-Rolling Software Upgrades A non-rolling upgrade allows you to perform a software upgrade from any previous revision to any higher revision or between operating system versions. For example, you may do a non-rolling upgrade from SGeRAC A.11.14 on HP-UX 11i v1 to A.11.16 on HP-UX 11i v2, given both are running the same architecture. The cluster cannot be running during a non-rolling upgrade, therefore it is necessary to halt the entire cluster in order to perform the upgrade.
10. Reinstall the cluster applications, such as RAC. 11. Restore the data.
B Blank Planning Worksheets This appendix reprints blank planning worksheets used in preparing the RAC cluster. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name: _____________________________________________________ Disk Group Name: __________________________________________________________ Physical Volume Name: _____________________________________________________ Physical Volume Name:__________________________________
Index A activation of volume groups in shared mode, 123 adding packages on a running cluster, 108 administration cluster and package states, 113 array replacing a faulty mechanism, 127, 128, 129 AUTO_RUN parameter, 108 AUTO_START_TIMEOUT in sample configuration file, 48, 87 B building a cluster CVM infrastructure, 59, 97 building an RAC cluster displaying the logical volume infrastructure, 47, 85 logical volume infrastructure, 41, 80 building logical volumes for RAC, 46, 84 C CFS, 54, 58, 93 deleting from
HEARTBEAT_IP in sample configuration file, 48, 87 high availability cluster defined, 15 O lock disk replacing a faulty mechanism, 130 logical volumes blank planning worksheet, 146 creating, 46, 84 creating for a cluster, 43, 63, 82, 101 creating the infrastructure, 41, 80 disk arrays, 45, 83 filled in planning worksheet, 32, 33, 75, 77 lssf using to obtain a list of disks, 42, 81 LV in sample package control script, 110 LVM creating on disk arrays, 45, 83 LVM commands exporting files, 47, 86 online hardw
physical volumes creating for clusters, 42, 81 filled in planning worksheet, 145 planning worksheets for logical volume planning, 32, 33, 75, 77 worksheets for physical volume planning, 145 planning worksheets blanks, 145 point to point connections to storage devices, 24 PVG-strict mirroring creating volume groups with, 42, 81 R RAC group membership services, 20 overview of configuration, 15 status, 117 RAC cluster defined, 15 removing packages on a running cluster, 108 removing Serviceguard Extension for
blanks, 145 150 Index
