Using the Oracle Toolkit in a HP Serviceguard Cluster ==================================================================== Revision: B.06.00 CONTENTS: A. Overview B. Supported versions C. Support For Oracle Database without ASM i. Setting Up the Application ii. Setting Up the Toolkit iii. Package Configuration Example D. Support for Oracle ASM instance and Oracle database with ASM i. Setting Up the Application ii. Setting Up the Toolkit iii. Package Configuration Example E.
For guidelines on creating modular packages, please refer to the sections C and D and the whitepaper "Modular package support in Serviceguard for Linux and ECM Toolkits" located at http://www.hp.com/go/hpux-serviceguard-docs -> HP Enterprise Cluster Master Toolkit This README file assumes the user has already installed the supported and compatible versions of Serviceguard, Oracle 10g or 11g, and the Enterprise Cluster Master Toolkit (ECMT), which contains the Oracle toolkit. B.
To use CFS, you must install the Serviceguard Storage Management Suite. appropriate version of HP For more details please refer to the Serviceguard Release Notes at: http://www.hp.com/go/hpux-serviceguard-docs -> HP Enterprise Cluster Master Toolkit NOTE: This toolkit supports Oracle 10g Release 2 ASM and 11g R1 and 11g R2 ASM on LVM only. Oracle 10g Release 2 ASM and 11g R1 and 11g R2 is not supported with CFS. C.
to ensure all nodes have the same configuration at any point in time. This is also the recommended configuration since there is no need for replication of changes made to the configuration file on a particular node. A password file (that is, orapw${SID_NAME}) must be created for each Oracle instance in the configuration directory. The password file is created via the utility 'orapwd'. Oracle supports multiple database instances running on a node simultaneously.
2. Make sure that the 'oracle' user has the same user id and group id on all nodes in the cluster. 3. Some of the possible configurations: Configuration of shared file system using LVM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a volume group, logical volume and file system to hold the necessary configuration information and symbolic links to the Oracle executables. This file system will be defined as ORACLE_HOME in the package control scripts.
$ vxdctl enable Create the directory /ORACLE_TEST0 on all nodes so that /dev/vx/dsk/DG0_ORACLE_TEST0/lvol1 can be mounted on that node if the package is to be run on the node. If you need help creating, importing, or managing the volume group or disk group and filesystem, please refer to the chapter titled "Building an HA Cluster Configuration" in the "Managing Serviceguard" manual for explicit instructions.
(NOTE: If you opted to store the configuration information on a local disk and propagate the information to all nodes, remember pfile/spfile, password file, and all control files and data files have to be on shared storage. For this set up, you will need to set up symbolic links to the pfile and password file from /home/oracle/dbs as follows: ln -s /ORACLE_TEST0/dbs/initORACLE_TEST0.ora \ ${ORACLE_HOME}/dbs/initORACLE_TEST0.ora ln -s /ORACLE_TEST0/dbs/orapwORACLE_TEST0.
link /ORACLE_TEST1 to /home/oracle. This configuration makes it possible to run several Oracle instances on one node, facilitating failover/failback of Oracle packages between nodes in the cluster.
/dev/vg01_ORACLE_TEST0/rlvol1 /dev/vg02_ORACLE_TEST0/rlvol1 /dev/vg02_ORACLE_TEST0/rlvol2 #Raw logical volume Oracle data #Raw logical volume Oracle data #Raw logical volume Oracle data or for use with Asynchronous disk access and file systems: /dev/vg01_ORACLE_TEST0/lvol1 /dev/vg02_ORACLE_TEST0/lvol1 /dev/vg02_ORACLE_TEST0/lvol2 #Logical volume Oracle data #Logical volume Oracle data #Logical volume Oracle data Please refer to your Oracle documentation is more appropriate for your environment.
hagetdbstatus.sh, hatimeoutdbhang.sh and SGAlert.sh) which work with each other to integrate Oracle database with the Serviceguard package control script. For modular packages, there is an Attribute Definition File(ADF) oracle, a Toolkit Module Script (tkit_module.sh) and a Toolkit Configuration File Generator Script(tkit_gen.sh) which work with the six scripts mentioned above for legacy packages, to integrate Oracle with the Serviceguard Master Control Script. These scripts are: - haoracle.
not specified, Oracle picks this up from the Oracle configuration directory $ORACLE_HOME/dbs. This parameter when configured overrides the default file/location. If both pfile and spfile are present and this parameter left unspecified, then ECM Oracle Toolkit will make use of pfile as the parameter file. To make use of spfile in the toolkit do not configure this parameter and remove the default pfile from its default location. MONITOR_PROCESSES: The names of all processes that should be executing.
MONITOR_INTERVAL: The time interval, in seconds, script will wait between checks to ensure that the Oracle instance is running. Default value is 30 seconds. TIME_OUT: The amount of time, in seconds, to wait for the Oracle abort to complete before killing the Oracle processes defined in MONITOR_PROCESSES.
shutdown using the abort option. If "immediate" is specified, the database is shutdown using the immediate option. ALERT_MAIL_ID: This parameter is used to specify the e-mail address for sending alerts. - Main Script (haoracle.sh) This script contains a list of internally used variables and functions that support the starting and stopping of an Oracle database or ASM instance. This script will be called by toolkit.
hagetdbstatus.sh and hatimeoutdbhang.sh to check the database status. - Alert Notification Script(SGAlert.sh) This script is used to send an email to the email address specified by the value of the ALERT_MAIL_ID package attribute, whenever there are critical problems with the package. - Interface Script (toolkit.sh) This script is the interface between control script and the Oracle toolkit. the Serviceguard package NOTE: The following three scripts are used only during the modular method of packaging.
include the listener name. For example: service_name oracle_listener_monitor_1 service_cmd "$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_monitor_listener " service_restart none service_fail_fast_enabled no service_halt_timeout 300 The above lines have to be repeated for each listener. The user should ensure that the listener name passed to this service_cmd must match to one of those specified in the LISTENER_NAME array.
- log - Log a message. A message is logged to the package log everytime a hang is detected. If the MONITOR_INTERVAL attribute is set to 30 seconds, then a message is logged to the package log file every 30 seconds. - alert - Send an alert mail. An alert mail is sent to the email address specified with the ALERT_MAIL_ID attribute. The mail is sent only the first time a database hang is detected. - failover - Failover the package to adoptive node. The default value for ACTION is 'failover'.
this database instance. Create a file system to hold the necessary configuration information and symbolic links to the Oracle executables. This file system will be used as ORACLE_HOME in the package control scripts. Since the volume group and file system have to be uniquely named within the cluster, use the name of the database instance ($SID_NAME) in the name.
$ sqlplus '/ as sysdba' SQL> startup SQL> exit Wait for few minutes, then check for the existence of processes (there should be several, identified by "ora_") Oracle $ ps -fu oracle Bring the database down, $ sqlplus '/ as sysdba' SQL> shutdown immediate SQL> exit If you are using LVM - unmount and deactivate the volume group, $ umount /ORACLE_TEST0 $ vgchange -a n /dev/vg0_ORACLE_TEST0 If you are using VxVM - unmount and deport the disk group, $ umount /ORACLE_TEST0 $ vxdg deport /dev/vx/dsk/DG0_ORACLE
- The Serviceguard package control script (ORACLE_TEST0.cntl). Below are some package control environment. examples of modifications to the Serviceguard script you need to make to customize to your If you are using LVM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VOLUME GROUPS Define the volume groups that are used by the Oracle instance.
LV[0]=/dev/vx/dsk/DG00_${SID_NAME} FS[0]=/${SID_NAME} EXAMPLE: LV[0]=/dev/vx/dsk/DG00_ORACLE_TEST0/lvol1 FS[0]=/ORACLE_TEST0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note: If you are using CFS mounted file systems you must NOT configure volume groups, logical volumes and file systems in the package control script but configure dependency on Serviceguard CFS packages.
To reset the restart counter execute the following command cmmodpkg [-v] [-n node_name] -R -s service_name package_name NOTE:- If listener monitoring is not required, configure a new service for listener. then do NOT Edit the customer_defined_run_cmds function to execute the toolkit.sh script with the start option. In the example below, the line "/etc/cmcluster/pkg/ORACLE_TEST0/toolkit.sh start" was added, and the ":" null command line deleted.
'/etc/cmcluster/pkg/${SID_NAME}/${SID_NAME}.conf' EXAMPLE: /etc/cmcluster/pkg/ORACLE_TEST0/ORACLE_TEST0.conf The configuration file should be edited as indicated by the comments in that file. The package name must be unique within the cluster.
Serviceguard CFS package(e.g. SG-CFS-MP-1, CFS mount point package). If the Oracle package is configured to depend on a Serviceguard CFS package, the Oracle package will run as long as the dependee package is running. If the dependee package fails, then the dependent Oracle package will also fail. To configure dependency of the Oracle package, you must set the following configurable parameters in the package configuration file.
LISTENER_RESTART= PFILE=${ORACLE_HOME}/dbs/init${SID_NAME}.
#cmmakepkg -m ecmt/oracle/oracle pkg.conf where, - 'ecmt/oracle/oracle' is the ECMT Oracle toolkit module name. - 'pkg.conf' is the name of the package configuration file ii. Configure the following Serviceguard parameters in the pkg.conf file. package_name - Set to any name desired. package_type - Set to "failover". Edit the service parameters if necessary. The service parameters are preset to service_name oracle_service service_cmd "$SGCONF/scripts/ecmt/oracle/tkit_module.
package configuration file. It also creates the toolkit configuration directory defined by TKIT_DIR parameter on all target nodes, if not already present and then creates the toolkit configuration file in it with the values specified in the pkg.conf file. For more information on modular packages, please refer the Whitepaper "Modular package support in Serviceguard for Linux and ECM Toolkits" at http://www.hp.
Serviceguard, please refer to the whitepaper "Support for Oracle 10gR2 and 11gR1 ASM with Serviceguard on HP-UX 11iv2 and 11iv3" available at: www.hp.com/go/hpux-serviceguard-docs -> HP Enterprise Cluster Master Toolkit The Oracle toolkit uses Multi-Node Package (MNP) and the package dependency feature to integrate Oracle ASM with HP Serviceguard. An MNP is used to encapsulate the per-node ASM instances with one or more Oracle single instance failover packages that are dependent on this MNP.
2. Install the ASM binaries on local storage in ASM_HOME on all the nodes. ASM can have its own home to keep the database and ASM homes separate and to have a simpler configuration. Install interim patches 7330611 and 7225720 on all the nodes (these patches are not required for 11gR2). Verify that the patch has been installed correctly on all the nodes. Set your current directory to where the patch is located.
Halt the database instance and listener and ASM instance, configured) on all the nodes. (if ii. Setting Up the Toolkit The internal structure of the Oracle Toolkit Scripts is the same for ASM Support as well. Please refer to section C for information about the various Oracle toolkit scripts. There are new variables introduced for ASM support, namely, INSTANCE_TYPE, ASM, ASM_HOME, ASM_USER, ASM_SID, ASM_DISKGROUP, ASM_VOLUME_GROUP and KILL_ASM_FOREGROUNDS. The following variables contained in haoracle.
groups used by the database instance. This parameter must be set only for the ASM database instance package. NOTE: For ASM instance package, no value must be set for this parameter. ASM_VOLUME_GROUP: This parameter gives the list of volume groups used in the disk groups for the ASM database instance. This parameter must be set only for the ASM database instance package. NOTE: For ASM instance package, no value must be set for this parameter. ASM_HOME: The home directory where ASM is installed.
is stored on the ASM diskgroup. This is true only for the ASM instance and not for the database instance that uses the ASM diskgroup. For an ASM instance, the ECM Toolkit cannot verify the existence of the spfile on the ASM diskgroup. So, it is mandatory that a pfile is created from the spfile on the local disk, at a location that ECMT can access. For 11gR2 ASM instance, the value of the PFILE attribute must point to the pfile that is created on the local disk.
instance package. MONITOR_INTERVAL: The time interval, in seconds, the script waits between checks to ensure that the Oracle instance is running. Default is 30 seconds. This parameter must be set in the ASM and database instance packages. TIME_OUT: waits during The amount of time, in seconds, the script package halt for the Oracle database shutdown to complete before killing the Oracle processes defined in MONITOR_PROCESSES.
USER_SHUTDOWN_MODE: This parameter is used to specify the instance shutdown mode only when a shutdown is initiated by the user and not due to a failure of a service. this parameter can take values "abort" or "immediate" only. If "abort" is specified, the instance is shutdown using the abort option. If "immediate" is specified, the instance is shutdown using the immediate option. The default value is "abort". This parameter can be set in the ASM and database instance package.
INSTANCE_TYPE=ASM ASM_HOME=/ASM_TEST0 ASM_USER=oracle ASM_SID=+ASM PFILE=${ASM_HOME}/dbs/init${ASM_SID}.
In the package control script asmpkg.cntl, edit the following parameters. Configure the package service: SERVICE_NAME[0]="ORACLE_ASM_SRV" SERVICE_CMD[0]="/etc/cmcluster/asm_package_mnp/toolkit.sh monitor" SERVICE_RESTART[0]="-r 2" Add in the customer_defined_run_cmds function: /etc/cmcluster/asm_package_mnp/toolkit.sh start Add in the customer_defined_halt_cmds function: if [ $SG_HALT_REASON = "user_halt" ]; then reason="user" else reason="auto" fi /etc/cmcluster/asm_package_mnp/toolkit.
Copy the framework scripts provided to this location. cp /opt/cmcluster/toolkit/oracle/* . - Edit the configuration file haoracle.conf for the database failover package to fit the individual's Oracle environment. Set the following parameters in the configuration file.
PARENT_ENVIRONMENT=yes CLEANUP_BEFORE_STARTUP=no USER_SHUTDOWN_MODE=abort Generate the database package configuration file and the control script In the database package directory and edit the parameters of these files as mentioned for the database package. # cmmakepkg -p db1pkg.conf # cmmakepkg -s db1pkg.cntl Edit the package configuration file db1pkg.conf as shown below: PACKAGE_NAME - Set to any name desired. PACKAGE_TYPE - Set to FAILOVER. RUN_SCRIPT /etc/cmcluster/db1_package/db1pkg.
If listener service is configured in the package configuration file, set the following parameters. SERVICE_NAME[1]="ORACLE_LSNR_SRV" SERVICE_CMD[1]="/etc/cmcluster/db1_package/toolkit.sh monitor_listener" SERVICE_RESTART[1]="-r 2" Configure the Package IP and the SUBNET. Add in the customer_defined_run_cmds function: /etc/cmcluster/db1_package/toolkit.
the toolkit scripts that were previously used in legacy packages will be installed in the /opt/cmcluster/toolkit/oracle directory. This directory is called the Installation directory. The user can copy these scripts to a configuration directory and define this location in the parameter "TKIT_DIR" in the modular package configuration file. Serviceguard uses the toolkit scripts in the configuration directory by default.
Configure the toolkit parameter TKIT_DIR. This parameter is synonymous to the legacy package directory (e.g. /etc/cmcluster/asm_package_mnp). On a cmapplyconf, TKIT_DIR will contain the toolkit configuration file haoracle.conf on all configured nodes. Configure the other toolkit parameters for the ASM package as mentioned in section D for the ASM legacy package. - Apply the package configuration using # cmapplyconf -P asmpkg.
service_halt_timeout 300 If the listener is not configured, comment the second set of service parameters which are used to monitor the listener. Edit the dependency parameters as shown: dependency_name asm_dependency dependency_condition =up dependency_location same_node Since LVM logical volumes are used in disk groups, specify the name(s) of the volume groups on which the ASM diskgroups reside on, for the attribute "vg". Configure the ip_subnet and ip_address parameters.
E. Modification of a legacy database package using older version of Oracle ECMT scripts to use the scripts provided for ASM support A customer can migrate from an older ECMT version database package to use the Oracle ECMT scripts provided for ASM support. The new legacy or modular database package may or may not use ASM. 1. Migration to a legacy database package not using ASM: - Halt the database package. - Copy the framework scripts provided in the bundle to the package directory on one node.
- Run the ASM MNP. - Copy the framework scripts provided in the bundle to the database instance package directory. this Configure the configuration file haoracle.conf file as mentioned in document for the database package. Create a new package ASCII file and control script or edit the existing package ASCII and control scripts in the database package directory. Configure the parameters as mentioned for the database package. Copy the scripts from the package directory to all the configured nodes.
"modular1.ascii" now has values for certain attributes that were present in the older toolkit configuration file. Edit the value for TKIT_DIR (where the new toolkit configuration file should be generated or where the toolkit scripts are copied to in case of a configuration directory mode of operation). Leave the INSTANCE_TYPE to the default value "database". Leave the value of ASM to the default value "no". The new toolkit parameters can be left as they are by default.
the To add the values to this modular package configuration file, from older toolkit configuration file, issue the following command: # cmmakepkg -i modular.ascii -m ecmt/oracle/oracle -t modular1.ascii "modular1.ascii" now has values for certain attributes which were present in the older toolkit configuration file.
less memory. From Oracle's point of view, we must make sure that any packaged instance can run on any node that has been specified in the corresponding package configuration. The Oracle shell script handles this situation in the following way: If node-specific tuning is required, set up a node-specific 'init.ora' file for each node in ${ORACLE_HOME}/dbs. This file should be named 'init${SID_NAME}.ora', and there should be one such file for each host. EXAMPLE: /ORACLE_TEST0/dbs/initORACLE_TEST0.ora.
/etc/hosts would contain names of services (database servers) in addition to real physical hostnames. From Serviceguard's point of view, these are relocatable hostnames. EXAMPLE: Assume a Serviceguard package for the Oracle instance ORACLE_TEST0, is assigned the IP address 192.10.10.1 Add the following entry to your hosts files: 192.10.10.1 ORACLE_TEST0 The 'ORACLE_TEST0' instance should now be reachable using the name 'ORACLE_TEST0' regardless of the node on which it is running.
By default Local OS Authentication is enabled for Oracle 10g and 11g with default value of "LOCAL_OS_AUTHENTICATION_listener_name = ON". The absence of this parameter in LISTENER.ORA file implies the feature is enabled. In case if it has been disabled, it can be re-enabled by commenting or removing "LOCAL_OS_AUTHENTICATION_listener_name = OFF" from LISTENER.ORA or by setting it to "LOCAL_OS_AUTHENTICATION_listener_name = ON".
- Re-start the Oracle database it prior to maintenance instance again if you manually stopped $ cd /etc/cmcluster/pkg/ORACLE_TEST0/ $ $PWD/toolkit.sh start - Allow monitoring scripts to continue normally as shown below: $ rm -f /etc/cmcluster/pkg/ORACLE_TEST0/oracle.debug Two messages "Starting Oracle toolkit monitoring again after maintenance" and "Starting Oracle Listener monitoring again after maintenance" appear in the Serviceguard Package Control script log.
