HP Serviceguard Developer’s Toolbox Version B.02.00.
© Copyright 2012 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Overview..................................................................................................4 Advantages.............................................................................................................................4 Dependencies..........................................................................................................................4 Prerequisites...........................................................................................................
1 Overview HP Serviceguard is a specialized software package for protecting mission-critical applications from a wide variety of hardware and software failures. HP Serviceguard monitors the health of each node in the cluster and rapidly responds to failures in a way that minimizes application downtime. HP provides Serviceguard Toolkits and Toolbox for integration of popular applications with HP Serviceguard.
Downloading the Toolbox HP partners and ISVs, can access Toolbox from www.hp.com/dspp—> Resources & Downloads —> Technologies & Tools —> High Availability -> Software/Products —> HP Serviceguard Developer's Toolbox. This Toolbox is also available as a free download from software depot at http:// h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=SGDTOOLBOX. Contact ha-toolkits@hp.com for questions or feedback about Toolbox.
2 Installing Serviceguard developer's Toolbox This section describes the procedure to install Serviceguard developer's Toolbox. 1. Untar and copy all the files into a directory. NOTE: The Toolbox contains two files in a tar file. Ensure that all the files are copied into the same directory. 2. Run setup.sh, and when prompted choose appropriate selection to install or uninstall. The following files are installed on the successful completion of the setup: • • tkit_module.
The Package configuration file contains the information necessary to start the application processes in the package, monitor the packages during operation, react to a failure, and halt the package when necessary. For more information about package configuration and management in HP-UX and Linux, see the latest version of Managing Serviceguard user guide at http://www.hp.com/go/ hpux-serviceguard-docs —> HP Serviceguard and http://www.hp.com/go/linux-serviceguard-docs respectively.
3 High Availability (HA) Toolkits HA Toolkits are a set of scripts that integrate popular applications with HP Serviceguard. Toolkits are available for both HP-UX and Linux platforms. ECMT is a bundle of Toolkits for the following popular applications on HP-UX: • Database Toolkits (Oracle's single instance database application including Oracle 9i, Oracle 10g R1, Oracle 10g R2 with ASM, Oracle 11gR1 and 11gR2, Postgre SQL, and MySQL) • Internet Toolkits (including HP Apache, HP Tomcat).
When the monitor function detects an application failure, it returns fail status that is a non-zero number to HP Serviceguard, thereby enabling it to take appropriate action (failover, restart and so on) based on the package configuration. • Attribute Definition File (toolbox.1) The cmmakepkg command uses the toolbox.1 file to include the application specific configuration variables into the package configuration file.
When the tkit_module.sh is called with the start argument by the Serviceguard master control script, it executes the application start script. The application start script is written by the user for the application intended to be integrated with Serviceguard. The application start script must return a zero if the application starts successfully, and a non-zero value if the application fails to start. When the tkit_module.sh is called with stop argument, it executes the application stop script.
package, so that package resources (including volume groups, relocatable IPs, and so on.) are intact and available. To continue monitoring by exiting the maintenance mode, remove the toolbox.debug file. Ensure that the application instance is running properly before exiting the maintenance mode, because the monitoring of the instance resumes after the package exits the maintenance mode. The default setting for this variable in the Toolkit template is yes.
The variable TKIT_MONITORING_ELEMENT specifies the actual object being monitored for the selected monitoring type. For example, if you choose to monitor the PID file of an application (/var/run/app/app.pid), then the configuration would be as follows: TKIT_MONITORING_TYPE pid_file TKIT_MONITORING_ELEMENT /var/run/app/app.
is stored in the PID file. That is, if the application for which this Toolkit is developed is MySQL, then APP_SERVER_PROC mysqld. • Toolkit Module Script (tkit_module.sh) The Toolkit module script contains the following functions that support validation, starting, stopping, and monitoring of application services: ◦ validate_function: validates parameters in the package environment. ◦ start_function: invokes the TKIT_APP_START script (to be developed by user) to start the application.
To use the test tool, untar the SG-toolbox-testtool.tar in to Toolkit configuration directory. The tool consists of two scripts named choapp.conf and choapp.sh. Configure the choapp.conf file as below: Variable Name Description CL_NAME The variable takes in the cluster name on which the test is being run as the value. For example, CL_NAME=TbxCluster PKG_DIR The variable takes in the absolute path of the Toolkit configuration directory.
A MySQL Toolkit scripts MySQL Toolkit is used to bring MySQL Database Server into HP Serviceguard environment. Package Configuration File (pkg.
# # # # # # "package_type" is the type of package. The package_type attribute specifies the behavior for this package. Legal values and their meanings are: failover package runs on one node at a time and if a failure # occurs it can switch to an alternate node. # # multi_node package runs on multiple nodes at the same time and # can be independently started and halted on # individual nodes.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # must have "node_fail_fast_enabled" set to "yes". Legal values for node_fail_fast_enabled: yes, no. node_fail_fast_enabled no "run_script_timeout" is the number of seconds allowed for the package to start. "halt_script_timeout" is the number of seconds allowed for the package to halt. If the start or halt function has not completed in the specified number of seconds, the function will be terminated. The default is "no_timeout".
# # Another policy is "min_package_node". This policy means # Serviceguard will select from the list of "node_name" entries the # node, which is running fewest packages when this package needs to # start. # # # Legal values for failover_policy: configured_node, min_package_node, site_preferred. failover_policy configured_node # "failback_policy" is the action to take when a package is not running # on its primary node.
# # # # # # monitoring the application. Note: if Maintenance flag is set to "no" then the above feature would not be available. Legal values for tkit/toolbox/toolbox/TKIT_MAINTENANCE_FLAG: yes, no. tkit/toolbox/toolbox/TKIT_MAINTENANCE_FLAG yes # # The interval (in seconds) between checking if application is are up and running. # The default setting is 5 seconds.
# # # # # # # # contains only alphanumeric characters, dot(.), dash(-), or underscore(_) in between. Maximum string length is 39 characters. Legal Legal Legal Legal values values values values for for for for service_cmd: service_restart: none, unlimited, (value > 0). service_fail_fast_enabled: yes, no. service_halt_timeout: (value >= 0).
# # # # # # # # # # # # # # # # # # # # # # # the same node. "all_nodes" means the package depended on must be down on all nodes in the cluster. NOTE: Within a package, you cannot specifiy more than one dependency on the same package. For example, pkg1 cannot have one same_node and one any_node dependency on pkg2.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 22 configuration file takes precedence. This allows you to set an overall default, but to override it for a particular package.
# # # # # # # # # # # # # # # # but not all. monitored_subnet 192.10.26.0 no monitored_subnet_access entry, hence this subnet is available on all nodes of the cluster. monitored_subnet 2001::/64 monitored_subnet_access full 2001::/64 is available on all nodes of the cluster. Legal values for monitored_subnet_access: partial, full. monitored_subnet 192.168.100.0 monitored_subnet_access full # IP subnets and addresses # # "ip_subnet" and "ip_address" specify subnets and # IP addresses used by this package.
# # # # # # # # # # # Volume Group Activation "vgchange_cmd" is the method of activation for LVM volume groups. Specify the method of activation for volume groups. Leave the default ("vgchange_cmd "vgchange -a y") if you want volume groups activated in default mode. "vgchange_cmd" replaces the legacy package control script parameter "vgchange". Legal values for vgchange_cmd: vgchange_cmd "vgchange -a y" # Volume Groups # # "vg" is used to specify which volume groups are used by this package.
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # Specify the number of unmount retries for each filesystem during package shutdown. The default is set to 1. fs_umount_retry_count must be set to 1 (default), if the underlying file system is of type Red Hat GFS. "fs_umount_retry_count" replaces the legacy package control script parameter "fs_umount_count".
# Legal values for fs_umount_opt: # Legal values for fs_fsck_opt: fs_name /dev/mysql_vg/lvol0 fs_directory /mysql_mnt fs_type “ext2” fs_mount_opt "-o rw" fs_umount_opt "" fs_fsck_opt "" # "pev_ defines an environment variable for the package. # # package environment variable # # You can define one or more variables which will be set as environment # variables for scripts identified by external script and external pre # script to use.
# | # STEP SG_SCRIPT_ACTION | STEP SG_SCRIPT_ACTION # | # START | START # VOLUME_GROUPS start | SERVICES stop # FILESYSTEMS start | IP_ADDRESSES stop # IP_ADDRESSES start | FILESYSTEMS stop # SERVICES start | VOLUME_GROUPS stop # END | END # # When a package includes external_pre scripts, the steps are executed # in the following order (for both validation and execution): # # START # /etc/cmcluster/packages/pkg1a/task0 start # /etc/cmcluster/packages/pkg1a/task1 start # /etc/cmcluster/packages/pkg1a/task2 s
# VOLUME_GROUPS # FILESYSTEMS # IP_ADDRESSES # /etc/cmcluster/packages/pkg1a/task0 start # /etc/cmcluster/packages/pkg1a/task1 start # /etc/cmcluster/packages/pkg1a/task2 start # SERVICES # DEFERRED_RESOURCES # END # # When the package halts, the steps are executed in the following order: # # START # DEFERRED_RESOURCES # SERVICES # /etc/cmcluster/packages/pkg1a/task2 stop # /etc/cmcluster/packages/pkg1a/task1 stop # /etc/cmcluster/packages/pkg1a/task0 stop # IP_ADDRESSES # FILESYSTEMS # VOLUME_GROUPS # END
This script is used by the Serviceguard master control script to start and stop packages that are created using the Toolbox. You can modify the monitor function, if required. ########################### # Source utility functions. ########################### . /etc/cmcluster.conf if [[ -z $SG_UTILS ]]; then SG_UTILS=$SGCONF/scripts/mscripts/utils.sh fi if [[ -f ${SG_UTILS} ]]; then .
;; esac } ################################################################## # # VALIDATE NOT EMPTY FUNCTION # ################################################################# function validate_not_empty { if [[ $# < 2 ]]; then sg_log 0 "$2 value" sg_log 0 "Usage : notempty VAR VALUE" sg_log 0 "ERROR: $1 is not defined" check_return 1 5 else if [ -z $2 ]; then sg_log 0 "ERROR: $1 is not defined" check_return 1 5 fi fi } ############################################### # The function which takes care of retu
echo "$(date '+%b %e %T') - Node \"$(hostname)\": Starting the monitoring process" while :; do # Maintenance code. This script will check whether the debug file # exists and only then Package gets into maintenance mode. Monitoring # is stopped completely at this moment.
else continue; fi fi done return 0; } ############################################################# # # MONITORING_PROCESS # ############################################################# function process_monitor { for k in ${APP_TKIT_MONITORING_ELEMENT[@]} do result=`UNIX95=yes ps -eo args | grep -i $k | grep -v grep` if [ -z "$result" ] ; then $ECHO "ERROR: The process $k is not running. Halting the monitor script.
exit $exit_code Application Start Script, (start_app.sh): This script must be written by the user to start the application that is being packaged using the Toolbox. The sample script given here is used to start MySQL server. #!/bin/sh # ############################################### # This script starts up the application # CONFIGURATION_FILE_PATH="/mysql/my.cnf" PID_FILE="/var/run/mysqld/mysqld.
then kill -s SIGTERM $pid return_value=$? if [[ $return_value != 0 ]]; then echo "ERROR: stop_app failed " return 1 fi else echo "$(date '+%b %e %T') - ERROR: stop_app failed: The $PID_FILE was corrupted.
Glossary ADF – Attribute Definition File CHO – Continuous Hours of Operation ECMT – Enterprise Cluster Master Toolkit HA – High Availability OS – Operating System PID – Process id RHEL – Red Hat Enterprise Linux SG – Serviceguard SLES – SUSE Enterprise Linux 35
Index H HA Toolkits, 8 HP Serviceguard HP Serviceguard Toolbox, 4 HP Serviceguard Toolkits, 4 T Toolbox Download, 5 Installation, 6 Toolkits Configuration, 10 Framework, 8 Integration, 6 Sample Scripts, 15 36 Index
