Managing Serviceguard 12th Edition, March 2006

Understanding Serviceguard Software Components
Responses to Failures
Chapter 3128
Responses to Package and Service Failures
In the default case, the failure of a failover package or of a service within
a package causes the failover package to shut down by running the
control script with the 'stop' parameter, and then restarting the package
on an alternate node. A package will fail if it is configured to have a
dependency on another package, and the dependency package fails. If the
package manager receives a report of an EMS (Event Monitoring
Service) monitor event showing that a configured resource dependency is
not met, the package fails and tries to restart on the alternate node.
If you wish, you can modify this default behavior by specifying that the
node should crash (TOC) before the transfer takes place. (In a very few
cases, Serviceguard will attempt to reboot the system prior to a TOC
when this behavior is specified.) If there is enough time to flush the
buffers in the buffer cache, the reboot is successful, and a TOC does not
take place. Either way, the system will be guaranteed to come down
within a predetermined number of seconds.
In cases where package shutdown might hang, leaving the node in an
unknown state, the use of a Failfast option can provide a quick failover,
after which the node will be cleaned up on reboot. Remember, however,
that when the node crashes, all packages on the node are halted
abruptly.
The settings of node and service failfast parameters during package
configuration will determine the exact behavior of the package and the
node in the event of failure. The section on “Package Configuration
Parameters” in the “Planning” chapter contains details on how to choose
an appropriate failover behavior.
Service Restarts
You can allow a service to restart locally following a failure. To do this,
you indicate a number of restarts for each service in the package control
script. When a service starts, the variable RESTART_COUNT is set in the
service's environment. The service, as it executes, can examine this
variable to see whether it has been restarted after a failure, and if so, it
can take appropriate action such as cleanup.