Disaster recovery rehearsal in Continentalclusters

Precautions

This section describes the precautions the operator has to follow while performing DR rehearsals.

Client access IP address at recovery cluster

During a DR rehearsal, Continentalclusters will start the rehearsal package which could be configured

to bring up the application instance at the recovery cluster. On finding that the application instance

started at the recovery cluster, clients would presume that a recovery has occurred, and would attempt

to connect to it and perform production transactions. This can lead to split brain situation where one

set of clients are connected to the application instance at the primary cluster while second set of

clients are connection to the application instance at the recovery cluster (which was started for

rehearsal). Hence during rehearsal, it the operator’s responsibility to ensure that rehearsal production

clients are prevented from accessing the application instance at the recovery cluster and attempt

production transactions. One way to prevent split brain is to prevent application access to clients,

which can be done by modifying the client access IP address at the recovery cluster during rehearsal.

For example, when rehearsal package is configured for Oracle Single Instance, ensure that the

rehearsal package IP address is different from that of the recovery package.

Cleanup of secondary mirror copy

Once rehearsal is completed and before the recovery groups are moved out of maintenance mode,

the operator should ensure that the rehearsal changes on the secondary mirror copy are cleaned up.

Note that during rehearsal, the rehearsal application could have invalidated the secondary mirror

copy with non-production I/O. Hence, before moving the recovery group out of maintenance, the

operator has to clean up the secondary mirror copy by synchronizing it with primary mirror copy or

restoring from the BC/BCV (in case the primary cluster fails during rehearsal). If not, recovery (via

cmrecovercl) or recovery package startup via cmrunpkg/cmmodpkg would potentially start up the

recovery package on data invalidated by rehearsal.

Cluster role switch during rehearsal

Using the Continentalclusters commands “cmswitchconcl/cmapplyconcl”, the recovery cluster role

can be changed to be the new primary cluster. Operators are responsible for ensuring that the

recovery groups are not in maintenance mode before attempting to switch cluster roles. This is can

potentially allow primary packages from being started on disks invalidated by the rehearsal at the

new primary cluster.

Rehearsing recovery of Oracle 9i RAC recovery groups

To rehearse recovery of an Oracle 9i RAC recovery group, ensure that all recovery groups defined

for that Oracle 9i RAC are in maintenance mode. In Continentalclusters, for the Oracle 9i RAC

application, an individual package is used for each RAC instance. Hence one recovery group for

each Oracle RAC instance recovery is defined. Therefore it is possible for users to put only a partial

set of the recovery groups into maintenance mode while the remaining recovery groups are out of

maintenance mode. Under these circumstances, Continentalclusters will not prevent users from starting

rehearsal on recovery groups in maintenance mode and recovery on those recovery group that are

not in maintenance mode, which can impact data integrity. The operator is responsible for ensuring

that either all or none of the recovery groups are in maintenance mode.