Building Disaster Recovery Serviceguard Solutions Using Metrocluster with 3PAR Remote Copy for Linux B.01.00.
Legal Notices © Copyright 2013 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Introduction...............................................................................................5 Overview of HP 3PAR concepts..................................................................................................5 Remote Copy pairs .............................................................................................................5 Remote Copy volume groups ................................................................................................
A Checklist and worksheet for configuring Metrocluster with 3PAR Remote Copy for Linux.....................................................................................................39 Disaster recovery checklist.......................................................................................................39 Cluster configuration worksheet................................................................................................39 Package configuration worksheet............................
1 Introduction This document describes the procedure to configure data replication solutions using HP 3PAR storage systems to provide disaster recovery for Serviceguard clusters over long distances. This chapter describes the HP 3PAR Remote Copy software and the additional files that integrate the HP 3PAR storage system with Metrocluster. Overview of HP 3PAR concepts The 3PAR storage systems are configured for use in data replication from one 3PAR storage system unit to another.
synchronization is manually initiated. If between two synchronizations, an area of the volume is written to multiple times, only the last write needs to be synchronized with the other storage system. Remote Copy target definitions As part of the Remote Copy setup process, you must create target definitions on each Remote Copy system. The target definitions are descriptions that exist on one system to identify a Remote Copy system.
• Metrocluster uses 3PAR CLI to communicate to the storage array. Ensure that port 5783 is not blocked by firewall. • When you upgrade the HP 3PAR storage system to any of the following 3PAR OS versions, then it impacts the HP Metrocluster 3PAR package. ◦ 2.3.1 MU5 Patch35 ◦ 3.1.1 MU3 Patch27 ◦ 3.1.2 MU3 Patch16 After the upgrade to the patches is complete, a new self-signed 2048-bit RSA SSL Certificate is created on the HP 3PAR Array.
NOTE: The maximum number of CLI connections to a 3PAR storage array is 64. Overview of a Metrocluster configuration A Metrocluster is configured with the nodes at Site A and Site B. When Site A and Site B form a Metrocluster, a third location is required where Quorum Server or arbitrator nodes must be configured. There is a 3PAR storage system at each site and they are connected to each other through Remote Copy links. An application is deployed in a Metrocluster by configuring it at both the sites.
Overview of Metrocluster 3PAR Remote Copy volume group monitor In a Metrocluster environment it is necessary to actively monitor the Remote Copy volume group state. If this is not done, it becomes difficult to find out the time when the application data became remotely unprotected for an extended period of time. Under such circumstances, the Metrocluster 3PAR Remote Copy volume group monitor provides the capability to monitor the status of the Remote Copy volume group used in a package.
2 Configuring an application in a Metrocluster environment Installing the necessary software Before a Metrocluster can be configured, make sure the following software is installed on all nodes: • Serviceguard for Linux A.11.20.00 or later • HP 3PAR InForm OS CLI • Metrocluster with 3PAR Remote Copy for Linux Creating the cluster NOTE: The file /etc/cmcluster.conf contains the mappings that resolve symbolic references to $SGCONF, $SGROOT, $SGLBIN, etc.
NODE_NAME SJC_2 SITE san_jose ........ Use cmviewcl command to view the list of sites that are configured in the cluster and their associated nodes. The following is a sample of the command, and the output: # cmviewcl -l node SITE_NAME san_francisco NODE STATUS STATE SFO_1 up running SFO_2 up running ......... SITE_NAME san_jose NODE STATUS STATE SJC_1 up running SJC_2 up running You can configure either of these failover policies for Metrocluster failover packages.
NOTE: When a TPVV is configured as a primary volume in a Remote Copy volume group, no data should be written on the secondary volume before adding it to the Remote Copy volume group, or it must match the primary volume. This enables the primary and secondary volumes to match during initial synchronization. The Remote Copy volume group can be created either using HP 3PAR Management Console GUI or using HP 3PAR CLI .
where: 4. ◦ -snp_cpg is the name of the copy group from which the snapshot space is allocated.You can use MC_CPG_RAID5 CPG that was created in step2 or any other CPG to allocate the space for the snapshot. ◦ -usr_aw is the allocation warning alert limit for the user space specified in percentage. This generates an alert when the user space of the volume exceeds a specified percentage of the volume’s size.
5. a. Create a Remote Copy volume group on the primary system and assign the created volume to that group. On primary storage system: cli% creatercopygroup –domain : cli% admitrcopyvv : where: • is the name of the Remote Copy volume group to be created. • is the recovery storage system. • is the name of the virtual volume created in step 3.
where: • is the name of the domain to which the new user will belong. If you are using domains, specify the name of an existing domain in your system. Specify 'all' as the domain name if you are not using any domain. NOTE: For Metrocluster operations the user must have the “edit” privileges. HP strongly recommends you to create Remote Copy volume groups and user with “edit” privileges for the Metrocluster operations under a 3PAR storage system domain.
# pvcreate -f /dev/sda1 2. Create the volume group on the source volume. # vgcreate --addtag $(uname -n) /dev/ /dev/sda1 # vgcreate --addtag $(uname -n) /dev/ 3. Create the logical volume. (XXXX indicates size in MB). # lvcreate -L XXXX /dev/ 4. Create a file system on the logical volume. # mke2fs -j /dev//lvol1 5. If required, deactivate the volume groups on the primary system and remove the tag. # vgchange -a n # vgchange --deltag $(uname -n) 6.
Perform the following steps on a node in a Metrocluster to configure a modular package using the Metrocluster 3PAR module on the command line: 1. Run the following command to create a modular package configuration file using the Metrocluster with 3PAR Remote Copy for Linux module dts/mc3parrc. # cmmakepkg –m dts/mc3parrc pkgName.config In this command, dts/mc3parrc is the Metrocluster module that needs to be included to create a Metrocluster package configuration file.
f. Specify the Remote Copy volume group name configured on the HP 3PAR storage system that resides in DC1, containing the disks used by the application. dts/3parrc/DC1_RC_VOLUME_GROUP “mcrcgroup” g. Specify the Remote Copy volume group name configured on the HP 3PAR storage system that resides in DC2, containing the disks used by the application. dts/3parrc/DC2_RC_VOLUME_GROUP “mcrcgroup.r2835” h. Specify the user on the HP 3PAR storage system that resides in DC1.
NOTE: The RESYNC_WAIT_TIMEOUT value must not be greater than the value of run_script_timeout parameter specified in the package configuration. Adjust the run_script_timeout value as necessary to consider the RESYNC_WAIT_TIMEOUT value in addition to the time taken for the full execution of each function in the package startup. m. To allow package to start up with non-current data, set AUTO_NONCURDATA to 1. dts/3parrc/AUTO_NONCURDATA 1 The legal values for this parameter are “0” (default value) or “1”.
For the failover_policy parameter, in Metrocluster failover packages you can use any of the Serviceguard defined failover policies. The site_preferred and site_preferred_manual failover policies are introduced in Serviceguard specifically for Metrocluster configurations. The site_preferred value implies that when a Metrocluster package needs to fail over, it fails over to a node in the same site as the node it last ran.
4. If the product Metrocluster with 3PAR Remote Copy for Linux is installed, you will be prompted to configure a Metrocluster package. Select the dts/mc3parrc module, and then click Next. Figure 3 Selecting Metrocluster module 5. You will be prompted next to include any other toolkit modules. In case your application being configured requires a Serviceguard toolkit, select the appropriate toolkit, otherwise move to the next screen. Figure 4 Select a Toolkit 6.
Figure 5 Configuring Package Name 7. Optional: Select additional modules required by the application. For Example, if the application uses LVM volume groups, select the volume_group module. Click Next. Figure 6 Additional Modules 8. 22 Review the node order in which the package will start and modify other attributes, if needed. Click Next.
Figure 7 Modify Other Attributes 9. Configure the attributes for a Metrocluster package. Make sure that you fill all the mandatory attributes (marked with *) accurately. Figure 8 Configuring the attributes for a Metrocluster package 10. Enter the values for other modules selected in step 7. 11. After you enter the values for all the modules, review all the inputs given to the various attributes in the final screen.
Figure 9 Applying the configuration 24 Configuring an application in a Metrocluster environment
3 Metrocluster Features Cluster verification Starting HP Serviceguard version A.11.20.20, the cmcheckconf -v command validates the cluster and the package configuration. Metrocluster uses this functionality to ensure the sanity of Metrocluster package configuration. HP recommends that you set up a cron job to regularly run the cmcheckconf command. For more information on cmcheckconf command, see the cmcheckconf (1m) manpage.
Table 2 Validating Metrocluster package (continued) # cmcheckconf -p Verify the replication status of the Remote Copy volume group based on the package configuration file. cmcheckconf [–v] # cmcheckconf When the package is up and running, the cmcheckconf command verifies the replication status. If the replication is not happening, the validation script displays the following warning message. WARNING: The replication is not happening.
Table 2 Validating Metrocluster package (continued) ERROR: DC2_RC_VOLUME_GROUP ${RC3PAR_DC2_RC_VOLUME_GROUP} is not the name of remote copy volume group ${RC3PAR_DC1_RC_VOLUME_GROUP} on the storage system ${RC3PAR_DC2_STORAGE_SYSTEM_NAME}. Verify whether the Remote Copy target cmcheckconf [–v] names matches with the actual Remote # cmcheckconf Copy target of the replication group as mentioned in the package configuration file.
4 Understanding failover/failback scenarios Failover/failback scenarios in a Metrocluster package The section describes a couple of rolling disaster scenarios. In the first scenario, the link had gone down previously and is now up. The data from primary volume group is being synced with remote Remote Copy volume group. The package has failed in the primary site and is now trying to start at the recovery site.
Table 3 Replication modes and failover scenarios (continued) Local RCVG Role Remote RCVG Role Replication State/Link Status Replication Mode Metrocluster Parameters Metrocluster Action happening from the remote storage system to the local storage system. Start the Remote Copy volume group manually before restarting the package.
Table 3 Replication modes and failover scenarios (continued) Local RCVG Role Remote RCVG Role Replication State/Link Status Replication Mode Metrocluster Parameters Metrocluster Action Error: The Remote Copy link is down and data in the local storage system may not be current. The user has set AUTO_NONCURDATA to "0” and has not created the FORCEFLAG file. To start the package forcefully using non current data, use FORCEFLAG file. Resolution in the location specified in the DTS_PKG_DIR parameter.
5 Administering a Metrocluster with 3PAR Remote Copy for Linux Administering a cluster that uses Metrocluster 3PAR Remote Copy While the package is running, a manual storage failover on Remote Copy volume group outside of Metrocluster software can cause the package to halt due to unexpected condition of the 3PAR Remote Copy virtual volumes. HP recommends that no manual storage failover be performed while the package is running.
Restoring replication after a failover When the Metrocluster package fails over to the remote site and the links are not up or the primary storage system is not up, Metrocluster issues the setrcopygroup failover command. This command changes the role of the Remote Copy volume group on the storage system in the recovery site from Secondary to Primary-Rev. In this role, the data is not replicated from the recovery site to the primary site.
Limitations of the rolling upgrade for Metrocluster The rolling upgrade for Metrocluster has the following limitations: • The cluster or package configuration cannot be modified until the rolling upgrade is complete. If the configuration must be edited, upgrade all nodes to the new release, modify the configuration file, and then copy it to all nodes in the cluster. • New features of the latest version of Metrocluster cannot be used until all nodes are upgraded to the latest version.
6 Troubleshooting Troubleshooting Metrocluster To troubleshoot problems with Metrocluster with 3PAR Remote Copy for Linux, you must understand HP 3PAR Remote Copy environments. See the Remote Copy User Guide for more information on Remote Copy configuration and volume group states.
Table 4 Error Messages and their Resolution (continued) Log Messages Cause file. The package is not allowed to start up. To start the package forcefully using non current data, use FORCEFLAG file. The Remote Copy volume group is in "Syncing" state and RESYNC_WAIT_TIMEOUT parameter is set to 0. The package is not allowed to start up. Resolution latest data in the local storage system. Restart the package.
Table 4 Error Messages and their Resolution (continued) Log Messages Cause Resolution storage system for the corresponding user. • The CLI connections are exhausted. For more information see “Managing CLI connections to 3PAR array” (page 38). 36 Not able to determine the status of the remote storage system. This might be because of CLI connectivity issues or because the remote storage system is down. The role of local Remote Copy volume group's is “Primary”.
Table 4 Error Messages and their Resolution (continued) Log Messages Cause Resolution Starting of the Remote Copy volume group The startrcopygroup command failed. Start the replication group using either the startrcopygroup command or the 3PAR Management Console. Restart the package. [] has failed to complete. This means that Remote Copy is not functioning between the primary and secondary volume groups.
Table 4 Error Messages and their Resolution (continued) Log Messages Cause Resolution NOTE: The password file must have the following format: <3parArrayUserName>_<3parArrayName>.pwf Managing CLI connections to 3PAR array The maximum number of CLI connections to a 3PAR storage array is 64. Metrocluster configuration or package startup operations uses CLI to connect and to get information about the remote copy groups from the storage array.
A Checklist and worksheet for configuring Metrocluster with 3PAR Remote Copy for Linux Disaster recovery checklist Use this checklist to make sure you have adhered to the disaster tolerant architecture guidelines for two main data centers and a third location configuration. Data centers A and B have the same number of nodes to maintain quorum in case an entire data center fails. Arbitrary nodes or Quorum Server nodes are located in a separate location from either of the primary data centers (A or B).
Member Timeout: _________________________________________________________ Network Polling Interval: _______________________________________________ AutoStart Delay: ________________________________________________________ Package configuration worksheet Use this package configuration worksheet either in place of, or in addition to the worksheet provided in the latest version of the Managing HP Serviceguard A.11.20.10 for Linux manual available at http://www.hp.com/go/linux-serviceguard-docs .
DC1 DC2 DC2 DC2 DC2 RC Target for DC2: ___________________________________________________ RC Volume Group: _____________________________________________________ Storage System User: _________________________________________________ Nodes List: __________________________________________________________ RC Target for DC1: ___________________________________________________ Package configuration worksheet 41
Glossary A—C 3PAR Remote Copy The 3PAR storage systems are configured for use in data replication from one 3PAR storage system unit to another. This type of physical data replication is a part of the Metrocluster with 3PAR Remote Copy for Linux. arbitrator Nodes in a disaster tolerant architecture that act as tie-breakers in case all of the nodes in a data center go down at the same time. These nodes are full members of the Serviceguard cluster and must conform to the minimum requirements.
LUN (Logical Unit Number) A SCSI term that refers to a logical disk device composed of one or more physical disk mechanisms, typically configured into a RAID level. M, N manual failover Failover requiring human intervention to start an application or service on another node. Metrocluster A Hewlett-Packard product that allows a customer to configure an Serviceguard cluster as a disaster recovery metropolitan cluster.
Index Symbols M 3PAR storage systems Remote Copy pair, 5 Metrocluster 3PAR Remote Copy, 5 configuration, 8 requirements, 6 modules, 22 A arrays cabled, 11 asynchronous, 5 B bidirectional configuration, 6 C P parameters, 30 password HP 3PAR storage systems, 15 privileges configure, 15 Cluster verification, 25 cluster multiple sites, 10 cmviewcl command, 11 command, 12 Common Provisioning Group (CPG) Remote Copy, 5 Virtual domains, 6 configuration environment, 8 Console, 14 console GUI, 12 R D targ
