HP 3PAR Cluster Extension Software Administrator Guide Abstract This guide contains detailed instructions for configuring and troubleshooting HP 3PAR Cluster Extension Software in a Windows environment. The intended audience has independent knowledge of related software and of the HP 3PAR storage system and its software.
© Copyright 2011 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 HP 3PAR Cluster Extension features...............................................................6 Integration into cluster software (Graphical user interface)..............................................................6 CLI for easy integration..............................................................................................................6 Disaster tolerance through geographical dispersion.......................................................................
Configuring HP 3PAR Cluster Extension resources........................................................................21 Setting Microsoft cluster-specific resource and service or application properties..........................22 Setting HP 3PAR Cluster Extension-specific resource properties.................................................25 Setting HP 3PAR Cluster Extension resource properties using the GUI........................................
Restrictions for customized implementations................................................................................51 Pre-execution and post-execution programs................................................................................51 Pre-execution return codes...................................................................................................52 Post-execution return codes..................................................................................................
1 HP 3PAR Cluster Extension features HP 3PAR Cluster Extension offers protection against application downtime due to a fault, failure, or site disaster by extending a local cluster between data centers over metropolitan distance. HP 3PAR Cluster Extension reinstates critical applications at a remote site within minutes of an adverse event, integrating your open-system clustering software and HP 3PAR Remote Copy to automate failover and failback between sites.
Figure 1 Physical replication using HP 3PAR Remote Copy HP 3PAR Remote Copy-mirrored disks have a read/write-enabled primary (local) disk and a read-only secondary (remote) disk. Current cluster software products cannot distinguish between read-only and write-enabled disks, and cannot enable disk access if the disk is not write-enabled during the server boot process.
No server reboot Disks on the server on both the primary and secondary sites are recognized during the initial system boot in a HP 3PAR Cluster Extension environment; therefore LUN presentation and LUN mapping changes are not necessary during failover or failback - for a truly hand-free disaster tolerant solution. No single point of failure Supreme redundancy — Identical configuration of established SAN infrastructure redundancies are implemented on a remote site.
(secondary) copy of the application disk set are not allowed from the same host. Configure your servers only in the zone that allows access to the local HP 3PAR storage system. • HP recommends a minimum of two cluster nodes per site. This allows for preferred local failover in case of server system failure. Local failover operations are faster than remote failover between HP 3PAR storage systems because you do not need to change the mirroring direction of the Remote Copy volume group members.
Figure 3 N-to-1 and 1-to-N configuration Processes and components For Windows Clustering, HP 3PAR Cluster Extension is shipped as a resource DLL. The installation of HP 3PAR Cluster Extension also includes a CLI for custom HP 3PAR Remote Copy integration. Customized solutions for failover application services must implement HP 3PAR Cluster Extension through the CLX CLI (clx3PARrun.exe) prior to the disk activation procedure on the server.
NOTE: HP 3PAR Remote Copy requires 3PAR Remote Copy licenses for all storage systems participating in a Remote Copy replication relationship. See the 3PAR InForm OS Concepts Guide for additional information on licensing and features. Figure 4 Relationship of primary versus secondary volume groups on primary and backup (secondary) storage system Remote Copy pairs Remote Copy configurations are based on the relationship between a storage system pair, also known as the Remote Copy pair.
Synchronous replication Using synchronous mode, all write requests from the server are first transferred to the remote storage system. After each I/O has been mirrored in the cache area of the remote storage system, it is acknowledged to the local storage system. The write request is then acknowledged to the server. User configuration file HP 3PAR Cluster Extension provides a user configuration file to customize failover behavior.
multiple disks across the cluster. The MNS resource ensures that the cluster's configuration data is stored locally on the nodes and that it is kept consistent across the different disks. This allows cluster topologies where the nodes do not need shared access to a quorum disk. (Number of nodes in the cluster/2) +1 This ensures that more than 50% of the nodes will have an up-to-date copy of the configuration information.
NOTE: File Share Witness support: • Windows Server 2003 (Service Pack 1 and later) supports only two node clusters when using File Share Witness. • Windows Server 2003 Service Pack 2 supports MNS with File Share Witness by default. • When using Microsoft Windows Server 2003 Service Pack 1 (SP1)-based server cluster or Windows Server 2003 R2-based server cluster, you must apply a Microsoft patch to support MNS with File Share Witness. See your Microsoft documentation for more information.
2 Configuring HP 3PAR Cluster Extension for Windows After installing HP 3PAR Cluster Extension, use the configuration tool to define the HP 3PAR Cluster Extension setup configuration. Use Cluster Administrator (Windows Server 2003), Failover Cluster Management (Windows Server 2008/2008 R2), or cluster commands in the CLI to add and configure resources. Integrating HP 3PAR Cluster Extension with Windows Clustering To integrate HP 3PAR Cluster Extension with Windows Clustering: 1.
Defining the HP 3PAR configuration information using the GUI To define configuration information using the GUI: 1. Open the configuration tool. For instructions, see “Starting the HP 3PAR Configuration Tool” (page 15). 2. 16 Click Add in the Array Configuration section to open the Add Array dialog box.
3. 4. 5. Enter the Array Network Name. Click Browse to select the password file. Click Connect to connect to the array. The array details (Array Name, Array Serial Number, Array Model, and Array ID) and the array target information appears on the Add Array dialog box. NOTE: If the Remote Copy link is down between the array and the defined targets, Cluster Extension will not be able to obtain the target array information. The value in the target array ID will be NA and a warning message is displayed.
NOTE: You must create or copy the array password file on all cluster nodes used by Cluster Extension. Ensure that the appropriate path is reflected in the configuration information of each node. During configuration, you will be prompted to confirm if you want to copy the configuration file on all cluster nodes. If you choose to copy the HP 3PAR Cluster Extension configuration file automatically, the same password file location is retained on all cluster nodes.
Exporting configuration settings using the GUI 1. Open the configuration tool. For instructions, see “Starting the HP 3PAR Configuration Tool” (page 15). 2. 3. 4. Click Export. When prompted, select a location, enter a file name, and then click Save. Click OK to save and close the configuration tool. Exporting configuration settings using the CLI 1. 2. Open a command window. Enter CLX3PARCONFIG EXPORT /FILE=filepath, where filepath specifies the location and file name to save the file.
CAUTION: Do not use the following characters in Cluster Extension resource names: \ / : * ? " < > |. Using these characters might affect the creation of the resourcename.online file, which is used for the HP 3PAR Cluster Extension resource health check mechanism. Adding a resource using the Cluster Administrator GUI (Windows Server 2003) Use the procedure in this section to add a resource using the Cluster Administrator GUI.
1. 2. Create two (or more) Remote Copy volume groups and create a resource for each Remote Copy volume group. Make every physical disk resource in the cluster service or application depend on both (all) resources within the cluster service or application. This ensures that any disk in the cluster service or application can go online only if both (all) Remote Copy volume groups are failed over correctly.
• Windows Server 2008/2008 R2 only: If a HP 3PAR Cluster Extension resource is not configured, the resource icon in the Failover Cluster Management GUI shows the message “not configured” next to the resource status. • The HP 3PAR Cluster Extension resource must be the first resource for all other disk resources. • The resource's PendingTimeout value must be equal to or greater than the value specified for the ResyncWaitTimeout attribute.
Table 1 Setting resource properties and values in the GUI Property Format Description Value Thorough Resource Health Check Interval (Windows Server 2008/2008 R2) Integer Used to poll “Alive” state for the resource. Also used by HP 3PAR Cluster Extension to perform HP 3PAR Remote Copy start operations if auto_recover policy is set for a particular Remote Copy volume group.Decreasing this value allows faster resource failure detection but also consumes more system resources.
Table 1 Setting resource properties and values in the GUI (continued) Property Format Description Value If a resource fails, attempt restart on current node – Period for restarts (Windows Server 2008/2008 R2) Integer Determines the amount of time for restart.
Setting HP 3PAR Cluster Extension-specific resource properties Changes to resource properties take effect when the resource is brought online again.
4. Make the necessary parameter changes, and then click OK. Using the resource configuration tool to set resource properties (Server Core and Hyper-V Server) To set resource properties for Server Core or Hyper-V Server, use the HP 3PAR Cluster Extension resource configuration tool. When using the resource configuration tool: • You must run the tool on a Server Core or Hyper-V cluster node. You cannot run the tool on a remote management station.
Configuring cluster node data center assignments To configure cluster node data center assignments from the cluster software Parameters tab or the resource configuration tool: 1. To add a cluster node: a. Click Add to open the Add Server Data center Assignment window. b. c. d. 2. 3. 4. Select a cluster node from the Pre-Configured host list. Assign the cluster node to a data center selected from the Data center list. Click OK to save and close the window.
b. c. d. 2. 3. 4. Select a storage system from the Storage System Name list. Assign the storage system to a data center selected from the Data center list. Click OK to save and close the window. To remove a storage system, select the storage system, and then click Remove. To modify a storage system, select the storage system, and then click Modify. Click Apply to finish your modifications and keep the window open, or click OK to save your changes and then close the Properties window.
Use this dialog box to configure the Remote Copy volume group behavior. 2. 3. • The settings in the Replication link up scenario section affect the failover behavior when the HP 3PAR Remote Copy link is up. • The settings in the Replication link down scenario section affect the failover behavior when the HP 3PAR Remote Copy link is down. Select RESYNCWAIT or FASTFAILBACK for the Application Startup Behavior. If you selected RESYNCWAIT in Step 2, set the RESYNCWAIT timeout value. • 4. 5. 6. 7.
2. 3. Update the PreExecScript, PostExecScript, and PostExecCheck values as needed, and then click OK. Click Apply to finish your modifications and leave the Properties window open, or OK to save your changes and close the window. Setting HP 3PAR Cluster Extension resource properties using the MMC If you are using Server Core or Hyper-V Server, you can manage a cluster remotely by using the MMC to run Failover Cluster Management.
Setting HP 3PAR Cluster Extension properties using a UCF You can use a UCF to configure certain HP 3PAR Cluster Extension properties for Windows. Properties that you can configure in a UCF include: • LogLevel • ClusterNotifyCheckTime • ClusterNotifyWaitTime • StatusRefreshInterval • HP3PARCLICommandTimeout IMPORTANT: If you plan to use the default values for these properties, no UCF is required. To configure properties using a UCF: 1. 2. 3. Take the HP 3PAR Cluster Extension resource offline.
5. Click OK to save your changes. Adding dependencies using Failover Cluster Management (Windows Server 2008/2008 R2) You can add dependencies with the GUI on a local node or by using the MMC to run the Failover Cluster Management application. 1. 2. 3. 4. 5. Open Failover Cluster Management. Select a service or application that has HP 3PAR Cluster Extension resource. Double-click a disk in the summary pane. Click the Dependencies tab, and then click Insert.
To add a dependency on a HP 3PAR Cluster Extension resource using the CLI, use the following command: cluster resource physical_disk_resource /adddependency:Cluster_Extension_3PAR_resource The following command adds a dependency on the HP 3PAR Cluster Extension clx_fileshare resource to the physical disk resource Disk_32b_00b: cluster resource Disk_32b_00b /adddependency:clx_fileshare Disaster-tolerant configuration example using a file share The following example assumes that your environment includes: •
Figure 6 Resource tree for service or application CLX_SHARE HP 3PAR Cluster Extension is configured as a single resource to enable read/write access to the physical disk resource used for the CLX_SHARE cluster group. The physical disk resource depends on the HP 3PAR Cluster Extension resource and can be brought online only when the HP 3PAR Cluster Extension resource is already online.
10. For Windows Server 2008/2008 R2 only: a. Click the Policies tab of the Failover Cluster Management Properties window. b. For the HP 3PAR Cluster Extension resource property response to resource failure, select If resource fails, do not restart. 11. Bring the service or application online on host3_DCB, and verify that the resource and all other CLX_SHARE service or application resources are brought online. 12. Take the service or application offline, and verify that all resources are stopped. 13.
Deleting a resource Deleting a running resource causes the resource and its dependents to go offline. CAUTION: Deleting a running resource does not remove the resource_name.online file. When deleting resources: • For Windows Server 2008/2008 R2, use the GUI or CLI. • For Server Core or Hyper-V Server, use the CLI or the MMC. • For Windows Server 2003, use the GUI or CLI. For more information on deleting resources, see your Microsoft documentation.
• When HP 3PAR Cluster Extension communicates with the HP 3PAR storage system, online operations could time out depending on the settings of the command time out configuration. • When a virtual volume resynchronization/ full synchronization is in progress, the software waits until the Remote Copy volume group is in a synced state if the ApplicationStartup resource property is set to RESYNCWAIT.
IMPORTANT: If there is any change in the configured IP address of the array or password file location and content, Cluster Extension will fail to communicate with the array and will not perform the required actions. At any point of time, you can launch the HP 3PAR Configuration tool and use the Connection Test option to verify connectivity and array credentials, and take necessary action.
An alternate method is to manually move all resources to another node in the cluster before stopping the cluster service. After all resources are successfully moved to another node, stop and then restart the cluster service; then, manually move back all resources. Logs If the HP 3PAR Cluster Extension log files need to be cleared and reset (for example, to reduce disk space usage), you can delete the files. HP 3PAR Cluster Extension automatically creates new log files.
3 User configuration file and HP 3PAR Cluster Extension objects Objects (also called properties in this document) define the HP 3PAR storage system environment and failover/failback behavior. You can customize HP 3PAR Cluster Extension objects in the user configuration file or in the cluster software. User configuration file For the CLI implementation of HP 3PAR Cluster Extension, the user configuration file UCF.cfg is used to configure application service-specific information. UCF.
Specifying object values You do not need to change the default settings unless you want to change the degree of protection for your Remote Copy volume group members. If you change an object, you may need to change additional objects as well. For example, if RESYNCWAIT has been specified for the ApplicationStartup object, the ResyncWaitTimeout object must be specified. Objects are supported according to the requirements or capabilities of the cluster software, as indicated in Table 3 (page 41).
COMMON Format tag Description Distinguishes between general (common) and application-specific objects. LogDir Format String Description (Optional) Defines the path to the HP 3PAR Cluster Extension log file. Default value Windows %ProgramFiles%\Hewlett-Packard\Cluster Extension 3PAR\ log LogLevel Format String Description (Optional) Defines the logging level used by HP 3PAR Cluster Extension. Valid values error (default): Logs only error messages for events that are unrecoverable.
Description Specifies the directory where HP 3PAR Cluster Extension searches for application-specific files, such as the force flag or online file. If ApplicationDir is set to a nonexistent drive, HP 3PAR Cluster Extension is unable to create the online file and cannot bring the resource online. Windows If ApplicationDir is not set, HP 3PAR Cluster Extension uses the local %HPCLX_3PAR_PATH% values as defined in the registry. Default values Windows Files resource_name.
systems for the service or application, the service or application will restart on configured nodes until one of the following conditions is met: • The resource is brought online in the remote data center. • The resource failed because the FailoverThreshold value has been reached. • The resource failed because the FailoverPeriod timeout value has been reached. CAUTION: Disable subsequent automated failover procedures for recovery failback operations.
ClusterNotifyWaitTime Format Integer Description Specifies the amount of time that HP 3PAR Cluster Extension will monitor for VM live migration state changes. Default value 5 seconds DC_A_Hosts (Required) Format List Description This space-separated list defines the cluster nodes in data center A. DC_B_Hosts (Required) Format List Description This space-separated list defines the cluster nodes in data center B.
Valid values YES NO (default) PostExecScript (Optional) Format String Description Specifies an executable with its full path name to be invoked after the failover action or failover procedure. You can include environment variables in path names. Do not specify arguments to be passed to the executable. PreExecScript (Optional) Format String Description Specifies an executable with its full path name to be invoked before the failover action or failover procedure.
Valid values NO YES(default) HP3PARCLICommandTimeout Format Integer Description Specifies timeout value (in seconds) for HP 3PAR storage system command execution. The timeout value is used whenever HP 3PAR Cluster Extension communicates with the HP 3PAR storage system. If the storage system is not responding within the configured timeout value, the command will time out and returns an error to the application. Default values 90 seconds Sample configuration file The following is a sample UCF.
4 CLI commands and utilities Using its CLI and command-line utilities, HP 3PAR Cluster Extension can be integrated with almost any cluster software for Windows operating systems.
Related information • clx3PARrun expects only the app_name parameter, which uniquely identifies an application service in the APPLICATION section of the user configuration file. • To force a Remote Copy volume group failover, use the –forceflag option. CAUTION: The forceflag option is an emergency switch used to manually activate your virtual volume.
Installing HP 3PAR InForm Command Line Interface (CLI) HP 3PAR Cluster Extension communicates with the HP 3PAR storage system using the HP 3PAR InForm Command Line Interface (CLI). Ensure that you install the supported version of HP 3PAR Command Line Interface (CLI) on a host where Clx3PARrun can be used. Creating and configuring the user configuration file The CLI expects as an argument the name configured as the APPLICATION tag value.
clx3PARrun, use the HP 3PAR Management Console GUI or CLI to check the progress of syncing. • Depending on the amount of data that is out of sync between the source and destination virtual volumes while the Remote Copy link(s) is unavailable, a resynchronization or a complete copy operation is initiated when the Remote Copy link(s) is re-established. The amount of delta data to be transferred between the source and destination virtual volume could be large enough for the copy process to take some time.
13. 14. 15. 16. 17. Remote Copy volume group status Remote Copy volume group virtual volume status AutoRecover Application directory path (ApplicationDir object) Log file location (LogDir object) Pre-executables and post-executables must supply a return code. The return code is used to determine whether a failover function is called. Pre-execution return codes Pre-execution return code Text Description 0 PRE_OK_TAKEOVER Pre-executable OK and failover action allowed.
5 Troubleshooting To troubleshoot problems with HP 3PAR Cluster Extension, one must understand HP 3PAR Remote Copy environments. See the Remote Copy User's Guide for more information on Remote Copy configuration and volume group states. See the HP 3PAR Cluster Extension log for any cluster specific information. However, it always keeps its own log files in its default log location.
Sample log file Line 1: The process begins for a Cluster Extension resource CLX03 using HP 3PAR Cluster Extension version number 1.00.00. Line 2: Cluster Extension internal configuration file version. Line 3 - 5: The warning message for a temporary license. Line 6: The host name where a Cluster Extension resource is being brought online in the data center ( A or B ) Line 7: The remote data center nodes status. Line 8-28 : Configuration details of the Cluster Extension resource before failover.
NOTE: Match the 3PAR storage system time with the cluster node time so that tasks can be easily mapped to Cluster Extension operations. For detailed log messages from HP 3PAR storage system, do the following: 1. Login with the 3parsvc user using HP 3PAR CLI. 2. Run showsys. This will indicate the master node on that system. 3. On the CLI shell, enter copyfile :/var/log/tpd/sysmgr /tmp/xxxsysmgr. If master node ID is 1, then copyfile 1:/var/log/tpd/sysmgr /tmp/sysmgr 4.
processes. (for example the cluster service) have not recognized the %HPCLX_3PAR_PATH% variable. To fix this problem, restart the cluster service. • If the required components of HP 3PAR Cluster Extension are not available, a file (clxlog.log ) is generated that can be used for troubleshooting. The file is usually in the working directory of the operating system (%SYSTEMROOT%).
Resource disk failure when HP 3PAR Cluster Extension is online In a cluster resource group, if the Cluster Extension resource is online but dependent disk resource is failing: • Ensure that the fibre channel connectivity between the servers and the storage systems is up and running. • Verify that the disk resource is dependent to the Cluster Extension resource.
NOTE: Host persona setting varies based on the Windows operating system installed on the cluster nodes, For more information, see the HP 3PAR Implementation Guide. fail_wrt_on_err and no_fail_wrt_on_err settings The fail_wrt_on_err and no_fail_wrt_on_err policies apply only to synchronous Remote Copy.
Cluster Extension Autopass troubleshooting Cluster Extension uses Autopass as a framework for licensing checks. Autopass provides Graphical User Interface and a Command Line Interface to perform licensing specific operations, and they are well integrated in to Cluster Extension. For the GUI, Autopass needs a compatible JRE version installed on the system. For the supported JRE version refer to Cluster Extension SPOCK.
6 Support and other resources Contacting HP For worldwide technical support information, see the HP support website: http://www.hp.
Typographic conventions Table 4 Document conventions Convention Element Blue text: Table 4 (page 61) Cross-reference links and e-mail addresses Blue, underlined text: http://www.hp.
Glossary 3PAR Remote Copy A product that allows you to copy virtual volumes from one HP 3PAR storage system to another. CLI Command-line interface. An interface comprised of various commands which are used to control operating system responses. cluster A group of controller nodes connected via the same storage server backplane. The nodes in a cluster operate as a unified system, separate from any other clusters that may share the same service processor. CLX Cluster Extension.
Index Symbols 3PAR Cluster Extension features, 6 integration, 6 3PAR Remote Copy Remote Copy pairs, 11 synchronous replication, 12 volume groups, 11 3PAR Virtual Domains, 58 A Adding dependencies Cluster Administrator, 31 APPLICATION description, 42 ApplicationDir description, 42 ApplicationStartup description, 43 array password file, 37 AutoFailbackType description, 24 Automatic Failback, 7 Automatic Failover, 7 B Basic Resource Health Check Interval description, 23 C CLI configuration, 6 CLI commands a
H help obtaining, 60 HP technical support, 60 HP 3PAR Cluster Extension components, 10 HP 3PAR Remote Copy support for HP 3PAR Cluster Extension, 10 HP3PARCLICommandTimeout description, 47 Hyper-V Live Migration, 36, 39 I importing configuration information, 18 integration with cluster software, 6 IsAlivePollInterval description, 23 L live migration, 31, 36, 39 log files Microsoft Cluster Service, 38 MSCS, 39 LogDir description, 42 LogLevel description, 42 UCF requirement, 31 LooksAlivePollInterval descri
StatusRefreshInterval description, 46 UCF requirement, 31 storage systems 1-to-1 configuration, 8 N-to-1 and 1-to-N configurations, 9 one-to-one configuration, 8 supported configurations, 8 symbols in text, 61 synchronous replication mode, 12 T technical support HP, 60 service locator website, 60 text symbols, 61 Thorough Resource Health Check Interval description, 23 Timing considerations, 50 Windows Clustering, 36 Troubleshooting 3PAR Cluster Extension logs, 53 Autopass, 59 Logs, 53 Start errors, 55 sysm
