HP StorageWorks HP PolyServe Matrix Server administration guide HP PolyServe Matrix Server 4.0.
Legal and notice information © Copyright 2001, 2010 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Quick start checklist ......................................................... 15 Cluster configuration steps .................................................................................. 15 2 Introduction to Matrix Server ............................................. 19 Product features ................................................................................................ Overview ...................................................................................................
Volume and filesystem limits ................................................................................ Cluster management applications ........................................................................ User authentication ............................................................................................ Authentication considerations ....................................................................... Start the Management Console .....................................................
Matrix Server license file .................................................................................... Upgrade the license file ............................................................................... Upgrade servers one-at-a-time ................................................................ Upgrade one server and export ............................................................. Refresh the license file .................................................................................
Show available volumes ........................................................................ Options for dynamic volumes ....................................................................... Show available subdevices .................................................................... Show dynamic volumes ......................................................................... Show properties for dynamic volumes ..................................................... Display unimported dynamic volumes .
Differences between Matrix Server and Microsoft utilities for volumes and filesystems ................................................................................................ Dynamic volumes ............................................................................... Reserved mount point assignments ........................................................ PSFS filesystems ..................................................................................
psfsdq command ...................................................................................... 139 psfsrq command ....................................................................................... 140 Examples ................................................................................................. 140 10 Manage hardware snapshots ........................................ 141 Supported snapshot methods ............................................................................
12 Configure event notifiers and view events ........................ 163 Overview ....................................................................................................... Event logs ................................................................................................ View event logs ........................................................................................ Event notifier services ................................................................................
Virtual hosts ....................................................................................... 187 Service and device monitors ................................................................ 187 14 Configure virtual hosts ................................................. 189 Overview ....................................................................................................... Cluster health and virtual host failover .........................................................
Custom scripts .......................................................................................... Scripts .............................................................................................. Use custom scripts to modify start/stop activities ..................................... Event severity ..................................................................................... Script ordering ...................................................................................
Integrate custom applications ............................................................................ Device monitor or service monitor? .............................................................. Built-in monitor or user-defined monitor? ....................................................... Example of using monitors with applications ................................................. An example of Start and Stop scripts ...........................................................
Restart the nodes ................................................................................ After the storage capacity upgrade ....................................................... Caveat .............................................................................................. Online replacement of a Fibre Channel switch ..................................................... Replace a Brocade FC switch .....................................................................
HP websites .................................................................................................... 277 Index ..............................................................................
1 Quick start checklist The following checklist is intended for new installations of Matrix Server and includes typical steps to configure the cluster. Cluster configuration steps The following checklist assumes that the installation and configuration steps described in the HP PolyServe Matrix Server installation guide have been completed. Action Description Review administrative considerations and restrictions.
Action Description Assign drive letters or mount paths to filesystems. To provide access to a PSFS filesystem, associate it with a drive letter or a mount path. See “Drive letters and mount paths” on page 117. Review snapshot feature if configured. If the snapshot feature is configured on the cluster, review the procedure to create snapshots. See “Create a snapshot or snapclone” on page 142.
Action Description Prepare for cluster security: Create administrative roles (optional). Create roles that allow or deny permission to perform cluster operations and assign users and groups to the roles. See Chapter 11 on page 149. Review the audit log feature. Matrix Server provides an audit trail of operations that change the configuration or state of the cluster. See “Matrix Server audit trail” on page 161. Prepare for user access: Configure SQL Server clients (optional).
Action Description Configure device monitors. Matrix Server provides built-in device monitors such as the SHARED_FILESYSTEM monitor and also allows you to create your own custom monitors. See “Add or modify a device monitor” on page 222.
2 Introduction to Matrix Server Matrix Server provides a cluster structure for managing a group of network servers and a Storage Area Network (SAN) as a single entity. Product features Matrix Server includes the following features: • Fully distributed data-sharing environment. The PSFS filesystem enables all servers in the cluster to directly access shared data stored on a SAN.
Overview The structure of a cluster The following example shows a cluster including a FibreChannel switch. The matrix includes these components: Servers. Each server must be running Matrix Server. Public LANs. A cluster can include up to four network interfaces per server. Each network interface can be configured to support multiple virtual hosts, which provide failover protection for Web, e-mail, file transfer, and other TCP/IP-based applications. Administrative Network.
Storage Area Network (SAN). The SAN includes Fibre Channel switches and RAID subsystems. Disks in a RAID subsystem are imported into the cluster and managed from there. After a disk is imported, you can create PSFS filesystems on it. Software components The Matrix Server software is installed on each server in the cluster. It includes kernel modules such as the PSFS filesystem, as well as several processes and drivers.
psd. Provides cluster-wide consistent device names among all servers. psv. Used by the Matrix Server Volume Manager, which creates, extends, recreates, or destroys dynamic volumes. Shared SAN devices Before a SAN disk can be used, the disk will need to be imported into the cluster. This step gives Matrix Server complete and exclusive control over access to the disk. During the import, the disk is given a unique global device name. The servers in the cluster use this name when they need to access the disk.
• Journaling and live crash recovery. Filesystem metadata operations are written to a journal before they are performed. If a server using the filesystem should crash during an operation, the journal is replayed and any journaled operations in progress at the time of the crash are completed. Users on other servers will experience only a slight delay in filesystem operations during the recovery.
After creating virtual hosts, you will need to configure your network applications to recognize them. When clients want to access a network application, they use the virtual host address instead of the address of the server where the application is running. Service and device monitors A service is a network service such as HTTP or FTP that is installed and configured on the servers in the cluster. Matrix Server can be configured to watch specific services with service monitors.
SNMP service The Matrix Server SNMP service provides tools that can be used to retrieve all matrix-wide state and status information. See “Using the Matrix Server SNMP service” on page 45 for more information. Cluster design guidelines Be sure to consider the following guidelines when planning the physical configuration of your cluster. Server memory Memory resources are consumed on each cluster server to manage the state necessary to preserve the coherency of shared filesystems.
Single FC port, dual FC switches, single FC fabric In this example, the fabric includes two FibreChannel switches. Servers 1–3 are connected to the first FC switch; servers 4–6 are connected to the second switch. The FC switches are connected to two RAID arrays, which contain multiported disks. If a switch fails, the servers connected to the other switch will survive and access to storage will be maintained.
iSCSI configuration This example shows an iSCSI configuration. The Microsoft iSCSI initiator is installed on each server. Ideally, a separate network should be used for connections to the iSCSI storage arrays.
Introduction to Matrix Server
3 Cluster administration Matrix Server can be administered either with the Management Console or from the command line. Administrative considerations and restrictions You should be aware of the following when managing Matrix Server. Network hostname resolution Normal operation of the matrix depends on a reliable network hostname resolution service. If the hostname lookup facility becomes unreliable, this can cause reliability problems for the running matrix.
Network changes The following network changes should not be made while Matrix Server is running. Attempting to do any of the following can adversely affect the entire matrix. If you need to perform these actions on a particular server, first stop Matrix Server on that server. Then make your change and restart Matrix Server. • • • • • Add or delete a network interface from the operating system. Change network properties on the Network Control Panel.
Windows authentication • Matrix Server uses NT credentials to authenticate users attempting to access data on PSFS filesystems. • If you are using HP PolyServe Software for Microsoft SQL Server, a DNS entry must exist for each Virtual SQL Server/IP address. The entry must include an A record and a [PTR] record for forward and reverse lookup. Windows authentication may fail if these records do not exist. Active Directory Matrix Server requires that Active Directory be configured.
and permissions to fail. Typically, when UAC is enabled the administrator group accounts do not have the same privileges as the system built-in administrator. • For best performance, HP recommends that you monitor the matrix from a separate administrative station rather than from a server in the matrix. The PolyServe Management Console can be installed on Windows systems outside the matrix. • Using search bots and running parallel searches across the matrix can have a negative impact on performance.
Volume and filesystem limits The following limits apply to volumes and filesystems: • The maximum volume size is 32 TB. • The maximum filesystem size is 32 TB for the 4 KB and 8 KB block sizes, and 128 TB for the 32 KB block size. • The maximum file size is 2 TB minus one byte. If an I/O operation would cause the size of a file to exceed this limit, the complete I/O operation will fail. • For both dynamic volumes and basic volumes, the maximum number of files/inodes is limited only by the available space.
When the PolyServe Management Console is used, credentials are processed in this order: • If a .matrixrc file exists, the user credentials specified in the file for the selected server are used. • If there is not a .matrixrc file or the file does not include user credentials, the credentials provided by single sign-on semantics are used. • If single sign-on fails, the user is prompted for a user name and password.
Connect to: Enter a matrix or server name or select a name from the drop-down list. When you connect to a server or matrix, it is added to the drop-down list. Click Clear History to delete the list. (Saved bookmarks will remain.) By default, the Connect window logs you onto the matrix using your OS user credentials. If you want to log on as another user, click As User: You can then supply the user name and password for that user as specified under “Authentication parameters and bookmarks” on page 35.
Password: Enter the user’s password. If you do not want to be prompted for the password again, click Remember this password. (For the password to be saved, you will also need to create a bookmark.) Add to bookmarks: Click this checkbox to create a bookmark for this connection. After clicking OK on the Authentication Parameters dialog, you can configure the bookmark on the Add Bookmark dialog. Specify a name for the cluster or select an existing name from the drop-down list.
The bookmark options are: • Add. This option opens the Add Bookmark dialog, allowing you to configure a new bookmark. • Delete. If a cluster is selected, this option removes the bookmark for that cluster. If a server is selected, the option removes just that server from the bookmark. • Rename. If a cluster is selected, this option allows you to rename that cluster. If a server is selected, you can replace that server with a different server in the cluster. After typing the new name, press Enter.
NOTE: If you are using a wildcard to match the servers in the cluster, the wildcard entry should appear after any server entries. You can use the move buttons to reorder the entries as necessary. (For information about wildcards, see the description of the .matrixrc file in the HP PolyServe Matrix Server command reference guide.) • Reset. This option is useful if you have checked Remember this password on the Authentication Parameters dialog.
The PolyServe Management Console The PolyServe Management Console can manipulate all entities in the matrix, including entities that are currently down. For example, you can change the configuration of a down server; it will learn its new configuration when it comes back up and rejoins the matrix. NOTE: For improved performance, the Management Console caches hostname lookups. If your DNS changes, you may need to restart the console so that it will reflect the new hostname.
Servers tab This tab lists the entire configuration of each server configured in the cluster, including the network interfaces on the server, any virtual hosts associated with those interfaces, any device monitors created on the server, and any PSFS filesystems mounted on the server.
Virtual Hosts tab The Virtual Hosts tab shows all virtual hosts in the cluster. For each virtual host, the window lists the network interfaces on which the virtual host is configured, any service monitors configured on that virtual host, and any device monitors associated with that virtual host.
Applications tab This view shows the application monitors configured in the cluster and provides the ability to manage and monitor them from a single screen. The tab uses a table format, with a column for each server in the matrix. The application monitors appear in the rows of the table. You can reorder the information on this tab or limit the information that is displayed.
Filesystems tab The Filesystems tab shows all PSFS filesystems in the cluster.
Matrix alerts The Alerts section at the bottom of the Management Console window lists errors that have occurred in matrix operations. Double click an alert message to see all of the information about the alert. For alerts affecting cluster components such as servers or monitors, double-click in the Source column to highlight the source of the error on the main Management Console window.
If you receive an alert telling you to reboot a server, the message will remain in the Alerts section until either Matrix Server is restarted on the rebooted server or the server is removed from the cluster. To view the current Alerts from the command line, use the mx alert status command. Matrix Server operations Many Matrix Server operations can be run in the background. These operations display a popup window allowing you to send the operation to the background.
in the Microsoft Windows 2003 distribution; however, it is not automatically installed. To perform the installation, complete these steps: 1. Open the Control Panel and select Add or Remove Programs. 2. Double-click Add/Remove Windows Components. On the Windows Components Wizard, select Management and Monitoring Tools and click Details. 3. On the Management and Monitoring window, select Simple Network Management Protocol and click OK and then click Next on the following window. 4.
Use the following command at the Command Prompt to see the software installed on specific servers: mx server listsoftware Start or stop Matrix Server By default, Matrix Server (the matrixserver service) starts automatically when the system is booted. This feature is controlled by the Startup dialog. If you do not want Matrix Server to start when the system is booted, use one of these methods to disable the service: • Open the Startup dialog and change the service from Automatic to Disabled.
• Open the Configure Matrix window (select Tools > Configuration Utility on the PolyServe Management Console or click Configure on the Matrix Server Connect dialog) and go to the Matrix-Wide Configuration tab. Select the server and then click Start Service. This method enables the service if it disabled. • Run the mx server start command. This method enables the service if it disabled. • Issue the command net start matrixserver from a command prompt.
Back up and restore the matrix configuration It is important to back up the cluster configuration whenever changes are made. You can then easily restore the cluster if necessary. If you are using Matrix Server disk quotas, you should also back up the Matrix Server quota information after backing up your PSFS filesystems. Back up and restore Matrix Server HP recommends that you back up Matrix Server whenever the matrix configuration or the membership partitions are changed.
• The mxds datastore, which contains cluster configuration information. The device database and the volume database are automatically backed up to the file conf\MP.backup whenever they are changed; your backups of the conf directory will always contain up-to-date copies of these databases. The mxds datastore is automatically backed up to conf\MPmxds.backup once-per-day.
External network port numbers The following port number is used for external connections to Matrix Server, such as connections from the Management Console and the mx utility. If Matrix Server is behind a firewall, it may be necessary to change the firewall rules to allow traffic for this port.
Cluster administration
4 Configure servers Before adding a server to a cluster, verify the following: • The server is connected to the SAN if it will be accessing PSFS filesystems. • The server is configured as a fully networked host supporting the services to be monitored. For example, if you want Matrix Server to provide failover protection for your Web service, the appropriate Web server software must be installed and configured on the servers. • If the hosts file has been modified, it should be consistent with the DNS.
5. 6. Go to the Matrix-Wide Configuration tab, select all servers except the server to which you are connected, and then click Export To. Restart the Matrix Server service on the existing nodes. Add a new server To add a new server to a cluster, complete the following steps: NOTE: If the server was previously used in a different Matrix Server matrix, remove the contents of the directory %SystemDrive%\Program Files\Polyserve\ Matrix Server\conf before you perform the following procedure. 1.
6. Connect to the matrix. Specify any server in the matrix on the Matrix Server Connect window. The new server now appears on the Management Console. In the following example, server ar53 has been added. 7. Check the drive mappings. When you add a server, the operating system may assign drive letters that do not match the assignments you have made for your shared filesystems on the other nodes.
Server: Enter the name or IP address of the server. Server Severity: When a server fails completely because of a power failure or other serious event, Matrix Server attempts to move any virtual hosts from the network interfaces on the failed server to backup network interfaces on healthy servers in the cluster.
Other server configuration procedures Delete a server When a server is deleted, it is removed from the cluster and from all Matrix Server applications. Applications hosted by the server may be adversely affected by the server's removal, and any applications that exclusively use the server will be destroyed. Before deleting a server, be sure to take the appropriate actions: • If the server is primary for a Virtual SQL Server, rehost the Virtual SQL Server to another node.
To enable servers from the command line, use this command: mx server enable ... Change the IP address for a server A server’s IP address can be changed without affecting the other servers in the cluster. The following procedure uses this scenario: three servers, S1, S2, and S3, with a single NIC. You want to change the IP address of server S2. 1. Stop Matrix Server on server S2. 2. Change the IP address of server S2. We will now identify the server as S2a. 3.
2. Go to the Matrix-Wide Configuration tab, click Add Server, and enter the hostname or IP address of the server on the input form. Then click OK to add the server to the Address column. 3. Select the server in the Address column and click Export To. The Last Operation Progress column will display status messages as the configuration is exported to the server. 4. Start Matrix Server on the server. The server will still be selected in the Address column. Click Start Service to start Matrix Server.
Upgrade one server and export This procedure requires that Matrix Server be stopped on all servers. Execute the procedure on one server in the matrix. 1. On one server, start the Management Console. Enter the IP address of the server on the Matrix Server Connect window and click Configure. NOTE: If there is a .matrixrc file on the system running mxconsole, you will see a Disconnect dialog instead of the Connection Parameters window. Select Logon to another matrix server and then click Configure. 2.
Management Console to determine the supported features for a particular server. Select the server on the Servers window, right-click, and select View Features. The Supported Features window lists all of the Matrix Server features. Any features not supported by your license are greyed out. Limit the servers that can join a matrix The network secret key controls server access to the matrix. When you create a matrix configuration, you will specify a password that is used as the network secret key.
server name will actually be served by the virtual host—basically, a failover-protected version of the server, with no difference in appearance to the clients. • Keep the existing name on the server. If you do not rename the server, clients will need to use the new virtual host name to benefit from failover protection. Clients can still access the server by its name, but those requests are not protected by Matrix Server.
The addresses on the domain name server are virtual_acmd1 and virtual_acmd2. Two virtual hosts have also been created with those names. The first virtual host uses acmd1 as the primary server and acmd2 as the backup. The second virtual host uses acmd2 as the primary and acmd1 as the backup. The DNS server is configured for round robin using the following A records: Address Time to Live Service Record Type IP Address www.acmd.com. 60 IN A 10.1.1.1 www.acmd.com. 60 IN A 10.1.1.
work; the other server is a backup in case the primary goes down or is taken offline for maintenance. Depending on your needs, more complicated combinations of DNS round robin and Matrix Server can be used, including cluster configurations with multiple failover-protected load-balanced servers.
5 Configure network interfaces When you add a server to the cluster, Matrix Server determines whether each network interface on that server meets the following conditions: • • • • The network interface is up and running. The network interface is multicast-capable. 802.3x Ethernet flow control is not used. Each network interface card (NIC) is on a separate network. Network interfaces meeting these conditions are automatically configured into the cluster.
If it cannot use that network, the PanPulse process on the new server will look for another network that all of the servers can use. If this attempt fails, the new server will attempt to communicate with the other servers over the main interface (the hostname interface) if possible; however, it may drop out of the cluster. Network topology The cluster should have a symmetrical network configuration; each server should be connected to the same set of networks.
The Management Console uses the following icons to specify whether the network interface allows or discourages administrative traffic. The network interface allows administrative traffic. A green checkmark indicates the current administrative network. A red X indicates that the interface is down. The network interface discourages administrative traffic. The network interface excludes administrative traffic.
If multiple interface failures occur on a server and there is not another network available for the administrative network, the server may drop out of the cluster. The remaining servers will continue to use the existing administrative network. Making network changes The following network changes should not be made while Matrix Server is running. Attempting to do any of the following can adversely affect the entire matrix.
Enable or disable a network interface for virtual hosting By default, all network interfaces are enabled for virtual hosting. To enable or disable a network interface for virtual hosting, select that interface on the Servers window, right-click, and then select either Enable Hosting or Disable Hosting. From the command line, use this command to disable virtual hosting: mx netif disable ... Use the following command to enable virtual hosting: mx netif enable ...
IP: Enter the IP address for the network interface. Netmask: Enter the net mask for the network interface. Allow Administrative Traffic: Specify whether the network interface can host administrative traffic for Matrix Server. The options are: • Allow. The default is to allow the traffic. • Discourage. This network will be used for administrative traffic only if no other network is available.
6 Configure the SAN SAN configuration includes the following: • Import or deport SAN disks. After a disk is imported, it can be used for PSFS filesystems. • Change the partitioning on SAN disks. • Display information about SAN disks. Overview SAN configuration requirements Be sure that your SAN configuration meets the requirements specified in the HP PolyServe Matrix Server installation guide. Storage Control Layer module The Storage Control Layer (SCL) module manages shared SAN devices.
Device identifiers and GPT disks When the SCL assigns device identifiers to the partitions on GPT disks, it skips the first partition because that partition cannot be used by Matrix Server. Therefore, the first identifier, psdXp1, is assigned to partition 2, the second identifier, psdXp2, is assigned to partition 3, and so on. Device database and membership partitions The SCL stores the device identifier and physical UID for each imported disk in a device database.
Maximum length for disk UIDs The maximum length for a Matrix Server disk UID is 128 characters. If a disk UID exceeds this limit, Matrix Server will not be able to import the disk into the matrix. If you are using a disk array such as iSCSI that allows you to specify a disk volume name as part of the UID, be sure that the total UID does not contain more than 128 characters. Restriction for disk partitions Do not delete disk partitions on LUNs accessed by Matrix Server.
• If servers from multiple matrices can access the SAN via a shared FC fabric, avoid importing the same disk into more than one cluster. Filesystem corruption can occur when different matrices attempt to share the same filesystem. • Only basic disks can be imported. Dynamic disks are not supported. • Shared disks must be partitioned before they are imported into the matrix. If there is not a partition table in place on the disk, you will not be able to access it via Matrix Server.
To import a disk from the command line, use the following command: mx disk import ... To determine the uuid for a disk, run the mx disk status command, which prints the uuid, the size, and a vendor string for each unimported SAN disk. You can also use the Disk Info window to import a disk. Deport SAN disks Deporting a disk removes it from cluster control. You cannot deport a disk that contains a membership partition.
To deport a disk from the command line, use the following command: mx disk deport ... To determine the uuid for the disk, run the mx disk status --imported command. You can also use the Disk Info window to deport a disk. Display local disk information The Disk Info window displays disk information from the viewpoint of the local server.
When you select a disk, the window displays information about the partitions on the disk. The window also lists any mount paths for PSFS filesystems. To import or deport a disk, select that disk and then click Import or Deport as appropriate. NOTE: Because the first partition on GPT disks cannot be used by Matrix Server, that partition is skipped when Matrix Server assigns device identifiers to the partitions.
available for filesystems. To see the window, select Storage > Storage Summary on the PolyServe Management Console or click Storage on the toolbar. The window shows the following information for each PSFS filesystem: • The label assigned to the filesystem. • The mount point or drive letter assigned to the filesystem. Click in the cell to see the mount point/drive letter for each server on which the filesystem is configured. • The volume used for the filesystem.
The window also lists the UIDs and sizes of the LUNs (both imported and unimported) that are available for filesystems. Display disk information with sandiskinfo The sandiskinfo command can display information for both imported and unimported SAN disks (or LUNs) and also for dynamic volumes. Under normal operations, the sandiskinfo output should be the same on all servers in the cluster.
Show partition information The -a option lists the partitions on each disk. When combined with -u, it displays partition information for unimported disks. # sandiskinfo -a Show local device information The -l option displays the local device name for each disk, as well as the default disk information. When combined with -u, it displays local device names for unimported disks. # sandiskinfo -al Show filesystem information The -f option displays existing PSFS filesystems on imported disks.
Subdevice: 20:00:00:04:cf:13:38:18::0/13 Subdevice: 20:00:00:04:cf:13:38:18::0/14 Size: Size: 490M 490M psd1p13 psd1p14 Show dynamic volumes The --dynvolumes option lists all dynamic volumes. # sandiskinfo --dynvolumes Dynamic Volume: psv1 Dynamic Volume: psv2 Dynamic Volume: psv3 Size: Size: Size: 2439M 490M 490M Stripe=Unstriped Stripe=Unstriped Stripe=Unstriped Show properties for dynamic volumes The --dynvol_properties [volname] option lists detailed properties for the specified dynamic volumes.
Configure the SAN
7 Configure dynamic volumes Matrix Server includes a Volume Manager that you can use to create, extend, recreate, or delete dynamic volumes. Dynamic volumes allow large filesystems to span multiple disks, LUNs, or storage arrays. Dynamic volumes can be deported from the matrix and later imported back into the original matrix or into another matrix. Overview Basic and dynamic volumes Volumes are used to store PSFS filesystems. There are two types of volumes: dynamic and basic.
subdevice, and then to the third subdevice. This method fills the subdevices at the same rate and may provide better performance. • Concatenation. When a dynamic volume is created without striping, data is written to the first subdevice until it is full. Then data is written to the second subdevice, and so on. Dynamic volume names The Cluster Volume Manager uses unique device names to control access to dynamic volumes. These names form the pathnames that servers use to access shared data.
• The subdevices used for a striped dynamic volume should be the same size. The Volume Manager uses the same amount of space on each subdevice in the stripeset. When a striped dynamic volume is created, the Volume Manager determines the size of the smallest specified subdevice and then uses only that amount of space on each subdevice. If the subdevices are different sizes, the additional space on the larger subdevices will be wasted. This limitation also applies when a striped dynamic volume is extended.
Filesystem: If you want Matrix Server to create a filesystem that will be placed on the dynamic volume, enter a label to identify the filesystem. If you do not want a filesystem to be created, remove the checkmark from Create filesystem after volume creation. If you are creating a filesystem, you can also select the options to apply to the filesystem. Click Options to see the Filesystem Option dialog, which allows you to select the block size for the filesystem and to configure quotas.
NOTE: If you selected a subdevice that is associated with an unimported volume, you will see a message reporting that the subdevice contains a volume signature. The message asks whether you want to destroy the affected unimported dynamic volume and reuse this subdevice in the new volume you are creating. Be sure that you do not need the unimported dynamic volume before doing this. If the unimported volume associated with the subdevice is in a “duplicate” state, the attempt to destroy the volume will fail.
The Stripe State reported in the “Dynamic Volume Properties” section will be one of the following: • Unstriped. The volume is concatenated and striping is not in effect. • Optimal. The volume has only one stripeset that includes all subdevices. Each subdevice is written to in turn. • Suboptimal. The volume has been extended and includes more than one stripeset. The subdevices in the first stripeset will be completely filled before writes to the next stripeset begin.
View stripeset information To see the contents of a stripeset, run mpdump.exe with no options from the Command Prompt. The command is in the directory Program Files\Polyserve\ MatrixServer\bin on the drive where you installed Matrix Server. Following is some sample output. Current Product MP Version: 2 Membership Partition Version: 2 Membership Partitions: 10:00:00:50:13:b3:41:66::63/2 (ONLINE) . . . Membership Partition Device Database (Version 1): UID:10:00:00:50:13:b3:41:66::63 Label:psd1 . . .
To extend a dynamic volume on the Management Console, select Storage > Dynamic Volume > Extend Volume and then select the volume that you want to extend. If a filesystem is on the volume, the Extend Dynamic Volume window shows information for both the dynamic volume and the filesystem. Dynamic Volume Properties: The current properties of this dynamic volume. Filesystem Properties: The properties for the filesystem located on this dynamic volume.
When you click OK, the dynamic volume will be extended. IMPORTANT: If you selected a subdevice that is associated with an unimported volume, you will see a message reporting that the subdevice contains a volume signature. The message asks whether you want to destroy the affected unimported dynamic volume and reuse this subdevice for the volume you are extending. Be sure that you do not need the unimported dynamic volume before doing this.
Recreate a dynamic volume Occasionally you may want to recreate a dynamic volume. For example, you might want to implement striping on a concatenated volume or, if a striped dynamic volume has been extended, you might want to recreate the volume to place all of the subdevices in the same stripe set. When a dynamic volume is recreated, the Cluster Volume Manager first deletes the volume and then creates it again using the subdevices and options that you select.
You can change or reorder the subdevices used for the volume and enable striping if desired. If you want to recreate the filesystem on the newly created volume, click Recreate Filesystem. To recreate a volume from the command line, run the dynvolume destroy command and then run the dynvolume create command.
The new dynamic volume will contain only the original subdevice; you can use the Extend Volume option to add other subdevices to the dynamic volume. NOTE: The new dynamic volume is unstriped. It is not possible to add striping to a converted dynamic volume. If you want to use striping, you will need to recreate the volume. To convert a basic volume, select the associated PSFS filesystem on the Management Console, right-click, and select Convert to Dynamic Volume.
Dynamic volume recovery The Dynamic Volume Recovery feature provides the ability to rebuild a dynamic volume from the LUNs originally in the volume. This feature can be used for purposes such as the following: • Move dynamic volumes from one cluster to another. Deport the dynamic volumes on the original cluster and then import them on the new cluster. • Recover dynamic volumes from mirrored LUNs for disaster recovery purposes. Import the dynamic volumes from the mirrored LUNs to the replacement cluster.
Select the dynamic volumes that you want to deport and click Deport on the toolbar. To deport dynamic volumes from the command line, use this command: mx dynvolume deport Import a dynamic volume When a dynamic volume is imported, the unimported LUNs associated with the volume will be imported and the psv binding, which Matrix Server uses to control access to the dynamic volume, will be created.
To import a dynamic volume, select Storage > Dynamic Volume > Import Volume. Select the dynamic volumes that you want to import and click Import on the toolbar. To import dynamic volumes from the command line, first use the following command to list the dynamic volumes that can be imported: mx dynvolume list --importable Locate the entry for the that you want to import. The appears in the first column of the output.
which was generated when the volume was created. Unimported volumes are either importable or unimportable. The following command lists all unimported volumes: mx dynvolume list --unimported Importable volumes All of the constituent parts for the volume are accessible, and the volume can be reassembled into a dynamic volume when it is imported into the cluster.
Reuse subdevices in unimportable volumes The subdevices in an unimportable volume can be reused in other dynamic volumes or filesystems after the original volume is destroyed. This step removes the volume signature from all of the subdevices, freeing them for other uses.
Configure dynamic volumes
8 Configure PSFS filesystems Matrix Server provides the PSFS filesystem. This direct-access shared filesystem enables multiple servers to concurrently read and write data stored on shared SAN storage devices. A journaling filesystem, PSFS provides live crash recovery.
Journaling filesystem When you initiate certain filesystem operations such as creating, opening, or moving a file or modifying its size, the filesystem writes the metadata, or structural information, for that event to a transaction journal. The filesystem then performs the operation. If a server using the filesystem should crash during an operation, the journal is replayed and any journaled operations in progress at the time of the crash are completed.
• Coordinates filesystem mounts, unmounts, and crash recovery operations. • Checks for cluster partitioning, which can occur when cluster network communications are lost but the affected servers can still access the SAN. If this situation occurs, Matrix Server determines which servers should continue to have SAN access and excludes the other servers from the SAN. This step prevents data corruption from occurring. • Communicates filesystem status to the PolyServe Management Console.
file; any attempts to allocate more space will fail. The file owner will need to remove files or reduce their size until the disk usage falls below the hard limit. Crash recovery When a server using a PSFS filesystem either crashes or stops communicating with the matrix, another server in the matrix will replay the filesystem journal to complete any transactions that were in progress at the time of the crash. Users on the remaining servers will notice a slight delay while the journal is replayed.
dynamic volumes and the filesystems created on those volumes. You can also view this information with the mx dynvolume command. The Windows Disk Manager utilities cannot display a Matrix Server dynamic volume with its associated psv name and volume information. Instead, the Disk Manager displays the local disk information for each subdevice included in the dynamic volume, with drive letter and/or mount point assignments shown on the first subdevice of the dynamic volume.
Using Microsoft Windows disk management utilities After a disk has been imported into a cluster, it can no longer be managed with the normal Windows Disk Manager/diskpart/format or other Microsoft utilities. To use these utilities to change, repartition, or reformat an imported disk, you must first deport the disk. All imported disks appear as “Read-only” in Disk Manager. After deporting a disk, you may need to refresh the Disk Manager view (by pressing F5) to see the disk change from Read-only to Online.
Create a filesystem A PSFS filesystem can be created on a basic volume (a psd device) or a dynamic volume (a psv device). You can create a filesystem from one server in the cluster using either the PolyServe Management Console or the command line. NOTE: The minimum size for a PSFS filesystem is 64 MB. Before creating a filesystem, verify the following: • The disk is partitioned appropriately.
Label: Enter a label that identifies the filesystem. NOTE: Although the “Disk Management” pane of the Microsoft Management Console (MMC) limits volume labels on non-NTFS filesystems to 11 characters, the Matrix Server tools allow PSFS filesystem labels to be up to 32 characters long. Available Volumes: This part of the window lists the basic or dynamic volumes that are currently unused. Subdevices belonging to unimported volumes that are on imported disks are also included.
NOTE: The Create a Filesystem window identifies volumes by their Matrix Server names such as psd1p2. To match these names to their local Windows names, open the Disk Info window (select the server on the Servers tab, right-click, and then select View Local Disk Info). IMPORTANT: If you selected a subdevice that is associated with an unimported volume, when you click OK to create the filesystem you will see a message reporting that the subdevice contains a volume signature.
The Quotas tab allows you to specify whether disk quotas should be enabled on this filesystem. You can enable or disable quotas on a filesystem at any time. (See “Enable or disable quotas” on page 135.) When you enable quotas, you can also set default hard and soft quotas and select other quota parameters. To enable quotas on the filesystem, check Enable quotas.
You can then set default hard and soft quotas for users on that filesystem. If you do not want a default limit, select Unlimited, which is the default. To assign a limit, select Limit and then specify the appropriate size in either kilobyes, megabytes, gigabytes, or terabytes. The defaults are rounded down to the nearest filesystem block. NOTE: The default user quotas apply to all users who do not have an individual quota assigned.
There are two options: • Static default quota. The default limits are explicitly assigned to the user. Subsequent changes to the default values for the filesystem do not affect the quota limits for the user. This is the default, and matches the NTFS policy. • Dynamic default quota. No explicit default limits are assigned to the user. Instead, the effective limits applied to the user are the default values for the filesystem at the time of each operation.
If you want to change this option after the filesystem is created, you will need to either disable quotas and then re-enable them with the changed option, or recreate the filesystem. Recreate a filesystem To reformat a filesystem, select the filesystem on the Filesystems window, right-click, and select Recreate. A message then appears stating that the Matrix Server needs to unmount the filesystem and any unsaved files will be lost. When you click Yes, the following window appears.
The psfsformat command Use this syntax: psfsformat [-fq] [-l
Create the filesystem with support for 8.3 short file names and name tunneling enabled. These features should be enabled only if you have a specific need for them, as the use of 8.3 files causes degradation in filesystem performance. The degradation is proportional to the number of 8.3 files created. If 8.3 SFN support is later disabled but 8.3 filenames exist, the performance degradation will continue. allowextchar Allow the extended character set to be used in 8.3 short file names.
Whether file operations that will result in exceeding a user’s hard limit are denied or allowed. noenforcehardlimit is the default. sparse-fullcost or sparse-realcost With sparse-fullcost, quota accounting for sparse files reflects the virtual size of the files. With sparse-realcost, quota accounting for sparse files reflects the actual allocation of filesystem space to the files. sparse-realcost is the default.
size will be calculated in bytes. (The default is rounded down to the nearest filesystem block.) [--defaultUserSoftLimit ] The default soft limit. unlimited specifies that there is no default. The optional size modifiers specify that the size is in kilobytes (K), megabytes (M), gigabytes (G), or terabytes (T). If a modifier is not specified, the size will be calculated in bytes. (The default is rounded down to the nearest filesystem block.
Assign Drive Letter: Matrix Server queries the servers in the matrix to determine the drive letters that are currently unused on all of the servers. You can assign any of these drive letters to the filesystem. NOTE: If, on a matrix server, the drive letter you selected is assigned for another purpose before the Assign Drive Letter operation is complete, the affected server will not be able to access the filesystem via the drive letter.
window lists each server that has mounted the filesystem and the mount path or drive letter assigned to it. You can also view this information from the command line. To see the assignments for a filesystem, use this command: mx fs queryassignments To see the assignments on specific servers, use this command: mx fs getdriveletter --server
After the drive letter or path has been unassigned, you will not be able to access the filesystem until you assign a new drive letter or path to it. To remove assignments from the command line, use this command: mx fs unassign Set permissions for filesystems accessed via mount points With mount point (or mount path) assignments, the PSFS filesystem, or volume, is mounted in an empty directory on an NTFS volume.
Click Properties, adjacent to the Type field, to see the properties for the mounted volume. Although this Properties dialog specifies that the filesystem is RAW, it is a PSFS filesystem. Click the Security tab to set the permissions for the root directory of the volume.
8.3 Short File Names and name tunneling By default, PSFS filesystems do not support the creation of 8.3 short file names (SFN) and name tunneling; however, support for these features can be enabled on specific filesystems. These features should be enabled only if you have a specific need for them, as the use of 8.3 files causes degradation in filesystem performance. The degradation is proportional to the number of 8.3 files created. If 8.3 SFN support is later disabled but 8.
psfscheck -e enable8dot3 -e allowextchar Determine status of 8.3 SFN on a filesystem The are several ways to determine whether 8.3 SFN is supported on a particular filesystem. The Features tab on the filesystem’s Properties window (select the filesystem, right-click, and select Properties) lists all supported features. For an example, see “Features tab” on page 126. The psfsinfo command specifies whether 8.3 support and the extended character set are enabled on the filesystem.
The psfsinfo command also reports ADS in the list of enabled features: psfsinfo psv1 Version: 11 BlkSize: 8192 FsSize: 52428792 DefaultUsrQuota: 0 DefaultGrpQuota: 0 DefaultSoftUsrQuota: 0 enable8dot3 = 1 allowextchar = 0 Features: FZBM QUOTA SPARSE_FILES EIGHT_THREE_PRIMED ADS Run-time flags: PSFS_RT_QUOTA_NO_ENFORCE PSFS_RT_QUOTA_STATIC_DEFAULT View or change filesystem properties To see information about a specific filesystem, select that filesystem, right-click, and select Properties.
Extend a mounted filesystem If the Volume allocation display shows that there is space remaining on the volume, you can use Extend Filesystem on the Properties window to increase the size of the PSFS filesystem to the maximum size of the volume. When you click Extend Filesystem, you will see a warning such as the following. When you click Yes, Matrix Server will extend the filesystem to use all of the available space.
Features tab The Features tab shows whether Full Zone Bit Maps (FZBM), quotas, sparse files, 8.3 short file names, and Alternate Data Streams (ADS) are enabled on the filesystem. Quotas tab The Quotas tab allows you to enable or disable quotas on the filesystem, to set the default hard limit for users and groups, and to configure other quota options. See “Filesystem options” on page 109 for more information about the quota options.
View filesystem status from the command line You can use the following mx command to see status information. mx fs status [--verbose] [--standard|--snapshots] The command lists the status of each filesystem. The --verbose option also displays the FS type (always PSFS), the size of the filesystem in KB, and the UUID of the parent disk. The --standard option shows only standard filesystems; the --snapshot option shows only snapshots.
NOTE: This option cannot be used to extend filesystems on disks containing a Matrix Server membership partition. Select the filesystem on the Management Console, right-click, and select Extend Volume. Matrix Server then determines whether the disk contains space that can be used to extend the volume or partition. On the Extend Basic Volume window, specify the amount of space that should be added to the filesystem.
When you click OK on the Confirm Extend window, Matrix Server deports the disk, resizes the filesystem partition by the specified size, reimports the disk, and then expands the filesystem to fill the additional space in the partition. IMPORTANT: If you used the Windows Disk Manager utility to assign drive letters or mount paths for the filesystem, you will need to reassign them on each node after the resize operation is complete.
The next example uses a mount path: psfssuspend c:\psfs_mount\ The psfssuspend command prevents modifications to the filesystem and forces any changed blocks associated with the filesystem to disk. The command performs these actions on all servers that have mounted the filesystem and then returns successfully. Any process attempting to modify a suspended filesystem will block until the filesystem is resumed.
• By the psd or psv name, such as psd2p2 or psv1 For a complete description of the options, see the HP PolyServe Matrix Server command reference guide. Perform a filesystem check If a filesystem is not unmounted cleanly, the journal will be replayed the next time the filesystem is mounted to restore consistency. You should seldom need to check the filesystem. However, if a filesystem was corrupted by a hardware or software failure, you can repair it with the psfscheck utility.
For more information about the check, click Details. If psfscheck locates errors that need to be repaired, it will display a message telling you to run the utility from the command line. For more information, see the HP PolyServe Matrix Server command reference guide.
Destroy a filesystem If necessary, you can remove a filesystem from the volume on which it is located. Select the filesystem on the Management Console, right-click, and select Delete. NOTE: Any snapshots of the filesystem must be destroyed before the filesystem is destroyed.
Configure PSFS filesystems
9 Manage disk quotas The PSFS filesystem supports disk quotas, which limit the amount of disk space on a filesystem that can be used for individual user’s files. Hard and soft filesystem limits The PSFS filesystem supports both hard and soft filesystem quotas. A hard quota specifies the maximum amount of disk space on a particular filesystem that can be used by files owned by the user.
NOTE: The enable/disable features provided with Windows utilities cannot be used to enable/disable quotas, as those features do not work with PSFS filesystems. These utilities will report messages such as “One or more settings were not changed.” Enable or disable quotas on the Management Console To enable or disable quotas on an existing filesystem, locate the filesystem on the Management Console, right-click, and select Properties. Then go to the Quotas tab on the Properties dialog.
You can also click Advanced Options to set the Logging & Enforcement options, the Quota Assignment Policy, and the File Accounting option. If the File Accounting Option has already been set on this filesystem and you want to change the accounting method, you will need to either disable quotas and then re-enable them with the changed option, or recreate the filesystem. For more information about the advanced options, see “Filesystem options” on page 109.
Quota GUI. The Windows Quota GUI can be accessed from Microsoft Windows Explorer. Right-click on the filesystem, select Properties, and then select the Quota tab. You can modify filesystem options from this window; however, you cannot disable quotas on the filesystem. The Quota tab does not appear for PSFS filesystems that do not have quotas enabled. The Quota Entries window. This window can be accessed via Microsoft Windows Explorer.
• The “Amount Used” column includes PSFS metadata as well as the space required for the user data in each user’s files. The space used may be different that it would be on another type of filesystem. • When you attempt to delete a user quota on the Quota Entries window, you will not see a list of the files owned by the user, as displayed for a NTFS filesystem. You will need to use another method to delete or change ownership of these files before deleting the quota.
The -f option specifies the file to which psfsdq will write its output. If the file already exists, it will be overwritten. If -f is not specified, psfsdq writes to stdout. filesystem is the psd or psv device used for the filesystem. psfsrq command This command restores the quota data generated by the psfsdq command to the specified PSFS filesystem. The command has this syntax: psfsrq [-f ] The -f option specifies the file that psfsrq should read to obtain the quota data.
10 Manage hardware snapshots Matrix Server provides support for taking hardware snapshots of PSFS filesystems. The snapshots provide a point-in-time image of a PSFS filesystem. Users or the Administrator can then use the Microsoft Shadow Copies of Shared Folders feature to recover individual files or whole volumes from the appropriate snapshot image. The subdevices containing the PSFS filesystems must reside on one or more storage arrays that are supported for snapshots.
To locate this software, contact your HP representative. To take hardware snapshots on HP StorageWorks 2000 Modular Smart Array (MSA2000) storage arrays, the latest version of firmware must be installed on the array controllers and the SSH Command Line Interface (CLI) service must be enabled on the array controllers. Also, note that a MSA2000 snapshot license is required. Only the filesystems located on Master Volumes (not Standard Volumes) are snapshot capable.
Label. The label is used to identify the snapshot on the Management Console. Share as Shadow Copy of Shared Folder. Check this box if you want users to be able to use the snapshot as a shadow copy. HP EVA Options. Snapshots initially consume storage space only to store pointers to the data in the source filesystem, growing in size when source filesystem data is changed.
Engenio snapshots The dialog asks for the following information: Label. The label is used to identify the snapshot on the Management Console. Share as Shadow Copy of Shared Folder. Check this box if you want users to be able to use the snapshot as a shadow copy. Engenio Options. The first time a snapshot is taken of a particular filesystem, the snapshot process creates a repository on disk that stores pointers to the data in the source filesystem.
To create a snapshot from the command line, first run the following command to determine the options available for the specified volume: mx snapshot showcreateopt Then run the following command to create the snapshot: mx snapshot create [--terse] [] The --terse option causes only the name of the snapshot volume to be printed on success. Errors during snapshot operations If the snapshot operation could not be completed, errors will be reported to the matrix event log.
To delete a snapshot from the command line, enter the following: mx snapshot destroy Snapclone devices, like regular filesystem LUNs, cannot be deleted from the Management Console. To delete snapclones, you must destroy the filesystem and/or volume, deport the LUN(s), and delete the LUN(s) using the appropriate array-specific utilities. Assign or remove drive letters or mount paths Drive letters or mount paths can be assigned or unassigned to snapshots in the same manner as filesystems.
NOTE: If the CIFS client workstation is running a different version of the Windows operating system, contact Microsoft to obtain the client pack. The client pack adds a “Previous Version” tab to the Windows Explorer Properties window for the virtual CIFS share. This tab lists the shadow copies, or previous versions, of the filesystem that are currently available. Users can select the appropriate version, view the contents, and copy or restore the needed files.
Manage hardware snapshots
11 Configure security features Matrix Server provides the following security features: • Role-Based Security. By default, the machine’s local Administrators group has full cluster rights and can perform all Matrix Server operations. You can use the RoleBased Security feature to create roles that allow or deny other users and groups the ability to perform specific cluster operations.
will authorize you only for creating and modifying filesystems. The deny status overrides the allow status. Matrix Server provides a built-in System Administrator role that includes all members of the machine local Administrators group. This group has permission to perform all cluster operations. In general, if a user in the System Administrator role belongs to another role that denies a cluster operation, the deny will override the permission granted to the System Administrator role.
Add a new role To define a new role, click Add to display the Role Properties window. Name: Enter a name for the new role. Role names cannot include the forward slash character (/). Enabled: By default, the role will be enabled when it is created. To disable the role, remove the checkmark. Resource: Use this pane to specify the rights that will apply to the new role. You can allow or deny the right to create, modify, or delete the cluster resources listed on the pane. The resources are as follows.
• Setup. Manipulate settings that affect the entire cluster configuration, including membership partitions, licensing, snapshot configuration, fencing configuration, servers, notification settings, and security roles. The Event Notification, Security, and Servers resources are subsets of this resource. • Event Notifications. Configure event notification settings. Create affects the ability to enable or disable notifiers.
• SQL. Manipulate SQL application objects including Virtual SQL Services, MSSQL, MSDTC, SQL instances, and SSAS instances. This resource is available when HP PolyServe Software for Microsoft SQL Server is installed. Create affects the ability to create new objects and to use the Multi-Node Installer for SQL.
When you select a template, the rights appropriate to that role will be marked with a checkmark to allow the right or an X to deny the right. You can adjust the rights as necessary. Then go to the Members tab to assign group or user accounts to the role. Assign accounts to a role The Members tab on the Role Properties window shows the user and group accounts that belong to the role.
Click Add to assign accounts to the role. The Enter an Account dialog then asks for the user or group to be added. Enter an account to add. Enter the name or ID for the user or group. Type. Specify whether you are adding a user account or a group account. Form. Specify whether you entered a name or an ID for the account.
Tips for specifying accounts When specifying accounts for a role, you should be aware of the following: • Matrix Server uses the contents of the access token created when you logged into the cluster to determine user and group identities. • To simplify Role-Based Security administration, specify groups instead of users wherever possible. • Specify groups that are valid for all servers in the cluster. Domain universal groups and domain global groups have access to all servers.
of the roles to which you belong. Allowed operations are indicated by a checkmark. Denied operations are indicated by an X. The Role memberships pane at the bottom of the My Rights tab shows the roles to which you belong. Roles that are disabled appear in italics. In the following example, the user has logged on as a member of the local Administrators group and is therefore a member of the default System Administrator role. This group is allowed to perform all cluster operations.
Other role-based security procedures Export or import roles The import and export features can used if you will be configuring a new cluster and want to use the Role-Based Security settings that you have configured on the existing cluster. Click Export to save the current settings to the file of your choice. (The default location is your home directory.) The file is written in XML format.
When configuring the new cluster, click Import to import the file containing the Role-Based Security settings. The imported settings will replace any current Role-Based Security settings. To import or export Role-Based Security settings from the command line, use these commands: mx role export [--permissionOnly] mx role import [--permissionOnly] The --permissionOnly option omits the list of role members from the import or export.
Delete a role When a role is deleted from the cluster configuration, the accounts belonging to the role will automatically lose their membership in that role. Roles are deleted on the Role Properties window. Select the role on the Role-Based Security Control Panel and click Edit to display the Role Properties window.
The --form option specifies whether you are entering the name or ID of the account (NAME is the default). The --type option specifies whether the account is for a user or group or is unknown (GROUP is the default). List roles assigned to an account Use this command: mx account listroles [--form ] [--type ] [--effective] [--noHeaders] [--csv] [--showBorder] [] The --type option specifies whether the account is for a user or group or is unknown (GROUP is the default).
Configure security features
12 Configure event notifiers and view events Matrix Server generates an event message when an error condition or failure occurs or when the status of the matrix changes. To provide an audit trail of matrix operations, a message is also generated when a user requests and is granted or denied authorization to perform a task. Event messages are logged and can be viewed either with the Matrix Event Viewer provided with the PolyServe Management Console or with command-line tools.
• Sends the message to the event notifier services configured on the server. If the message has been selected to trigger a notifier service, the appropriate action will take place (send an SNMP trap, send email, or run a script). • Sends the message to all servers in the matrix. The servers, including the server where the event occurred, copy the message into their own matrix logs.
Matrix event viewer The Matrix Event Viewer shows the messages in the matrix log for a particular server. To open the Viewer, select a server on the Management Console, right-click, and select View Events. The title bar shows the last time that the Event Viewer was updated. Click Refresh to update the display. By default, the Event Viewer shows the last 1000 messages in the matrix log. To display a different number of messages, select Viewer > Max Events to Display.
View event details To view all of the information for a particular event, double-click that event on the Event Viewer. The Event Properties window shows the information. Filter the event output The Event Viewer includes three filters that can be used to limit the events that are displayed: • Search All. This filter allows you to enter text to be matched. The Event Viewer will show only those events that include the text in any of the event fields. • Timestamp.
View events from the command prompt Matrix Server provides commands that can be used to view the matrix log on a particular server and to view outstanding alerts.
The severity values are INFO, WARNING, ERROR, CRITICAL. If you specify more than one alert level, use commas to separate the levels. The remaining options determine the format of the output. --noHeaders omits column headers, --csv prints output in comma-separated format, and --showborder displays borders in the output. Windows Event Viewer You can view the matrix events written to the Windows event log by using the Windows Event Viewer.
The Control Panel opens on the Event Definition tab, which lists all events defined in the Matrix Server event catalog. The Event Definition tab provides a Search All filter that lists messages matching the specified term. You can also select one or more severity levels to be matched.
• Select a message row, right-click, and then set or clear that message for the appropriate notifier services. To add or remove notifier events from the command line, use these commands. If you do not specify a service, the events will be added or removed from all services. You can specify individual event IDs or a range of IDs to be added. Use commas to separate the values, for example: 100, 300-400,555.
To define a new SNMP trap forwarding target, click Add New Target. Target. Enter either the hostname or the IP address of the SNMP trap forwarding target. The trap-forwarding destination port is the IANA registered port for snmptrap (162/udp). Community. Enter the community string that is used to access the target. The default is public. Disable the SNMP trap forwarding service. This checkbox can be used to enable or disable the service as necessary.
Traps are sent as SNMP v1 traps by the server that logged the event. The trap ID is the Matrix Server message ID. You can change the information for an existing target by selecting it from the SNMP Trap Forwarding Table and then clicking Edit Target. To remove a target, select it from the table and click Remove Target.
From Email address. Enter the email address that will be specified as the sender of the notification emails. If this option is not included, the server name will be used as the sender. To Email address. Enter the email addresses to which event notifier email should be sent. If multiple addresses will be specified, use semicolons to separate the addresses. Subject line. Select the amount of information that will appear in the Subject line of the email.
[--subject ] [--omitdesc] [--smtpport ] [--smtpuser ] [--smtppass ] Configure the script notifier service To configure the script notifier service, select the Script Notification Settings tab. This service runs a script when an event configured for the service occurs. You can specify only one script. This service is similar to the Notifiers provided in earlier releases of Matrix Server. Script.
For more information about notifier scripts, see “Using custom notifier scripts” on page 176. View configurations from the command line The following command can be used to view the events configured for one or more notifier services: mx eventnotifier viewconfig [--snmp] [--email] [--script] With no options, the command displays the configured events for all of the notifier services.
To restore event settings from the command line, use this command: mx eventnotifier restoreevents [--snmp] [--email] [--script] Import or export the notifier event settings The import and export features and can used if you will be configuring a new matrix and want to use the notifier event settings that you have configured on the existing matrix. Click Export Definitions to save the current settings to the file of your choice. (The default is your home directory.
• Event details, formatted in XML, are passed to the standard input (stdin) of the script or program. Script requirements For the script to work properly, the following requirements must be met: • The script or program must be accessible from each node in the matrix. It is recommended that an identical copy of the script or program be placed on local storage on each node to ensure that it will always be available. • The script must be able to be executed on each node.
Script notifier test message.
13 Matrix operations on the Applications tab The Applications tab on the Management Console shows all Matrix Server applications and resources configured in the matrix and enables you to manage and monitor them from a single screen. Applications overview An application provides a way to group associated matrix resources (virtual hosts, service monitors, and device monitors) so that they can be treated as a unit.
and resources appear in the rows of the table. (Double-click a resource to see its properties.) The servers on which the resources are configured appear in the columns. You can change the order of the server columns by dragging a column to another location. You can also resize the columns. The cells indicate whether a resource is deployed on a particular server, as well as the current status of the resource. If a cell is empty, the resource is not deployed on that server.
Up/Okay Starting Stopping Down Unknown Virtual hosts and single-active monitors use the following icons to indicate the primary and backups. Multi-active monitors use the same icons but do not include the primary or backup indication. The green arrow changes as shown above when the status of the virtual host or monitor changes. The green dot changes color as the status changes. When an error occurs on a single-active monitor, a red E is added to the green dot.
In the following example, the status for most of the applications is OK because clients are accessing the application through the primary server. However, the status of application 99.11.14.213 is “Warning” because the associated Virtual SQL Server has failed over to the first backup server. Filter the applications display You can use filters to limit the information appearing on the Application tab.
Name: Specify a name for this filter. On the Type tab shown above, select the types of virtual hosts, service monitors, and device monitors that you want to see. If additional HP Software products are installed, devices for those products will appear on the Type tab. Click the State tab to select specific states that you are interested in viewing. (The Applications tab will be updated immediately.
Click OK to close the filter. The filter then appears as a separate tab and will be available to you when you connect to any matrix. To modify an existing filter, select that filter, right-click, and select Edit Filter. To remove a filter, select the filter, right-click, and select Delete Filter.
The following drag and drop operations are allowed. Applications These operations are allowed only for applications that include at most only one virtual host. • Assign an application to a server. Drag the application from the Name column to the empty cell for the server. • Move an application to a new server and make that server the primary for the associated virtual host. Drag the application from the server cell where it is currently primary to the empty server cell.
backups will be reordered as necessary. (The virtual host cannot be removed via drag and drop if it is configured on only one server.) Device monitors • Add a device monitor to a server. Drag the device monitor from the Name column to the empty server cell. If the device monitor is single-active, the server will become a backup for the device monitor. If the device monitor is multi-active, it will become active on the new server.
Enable an application on a server: Left-click on the cell corresponding to the application and server. Then right-click and select Enable. Disable an application on a server: Left-click on the cell corresponding to the application and server. Then right-click and select Disable. Servers To add a resource such as a service or device monitor to a server, left-click in the cell for that server and resource, right-click, and select Add To Resource.
To perform these procedures on the Applications tab, left-click on the cell corresponding to the monitor and server. Then right-click and select the appropriate operation from the menu. For more information about using these procedures on service monitors, see “Add or modify a service monitor” on page 207 and “Other configuration procedures” on page 215.
14 Configure virtual hosts Matrix Server uses virtual hosts to provide failover protection for servers and network applications. Overview A virtual host is a hostname/IP address configured on a set of network interfaces. Each interface must be located on a different server. The first network interface configured is the primary interface for the virtual host. The server providing this interface is the primary server.
can transfer the virtual host to a backup interface on another server and the network traffic will continue. The following example shows the virtual host failover mechanism. Servers A and B provide the http service, which is accessed with the virtual host www.xvz.com. Each server is configured with an http service monitor and a GATEWAY device monitor to ping the router. The failover operation to another network interface has minimal impact on clients.
• When planning the virtual hosts needed for your cluster, first determine the network services that will be available to your clients. Then determine the IP addresses for those services. You will need to create a virtual host for each IP address. • Choose virtual hostnames that differ from your actual server names. Virtual hosts are independent of specific servers, and their names should be independent as well.
Virtual Host: Enter a hostname or an IP address for the virtual host. The IP address must be higher than the system's primary address. Application name: An “application” provides a way to group virtual hosts and related service and device monitors on the Applications tab. All of the Matrix Server resources associated with the application can then be treated as a unit. You can specify a new application name, select an existing application name, or leave this field blank.
As part of configuring a virtual host, you will need to select network interfaces located on the servers that can be used for the virtual host. The interfaces are placed in order: primary, backup #1, backup #2, and so on. The ClusterPulse process considers the “health” of the servers providing those interfaces when determining where to place a virtual host. The status and enablement of each monitor contributes to the health of a server with respect to the virtual host.
To add or update a virtual host from the command line, use the following command. The first network interface specified is the primary interface and the additional interfaces are backups. mx vhost add [--application ] [--policy autofailback|nofailback] [--activitytype single|always] ([ ...][ALL_SERVERS ]) mx vhost update [--policy autofailback|nofailback] ...
mx vhost enable Rehost a virtual host You can use the Rehost option to modify the configuration of a virtual host. For example, you might want to change the primary for the virtual host or reorder the backups. To use this option, select the virtual host, right-click, and then select Rehost. The Virtual Host Rehost window appears. When you make your changes and click OK, you will see a message warning that this action may cause a disruption of service.
Delete a virtual host Select the virtual host to be deleted on either the Servers window or the Virtual Hosts window, right-click, and select Delete. Any service monitors configured on that virtual host are also deleted. To delete a virtual host from the command line, use this command: mx vhost delete ...
also contribute to a server’s health calculation. When a server is completely “healthy,” all of the services associated with the virtual host are up and enabled. When certain events occur on the server where a virtual host is located, the ClusterPulse process will attempt to fail over the virtual host to another server configured for that virtual host. For example, if the server goes down, ClusterPulse will check the health of the other servers and then determine the best location for the virtual host.
4. If the virtual host is currently active on a server and that virtual host has the NOFAILBACK policy, then this active server is moved to the head of the list of preferred servers to be considered. 5. From this list, ClusterPulse tries to find a server with all services up and enabled. If ClusterPulse finds a server meeting these conditions, it will use it, preferring servers earlier in the list of servers configured for the virtual host. 6.
• AUTORECOVER. The virtual host fails over when a monitor probe fails. When the service or device is recovered on the original node, failback occurs according to the virtual host’s failback policy. • NOAUTORECOVER. The virtual host fails over when a monitor probe fails and the monitor is disabled on the original node, preventing automatic failback.
When the health of one of these servers is equal to or greater than the backup server where the virtual host currently resides, the virtual host automatically attempts to fail over to that server. NOFAILBACK NOFAILBACK is intended to minimize failovers. The virtual host remains active on the backup server until a healthier server (it’s health is greater than the backup server) becomes available, at which point the virtual host will fail over to that server.
Virtual Host Policy NOFAILBACK Monitor Probe Severity Behavior when Probe Reports DOWN NOAUTORECOVER Failover occurs and the monitor is disabled on the original server. When the monitor is reenabled, failback occurs. NOFAILOVER Failover does not occur. AUTORECOVER Failover occurs. The virtual host remains on the backup server until a “healthier” server is available. NOAUTORECOVER Failover occurs and monitor is disabled on the original server.
Configure virtual hosts
15 Configure service monitors Service monitors are typically used to monitor a network service such as HTTP or FTP. If a service monitor indicates that a network service is not functioning properly on the primary server, Matrix Server can transfer the network traffic to a backup server that also provides that network service. Overview Before creating a service monitor for a particular service, you will need to configure that service on your servers.
move the virtual host. The virtual host may then remain active on a server with a failed monitored service. Matrix Server will avoid this situation if there is a healthy server configured and available, but sometimes it may not be possible in a multiple-failure scenario. Types of service monitors Matrix Server supports the following service monitors. When configuring a monitor, you will need to supply the following information.
successful, the monitor sends the string QUIT\n to the server. If there are no errors, the service status remains Up. If an error occurs, the status is set to Down. To create a FTP site that can be hosted on a Matrix Server virtual host, use the following configuration. On the Internet Information Services (IIS) 6.0 Manager (click Start > All Programs > Administrative tools), right click the FTP site and select Properties.
the monitor, you will need to indicate whether dependent services of the NT service should also be started and stopped. When the monitored NT service is started, it is provided with four environment variables that are set as follows: • COMPUTERNAME = • _CLUSTER_NETWORK_FQDN_ = . • _CLUSTER_NETWORK_HOSTNAME_ = • _CLUSTER_NETWORK_NAME_ == The monitor probe queries the status of the NT service.
NOTE: Service monitors using TCP port numbers must be assigned distinct port numbers. For example, the built-in FTP service monitor and a generic TCP monitor cannot both attempt to use TCP port 21. You do not need to specify an IP address because a service monitor is associated with a virtual host. Matrix Server uses that virtual host IP address in combination with the TCP port number to create network TCP connections for service monitors.
Virtual Host: The service monitor is assigned to this virtual host. If the virtual host is associated with a Matrix Server application, the service monitor will also be associated with that application. Port: Matrix Server supplies the default port number for the service you select. If your service uses a port other than the default, enter that port number here. NOTE: This field does not appear if you are creating an NTSERVICE monitor.
that the service is functioning correctly. If you do not specify a URL, the probe operation will connect to the standard Web server port and wait for a response. If there is a response, the monitor assumes that the service is operating correctly. • NTSERVICE monitor. If you want services dependent on the monitored NT service to also be started when the NT service is started, click Start dependent services. Then, at the Start Service Retry Count prompt, specify the number of attempts that should be made.
Service monitor policy The Policy tab lets you specify the failover behavior of the service monitor and set its service priority. Timeout and failure severity This setting works with the virtual host policy (either AUTOFAILBACK or NOFAILBACK) to determine what happens when a probe of a monitored service fails. The default policies (AUTOFAILBACK for the virtual host and AUTORECOVERY for the monitor) cause ClusterPulse to fail over the associated virtual host to a backup network interface on another server.
AUTORECOVER. This is the default. The virtual host fails over when a monitor probe fails. When the service is recovered on the original node, failback occurs according to the virtual host’s failback policy. NOAUTORECOVER. The virtual host fails over when a monitor probe fails and the monitor is disabled on the original node, preventing automatic failback. When the monitor is reenabled, failback occurs according to the virtual host’s failback policy.
If the virtual host fails over to a backup node, the monitor instance on the original node becomes inactive and the probe is no longer run on that node. Matrix Server activates the virtual host on the new node, which causes the monitor instance on that node to change status from “standby” to “active.” The monitor then begins probing on that node and the Start script starts the application.
Start script. Runs as a service is becoming active on a server. Stop script. Runs as a service is becoming inactive on a server. When a monitor is instantiated for a service (because the ClusterPulse process is starting or the configuration has changed), Matrix Server selects the best server to make the service active. The Start script is run on this server. On all other servers configured for the monitor, the Stop script is run to ensure that the service is not active.
If you want to reverse this order, preface the Stop script with the prefix [post] on the Scripts tab. Event severity If a Start or Stop script fails or times out, a monitor event is created on the node where the failure or timeout occurred. Configuration errors can also cause this behavior. You can view these events on the PolyServe Management Console and clear them from the Console or command line after you have fixed the problems that caused them.
do not depend on strict ordering of Start and Stop scripts. Assuming that it is safe to run the scripts in parallel (which depends on your application), this setting can also increase the chances of a successful failover because Matrix Server does not have to wait for the failing server to finish running its Stop script.
To enable a service monitor from the command line, use this command: mx service enable ... Clear service monitor errors From the Management Console, select the service monitor where the error occurred, and then, on the node where the error occurred, right-click, and select Clear Last Error. To clear a monitor error from the command line, use this command: mx service clear ...
16 Configure device monitors Matrix Server provides built-in device monitors that can be used to watch local disks, gateway devices, or an NT service, or to monitor access to a SAN disk partition containing a PSFS filesystem. You can also create custom device monitors. Overview A device monitor is configured on one or more servers in the matrix. Depending on the type of monitor, it can be active on all servers on which it is configured, or on only one server.
Activity types for device monitors The activity type specifies where the device monitor can be active. The activity type can be one of the following: • Single-Active. The monitor is active on only one of the selected servers. Upon server failure, the monitor will fail over to an active server unless all associated service and device monitors are down. (“Associated” service and device monitors are those monitors that are associated with the same virtual host as this device monitor.) • Single-Always-Active.
When you configure the monitor, you will be asked for the network address of a router or other gateway probe device. The IP address of the device should be on a different subnet than the servers in the matrix. The device should respond to a ping and should be visible on each server. After the gateway device monitor is configured on a server, it pings the gateway device periodically.
A SHARED_FILESYSTEM device monitor is associated with a specific PSFS filesystem. The monitor probes the filesystem at periodic intervals to determine the filesystem health. The probe attempts to open and read a file located on the filesystem. When you configure the monitor, you can specify the file to be read, or you can let Matrix Server create a file for this purpose. A SHARED_FILESYSTEM device monitor attempts to be active on each server specified in its configuration.
• For a single-active monitor, the active virtual hosts are failed over to a backup server and the monitor becomes active on that server. See the “Device Monitor Activeness Policy,” below, for details about where the monitor will be made active. • For a multi-active monitor, the virtual hosts are failed over to a backup server. The monitor remains active on the same set of servers.
Add or modify a device monitor Select the appropriate option from the PolyServe Management Console: • To add a new device monitor, select a server to be associated with the monitor from the Servers window, right-click, and select Add Device Monitor (or click Device on the toolbar). Then configure the device monitor on the New Device Monitor window. • To update an existing device monitor, select the monitor on the Servers window, right-click, and select Properties.
Device type: Select the appropriate device type (DISK, GATEWAY, NTSERVICE, SHARED_FILESYSTEM, or CUSTOM). See “Overview” on page 217 for a description of these monitors. Timeout and Frequency: These fields are set to the default values for the type of device you have selected. Change them as needed. Additional parameters: Depending on the type of monitor you are creating, you will be asked for an additional parameter. • DISK monitor. Specify the partition to be monitored.
To add a device monitor from the command line, use this command: mx device add --servers ,,... [--type CUSTOM|DISK|GATEWAY| SHARED_FILESYSTEM| NTSERVICE] [--application ] [--timeout ] [--frequency ] [--parameters ] [other-arguments] ... Advanced settings for device monitors You can use the advanced settings to fine-tune device monitors.
The Probe Severity setting works with the virtual host policy (either AUTOFAILBACK or NOFAILBACK) to determine what happens when a monitored device fails. The default policies (AUTOFAILBACK for the virtual host and AUTORECOVERY for the device monitor) cause ClusterPulse to fail over the associated virtual hosts to a backup network interface on another server when the monitor probe fails. When the device is restored, ClusterPulse fails back the virtual hosts to the network interface on the original server.
This option is useful when integrating Matrix Server with a custom application where certain application-specific actions must be taken before failback can occur. For more information on the interaction between the Probe Severity attribute and the virtual host failback policy, see “Virtual hosts and failover protection” on page 23.
When a monitor is instantiated for a device (because the ClusterPulse process is starting or the configuration has changed), Matrix Server selects the best server to make the device active. The Start script is run on this server. The Stop script is run on all other servers configured for the monitor to ensure that the device is not active on those servers. Start scripts must be robust enough to run when the device is already started, without considering this to be an error.
If you want to reverse this order, preface the Stop script with the prefix [post] on the Scripts tab. Event severity If a Start or Stop script fails or times out, a monitor event is created on the node where the failure or timeout occurred. Configuration errors can also cause this behavior. You can view these events on the PolyServe Management Console and clear them from the Console or command line after you have fixed the problems that caused them.
PARALLEL. The strict ordering sequence for Stop and Start scripts is not enforced. The scripts run in parallel across the cluster as a shared device or virtual host is in transition. The PARALLEL configuration can speed up failover time for services and devices that do not depend on strict ordering of Start and Stop scripts.
For example, if you are creating a SHARED_FILEYSTEM monitor to check access to a SAN partition containing a PSFS filesystem, select the virtual hosts associated with applications that access data on that PSFS filesystem. As another example, you might have a DISK monitor for a disk containing Web and FTP files. If the disk fails, you want Matrix Server to fail over the virtual hosts for these services.
• Single-Active. The monitor is active on only one of the selected servers. Upon server failure, the monitor will fail over to an active server unless all associated service and device monitors are down. (“Associated” service and device monitors are those monitors that are associated with the same virtual host as this device monitor.) • Single-Always-Active. The monitor is active on only one of the selected servers.
Delete a device monitor From the Management Console, select the device monitor to be deleted, right-click, and select Delete. To delete a device monitor from the command line, use this command: mx device delete ... Disable a device monitor From the Management Console, select the device monitor to be disabled, right-click, and select Disable. To disable a device monitor from the command line, use this command: mx device disable ...
17 Advanced monitor topics The topics described here provide technical details about Matrix Server monitor operations. This information is not required to use Matrix Server in typical configurations; however, it may be useful if you want to design custom scripts and monitors, to integrate Matrix Server with custom applications, or to diagnose complex configuration problems.
host is configured on the Primary. At i3, the monitor start script begins on the Primary and probing begins on the backups. At i4, probing begins on the Primary. When a failure occurs on the Primary, the virtual host needs to fail over to a backup. Matrix Server now looks for the best location for the virtual host. Because the probe status on the first backup is “down,” Matrix Server selects the second backup, where the probe status is “up.” At i5 in the following example, the probe fails on the Primary.
Custom device monitors A custom device monitor is associated with a list of servers and a list of virtual hosts configured on those servers. A custom device monitor can be active on only one server at a time. On each server, the monitor uses a probe mechanism to determine whether the service is active. The probe mechanism is in one of the following states on each server: Up, Down, Unknown, Timeout. A custom device monitor also has an activity status on each server.
The following example shows the state transitions for a custom device monitor and its associated virtual host. The device monitor uses the default values for autorecover, priority, and serial script ordering, and has a Start and Stop script defined. The virtual host is configured with a Primary network interface and two backup interfaces. There is also a service monitor defined on the virtual host.
Integrate custom applications There are many ways to integrate custom applications with Matrix Server: • Use service monitors or device monitors to monitor the application • Use a predefined monitor or your own user-defined monitor • Use Start, Stop, and Recovery scripts HP PolyServe Matrix Server administration guide 237
Following are some examples of these strategies. Device monitor or service monitor? When deciding whether to monitor your custom application with a service monitor or a device monitor, first determine whether the application provides a service over the network directly to the outside world. If so, you should consider using a service monitor. Because a service monitor is associated with a virtual host, it represents an IP address provided to the outside world to access your services.
NOTE: The custom monitor dialog prompts you for a service monitor name and not a port because you may be writing a monitor for an application that does not provide network services and therefore needs no port. Example of using monitors with applications In this example, we will use service monitors with a custom application called myservice. This application provides some facilities to clients who connect to port 2468 and speak a protocol.
Script environment variables When you specify a script for a service or device monitor, Matrix Server sets the following environment variables for that script. MX_METHOD=(START|STOP|RECOVER|PROBE) The type of script (Start, Stop, Recovery, or probe). MX_ACTIVE_STATE=(ACTIVE|INACTIVE) Whether the script is being run on an active instance of the object. For example, if a service-monitor script is being run on the server that currently has the active virtual host, its state will be ACTIVE.
18 SAN maintenance The following information and procedures apply to SANs used with Matrix Server. Server access to the SAN When a server is either added to the cluster or rebooted, Matrix Server needs to take some administrative actions to make the server a full member of the cluster with access to the shared filesystems on the SAN. During this time, the PolyServe Management Console reports the message “Joining cluster” for the server.
Membership partitions Matrix Server uses a set of membership partitions to control access to the SAN and to store the mxds datastore and the device naming database. (The mxds datastore contains cluster configuration information and the device naming database includes the global device names for SAN disks imported into the cluster.) Typically the membership partitions are created when you install Matrix Server. You can add, replace, or repair membership partitions as necessary.
Any of these messages can appear in the “SANlock State” column. held by SDMP administrator The SANlock was most recently held by the SDMP administrator of the cluster to which the host where mxsanlk was run belongs. trying to lock, last held by host X.X.X.X The SANlock was most recently held by host X.X.X.X and may still be held by that host. The host on which mxsanlk was run is trying to acquire the SANlock. cannot access The host on which mxsanlk was run is unable to access the SANlock.
lock is corrupt, will repair This transitional state occurs after the SDMP has detected that the SANlock has been corrupted but before it has repaired the SANlock. trying to lock (lock is corrupt, will repair) The host on which mxsanlk was run is trying to acquire the SANlock. The SANlock was corrupted but will be repaired. locked (lock is corrupt, will repair) The host on which mxsanlk was run holds the lock. The SANlock was corrupted but will be repaired.
• The mprepair command can be used to display information about membership partitions and to perform all operations affecting membership partitions. Matrix Server must be offline on all nodes when this command is used. Online operations When Matrix Server is running, the Add, Repair, and Replace options on the Storage Settings tab and the mx config mp set and repair commands can be used only in the following circumstances: • A disk containing a membership partition is out-of-service.
Membership partition states The Storage Settings tab reports the state of each membership partition. The possible states are: • OK. The membership partition is functioning correctly. • FENCED. The server has been fenced and cannot access the SAN. Start Matrix Server if it is not running or reboot the server. • NOT_FOUND. Matrix Server cannot find the device containing the membership partition. Check the device for hardware problems.
• CID_MISMATCH. The Cluster-ID is out-of-sync among the membership partitions and must be reset. Repair a membership partition When Matrix Server is started on a server, it checks the status of the membership partitions and reports any invalid partitions. If a single membership partition is found to be corrupt, you can repair that partition while Matrix Server is running. (If more than one partition is corrupt, Matrix Server must be offline when the partitions are repaired.
When you select a partition and click Replace, you will see a confirmation message describing the replace operation. A message also appears when the replace operation is complete. NOTE: The Replace option on the Storage Settings tab is available only when Matrix Server is running. Add membership partitions When you initially configure the matrix, you can add either one or three membership partitions, as described in the HP PolyServe Matrix Server installation guide.
When you have completed your selections, click Apply at the bottom of the Configure Matrix window. Remove membership partitions Before using this option, verify that Matrix Server is offline on all nodes in the cluster. When you click Remove, a dialog listing the current membership partitions is displayed. Select the partition that you want to delete. When you have completed the dialog, click Apply at the bottom of the Configure Matrix window.
--force Force membership partition replacements to occur even if the partitions to be replaced cannot be invalidated. --reuse Allow disks that contain existing volume information to be reused. (The existing data is destroyed.) Repair a membership partition This command resilvers the specified membership partition. mx config mp repair [--reuse] The --reuse option allows disks that contain existing volume information to be reused. (The existing data is destroyed.
to the SAN, it is important that each server in the cluster have the same view of where the partitions are located. However, it is possible for these views to get out of sync if changes to the membership partition configuration are made when all servers are not accessible. Therefore, before a server can negotiate for control of the SAN, it must have a local MP list that matches the membership lists stored on each of the membership partitions, and each membership partition must be in the Active state.
CORRUPT. The partition is not valid. You will need to resilver the partition. This step copies the membership data from a valid membership partition to the corrupted partition. NOTE: The membership partition may have become corrupt because it was used by another application. Before resilvering, verify that it is okay to overwrite any existing data on the partition. RESILVER. The membership partition is valid but its MP list does not match the server’s local MP list.
Export configuration changes When you change the membership partition configuration with mprepair, it updates the membership list on the local server. It also updates the lists on the disks containing the membership partitions specified in the local MP file. After making changes with mprepair, you will need to export the configuration to the other servers in the cluster. To do this, start Matrix Server on the server where you ran mprepair and then connect to the PolyServe Management Console.
Host ID: 10.10.30.4 fencestatus=0 SAN Loc:10:00:00:00:c9:2d:27:7d::0 Host ID: 10.10.30.3 fencestatus=0 SAN Loc:10:00:00:00:c9:2d:27:78::0 idstatus=0 (switch=fcswitch5) idstatus=0 (switch=fcswitch5) Search the SAN for membership partitions. To search the SAN for all partitions that appear to be membership partitions, enter this command: mprepair --search_mps The output includes each membership partition found by the search and specifies whether it is active or inactive.
indicates the membership partition to be resilvered. is the UUID for the device and is the number of the partition on the device. The membership partition is resilvered from a known valid membership partition. Without the --force option, the command will fail if mprepair cannot find two valid membership partitions to use as a source for the resilver operation. The --force option overrides this requirement and causes the resilver operation to take place.
1. Stop Matrix Server: net stop matrixserver 2. Disable the Matrix Server service: mxservice -uninstall 3. Remove the psd driver from the driver stack: psdcoinst -uninstall 4. Reboot the server. The server will come up without Matrix Server and the psd driver. 5. Make the necessary change to the HBA or driver. Reboot the system if the HBA installation prompts you to do so. 6. (Optional.) Verify that the new HBA is supported: mxcheck 7. Update the registry: regedit /s psd.reg 8.
2. Open the PolyServe Management Console Connect window, enter the login credentials for a server in the matrix, and click Configure button on the Connect window. 3. On the Configure Matrix window, select the Fencing tab and then configure the appropriate fencing method. See the HP PolyServe Matrix Server installation guide for more information about this tab. 4.
Reboot ASAP as it stopped cluster network communication at date/time but attempts to exclude it from the SAN were unsuccessful! Rebooting it will Alternatively, if confirmed to have to restore normal allow normal cluster operation to continue. the server cannot be rebooted, but can be no access to the SAN, run 'mx server markdown ' cluster operation. The following example shows the operation of the command: $ mx server markdown 99.10.20.
Operator error may have caused filesystem corruption! 99.10.20.4 returned to the cluster without being rebooted, even though the operator verified that 99.10.20.4 was down. It is recommended that all filesystems that 99.10.20.4 had mounted be checked for corruption. This alert will display for 48 hours.
Membership partition timeout The membership partition timeout should be increased to 120 seconds (120000ms). This value is set in the registry. Complete the following steps: Start regedit and navigate to the following registry key: [HKEY_LOCAL_MACHINE\SOFTWARE\PolyServe\MatrixServer\mxservice\Started Processes\sanpulse] Double-click ProgramArguments and, on the Edit String dialog, enter -o sdmp_io_timeout=120000 as the Value data.
psd_timeout 180 Restart the nodes After changing the timeouts on all nodes, stop and restart Matrix Server on each node. The stop/restart can be performed on one node at a time. After the storage capacity upgrade When the disk upgrade is complete, restore the original scl.conf file and remove the registry value that you added. After this changes are made on all nodes, stop and restart Matrix Server on each node. The stop/restart can be performed on one node at a time.
Consult your switch documentation for the appropriate replacement procedure, keeping in mind that the above requirements must be met. However, if this documentation is not available, the following procedures describe a method to replace a switch. Replace a Brocade FC switch To replace a Brocade Fibre Channel switch, complete these steps: 1. Stop Matrix Server on any servers that are connected only to the original switch. 2. If possible, save the configuration information on the original switch.
14. Set the Ethernet IP address on the new switch to the IP address of the original switch. Use the ipAddrSet command. If you had connected to the switch over the Ethernet interface, the session will be disconnected and you will need to log back into the switch with the IP address you just configured. 15. Enable the switch using the switchEnable command. The new switch should now connect to the rest of the fabric. Verify that the HBAs on the servers log into the new switch in the expected locations.
5. Make the switch operating mode and domain ID acceptable to the original fabric. This can be done either by consulting the fabric or by taking the values from the data saved in step 2. This procedure might include changing the default zone setting as directed by EWS when changing interoperation mode. Any existing zone configuration on the new switch should be removed to allow the fabric to properly communicate current zoning when the switch joins the fabric. 6.
19 Other cluster maintenance This chapter describes how to perform the following activities: • Collect Matrix Server log files with mxcollect • • • • Check the server configuration Disable a server for maintenance Troubleshoot a cluster Troubleshoot service and device monitors Collect log files with mxcollect The mxcollect utility collects error event logs that can be useful for diagnosing technical issues with Matrix Server.
Upload mxcollect files to HP Support After running mxcollect, you can upload the resulting files to HP Support. Contact HP Support for more information. Check the server configuration The Matrix Server mxcheck utility can be used to verify that a server meets the configuration requirements for Matrix Server. The utility is run automatically whenever Matrix Server is booted. You can also run mxcheck manually.
Troubleshoot cluster problems Matrix Server fails to start If the Matrix Server service fails to start on a server, check that the domain name of the server is configured in the DNS suffix list. The DNS suffix list is configurable via the Advanced TCP/IP Properties. The server status is “Down” If a server is running but Matrix Server shows it as down, follow these diagnostic steps: 1. Verify that the server is connected to the network. 2.
A virtual host is inaccessible If a site is inaccessible but Matrix Server indicates that it is okay, verify that the service is running and that the actual data exists on the server providing the network interface currently used by the virtual host. Matrix Server exits immediately If the ClusterPulse process exits immediately on starting, first determine whether the process is running.
“Down” status The “Down” status indicates that the monitor finished its probe but it did not complete successfully. Depending on the monitor type (such as HTTP or SMTP), the service monitor probe may involve more than being able to connect to the network service. For many built-in service monitors, Matrix Server may conclude that the monitor is down even if the TCP connection succeeds.
CONFIG_ERROR. A script must exist and be executable by root. This condition is checked for probe, Start, Stop, and Recovery scripts each time an attempt is made to execute the script. SCRIPT_SYSERR. The monitor_agent tried to fork a process to execute the script but the fork system call failed. This condition can occur when Matrix Server is trying to execute probe, Start, Stop, or Recovery scripts. START_TIMEOUT. A Start script was executed but it did not complete within the specified timeout period.
NOTE: An error on a monitor may still be indicated after correcting the problem with the Start, Stop, Recovery, or probe script. Errors can be cleared only with the Management Console or the appropriate mx command. An error will not be automatically cleared by the ClusterPulse process. The date and time of the script event are available in the cluster log file.
“Transitioning” activity The “Transitioning” activity indicates that the monitor state is on its way to becoming ACTIVE or INACTIVE (or starting or stopping, if a Start or Stop script is present).
A Matrix Server Management Console icons The PolyServe Management Console uses the following icons. Matrix Server entities The following icons represent the Matrix Server entities. If an entity is disabled, the color of the icon becomes less intense.
Additional icons are added to the entity icon to indicate the status of the entity. The following example shows the status icons for the server entity. The status icons are the same for all entities and have the following meanings. Icon Status No icon Up/Okay Starting Stopping Down Unknown Monitor probe status The following icons indicate the status of service monitor and device monitor probes. If the monitor is disabled, the color of the icons is less intense.
Management Console alerts The PolyServe Management Console uses the following icons to indicate the severity of the messages that appear in the Alert window. Critical. Error. Warning. Informational.
Matrix Server Management Console icons
B Support and other resources HP technical support For worldwide technical support information, see the HP support website: http://www.hp.
• http://www.hp.
Index Symbols TCP service monitor, A accounts assign to role, 154 administrative network allow, discourage or exclude traffic, 68 defined, 20 failover, 67 network topology, 66 requirements for, 65 select, 65 alerts Alerts pane on Management Console, 44 display on Management Console, 44 icons shown on Management Console, 275 Applications tab drag and drop operations, 184 filter applications, 182 format, 182 icons, 180 manage application monitors, 187 manage applications, 184 menu operations, 186 modify dis
bookmarks add, 37 clear password information, 38 create, 36 delete, 37 manage, 36 rename, 37 reorder, 37 set default, 37 synchronize server bookmarks, 38 C checklist, quick start, 15 cluster features supported on server, 60 cluster configuration SAN restrictions, 30 cluster configurations Fibre Channel, 25 iSCSI, 27 Cluster-ID, reset, 255 ClusterPulse defined, 21 failover, 196 configuration virtual host , back up, 49 device monitor, 222 network interface, 65 PSFS filesystems, SAN disks, 71 server, 55 servi
dynamic volumes concatenated, 84 convert from basic, 93 create, 85 defined; , delete, 91 deport, 95 display unimported volumes, 81 extend, 89 guidelines, 84 import, 96 names, 84 properties, 87 recovery of, 95 recreate, 92 striped defined, 83 stripe state, 88 stripeset, 89 subdevices, 83 E email notifier service, configure, 172 error messages PSFS filesystem, 241 errors service monitor clear from Console, 216 event log audit trail, 161 Event Notification Control Panel, 168 event notifier services configure
filesystem, PSFS access, 101 create, 107 create with mx fs command, 116 extend, 125 features, 101 features, configured, 126 journal, 102 properties; filesystem, PSFS relabel, 124 quotas, 135 quotas, enable, 110 quotas.
mprepair clear host registry, 255 display membership partitions, 253 export changes to other nodes, 253 inactivate a membership partition, 252 inactivate membership partition, 255 membership partition file, 250 options, 253 repair a membership partition, 254 reset Cluster-ID, 255 resilver membership partitions, 254 search SAN for membership partitions, 254 view membership partition status, 251 mx server markdown command, 257 mxcheck utility, 266 mxcollect utility, 265 mxds datastore backup and restore, 49 m
Role-Based Security roles modify, account commands, 160 Control Panel, 150 overview, 149 resources, 151 rights, allow or deny, 153 roles add, 151 assign accounts, 154 assign rights manually, 153 assign rights with a template, 153 delete, 160 enable or disable, 159 export or import, 158 rename, 159 view from command line, 160 view rights, 156 S SAN SAN ownership locks, 242 server access, 241 SAN (storage area network), 21 sandiskinfo utility disk options, 79 display unimported volumes, 81 dynamic volume opt
service monitor configuration advanced settings probe type, scripts, service priority, timeout and failure severity, add or update, 207 delete, 215 disable, 215 enable, 215 service monitor types custom, 207 FTP, 204 HTTP, 205 HTTPS, 205 NTSERVICE, 205 SMTP, 206 TCP, 206 shared disks import, 73 SHARED_FILESYSTEM device monitor, 219 SMDS (Shared Memory Data Store), 23 SMTP service monitor, 206 snapclone, 142 snapshots assign drive letter/mount path, 144 create from command line, 145 create from Management Con
volumes dynamic volume recovery, 95 import, 96 importable, 98 unimportable, 98 unimported, 97 volumes, basic or dynamic, 83 W websites HP , HP Subscriber's Choice for Business, 277 286
