Veritas Storage Foundation™ 5.
Legal Notices © Copyright 2005-2008 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents 1. Technical Overview Overview of Cluster File System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Cluster File System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Cluster File System Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Group Lock Manager . . . . . . . . . . . . . . .
Contents Distributing the Load on a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Snapshots for Cluster File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cluster Snapshot Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Preface The Veritas Storage Foundation™ 5.0 Cluster File System Administration Guide Extracts for the HP Serviceguard Storage Management Suite contains information extracted from the Veritas Storage Foundation™ Cluster File System Administration Guide - 5.0 - HP-UX, which has been modified to support the HP Serviceguard Storage Management Suite bundles that include the Veritas Storage Foundation™ Cluster File System by Symantec and the Veritas Storage Foundation™ Cluster Volume Manager by Symantec.
1 Technical Overview This chapter includes the following topics: • “Overview of Cluster File System Architecture” on page 8 • “VxFS Functionality on Cluster File Systems” on page 9 • “Benefits and Applications” on page 12 HP Serviceguard Storage Management Suite (SG SMS) bundles provide several options for clustering and storage. The information in this document applies to the SG SMS bundles that include the Veritas Storage Foundation™ 5.
Technical Overview Overview of Cluster File System Architecture Overview of Cluster File System Architecture CFS allows clustered servers to mount and use the same file system simultaneously, as if all applications using the file system are running on the same server. CVM makes logical volumes and raw device applications accessible throughout a cluster. Cluster File System Design Beginning with version 5.
Technical Overview VxFS Functionality on Cluster File Systems VxFS Functionality on Cluster File Systems The HP Serviceguard Storage Management Suite uses the Veritas File System (VxFS).
Technical Overview VxFS Functionality on Cluster File Systems Table 1-1 CFS Supported Features (Continued) Features and Commands Supported on CFS Disk Layout Versions CFS supports only disk layout Version 6 and Version 7. Cluster mounted file systems can be upgraded. A local mounted file system can be upgraded, unmounted, and mounted again, as part of a cluster. Use the fstyp -v special_device command to ascertain the disk layout version of a VxFS file system.
Technical Overview VxFS Functionality on Cluster File Systems Table 1-2 CFS Unsupported Features (Continued) Features and Commands Not Supported on CFS Chapter 1 Commands that Depend on File Access Times File access times may appear different across nodes because the atime file attribute is not closely synchronized in a cluster file system. Utilities that depend on checking access times may not function reliably. Nested Mounts HP Serviceguard does not support CFS nested mounts.
Technical Overview Benefits and Applications Benefits and Applications The following sections describe CFS benefits and some applications. Advantages To Using CFS CFS simplifies or eliminates system administration tasks resulting from hardware limitations: • The CFS single file system image administrative model simplifies administration by allowing all file system management operations, resizing, and reorganization (defragmentation) to be performed from any node.
Technical Overview Benefits and Applications When To Use CFS You should use CFS for any application that requires file sharing, such as for home directories, web pages, and for cluster-ready applications. CFS can also be used when you want highly available standby data in predominantly read-only environments, or when you do not want to rely on NFS for file sharing. Almost all applications can benefit from CFS. Applications that are not “cluster-aware” can operate and access data from anywhere in a cluster.
Technical Overview Benefits and Applications 14 Chapter 1
2 Cluster File System Architecture This chapter includes the following topics: Chapter 2 • “Role of Component Products” on page 16 • “About CFS” on page 17 • “About Veritas Cluster Volume Manager Functionality” on page 21 15
Cluster File System Architecture Role of Component Products Role of Component Products The HP Serviceguard Storage Management Suite bundles that include CFS also include the Veritas™ Volume Manager by Symantec (VxVM) and it's cluster component, the Veritas Storage Foundation™ Cluster Volume Manager by Symantec (CVM). The following sections introduce cluster communication, membership ports, and CVM functionality.
Cluster File System Architecture About CFS About CFS If the CFS primary node fails, the remaining cluster nodes elect a new primary node. The new primary node reads the file system intent log and completes any metadata updates that were in process at the time of the failure. Application I/O from other nodes may block during this process and cause a delay. When the file system becomes consistent again, application processing resumes.
Cluster File System Architecture About CFS Table 2-1 Primary and Secondary Mount Options Secondary: ro Secondary: rw Secondary: ro, crw Primary: rw X X Primary: ro, crw X X Primary: ro X Mounting the primary node with only the -o cluster,ro option prevents the secondary nodes from mounting in the read-write mode. Note that mounting the primary node with the rw option implies read-write capability throughout the cluster.
Cluster File System Architecture About CFS Cluster File System Backup Strategies The same backup strategies used for standard VxFS can be used with CFS, because the APIs and commands for accessing the namespace are the same. File system checkpoints provide an on-disk, point-in-time copy of the file system.
Cluster File System Architecture About CFS Synchronizing Time on Cluster File Systems CFS requires that the system clocks on all nodes are synchronized using some external component such as the Network Time Protocol (NTP) daemon. If the nodes are not in sync, timestamps for creation (ctime) and modification (mtime) may not be consistent with the sequence in which operations actually happened.
Cluster File System Architecture About Veritas Cluster Volume Manager Functionality About Veritas Cluster Volume Manager Functionality CVM supports up to 8 nodes in a cluster to simultaneously access and manage a set of disks under VxVM control (VM disks). The same logical view of the disk configuration and any changes are available on each node. When the cluster functionality is enabled, all cluster nodes can share VxVM objects.
Cluster File System Architecture About Veritas Cluster Volume Manager Functionality To the cmvx daemon, all nodes are the same. VxVM objects configured within shared disk groups can potentially be accessed by all nodes that join the cluster. However, the cluster functionality of VxVM requires one node to act as the master node; all other nodes in the cluster are slave nodes. Any node is capable of being the master node, which is responsible for coordinating certain VxVM activities.
Cluster File System Architecture About Veritas Cluster Volume Manager Functionality Whether all members of the cluster have simultaneous read and write access to a cluster-shareable disk group depends on its activation mode setting as described in Table 2-2, “Activation Modes for Shared Disk Groups.” The data contained in a cluster-shareable disk group is available as long as at least one node is active in the cluster. The failure of a cluster node does not affect access by the remaining active nodes.
Cluster File System Architecture About Veritas Cluster Volume Manager Functionality The following table summarizes the allowed and conflicting activation modes for shared disk groups: Table 2-3 Allowed and conflicting activation modes Disk group activated in cluster as: Attempt to activate disk group on another node as: exclusivewrite readonly sharedread sharedwrite exclusivewrite Fails Fails Succeeds Fails readonly Fails Succeeds Succeeds Fails sharedread Succeeds Succeeds Succeeds Suc
Cluster File System Architecture About Veritas Cluster Volume Manager Functionality contact a disk, it should contact another node to check on the disk’s status. If the disk fails, no node can access it and the nodes can agree to detach the disk. If the disk does not fail, but rather the access paths from some of the nodes fail, the nodes cannot agree on the status of the disk.
Cluster File System Architecture About Veritas Cluster Volume Manager Functionality 26 Chapter 2
3 Cluster File System Administration The following HP Serviceguard Storage Management Suite bundles include the Veritas Storage Foundation™ 5.0 Cluster File System (CFS) and Cluster Volume Manager (CVM) by Symantec: • Bundles T2775CA, T2776CA, and T2777CA of the HP Serviceguard Storage Management Suite version A.02.00 for HP-UX 11i v2 • Mission Critical Operating Environment (MCOE) bundles T2795CA, T2796CA, and T2797CA of the HP Serviceguard Storage Management Suite version A.02.
Cluster File System Administration Topics in this chapter include: 28 • “Cluster Messaging - GAB” on page 29 • “Cluster Communication - LLT” on page 30 • “Volume Manager Cluster Functionality Overview” on page 31 • “Cluster File System Overview” on page 32 • “Cluster File System Administration” on page 34 • “Snapshots for Cluster File Systems” on page 37 Chapter 3
Cluster File System Administration Cluster Messaging - GAB Cluster Messaging - GAB GAB provides membership and messaging services for clusters and for groups of applications running on a cluster. The GAB membership service provides orderly startup and shutdown of clusters. GAB is automatically configured initially when Serviceguard is installed, and GAB is also automatically configured each time the CFS package is started. For more information, see the gabconfig(1m) manual page.
Cluster File System Administration Cluster Communication - LLT Cluster Communication - LLT LLT provides kernel-to-kernel communications and monitors network communications. The LLT files /etc/llthosts and /etc/llttab can be configured to set system IDs within a cluster, set cluster IDs for multiple clusters, and tune network parameters such as heartbeat frequency. LLT is implemented so events such as state changes are reflected quickly, which in turn enables fast responses.
Cluster File System Administration Volume Manager Cluster Functionality Overview Volume Manager Cluster Functionality Overview The Veritas™ Cluster Volume Manager (CVM) component of the Veritas™ Volume Manager by Symantec (VxVM) allows multiple hosts to concurrently access and manage a given set of logical devices under VxVM control. A VxVM cluster is a set of hosts sharing a set of devices; each host is a node in the cluster. The nodes are connected across a network.
Cluster File System Administration Cluster File System Overview Cluster File System Overview With respect to each shared file system, a cluster includes one primary node, and up to 7 secondary nodes. The primary and secondary designation of nodes is specific to each file system, not the hardware. It is possible for the same cluster node be primary for one shared file system, while at the same time it is secondary for another shared file system.
Cluster File System Administration Cluster File System Overview Asymmetric Mounts Asymmetric mounts allow shared file systems to be mounted with different read/write capabilities. One node in the cluster can mount read-write, while other nodes mount read-only. You can specify the cluster read-write (crw) option when you first mount the file system. The first column in the following table shows the mode in which the primary is mounted.
Cluster File System Administration Cluster File System Administration Cluster File System Administration This section describes some of the major aspects of cluster file system administration and the ways that it differs from single-host VxFS administration.
Cluster File System Administration Cluster File System Administration Any HP-UX command that can write to a raw device must be used carefully in a shared environment to prevent data from being corrupted. For shared VxVM volumes, CFS provides protection by reserving the volumes in a cluster to prevent VxFS commands, such as fsck and mkfs, from inadvertently damaging a mounted file system from another node in a cluster.
Cluster File System Administration Cluster File System Administration command can define the order in which primaryship is assumed if the current primary node fails. After setup, the policy is in effect as long as one or more nodes in the cluster have the file system mounted.
Cluster File System Administration Snapshots for Cluster File Systems Snapshots for Cluster File Systems A snapshot provides a consistent point-in-time image of a VxFS file system. A snapshot can be accessed as a read-only mounted file system to perform efficient online backups. Snapshots implement copy-on-write semantics that incrementally copy data blocks when they are overwritten on the “snapped” file system.
Cluster File System Administration Snapshots for Cluster File Systems version 7 layout 104857600 sectors, 52428800 blocks of size 1024, log size 16384 blocks unlimited inodes, largefiles not supported 52428800 data blocks, 52399152 free data blocks 1600 allocation units of 32768 blocks, 32768 data blocks 2.
4 Cluster Volume Manager Administration A cluster consists of a number of hosts or nodes that share a set of disks. The main benefits of cluster configurations are: • Availability—If one node fails, the other nodes can still access the shared disks. When configured with suitable software, mission-critical applications can continue running by transferring their execution to a standby node in the cluster.
Cluster Volume Manager Administration Overview of Cluster Volume Management Overview of Cluster Volume Management Tightly coupled cluster systems have become increasingly popular in enterprise-scale mission-critical data processing. The main advantage clusters offer is protection against hardware failure. If the master node fails or otherwise becomes unavailable, applications can continue to run by transferring their execution to standby nodes in the cluster.
Cluster Volume Manager Administration Overview of Cluster Volume Management Figure 4-1 Example of a 4-Node Cluster Redundant Private Network Node 0 Master Node 1 Slave Node 2 Slave Node 3 Slave Redundant Fibre Channel Connectivity Cluster-Shareable Disks Cluster-Shareable Disk Groups To the cmvx daemon, all nodes are the same. VxVM objects configured within shared disk groups can potentially be accessed by all nodes that join the cluster.
Cluster Volume Manager Administration Overview of Cluster Volume Management In a cluster, most disk groups are shared. Disks in a shared disk group are accessible from all nodes in a cluster, allowing applications on multiple cluster nodes to simultaneously access the same disk. A volume in a shared disk group can be simultaneously accessed by more than one node in the cluster, subject to licensing and disk group activation mode restrictions.
Cluster Volume Manager Administration Overview of Cluster Volume Management The activation mode of a disk group controls volume I/O from different nodes in the cluster. It is not possible to activate a disk group on a cluster node, if it is activated in a conflicting mode on another node in the cluster. Table 4-1 Activation Modes for Shared Disk Groups Activation Mode Description exclusive-write (ew) The node has exclusive write access to the disk group.
Cluster Volume Manager Administration Overview of Cluster Volume Management • exclusive-write • read-only • shared-read • shared-write • off To view the activation mode setting for each of your shared disk groups, enter: # cfsdgadm display When a shared disk group is created or imported, it is activated in the specified mode. When a node joins the cluster, all shared disk groups accessible from the node are activated in the specified mode.
Cluster Volume Manager Administration Overview of Cluster Volume Management • leave—The master node panics instead of disabling the disk group if a log update fails for a user or kernel initiated transaction (including first write or final close). If the failure to access the log copies is global, all nodes panic in turn as they become the master node.
Cluster Volume Manager Administration Recovery in a CVM Environment Recovery in a CVM Environment In a Cluster Volume Manager environment, when one set of mirrored disks fails and gets replaced, vxreattach fails to recognize the replaced disk.
A Troubleshooting This appendix contains the following topics: Appendix A • “Installation Issues” on page 48 • “Cluster File System Problems” on page 50 47
Troubleshooting Installation Issues Installation Issues If you encounter any issues installing CFS, refer to the following paragraphs for typical problems and their solutions. You can also refer to the HP Serviceguard Storage Management Suite Release Notes and the HP Serviceguard Storage Management Suite Read Before Installing document, if you encounter an issue that is not included here. Incorrect Permissions for Root on Remote System The permissions are inappropriate.
Troubleshooting Installation Issues If a system cannot access the software source depot, either swagentd is not running on the target system or the swlist command cannot see the source depot. Correct /etc/{hosts, nsswitch.conf} and continue from here Continue? [Y/N] : Suggested solutions: check that swagentd is running. Check whether there is an entry for the target system in /etc/hosts. If there is no entry, then ensure the hosts file is not the primary lookup for the “hosts” entry in the /etc/nsswitch.
Troubleshooting Cluster File System Problems Cluster File System Problems If there is a device failure or controller failure to a device, the file system may become disabled cluster-wide. To address this problem, unmount the file system on all of the nodes, then run a full fsck. When the file system check completes, mount all nodes again. When the file system check completes, use cfsmount to mount the file system cluster-wide.
Troubleshooting Cluster File System Problems mount: slow The node may be in the process of joining the cluster.
Troubleshooting Cluster File System Problems 52 Appendix A
