Veritas Volume Manager 5.1 SP1 Troubleshooting Guide HP-UX 11i v3 HP Part Number: 5900-1507 Published: April 2011 Edition: 1.
© Copyright 2011 Hewlett-Packard Development Company, L.P. Legal Notices Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Technical Support ............................................................................................... 4 Chapter 1 Recovering from hardware failure ..................................... 9 About recovery from hardware failure ............................................... 9 Listing unstartable volumes ........................................................... 10 Displaying volume and plex states ................................................... 11 The plex state cycle ..............
8 Contents Recovering from copy-on-write failure ............................................. 39 Recovering from I/O errors during resynchronization ........................ 40 Recovering from I/O failure on a DCO volume ................................... 40 Chapter 3 Recovering from boot disk failure ................................... 41 VxVM and boot disk failure ............................................................ Recovering a system by booting from a VxVM root disk mirror .............
Chapter 1 Recovering from hardware failure This chapter includes the following topics: ■ About recovery from hardware failure ■ Listing unstartable volumes ■ Displaying volume and plex states ■ The plex state cycle ■ Recovering an unstartable mirrored volume ■ Recovering an unstartable volume with a disabled plex in the RECOVER state ■ Forcibly restarting a disabled volume ■ Clearing the failing flag on a disk ■ Reattaching failed disks ■ Failures on RAID-5 volumes ■ Recovering from a
10 Recovering from hardware failure Listing unstartable volumes If a volume has a disk I/O failure (for example, because the disk has an uncorrectable error), VxVM can detach the plex involved in the failure. I/O stops on that plex but continues on the remaining plexes of the volume. If a disk fails completely, VxVM can detach the disk from its disk group. All plexes on the disk are disabled. If there are any unmirrored volumes on a disk when it is detached, those volumes are also disabled.
Recovering from hardware failure Displaying volume and plex states Displaying volume and plex states To display detailed information about the configuration of a volume including its state and the states of its plexes, use the vxprint command. To display volume and plex states ◆ Type the following command: # vxprint [-g diskgroup] -hvt [volume ...
12 Recovering from hardware failure The plex state cycle Main plex state cycle Figure 1-1 Start up (vxvol start) PS: CLEAN PS: ACTIVE PKS: DISABLED PKS: ENABLED Shut down (vxvol stop) PS = plex state PKS = plex kernel state For more information about plex states, see the Veritas Volume Manager Administrator’s Guide. At system startup, volumes are started automatically and the vxvol start task makes all CLEAN plexes ACTIVE. At shutdown, the vxvol stop task marks all ACTIVE plexes CLEAN.
Recovering from hardware failure The plex state cycle 13 Additional plex state transitions Figure 1-2 Create plex PS: EMPTY PKS: DISABLED Initialize plex (vxvol init clean) Start up (vxvol start) PS: CLEAN PKS: DISABLED PS: ACTIVE PKS: DISABLED After crash and reboot (vxvol start) Recover data (vxvol resync) Take plex offline (vxmend off) PS: ACTIVE PKS: ENABLED PS: OFFLINE PKS: DISABLED Resync data (vxplex att) Shut down (vxvol stop) Put plex online (vxmend on) Uncorrectable I/O failure P
14 Recovering from hardware failure Recovering an unstartable mirrored volume Recovering an unstartable mirrored volume A system crash or an I/O error can corrupt one or more plexes of a mirrored volume and leave no plex CLEAN or ACTIVE. You can mark one of the plexes CLEAN and instruct the system to use that plex as the source for reviving the others.
Recovering from hardware failure Recovering an unstartable volume with a disabled plex in the RECOVER state Recovering an unstartable volume with a disabled plex in the RECOVER state A plex is shown in the RECOVER state if its contents are out-of-date with respect to the volume. This can happen if a disk containing one or more of the plex’s subdisks has been replaced or reattached. If a plex is shown as being in this state, it can be recovered by using the vxmend and vxvol commands.
16 Recovering from hardware failure Clearing the failing flag on a disk replacing the failed disk. Any volumes that are listed as Unstartable must be restarted using the vxvol command before restoring their contents from a backup. To forcibly restart a disabled volume ◆ Type the following command: # vxvol [-g diskgroup] -o bg -f start volume The -f option forcibly restarts the volume, and the -o bg option resynchronizes its plexes as a background task.
Recovering from hardware failure Reattaching failed disks To clear the failing flag on a disk 1 Use the vxdisk list command to find out which disks are failing: # vxdisk list DEVICE c1t1d0 c1t1d0 c1t1d0 . . .
18 Recovering from hardware failure Failures on RAID-5 volumes To reattach a failed disk 1 Use the vxdisk list command to see which disks have failed, as shown in the following example: # vxdisk list DEVICE c1t1d0 c1t2d0 - 2 TYPE auto:simple auto:simple - DISK mydg01 mydg02 mydg03 mydg04 GROUP mydg mydg mydg mydg STATUS online online failed was: c1t3d0 failed was: c1t4d0 Once the fault has been corrected, the disks can be reattached by using the following command to rescan the device list: # /usr/
Recovering from hardware failure Failures on RAID-5 volumes synchronization occurs because the status of writes that were outstanding at the time of the failure cannot be determined. If a loss of sync occurs while a RAID-5 volume is being accessed, the volume is described as having stale parity. The parity must then be reconstructed by reading all the non-parity columns within each stripe, recalculating the parity, and writing out the parity stripe unit in the stripe.
20 Recovering from hardware failure Failures on RAID-5 volumes ...
Recovering from hardware failure Failures on RAID-5 volumes Default startup recovery process for RAID-5 VxVM may need to perform several operations to restore fully the contents of a RAID-5 volume and make it usable. Whenever a volume is started, any RAID-5 log plexes are zeroed before the volume is started. This prevents random data from being interpreted as a log entry and corrupting the volume contents.
22 Recovering from hardware failure Failures on RAID-5 volumes Parity resynchronization and stale subdisk recovery are typically performed when the RAID-5 volume is started, or shortly after the system boots. They can also be performed by running the vxrecover command. See “Unstartable RAID-5 volumes” on page 25. If hot-relocation is enabled at the time of a disk failure, system administrator intervention is not required unless no suitable disk space is available for relocation.
Recovering from hardware failure Failures on RAID-5 volumes sd disk02-01 sd disk03-01 ... r5vol-01 r5vol-01 disk02 disk03 0 0 102400 102400 1/0 2/0 c2t10d0 c2t11d0 dS ENA This output lists the volume state as NEEDSYNC, indicating that the parity needs to be resynchronized. The state could also have been SYNC, indicating that a synchronization was attempted at start time and that a synchronization process should be doing the synchronization.
24 Recovering from hardware failure Failures on RAID-5 volumes To reattach a failed RAID-5 log plex ◆ Type the following command: # vxplex -g diskgroup att r5vol r5vol-plex Recovering a stale subdisk in a RAID-5 volume Stale subdisk recovery is usually done at volume start time. However, the process doing the recovery can crash, or the volume may be started with an option such as -o delayrecover that prevents subdisk recovery.
Recovering from hardware failure Failures on RAID-5 volumes 25 Only the third case can be overridden by using the -o force option. Subdisks of RAID-5 volumes can also be split and joined by using the vxsd split command and the vxsd join command. These operations work the same way as those for mirrored volumes. RAID-5 subdisk moves are performed in the same way as subdisk moves for other volume types, but without the penalty of degraded redundancy.
26 Recovering from hardware failure Failures on RAID-5 volumes Figure 1-3 Invalid RAID-5 volume disk00-00 disk01-00 disk02-00 disk03-00 disk04-00 disk05-00 W Data Data Parity W X Data Parity Data X Y Parity Data Data Y Z Data Data Parity Z RAID-5 plex There are four stripes in the RAID-5 array. All parity is stale and subdisk disk05-00 has failed. This makes stripes X and Y unusable because two failures have occurred within those stripes.
Recovering from hardware failure Recovering from an incomplete disk group move To forcibly start a RAID-5 volume with stale subdisks ◆ Specify the -f option to the vxvol start command. # vxvol [-g diskgroup] -f start r5vol This causes all stale subdisks to be marked as non-stale. Marking takes place before the start operation evaluates the validity of the RAID-5 volume and what is needed to start it.
28 Recovering from hardware failure Recovery from failure of a DCO volume Automatic recovery depends on being able to import both the source and target disk groups. However, automatic recovery may not be possible if, for example, one of the disk groups has been imported on another host. To recover from an incomplete disk group move 1 Use the vxprint command to examine the configuration of both disk groups. Objects in disk groups whose move is incomplete have their TUTIL0 fields set to MOVE.
Recovering from hardware failure Recovery from failure of a DCO volume and the badlog flag is set on the DCO. All further writes to the volume are not tracked by the DCO.
30 Recovering from hardware failure Recovery from failure of a DCO volume You can use such output to deduce the name of a volume’s DCO (in this example, vol1_dco), or you can use the following vxprint command to display the name of a volume’s DCO: # vxprint [-g diskgroup] -F%dco_name volume You can use the vxprint command to check if the badlog flag is set for the DCO of a volume as shown here: # vxprint [-g diskgroup] -F%badlog dco_name This command returns the value on if the badlog flag is set.
Recovering from hardware failure Recovery from failure of a DCO volume 3 Restart the DCO volume using the following command: # vxvol [-g diskgroup] start dco_log_vol For the example output, the command would take this form: # vxvol -g mydg start vol1_dcl 31
32 Recovering from hardware failure Recovery from failure of a DCO volume 4 Use the vxassist snapclear command to clear the FastResync maps for the original volume and for all its snapshots. This ensures that potentially stale FastResync maps are not used when the snapshots are snapped back (a full resynchronization is performed). FastResync tracking is re-enabled for any subsequent snapshots of the volume.
Recovering from hardware failure Recovery from failure of a DCO volume 5 To snap back the snapshot volume on which you performed a snapclear, use the following command (after using the vxdg move command to move the snapshot plex back to the original disk group, if necessary): # vxplex -f [-g diskgroup] snapback volume snapvol_plex For the example output, the command would take this form: # vxplex -f -g mydg snapback vol1 vol1-03 You cannot use the vxassist snapback command because the snapclear operatio
34 Recovering from hardware failure Recovery from failure of a DCO volume 4 Start the volume using the vxvol command: # vxvol [-g diskgroup] start volume For the example output, the command would take this form: # vxvol -g mydg start vol1 5 Prepare the volume again using the following command: # vxsnap [-g diskgroup] prepare volume [ndcomirs=number] \ [regionsize=size] [drl=yes|no|sequential] \ [storage_attribute ...
Chapter 2 Recovering from instant snapshot failure This chapter includes the following topics: ■ Recovering from the failure of vxsnap prepare ■ Recovering from the failure of vxsnap make for full-sized instant snapshots ■ Recovering from the failure of vxsnap make for break-off instant snapshots ■ Recovering from the failure of vxsnap make for space-optimized instant snapshots ■ Recovering from the failure of vxsnap restore ■ Recovering from the failure of vxsnap reattach or refresh ■ Recove
36 Recovering from instant snapshot failure Recovering from the failure of vxsnap make for full-sized instant snapshots To recover from the failure of the vxsnap prepare command ◆ Type the following command: # vxedit [-g diskgroup] rm DCO_volume Alternatively, the DCO volume is removed automatically when the system is next restarted. When the DCO volume has been removed, run the vxsnap prepare command again.
Recovering from instant snapshot failure Recovering from the failure of vxsnap make for break-off instant snapshots 4 Prepare the snapshot volume again for snapshot operations. Enter the following command: # vxsnap [-g diskgroup] prepare snapshot_volume 5 Clear the volume’s tutil0 field (if it is set). Enter the following command.
38 Recovering from instant snapshot failure Recovering from the failure of vxsnap restore To recover from the failure of the vxsnap make command for space-optimized instant snapshots ◆ Type the following command: # vxedit [-g diskgroup] rm snapshot_volume Alternatively, the snapshot volume is removed automatically when the system is next restarted.
Recovering from instant snapshot failure Recovering from copy-on-write failure 3 Use the vxsnap command to dissociate the volume from the snapshot hierarchy: # vxsnap [-g diskgroup] dis volume 4 Use the following command to start the volume: # vxvol [-g diskgroup] start volume 5 Re-run the failed reattach or refresh command. This results in a full resynchronization of the volume. Alternatively, remove the snapshot volume and recreate it if required.
40 Recovering from instant snapshot failure Recovering from I/O errors during resynchronization Recovering from I/O errors during resynchronization Snapshot resynchronization (started by vxsnap syncstart, or by specifying sync=on to vxsnap) stops if an I/O error occurs, and displays the following message on the system console: VxVM vxsnap ERROR V-5-1-6840 Synchronization of the volume volume stopped due to I/O error After correcting the source of the error, restart the resynchronization operation.
Chapter 3 Recovering from boot disk failure This chapter includes the following topics: ■ VxVM and boot disk failure ■ Recovering a system by booting from a VxVM root disk mirror ■ Recovering a system by booting from recovery media ■ Initiating VxVM Maintenance Mode Boot ■ Recovery by reinstallation ■ Recovering a system with VxVM boot disk under native multi-pathing VxVM and boot disk failure Veritas Volume Manager (VxVM) protects systems from disk and other hardware failures and helps you to
42 Recovering from boot disk failure Recovering a system by booting from a VxVM root disk mirror Recovering a system by booting from a VxVM root disk mirror If a failed primary boot disk is under VxVM control and is mirrored, it must be replaced. To recover a system by booting from a VxVM root disk mirror 1 Replace the failed disk. Depending on the system hardware, this may require you to shut down and power off the system.
Recovering from boot disk failure Recovering a system by booting from recovery media If these methods fail, use the recovery by reinstallation procedure. See “Recovery by reinstallation” on page 46. Starting VxVM after booting from recovery media You can use the vx_emerg_start utility to start VxVM after booting a system from recovery media. This command allows a rootable VxVM configuration to be repaired in the event of a catastrophic failure.
44 Recovering from boot disk failure Recovering a system by booting from recovery media Recovering the root volume after VxVM emergency startup To recover the root volume after VxVM emergency startup 1 After you have used vx_emerg_start to start VxVM, use the vxprint command to determine the configuration state. One common problem is that all the plexes of the root volume, rootvol, are stale.
Recovering from boot disk failure Initiating VxVM Maintenance Mode Boot Fixing a missing or corrupt /etc/vx/volboot file The following messages may be displayed at boot time if the /etc/vx/volboot file is missing or its contents are incorrect: vxvm:vxconfigd: ERROR: enable failed: Volboot file not loaded transactions are disabled.
46 Recovering from boot disk failure Recovery by reinstallation To initiate VxVM MMB ◆ Boot the system from the primary or secondary boot device, and enter the following command at the ISL boot prompt: hpux -vm This causes the bootloader and the HP-UX kernel to take alternate actions in finding the parts of the system that are required to boot the system. The bootloader starts reading the stand file system at 1KB block 3168 instead of locating it from the LIF label.
Recovering from boot disk failure Recovering a system with VxVM boot disk under native multi-pathing See “Recovering a system by booting from a VxVM root disk mirror” on page 42. See “Recovering a system by booting from recovery media” on page 42. Recovering a system with VxVM boot disk under native multi-pathing If a system with VxVM boot disk under native multi-pathing is unable to boot, boot the system from an alternate boot disk and use the following recovery procedures.
48 Recovering from boot disk failure Recovering a system with VxVM boot disk under native multi-pathing To recover the system from such failure, where krs value is displayed as 0, boot the system from an alternate boot disk and follow the steps given below: 1 # mkdir /mnt1 2 # fsck /dev/vx/dsk/dgname/rootvol where dgname is the name of the diskgroup containing the affected boot disk.
Chapter 4 Logging commands and transactions This chapter includes the following topics: ■ Command logs ■ Transaction logs ■ Association of command and transaction logs ■ Associating CVM commands issued from slave to master node Command logs The vxcmdlog command allows you to log the invocation of other Veritas Volume Manager (VxVM) commands to a file. The following examples demonstrate the usage of vxcmdlog: vxcmdlog -l List current settings for command logging.
50 Logging commands and transactions Command logs Command lines are logged to the file, cmdlog, in the directory /etc/vx/log. This path name is a symbolic link to a directory whose location depends on the operating system. If required, you can redefine the directory which is linked. If you want to preserve the settings of the vxcmdlog utility, you must also copy the settings file, .cmdlog, to the new directory. Warning: The .cmdlog file is a binary and should not be edited.
Logging commands and transactions Transaction logs Most command scripts are not logged, but the command binaries that they call are logged. Exceptions are the vxdisksetup, vxinstall, and vxdiskunsetup scripts, which are logged. If there is an error reading from the settings file, command logging switches to its built-in default settings. This may mean, for example, that logging remains enabled after being disabled using vxcmdlog -m off command.
52 Logging commands and transactions Transaction logs maximum size, the current transaction log file, translog, is renamed as the next available historic log file, translog.number, where number is an integer from 1 up to the maximum number of historic log files that is currently defined, and a new current log file is created. A limited number of historic log files is preserved to avoid filling up the file system.
Logging commands and transactions Association of command and transaction logs Association of command and transaction logs The Client and process IDs that are recorded for every request and command assist you in correlating entries in the command and transaction logs.
54 Logging commands and transactions Associating CVM commands issued from slave to master node For example, on the slave node, you run the following command, to create a volume in a shared disk group. CVM ships the command to the master node, and CVM executes the command on the master node.
Logging commands and transactions Associating CVM commands issued from slave to master node The output from the example shows a match at line 7310 in the command log.
56 Logging commands and transactions Associating CVM commands issued from slave to master node Clid = 27741, PID = 475212, Part = 0, Status = 0, Abort Reason = 0 CMDSHIP_RESPONSE SlaveCLID = 5302 SlaveCMID = 0 ExitCode = 12 Flags = 1 stdoutlen = 0 stderrlen = 98 Response = VxVM vxassist ERROR V-5-1-10127 creating volume shared-vol1: Record already exists in disk group DROPPED
Chapter 5 Backing up and restoring disk group configurations This chapter includes the following topics: ■ About disk group configuration backup ■ Backing up a disk group configuration ■ Restoring a disk group configuration About disk group configuration backup Disk group configuration backup and restoration allows you to backup and restore all configuration data for Veritas Volume Manager (VxVM) disk groups, and for VxVM objects such as volumes that are configured within the disk groups.
58 Backing up and restoring disk group configurations Backing up a disk group configuration vxconfigbackup and vxconfigrestore, are provided for backing up and restoring a VxVM configuration for a disk group.
Backing up and restoring disk group configurations Restoring a disk group configuration the five most recent backups are preserved. If required, you can also back up a disk group configuration by running the vxconfigbackup command. The following files record disk group configuration information: /etc/vx/cbr/bk/diskgroup.dgid/dgid.dginfo Disk group information. /etc/vx/cbr/bk/diskgroup.dgid/dgid .diskinfo Disk attributes. /etc/vx/cbr/bk/diskgroup.dgid/dgid .binconfig Binary configuration copy.
60 Backing up and restoring disk group configurations Restoring a disk group configuration Warning: None of the disks or VxVM objects in the disk group may be open or in use by any application while the restoration is being performed. You can choose whether or not any corrupted disk headers are to be reinstalled at the precommit stage. If any of the disks’ private region headers are invalid, restoration may not be possible without reinstalling the headers for the affected disks.
Backing up and restoring disk group configurations Restoring a disk group configuration To perform the commit operation ◆ To commit the changes that are required to restore the disk group configuration, use the following command: # /etc/vx/bin/vxconfigrestore -c [-l directory] \ {diskgroup | dgid} If no disk headers are reinstalled, the configuration copies in the disks’ private regions are updated from the latest binary copy of the configuration that was saved for the disk group.
62 Backing up and restoring disk group configurations Restoring a disk group configuration 1047336696.19.xxx.veritas.com 1049135264.31.xxx.veritas.com The solution is to specify the disk group by its ID rather than by its name to perform the restoration. The backup file, /etc/vx/cbr/bk/diskgroup. dgid/ dgid.dginfo, contains a timestamp that records when the backup was taken.
Chapter 6 Restoring a previous array support library This chapter includes the following topics: ■ Downgrading the array support Downgrading the array support The array support is available in a single depot, VRTSaslapm, that includes Array Support Libraries (ASLs) and Array Policy Modules (APMs). Each major release of Veritas Volume Manager includes the supported VRTSaslapm depot, which is installed as part of the product installation.
64 Restoring a previous array support library Downgrading the array support
Chapter 7 Error messages This chapter includes the following topics: ■ About error messages ■ How error messages are logged ■ Types of messages About error messages Informational, failure, and other error messages may be displayed on the console by the Veritas Volume Manager (VxVM) configuration daemon (vxconfigd), the VxVM kernel driver, vxio, and the various VxVM commands. These messages may indicate errors that are infrequently encountered and difficult to troubleshoot.
66 Error messages How error messages are logged vxconfigd also supports the use of syslog to log all of its regular console messages. When this is enabled, all console output is directed through the syslog interface. syslog and log file logging can be used together to provide reliable logging to a private log file, along with distributed logging through syslogd. Note: syslog logging is enabled by default. Debug message logging is disabled by default.
Error messages Types of messages 67 To configure logging in the startup script ◆ Comment-out or uncomment any of the following lines to enable or disable the corresponding feature in vxconfigd: opts="$opts -x syslog" # use syslog for console messages #opts="$opts -x log" # messages to vxconfigd.log #opts="$opts -x logfile=/foo/bar" # specify an alternate log file #opts="$opts -x timestamp" # timestamp console messages # To turn on debugging console output, uncomment the following line.
68 Error messages Types of messages provide a list of the more common errors, a detailed description of the likely cause of the problem together with suggestions for any actions that can be taken. Messages have the following generic format: product component severity message_number message_text For Veritas Volume Manager, the product is set to VxVM. The component can be the name of a kernel module or driver such as vxdmp, a configuration daemon such as vxconfigd, or a command such as vxassist.
Error messages Types of messages NOTICE A notice message indicates that an error has occurred that should be monitored. Shutting down the system is unnecessary, although you may need to take action to remedy the fault at a later date. The following is an example of such a message: VxVM vxio NOTICE V-5-0-252 read error on object subdisk of mirror plex in volume volume (start offset, length length) corrected. INFO An informational message does not indicate an error, and requires no action.
70 Error messages Types of messages VxVM vxio WARNING V-5-0-2 object_type object_name blockoffset:Uncorrectable write error ... Description: A read or write operation from or to the specified Veritas Volume Manager object failed. An error is returned to the application. Recommended action: These errors may represent lost data. Data may need to be restored and failed media may need to be repaired or replaced.
Error messages Types of messages An attempt is being made to disable the one remaining active path to the root disk controller. Recommended action: The path cannot be disabled. V-5-0-55 VxVM vxio WARNING V-5-0-55 Cannot find device number for boot_path vxvm vxdmp WARNING V-5-0-55 Cannot find device number for boot_path Description: The boot path retrieved from the system PROMs cannot be converted to a valid device number. Recommended action: Check your PROM settings for the correct boot string.
72 Error messages Types of messages Check for other console error messages that may provide additional information about the failure. V-5-0-108 VxVM vxio WARNING V-5-0-108 Device major, minor: Received spurious close Description: A close was received for an object that was not open. This can only happen if the operating system is not correctly tracking opens and closes. Recommended action: No action is necessary; the system will continue.
Error messages Types of messages Description: A path has been marked disabled in the DMP database. This path is controlled by the DMP node indicated by the specified device number. This may be due to a hardware failure. Recommended action: Check the underlying hardware if you want to recover the desired path. V-5-0-144 VxVM vxio WARNING V-5-0-144 Double failure condition detected on RAID-5 volume Description: I/O errors have been received in more than one column of a RAID-5 volume.
74 Error messages Types of messages V-5-0-146 VxVM vxdmp NOTICE V-5-0-146 enabled controller controller_name connected to disk array disk_array_serial_number Description: All paths through the controller connected to the disk array are enabled. This usually happens if a controller is enabled after maintenance. Recommended action: No recovery procedure is required. V-5-0-147 VxVM vxdmp NOTICE V-5-0-147 enabled dmpnode dmpnode_device_number A DMP node has been marked enabled in the DMP database.
Error messages Types of messages Recommended action: Use the vxdisk -s list command on the master node to see what disks should be visible to the slave node. Then check that the operating system and VxVM on the failed node can also see these disks. If the operating system cannot see the disks, check the cabling and hardware configuration of the node. If only VxVM cannot see the disks, use the vxdctl enable command to make it scan again for the disks.
76 Error messages Types of messages V-5-0-181 VxVM vxio WARNING V-5-0-181 Illegal vminor encountered Description: An attempt was made to open a volume device (other than the root volume device) before vxconfigd loaded the volume configuration. Recommended action: No recovery procedure is required. Under normal startup conditions, this message should not occur. If necessary, start VxVM and re-attempt the operation.
Error messages Types of messages To restore RAID-5 logging to a RAID-5 volume, create a new log plex and attach it to the volume. V-5-0-216 VxVM vxio WARNING V-5-0-216 mod_install returned errno Description: A call made to the operating system mod_install function to load the vxio driver failed. Recommended action: Check for additional console messages that may explain why the load failed.
78 Error messages Types of messages V-5-0-244 VxVM vxdmp NOTICE V-5-0-244 Path failure on major/minor Description: A path under the control of the DMP driver failed. The device major and minor numbers of the failed device is supplied in the message. Recommended action: No recovery procedure is required. V-5-0-249 VxVM vxio WARNING V-5-0-249 RAID-5 volume entering degraded mode operation Description: An uncorrectable error has forced a subdisk to detach.
Error messages Types of messages If the volume is not mirrored, this message indicates that some data could not be read. The file system or other application reading the data may report an additional error, but in either event, data has been lost. The volume can be partially salvaged and moved to another location if desired. This message may also appear during a plex detach operation in a cluster. In this case, no action is required.
80 Error messages Types of messages Recommended action: Check for obvious problems with the disk (such as a disconnected cable). If hot-relocation is enabled and the disk is failing, recovery from subdisk failure is handled automatically. V-5-1-90 VxVM vxconfigd ERROR V-5-1-90 mode: Unrecognized operating mode Description: An invalid string was specified as an argument to the -m option. Valid strings are: enable, disable, and boot. Recommended action: Supply a correct option argument.
Error messages Types of messages in its usual location. This may be a serious problem for the general running of your system. Recommended action: If the /usr file system is not mounted, you need to determine how to get it mounted. If the rm utility is missing, or is not in the /usr/bin directory, restore it.
82 Error messages Types of messages This error indicates that the volume cannot be started because it does not contain any valid plexes. This can happen, for example, if disk failures have caused all plexes to be unusable. It can also happen as a result of actions that caused all plexes to become unusable (for example, forcing the dissociation of subdisks or detaching, dissociation, or offlining of plexes). Recommended action: It is possible that this error results from a drive that failed to spin up.
Error messages Types of messages V-5-1-123 VxVM vxconfigd ERROR V-5-1-123 Disk group group: Disabled by errors Description: This message indicates that some error condition has made it impossible for VxVM to continue to manage changes to a disk group. The major reason for this is that too many disks have failed, making it impossible for vxconfigd to continue to update configuration copies. There should be a preceding error message that indicates the specific error that was encountered.
84 Error messages Types of messages V-5-1-134 VxVM vxconfigd ERROR V-5-1-134 Memory allocation failure Description: This implies that there is insufficient memory to start VxVM. Recommended action: This error should not normally occur, unless your system has very small amounts of memory. Adding swap space will probably not help because this error is most likely to occur early in the boot sequence, before swap areas have been added.
Error messages Types of messages ■ Case 1: cannot open /dev/vx/config: reason The /dev/vx/config device could not be opened. vxconfigd uses this device to communicate with the Veritas Volume Manager kernel drivers. The most likely reason is “Device is already open.” This indicates that some process (most likely vxconfigd) already has /dev/vx/config open. Less likely reasons are “No such file or directory” or “No such device or address.
86 Error messages Types of messages If hot-relocation is enabled, Veritas Volume Manager objects affected by the disk failure are taken care of automatically. Mail is sent to root indicating what actions were taken by VxVM, and what further actions you should take. V-5-1-480 VxVM vxconfigd ERROR V-5-1-480 Cannot reset VxVM kernel: reason Description: The -r reset option was specified to vxconfigd, but the VxVM kernel drivers could not be reset. The most common reason is “A virtual disk device is open.
Error messages Types of messages The DRL or RAID-5 log for the named volume was detached as a result of a disk failure, or as a result of the administrator removing a disk with vxdg -k rmdisk. A failing disk is indicated by a “Detached disk” message. Recommended action: If the log is mirrored, hot-relocation tries to relocate the failed log automatically. Use either vxplex dis or vxsd dis to remove the failing logs. Then, use vxassist addlog to add a new log to the volume. See the vxassist(1M) manual page.
88 Error messages Types of messages The specified volume was detached as a result of a disk failure, or as a result of the administrator removing a disk with vxdg -k rmdisk. A failing disk is indicated by a “Detached disk” message. Unless the disk error is transient and can be fixed with a reboot, the contents of the volume should be considered lost. Recommended action: Contact Veritas Technical Support.
Error messages Types of messages This message can usually be ignored. If you want to use the disk on this system, use vxdiskadd to add the disk. Do not do this if the disk really is shared with other systems. V-5-1-546 VxVM vxconfigd WARNING V-5-1-546 Disk disk in group group: Disk device not found Description: No physical disk can be found that matches the named disk in the given disk group. This is equivalent to failure of that disk.
90 Error messages Types of messages If the disks should be imported into the group, this must be done by adding the disk to the group at a later stage, during which all configuration information for the disk is lost. V-5-1-557 VxVM vxconfigd ERROR V-5-1-557 Disk disk, group group, device device: not updated with new host ID Error: reason Description: This can result from using vxdctl hostid hostid to change the Veritas Volume Manager host ID for the system.
Error messages Types of messages Typically, unless a disk group was disabled due to transient errors, there is no way to repair a disabled disk group. The disk group may have to be reconstructed from scratch. If the disk group was disabled due to a transient error such as a cabling problem, then a future reboot may not automatically import the named disk group, due to the change in the system’s Veritas Volume Manager host ID.
92 Error messages Types of messages yield further errors resulting from inability to access the volume when mounting the file system. Recommended action: If the error is clearly caused by excessive disk failures, then you may have to recreate the disk group configuration, and restore the contents of any volumes from a backup. See “Restoring a disk group configuration” on page 59. There may be other error messages that appear which provide further information.
Error messages Types of messages V-5-1-577 VxVM vxconfigd WARNING V-5-1-577 Disk group group: Disk group log may be too small Log size should be at least number blocks Description: The log areas for the disk group have become too small for the size of configuration currently in the group.
94 Error messages Types of messages V-5-1-583 VxVM vxconfigd ERROR V-5-1-583 Disk group group: Reimport of disk group failed: reason Description: After vxconfigd was stopped and restarted (or disabled and then enabled), VxVM failed to recreate the import of the indicated disk group. The reason for failure is specified. Additional error messages may be displayed that give further information describing the problem.
Error messages Types of messages # vxdisk clearimport devicename... To clear the locks during import, use the following command: # vxdg -C import diskgroup Warning: Be careful when using the vxdisk clearimport or vxdg -C import command on systems that have dual-ported disks. Clearing the locks allows those disks to be accessed at the same time from multiple hosts and can result in corrupted data.
96 Error messages Types of messages V-5-1-663 VxVM vxconfigd WARNING V-5-1-663 Group group: Duplicate virtual device number(s): Volume volume remapped from major,minor to major,minor ... Description: The configuration of the named disk group includes conflicting device numbers. A disk group configuration lists the recommended device number to use for each volume in the disk group.
Error messages Types of messages V-5-1-768 VxVM vxconfigd NOTICE V-5-1-768 Offlining config copy number on diskdisk: Reason: reason Description: An I/O error caused the indicated configuration copy to be disabled. This is a notice only, and does not normally imply serious problems, unless this is the last active configuration copy in the disk group.
98 Error messages Types of messages V-5-1-1049 VxVM vxconfigd ERROR V-5-1-1049 System boot disk does not have a valid rootvol plex Please boot from one of the following disks: DISK diskname MEDIA DEVICE device BOOT boot COMMAND vx-diskname... Description: The system is configured to use a volume for the root file system, but was not booted on a disk containing a valid mirror of the root volume. Disks containing valid root mirrors are listed as part of the error message.
Error messages Types of messages disk group. This should happen only as a result of direct manipulation by the administrator. ■ Case 2: The system somehow has a duplicate boot disk group, one of which contains the /usr file system volume and one of which does not (or uses a different volume name), and vxconfigd somehow chose the wrong boot disk group.
100 Error messages Types of messages Regular startup of vxconfigd failed. This error can also result from the command vxdctl enable. This message has several variations, described below: VxVM vxconfigd ERROR V-5-1-1589 enable failed: aborting The failure was fatal and vxconfigd was forced to exit. The most likely cause is that the operating system is unable to create interprocess communication channels to other utilities.
Error messages Types of messages VxVM vxconfigd ERROR V-5-1-1589 enable failed: transactions are disabled Evaluate the error messages to determine the root cause of the problem. Make changes suggested by the errors and then try rerunning the command. If the “Errors in some configuration copies” error occurs again, that may indicate the real problem lies with the configuration copies in the disk group. See “Restoring a disk group configuration” on page 59.
102 Error messages Types of messages Description: The vxconfigd daemon is not responding properly in a cluster. Recommended action: Stop and restart the vxconfigd daemon on the node indicated. V-5-1-2274 VxVM vxconfigd ERROR V-5-1-2274 volume:vxconfigd cannot boot-start RAID-5 volumes Description: A volume that vxconfigd should start immediately upon booting the system (that is, the volume for the /usr file system) has a RAID-5 layout. The /usr file system should never be defined on a RAID-5 volume.
Error messages Types of messages a failure related to reading the file /var/vxvm/tempdb/group. This is a temporary file used to store information that is used when recovering the state of an earlier vxconfigd. The file is recreated on a reboot, so this error should never survive a reboot. Recommended action: If you can reboot the system, do so. If you do not want to reboot, then use the following procedure.
104 Error messages Types of messages This warning may safely be ignored. Recommended action: No recovery procedure is required. V-5-1-2824 VxVM vxconfigd ERROR V-5-1-2824 Configuration daemon error 242 Description: A node failed to join a cluster, or a cluster join is taking too long. If the join fails, the node retries the join automatically. Recommended action: No action is necessary if the join is slow or a retry eventually succeeds.
Error messages Types of messages One of the disk groups specified in a disk group move, split or join operation is currently involved in another unrelated disk group move, split or join operation (possibly as the result of recovery from a system failure). Recommended action: Use the vxprint command to display the status of the disk groups involved.
106 Error messages Types of messages V-5-1-2870 VxVM vxdg ERROR V-5-1-2870 volume: Volume or plex device is open or mounted Description: An attempt was made to perform a disk group move, split or join on a disk group containing an open volume. Recommended action: It is most likely that a file system configured on the volume is still mounted. Stop applications that access volumes configured in the disk group, and unmount any file systems configured in the volumes.
Error messages Types of messages Correct the name of the disk object specified in the disk group move or split operation. V-5-1-2911 VxVM vxdg ERROR V-5-1-2911 diskname: Disk is not usable Description: The specified disk has become unusable. Recommended action: Do not include the disk in any disk group move, split or join operation until it has been replaced or repaired.
108 Error messages Types of messages The requested disk group move, split or join operation would leave the disk group without any configuration copies. Recommended action: No action is required. The operation is not supported. V-5-1-2935 VxVM vxassist ERROR V-5-1-2935 No more space in disk group configuration. Description: There is no more space in the disk group’s configuration database for VxVM object records.
Error messages Types of messages V-5-1-3022 VxVM vxconfigd ERROR V-5-1-3022 Cannot find disk on slave node Description: A slave node in a cluster cannot find a shared disk. This is accompanied by the syslog message: VxVM vxconfigd ERROR V-5-1-2173 cannot find disk disk Recommended action: Make sure that the same set of shared disks is online on both nodes.
110 Error messages Types of messages vxconfigd was not able to create a portal for communication with the vxconfigd on the other node. This may happen in a degraded system that is experiencing shortages of system resources such as memory or file descriptors. Recommended action: If the system does not appear to be degraded, stop and restart vxconfigd, and try again.
Error messages Types of messages During the slave join protocol, a message without data was received from the master. This message is only likely to be seen in the case of an internal VxVM error. Recommended action: Contact Veritas Technical Support. V-5-1-3033 VxVM vxconfigd ERROR V-5-1-3033 Join in progress Description: An attempt was made to import or deport a shared disk group during a cluster reconfiguration. Recommended action: Retry when the cluster reconfiguration has completed.
112 Error messages Types of messages V-5-1-3046 VxVM vxconfigd ERROR V-5-1-3046 Node activation conflict Description: The disk group could not be activated because it is activated in a conflicting mode on another node in a cluster. Recommended action: Retry later, or deactivate the disk group on conflicting nodes. V-5-1-3049 VxVM vxconfigd ERROR V-5-1-3049 Retry rolling upgrade Description: An attempt was made to upgrade a cluster to a higher protocol version when a transaction was in progress.
Error messages Types of messages Use the -o expand option to vxdg listmove to produce a self-contained list of objects. V-5-1-3212 VxVM vxconfigd ERROR V-5-1-3212 Insufficient DRL log size: logging is disabled. Description: A volume with an insufficient DRL log size was started successfully, but DRL logging is disabled and a full recovery is performed. Recommended action: Create a new DRL of sufficient size. V-5-1-3243 VxVM vxdmpadm ERROR V-5-1-3243 The VxVM restore daemon is already running.
114 Error messages Types of messages Description: Checking for the current protocol version (using vxdctl protocol version) does not work if the node is not in a cluster. Recommended action: Bring the node into the cluster and retry. V-5-1-3689 VxVM vxassist ERROR V-5-1-3689 Volume record id rid is not found in the configuration. Description: An error was detected while reattaching a snapshot volume using snapback.
Error messages Types of messages file should contain a supported protocol version before trying to bring the node into the cluster. Recommended action: Verify the supported cluster protocol versions using the vxdctl protocolversion command. The volboot file should contain a supported protocol version before trying to bring the node into the cluster. Run vxdctl init to write a valid protocol version to the volboot file. Restart vxconfigd and retry the join.
116 Error messages Types of messages V-5-1-4277 VxVM vxconfigd ERROR V-5-1-4277 cluster_establish: CVM protocol version out of range Description: When a node joins a cluster, it tries to join at the protocol version that is stored in its volboot file. If the cluster is running at a different protocol version, the master rejects the join and sends the current protocol version to the slave. The slave re-tries with the current version (if that version is supported on the joining node), or the join fails.
Error messages Types of messages salcontact=no V-5-1-4625 VxVM vxassist WARNING V-5-1-4625 SAL authentication failed... Description: The SAN Access Layer (SAL) rejects the credentials that are supplied by the vxassist command. Recommended action: If connection to SAL is desired, use the vxspcshow command to set a valid user name and password.
118 Error messages Types of messages Description: An attempt was made to snap back a detached plex. Recommended action: Reattach the snapshot plex to the snapshot volume. V-5-1-5162 VxVM vxplex ERROR V-5-1-5162 Plexes do not belong to the same snapshot volume. Description: An attempt was made to snap back plexes that belong to different snapshot volumes. Recommended action: Specify the plexes in separate invocations of vxplex snapback.
Error messages Types of messages ■ Case 1: Some arrays such as EMC and HDS provide mirroring in hardware. When a LUN pair is split, depending on how the process is performed, this can result in two disks that have the same disk identifier and UDID value. See "Handling Disks with Duplicated Identifiers" in the "Creating and Administering Disk Groups" chapter of the Veritas Volume Manager Administrator’s Guide for full details of how to deal with this condition.
120 Error messages Types of messages or vxmend. Verify the running state of the vxconfigd process. If necessary, restart the vxconfigd process. V-5-1-15861 VxVM command ERROR V-5-1-15861 Command is not supported for command shipping. Operation must be executed on master. Description: On the slave, when you run a command that changes the configuration for a shared disk group, CVM ships the command to the master for execution.
Error messages Types of messages Description: The message occurs during execution of a shared disk group configuration change command that is shipped from slave on master. The message indicates that either the vxconfigd process exited on the master node, the master node left the cluster, or the slave node that issued the command left the cluster. Recommended action: Verify the state of the VxVM objects using query commands such as vxprint or vxdisk list.
122 Error messages Types of messages
Index Symbols .cmdlog file 49 .translog file 51 /etc/vx/cbr/bk/diskgroup.dgid dgid .binconfig file 59 dgid .cfgrec file 59 dgid .diskinfo file 59 dgid.dginfo file 59 /etc/vx/log logging directory 49, 51 /sbin/init.d/vxvm-sysboot file 66 /stand/rootconf file 45 /var/adm/configd.log file 65 /var/log/syslog/syslog.
124 Index DMP fixing duplicated disk IDs 118 E emergency startup 43 EMPTY plex state 11 ENABLED plex kernel state 11 ENABLED volume kernel state 21 ERROR messages 68 error messages A virtual disk device is open 86 All transactions are disabled 83 Already at highest version 114 Attempt to disable controller failed 113 Attempt to enable a controller that is not available 102 Cannot assign minor 110 Cannot auto-import group 57, 91 Cannot find disk on slave node 109 Cannot kill existing daemon 101 cannot ope
Index error messages (continued) Return from cluster_establish is Configuration daemon error 104 Rootdg cannot be imported during boot 45 Skip disk group with duplicate name 92 some subdisks are unusable and the parity is stale 25 startup script 66 Synchronization of the volume stopped due to I/O error 40 System boot disk does not have a valid rootvol plex 98 System startup failure 84 The VxVM restore daemon is already running 113 There are two backups that have the same diskgroup name with different diskg
126 Index notice messages added disk array 70 Attempt to disable controller failed 70 Detached disk 82 Detached log for volume 86 Detached plex in volume 87 Detached subdisk in volume 87 Detached volume 87 disabled controller connected to disk array 72 disabled dmpnode 72 disabled path belonging to dmpnode 72 enabled controller connected to disk array 74 enabled dmpnode 74 enabled path belonging to dmpnode 74 Offlining config copy 97 Path failure 78 read error on object 79 removed disk array 79 Rootdisk h
Index syslog error log file 66 system reinstalling 46 system failures 18 T transactions associating with commands 53 logging 51 translog file 51 TUTIL0 field clearing MOVE flag 27 U udid_mismatch flag 118 V V-5-0-106 71 V-5-0-108 72 V-5-0-110 72 V-5-0-111 72 V-5-0-112 72 V-5-0-144 73 V-5-0-145 73 V-5-0-146 74 V-5-0-147 74 V-5-0-148 74 V-5-0-164 74 V-5-0-166 75 V-5-0-168 75 V-5-0-181 76 V-5-0-194 76 V-5-0-196 76 V-5-0-2 69 V-5-0-207 76 V-5-0-216 77 V-5-0-237 77 V-5-0-243 77 V-5-0-244 78 V-5-0-249 78 V-5-
128 Index V-5-1-3024 109 V-5-1-3025 109 V-5-1-3030 110 V-5-1-3031 110 V-5-1-3032 110 V-5-1-3033 111 V-5-1-3034 111 V-5-1-3042 111 V-5-1-3046 112 V-5-1-3049 112 V-5-1-3050 112 V-5-1-3091 112 V-5-1-3212 113 V-5-1-3243 113 V-5-1-3362 113 V-5-1-3486 113 V-5-1-3689 114 V-5-1-3828 114 V-5-1-3848 114 V-5-1-4220 115 V-5-1-4267 115 V-5-1-4277 116 V-5-1-4551 116 V-5-1-4620 116 V-5-1-4625 117 V-5-1-480 86 V-5-1-484 86 V-5-1-5150 117 V-5-1-5160 117 V-5-1-5161 117 V-5-1-5162 118 V-5-1-525 86 V-5-1-526 87 V-5-1-527 87
Index vxedit clearing a disk failing flag 16 vxinfo command 10 vxmend command 14 vxplex command 23 vxprint displaying volume and plex states 11 vxreattach reattaching failed disks 17 vxsnap make recovery from failure of 36 vxsnap prepare recovery from failure of 35 vxsnap reattach recovery from failure of 38 vxsnap refresh recovery from failure of 38 vxsnap restore recovery from failure of 38 vxtranslog controlling transaction logging 51 VxVM emergency startup 43 RAID-5 recovery process 21 starting after b
