VERITAS Volume Manager™ 3.
Disclaimer The information contained in this publication is subject to change without notice. VERITAS Software Corporation makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. VERITAS Software Corporation shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this manual.
Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vii Audience and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vii Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Recovery from DCO Volume Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Chapter 2. Recovery from Boot Disk Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 VxVM Boot Disk Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vxdmp Notice Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 vxdmpadm Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 vxplex Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Cluster Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi VERITAS Volume Manager Troubleshooting Guide
Preface Introduction The VERITAS Volume ManagerTM Troubleshooting Guide provides information about how to recover from hardware failure, and how to understand and deal with VERITAS Volume Manager (VxVM) error messages during normal operation. For detailed information about VERITAS Volume Manager and how to use it, refer to the VERITAS Volume Manager Administrator’s Guide.
Organization Organization This guide is organized as follows: ◆ Recovery from Hardware Failure ◆ Recovery from Boot Disk Failure ◆ Error Messages Related Documents The following documents provide information related to the Volume Manager: viii ◆ VERITAS Volume Manager Installation Guide ◆ VERITAS Volume Manager Release Notes ◆ VERITAS Volume Manager Hardware Notes ◆ VERITAS Volume Manager Administrator’s Guide ◆ VERITAS Volume Manager (UNIX) User’s Guide — VEA ◆ VERITAS Volume Manager ma
Conventions Conventions The following table describes the typographic conventions used in this guide. Typeface Usage Examples monospace Computer output, file contents, files, directories, software elements such as command options, function names, and parameters Read tunables from the /etc/vx/tunefstab file. New terms, book titles, emphasis, variables to be replaced by a name or value See the User’s Guide for details.
Getting Help Getting Help If you have any comments or problems with VERITAS products, contact VERITAS Technical Support: ◆ U.S. and Canadian Customers: 1-800-342-0652 ◆ International Customers: +1 (650) 527-8555 ◆ Email: support@veritas.com For license information (U.S. and Canadian Customers): ◆ Phone: 1-925-931-2464 ◆ Email: license@veritas.com ◆ Fax: 1-925-931-2487 For software updates: ◆ Email: swupdate@veritas.
1 Recovery from Hardware Failure Introduction VERITAS Volume Manager (VxVM) protects systems from disk and other hardware failures and helps you to recover from such events. This chapter describes recovery procedures and information to help you prevent loss of data or system access due to disk and other hardware failures. If a volume has a disk I/O failure (for example, because the disk has an uncorrectable error), VxVM can detach the plex involved in the failure.
Understanding the Plex State Cycle Understanding the Plex State Cycle Changing plex states are part of normal operations, and do not necessarily indicate abnormalities that must be corrected. A firm understanding of the various plex states and their interrelationship is necessary if you want to be able to perform the recovery procedure described in this chapter. The figure “Main Plex State Cycle” shows the main transitions that take place between plex states in VxVM.
Understanding the Plex State Cycle Additional Plex State Transitions Create plex PS: EMPTY PKS: DISABLED PS: ACTIVE PKS: DISABLED After crash and reboot (vxvol start) Initialize plex (vxvol init clean) Start up (vxvol start) PS: CLEAN PKS: DISABLED Shut down (vxvol stop) Recover data (vxvol resync) Take plex offline (vxmend off) PS: ACTIVE PKS: ENABLED PS: OFFLINE PKS: DISABLED Resync data (vxplex att) Put plex online (vxmend on) Uncorrectable I/O failure PS = Plex State Resync PS: IOFAIL fails P
Listing Unstartable Volumes Listing Unstartable Volumes An unstartable volume can be incorrectly configured or have other errors or conditions that prevent it from being started. To display unstartable volumes, use the vxinfo command. This displays information about the accessibility and usability of volumes: # vxinfo [-g diskgroup] [volume ...
Reattaching Disks 2. To recover the other plexes in a volume from the CLEAN plex, the volume must be disabled, and the other plexes must be STALE. If necessary, make any other CLEAN or ACTIVE plexes STALE by running the following command on each of these plexes in turn: # vxmend fix stale plex 3.
Failures on RAID-5 Volumes Failures on RAID-5 Volumes Failures are seen in two varieties: system failures and disk failures. A system failure means that the system has abruptly ceased to operate due to an operating system panic or power failure. Disk failures imply that the data on some number of disks has become unavailable due to a system failure (such as a head crash, electronics failure on disk, or disk controller failure).
Failures on RAID-5 Volumes Disk Failures Disk failures can cause the data on a disk to become unavailable. In terms of a RAID-5 volume, this means that a subdisk becomes unavailable. This can occur due to an uncorrectable I/O error during a write to the disk. The I/O error can cause the subdisk to be detached from the array or a disk being unavailable when the system is booted (for example, from a cabling problem or by having a drive powered down).
Failures on RAID-5 Volumes failure. In the output of the vxprint -ht command, failure within a RAID-5 log plex is indicated by the plex state being shown as BADLOG rather than LOG. This is shown in the following display, where the RAID-5 log plex r5vol-11 has failed: V NAME RVG KSTATE STATE LENGTH PL NAME VOLUME KSTATE STATE LENGTH SD NAME PLEX DISK DISKOFFSLENGTH SV NAME PLEX VOLNAME NVOLLAYRLENGTH ...
Failures on RAID-5 Volumes Caution The -o unsafe start option is considered dangerous, as it can make the contents of the volume unusable. Using it is not recommended. 2. Any existing log plexes are zeroed and enabled. If all logs fail during this process, the start process is aborted. 3. If no stale subdisks exist or those that exist are recoverable, the volume is put in the ENABLED volume kernel state and the volume state is set to ACTIVE. The volume is now started.
Failures on RAID-5 Volumes Note Following severe hardware failure of several disks or other related subsystems underlying a RAID-5 plex, it may be impossible to recover the volume using the methods described in this chapter. In this case, remove the volume, recreate it on hardware that is functioning correctly, and restore the contents of the volume from a backup. Parity Resynchronization In most cases, a RAID-5 array does not have stale parity.
Failures on RAID-5 Volumes Parity is regenerated by issuing VOL_R5_RESYNC ioctls to the RAID-5 volume. The resynchronization process starts at the beginning of the RAID-5 volume and resynchronizes a region equal to the number of sectors specified by the -o iosize option. If the -o iosize option is not specified, the default maximum I/O size is used. The resync operation then moves onto the next region until the entire length of the RAID-5 volume has been resynchronized.
Failures on RAID-5 Volumes Recovery After Moving RAID-5 Subdisks When RAID-5 subdisks are moved and replaced, the new subdisks are marked as STALE in anticipation of recovery. If the volume is active, the vxsd command may be used to recover the volume. If the volume is not active, it is recovered when it is next started. The RAID-5 volume is degraded for the duration of the recovery operation. Any failure in the stripes involved in the move makes the volume unusable.
Failures on RAID-5 Volumes When this occurs, the vxvol start command returns the following error message: vxvm:vxvol: ERROR: Volume r5vol is not startable; RAID-5 plex does not map entire volume length. At this point, the contents of the RAID-5 volume are unusable. Another possible way that a RAID-5 volume can become unstartable is if the parity is stale and a subdisk becomes detached or stale.
Failures on RAID-5 Volumes Forcibly Starting RAID-5 Volumes You can start a volume even if subdisks are marked as stale. For example, if a stopped volume has stale parity and no RAID-5 logs and a disk becomes detached and then reattached. The subdisk is considered stale even though the data is not out of date (because the volume was in use when the subdisk was unavailable) and the RAID-5 volume is considered invalid.
Recovering from Incomplete Disk Group Moves Recovering from Incomplete Disk Group Moves If the system crashes or a subsystem fails while a disk group move, split or join operation is being performed, VxVM attempts either to reverse or to complete the operation when the system is restarted or the subsystem is repaired. Whether the operation is reversed or completed depends on how far it had progressed. Automatic recovery depends on being able to import both the source and target disk groups.
Recovery from DCO Volume Failure Recovery from DCO Volume Failure Persistent FastResync uses a data change object (DCO) log volume to perform tracking of changed regions in a volume. If an error occurs while reading or writing a DCO volume, it is detached and the badlog flag is set on the DCO. (You can use one of the options -a, -F or -m to vxprint to check if the badlog flag is set on a DCO.) All further writes to the volume are not tracked by the DCO.
Recovery from DCO Volume Failure 5. To snap back the snapshot volume on which you performed a snapclear in the previous step, use the following command (after using the vxdg move command to move the snapshot volume back to the original disk group, if necessary): # vxplex -f -g diskgroup snapback volume snapvol_plex Note You cannot use vxassist snapback because the snapclear operation removes the snapshot association information.
Recovery from DCO Volume Failure 18 VERITAS Volume Manager Troubleshooting Guide
2 Recovery from Boot Disk Failure Introduction VERITAS Volume Manager (VxVM) protects systems from disk and other hardware failures and helps you to recover from such events. This chapter describes recovery procedures and information to help you prevent loss of data or system access due to the failure of the boot (root) disk. For information about recovering volumes and their data on non-boot disks, see “Recovery from Hardware Failure” on page 1.
Starting VxVM after Booting from Recovery Media If there is a failure to boot from the VxVM boot disk on HP-UX 11i, and no bootable root mirror is available, use one of the following methods to recover. The method you choose depends on the type of failure: ◆ If the kernel is corrupted or the files required for booting are missing, boot the system from the recovery media and use the vx_emerg_start utility to start VxVM as described in “Starting VxVM after Booting from Recovery Media” on page 20.
Correcting Boot Problems in VxVM Maintenance Mode When this happens, the root volume can usually be repaired by using the following command: # vxvol -g rootdg -f start rootvol If the root volume is mirrored, recovery is started. Wait until recovery completes and the command exits.
Correcting Boot Problems in VxVM Maintenance Mode Several conditions can prevent the system from being booted normally as indicated by error messages at boot time.
Correcting Boot Problems in VxVM Maintenance Mode d. Create a LABEL file on the root disk and display its contents: # mkboot -l /dev/rdsk/c4t8d0s2 # vxvmboot -v /dev/rdsk/c4t8d0s2 LIF Label File @ (1k) block # 846 on LVM Disk \ /dev/rdsk/c4t8d0s2: Label Entry: 0, Boot Volume start: 2912; length: 3992 MB e.
Correcting Boot Problems in VxVM Maintenance Mode Missing or Corrupt /etc/vx/volboot File The following messages may be displayed at boot time if the /etc/vx/volboot file is missing or its contents are incorrect: vxvm:vxconfigd: ERROR: enable failed: Volboot file not loaded transactions are disabled.
Correcting Boot Problems in VxVM Maintenance Mode Missing or Corrupt ioconfig or Device Files The following messages may be displayed at boot time if the /stand/ioconfig file is missing, its contents have been corrupted, or it does not correctly describe the devices in the system: ioconfig_lock: Cannot open /etc/ioconfig with specified flags.
Correcting Boot Problems in VxVM Maintenance Mode d. Re-install the device special files: # /sbin/insf -e insf: Unable to lock /etc/ioconfig - : /etc/ioconfig - \ permission denied. insf: Installing special files for btlan instance 0 address \ 0/16/1/1/0 . . . e. Check that the block and raw disk device files have been re-installed: # ls -lt /dev/dsk total 0 brw-r----- 1 bin sys . . . brw-r----- 1 bin sys # ls -l /dev/rdsk total 0 crw-r----- 1 bin sys . . . crw-r----- 1 bin sys f.
Recovery by Reinstallation h. Start, mount and list all the volumes: # vxvol startall # mountall # mount / on /dev/vx/dsk/rootdg/rootvol log on Wed Apr 17 14:15:29 2002 /var on /dev/vx/dsk/rootdg/varvol delaylog,nodatainlog on ... /usr on /dev/vx/dsk/rootdg/usrvol delaylog,nodatainlog on ... /tmp on /dev/vx/dsk/rootdg/tmpvol delaylog,nodatainlog on ... /stand on /dev/vx/dsk/rootdg/standvol delaylog,nodatainlog on ... /opt on /dev/vx/dsk/rootdg/optvol delaylog,nodatainlog on ...
Recovery by Reinstallation General Reinstallation Information This section describes procedures used to reinstall VxVM and preserve as much of the original configuration as possible after a failure. Note System reinstallation destroys the contents of any disks that are used for reinstallation. All VxVM-related information is removed during reinstallation. Data removed includes data in private areas on removed disks that contain the disk identifier and copies of the VxVM configuration.
Recovery by Reinstallation Reinstalling the System and Recovering VxVM To reinstall the system and recover the VERITAS Volume Manager configuration, use the following procedure. These steps are described in detail in the sections that follow: 1. “Prepare the System for Reinstallation” on page 29. Replace any failed disks or other hardware, and detach any disks not involved in the reinstallation. 2. “Reinstall the Operating System” on page 30.
Recovery by Reinstallation Reinstall the Operating System Once any failed or failing disks have been replaced and disks not involved with the reinstallation have been detached, reinstall the operating system as described in your operating system documentation. Install the operating system prior to installing VxVM. Ensure that no disks other than the root disk are accessed in any way while the operating system installation is in progress.
Recovery by Reinstallation 5. When the system comes up, bring the system to single-user mode using the following command: # exec init S 6. When prompted, enter the password and press Return to continue. 7. Remove files involved with installation that were created when you loaded VxVM but are no longer needed using the following command: # rm -rf /etc/vx/reconfig.d/state.d/install-db 8. Start some VERITAS Volume Manager I/O daemons using the following command: # vxiod set 10 9.
Recovery by Reinstallation If the root disk (or another disk) was involved with the reinstallation, any volumes or mirrors on that disk (or other disks no longer attached to the system) are now inaccessible. If a volume had only one plex contained on a disk that was reinstalled, removed, or replaced, then the data in that volume is lost and must be restored from backup.
Recovery by Reinstallation Note Your system may use a device name that differs from the examples. For more information on device names, see the chapter “Administering Disks” in the VERITAS Volume Manager Administrator’s Guide. The display shows that the reinstalled root device, c0t0d0, is not associated with a VM disk and is marked with a status of error. The disks disk02 and disk03 were not involved in the reinstallation and are recognized by VxVM and associated with their devices (c0t1d0 and c0t2d0).
Recovery by Reinstallation 4. Because v01-01 was the only plex of the volume, the volume contents are irrecoverable except by restoring the volume from a backup. The volume must also be removed. If a backup copy of the volume exists, you can restore the volume later. Keep a record of the volume name and its length, as you will need it for the backup procedure. Remove irrecoverable volumes (such as v01) using the following command: # vxedit -r rm v01 5.
Recovery by Reinstallation v pl sd pl sd v03 fsgen v03-01 v03 disk02-01 v03-01 v03-02 v03 disk01-04 v03-02 DISABLEDACTIVE 0720 DISABLEDACTIVE 30720 disk01 620544 30720 DISABLEDNODEVICE30720 disk03 262144 30720 SELECT CONCAT 0 CONCAT 0 c1t5d5 c1t5d6 RW ENA RW DIS This volume has two plexes, v03-01 and v03-02. The first plex (v03-01) does not use any space on the invalid disk, so it can still be used. The second plex (v03-02) uses space on invalid disk disk01 and has a state of NODEVICE.
Recovery by Reinstallation For example, to recreate the volumes v01 and v02, use the following command: # vxassist make v01 24000 # vxassist make v02 30720 layout=stripe nstripe=3 Once the volumes are created, they can be restored from backup using normal backup/restore procedures. Recreate any plexes for volumes that had plexes removed as part of the volume cleanup.
3 Error Messages Introduction This chapter provides information on error messages associated with the VERITAS Volume Manager (VxVM) configuration daemon (vxconfigd), the kernel, and other utilities. It covers most informational, failure, and error messages displayed on the console by vxconfigd, and by the VERITAS Volume Manager kernel driver, vxio. These include some errors that are infrequently encountered and difficult to troubleshoot.
Configuring Logging in the Startup Script There are 9 possible levels of debug logging; 1 provides the least detail, and 9 the most. To enable syslog logging of console output, specify the option -x syslog to vxconfigd as shown here: # vxconfigd [-x [1-9]] -x syslog Messages with a priority higher than Debug are written to /var/adm/syslog/syslog.log, and all other messages are written to /var/adm/configd.log.
Understanding Error Messages Understanding Error Messages VxVM is fault-tolerant and resolves most problems without system administrator intervention. If the configuration daemon (vxconfigd) recognizes the actions that are necessary, it queues up the transactions that are required. VxVM provides atomic changes of system configurations; either a transaction completes fully, or the system is left in the same state as though the transaction was never attempted.
Kernel Warning Messages vxvm:vxio:WARNING: check_ilocks: overlapping ilocks: offset for length, offset for length vxvm:vxio:WARNING: check_ilocks: stranded ilock on object_name start offset len length ◆ Description: These internal errors do not occur unless there is a bug in VxVM. ◆ Action: Contact Customer Support.
Kernel Warning Messages vxvm:vxio:WARNING: Failed to log the detach of the DRL volume volume ◆ Description: An attempt failed to write a kernel log entry indicating the loss of a DRL volume. The attempted write to the log failed either because the kernel log is full, or because of a write error to the drive. The volume becomes detached. ◆ Action: Messages about log failures are usually fatal, unless the problem is transient.
Kernel Warning Messages vxvm:vxio:WARNING: mod_install returned errno ◆ Description: A call made to the operating system mod_install function to load the vxio driver failed. ◆ Action: Check for additional console messages that may explain why the load failed. Also check the console messages log file for any additional messages that were logged but not displayed on the console.
Kernel Warning Messages vxvm:vxio:WARNING: RAID-5 volume entering degraded mode operation ◆ Description: An uncorrectable error has forced a subdisk to detach. At this point, not all data disks exist to provide the data upon request. Instead, parity regions are used to regenerate the data for each stripe in the array. Consequently, access takes longer and involves reading from all drives in the stripe. ◆ Action: Check for other console error messages that indicate the cause of the failure.
Kernel Notice Messages If this error occurs often, but never leads to a plex detach, there may be a marginally defective region on the disk at the position shown. It may eventually be necessary to remove data from this disk (see the vxevac(1M) manual page) and then to reformat the drive. Kernel Notice Messages A notice message indicates that an error has occurred that should be monitored. Shutting down the system is unnecessary, although you may need to take action to remedy the fault at a later date.
vxassist Error Messages vxassist Error Messages An error message from the vxassist command indicates that the requested operation cannot be performed. Follow the recommended course of action given below. vxvm:vxassist: ERROR: Insufficient number of active snapshot mirrors in snapshot_volume. ◆ Description: An attempt to snap back a specified number of snapshot mirrors to their original volume failed. ◆ Action: Specify a number of snapshot mirrors less than or equal to the number in the snapshot volume.
vxconfigd Fatal Error Messages vxconfigd Fatal Error Messages A fatal error message from the configuration daemon, vxconfigd, indicates a severe problem with the operation of VxVM that prevents it from running.
vxconfigd Error Messages vxvm:vxconfigd: ERROR: Cannot get record record_name from kernel: reason ◆ Description: These internal errors should not occur unless there is a bug in VxVM. ◆ Action: Contact Customer Support. vxvm:vxconfigd: ERROR: Cannot kill existing daemon, pid=process_ID ◆ Description: The -k (kill existing vxconfigd process) option was specified, but a running configuration daemon process could not be killed.
vxconfigd Error Messages ◆ - The VERITAS Volume Manager package installation did not complete correctly. - The device node was removed by the administrator or by an errant shell script. Action: If the reason is “Device is already open,” stop or kill the old vxconfigd by running the command: # vxdctl -k stop For other failure reasons, consider re-adding the base VERITAS Volume Manager package. This will reconfigure the device node and re-install the VERITAS Volume Manager kernel device drivers.
vxconfigd Error Messages ◆ Description: These errors indicate that the volume cannot be started because the volume contains no valid plexes. This can happen, for example, if disk failures have caused all plexes to be unusable. It can also happen as a result of actions that caused all plexes to become unusable (for example, forcing the dissociation of subdisks or detaching, dissociation, or offlining of plexes). ◆ Action: It is possible that this error results from a drive that failed to spin up.
vxconfigd Error Messages ◆ Action: Manual use of the vxdg recover command may be required to clean the disk group to be imported. See “Recovering from Incomplete Disk Group Moves” on page 15 for more information. vxvm:vxconfigd: ERROR: Differing version of vxconfigd installed ◆ Description: A vxconfigd daemon was started after stopping an earlier vxconfigd with a non-matching version number. This can happen, for example, if you upgrade VxVM and then run vxconfigd without first rebooting.
vxconfigd Error Messages a. Ensure that no vxvol, vxplex, or vxsd processes are running. Use ps -e to search for such processes, and use kill to kill any that you find. You may have to run kill twice to make these processes go away. Killing utilities in this way may make it difficult to make administrative changes to some volumes until the system is rebooted. b.
vxconfigd Error Messages The most common reason for auto-import failures is excessive numbers of disk failures, making it impossible for VxVM to find correct copies of the disk group configuration database and kernel update log. Disk groups usually have enough copies of this configuration information to make such import failures unlikely.
vxconfigd Error Messages ◆ Description: Two disk groups with the same name are tagged for auto-importing by the same host. Disk groups are identified both by a simple name and by a long unique identifier (disk group ID) assigned when the disk group is created. Thus, this error indicates that two disks indicate the same disk group name but a different disk group ID.
vxconfigd Error Messages vxvm:vxconfigd: ERROR: Disk group group: update failed: reason ◆ Description: I/O failures have prevented vxconfigd from updating any active copies of the disk group configuration. This usually implies a large number of disk failures.
vxconfigd Error Messages Evaluate the error messages to determine the root cause of the problem. Make changes suggested by the errors and then try rerunning the command. vxvm:vxconfigd: ERROR: Failed to store commit status list into kernel: reason vxvm:vxconfigd: ERROR: GET_VOLINFO ioctl failed: reason vxvm:vxconfigd: ERROR: Get of current rootdg failed: reason ◆ Description: These internal errors should not occur unless there is a bug in VxVM. ◆ Action: Contact Customer Support.
vxconfigd Error Messages the database copy to find the list of disk access records for disks contained in the group. These disks are then examined to ensure that they contain the same database copy. The algorithm expects to gain convergence on the set of disks and on the database copies that they contain. If a loop is entered and convergence cannot be reached, this message is displayed and the root disk group importation fails.
vxconfigd Error Messages vxvm:vxconfigd: ERROR: found in kernel vxvm:vxconfigd: ERROR: reconfiguration: reason vxvm:vxconfigd: ERROR: reason vxvm:vxconfigd: ERROR: Unexpected configuration tid for group group Unexpected error during volume volume Unexpected error fetching disk for disk volume: Unexpected values stored in the kernel ◆ Description: These internal errors should not occur unless there is a bug in VxVM. ◆ Action: Contact Customer Support.
vxconfigd Warning Messages (causing the apparent access order of the two disk groups to be reversed). This can also happen if some disk group was deported and renamed to rootdg with locks given to this host. ◆ Action: In case 1, boot the system on a CD-ROM or networking-mounted root file system. If the root file system is defined on a volume, then start and mount the root volume. If the root file system is not defined on a volume, mount the root file system directly.
vxconfigd Warning Messages vxvm:vxconfigd: WARNING: Cannot exec /usr/bin/rm to remove directory: reason ◆ Description: The given directory could not be removed because the /usr/bin/rm utility could not be executed by vxconfigd. This is not a serious error. The only side effect of a directory not being removed is that the directory and its contents continue to use space in the root file system.
vxconfigd Warning Messages ◆ Description: This error only happens for volumes that are started automatically by vxconfigd at system startup. The plex is being detached as a result of I/O failure, disk failure during startup or prior to the last system shutdown or crash, or disk removal prior to the last system shutdown or crash. ◆ Action: To ensure that the file system retains the same number of active mirrors, remove the given plex and add a new mirror using the vxassist mirror operation.
vxconfigd Warning Messages ◆ Description: This internal error should not occur unless there is a bug in VxVM. ◆ Action: Contact Customer Support. vxvm:vxconfigd: WARNING: Disk disk names group group, but group ID differs ◆ Description: As part of a disk group import, a disk was discovered that had a mismatched disk group name and disk group ID. This disk is not imported. This can only happen if two disk groups have the same name but have different disk group ID values.
vxconfigd Warning Messages vxvm:vxconfigd: WARNING: Disk group group is disabled, disks not updated with new host ID ◆ Description: As a result of failures, the named disk group has become disabled. Earlier error messages should indicate the cause. This message indicates that disks in that disk group were not updated with a new VERITAS Volume Manager host ID. This warning message should result only from a vxdctl hostid operation.
vxconfigd Notice Messages vxvm:vxconfigd: WARNING: Group group: Duplicate virtual device number(s): Volume volume remapped from major,minor to major,minor ... ◆ Description: The configuration of the named disk group includes conflicting device numbers. A disk group configuration lists the recommended device number to use for each volume in the disk group. If two volumes in two disk groups happen to list the same device number, then one of the volumes must use an alternate device number.
vxconfigd Notice Messages ◆ Action: If hot-relocation is enabled, VERITAS Volume Manager objects affected by the disk failure are taken care of automatically. Mail is sent to root indicating what actions were taken by VxVM and what further actions the administrator should take. vxvm:vxconfigd: NOTICE: Detached log for volume volume ◆ Description: The DRL or RAID-5 log for the named volume was detached as a result of a disk failure, or as a result of the administrator removing a disk with vxdg -k rmdisk.
vxconfigd Notice Messages ◆ Action: Consider replacing the indicated disk, since this error implies that the disk has deteriorated to the point where write errors cannot be repaired automatically. The error can also result from transient problems with cabling or power. vxvm:vxconfigd: NOTICE:Unable to resolve duplicate diskid.
vxdg Error Messages - Case 2: Some arrays such as EMC and HDS provide mirroring in hardware. When a LUN pair is split, depending on how the process is performed, this may result in two disks with the same disk ID. Check with your array vendor to make sure that you are using the correct split procedure. If you know which LUNs you want to use, choose which path to exclude, and then either edit the file /etc/vx/vxvm.
vxdg Error Messages vxvm: vxdg: ERROR: diskgroup: Configuration too large for configuration copies ◆ Description: The disk group’s configuration database is too small to hold the expanded configuration after a disk group move or join operation. ◆ Action: None. vxvm: vxdg: ERROR: diskgroup: Disk group does not exist ◆ Description: The disk group does not exist or is not imported ◆ Action: Use the correct name, or import the disk group and try again.
vxdg Error Messages vxvm: vxdg: ERROR: diskdevice: Request crosses disk group boundary ◆ Description: The specified disk device is not configured in the source disk group for a disk group move or split operation. ◆ Action: Correct the name of the disk object specified in the disk group move or split operation. vxvm: vxdg: ERROR: diskgroup: split failed: Error in cluster processing ◆ Description: The host is not the master node in the cluster. ◆ Action: Perform the operation from the master node.
vxdmp Notice Messages vxvm: vxdg: ERROR: vxdg listmove sourcedg targetdg failed vxvm:vxdg: ERROR: diskname : Disk not moving, but subdisks on it are ◆ Description: Some volumes have subdisks that are not on the disks implied by the supplied list of objects. ◆ Action: Use the -o expand option to vxdg listmove to produce a self-contained list of objects.
vxdmp Notice Messages vxvm:vxdmp:NOTICE:disabled controller controller_name connected to disk array disk_array_serial_number ◆ Description: All paths through the controller connected to the disk array are disabled. This usually happens if a controller is disabled for maintenance. ◆ Action: None. vxvm:vxdmp:NOTICE:disabled dmpnode dmpnode_device_number ◆ Description: A DMP node has been marked disabled in the DMP database. It will no longer be accessible for further IO requests.
vxdmpadm Error Messages vxvm:vxdmp:NOTICE: Path failure on major/minor ◆ Description: A path under the control of the DMP driver failed. The device major and minor numbers of the failed device is supplied in the message. ◆ Action: None. vxvm:vxdmp:NOTICE:removed disk array disk_array_serial_number ◆ Description: A disk array has been disconnected from the host, or some hardware failure has resulted in the disk array becoming inaccessible to the host.
vxplex Error Messages vxplex Error Messages An error message from the plex administration utility, vxplex, indicates a problem with the requested operation. vxvm:vxplex: ERROR: Plex plex not associated with a snapshot volume. ◆ Description: An attempt was made to snap back a plex that is not from a snapshot volume. ◆ Action: Specify a plex from a snapshot volume. vxvm:vxplex: ERROR: Plex plex not attached. ◆ Description: An attempt was made to snap back a detached plex.
Cluster Error Messages Cannot find disk on slave node ◆ Description: A slave node in a cluster cannot find a shared disk. This is accompanied by the syslog message: vxvm:vxconfigd cannot find disk disk ◆ Action: Make sure that the same set of shared disks is online on both nodes. Examine the disks on both the master and the slave with the command vxdisk list and make sure that the same set of disks with the shared flag is visible on both nodes. If not, check the connections to the disks.
Cluster Error Messages Error in cluster processing ◆ Description: This may be due to an operation inconsistent with the current state of a cluster (such as an attempt to import or deport a shared disk group to or from the slave). It may also be caused by an unexpected sequence of commands from vxclust. ◆ Action: Make sure that the operation can be performed in the current environment.
Cluster Error Messages Join not allowed now ◆ Description: A slave attempted to join a cluster when the master was not ready. The slave will retry automatically. If the retry succeeds, the following message appears: vxclust: slave join complete ◆ Action: No action is necessary if the join eventually completes. Otherwise, investigate the cluster monitor on the master. Master sent no data ◆ Description: During the slave join protocol, a message without data was received from the master.
Cluster Error Messages NOTICE: vol_kmsg_send_wait_callback: got error 22 ◆ Description: This message may appear during a plex detach operation on a slave in a cluster. Action: None required. Retry rolling upgrade ◆ Description: An attempt was made to upgrade the cluster to a higher protocol version when a transaction was in progress. ◆ Action: Retry at a later time.
Cluster Error Messages Upgrade operation failed: Version out of range for at least one node ◆ Description: Before trying to upgrade a cluster by running vxdctl upgrade, all nodes should be able to support the new protocol version. An upgrade can fail if at least one of them does not support the new protocol version. ◆ Action: Make sure that the VERITAS Volume Manager package that supports the new protocol version is installed on all nodes and retry the upgrade.
Cluster Error Messages WARNING: vxvm:vxio: Plex plex detached from volume volume ◆ Description: This message may appear during a plex detach operation in a cluster. ◆ Action: None required. WARNING: vxvm:vxio: read error on plex plex of shared volume volume offset offset length length 78 ◆ Description: This message may appear during a plex detach operation on the master in a cluster. ◆ Action: None required.
Index re-installing 26 DISABLED plex kernel state 2, 8 disk group recovery from failed move, split or join 15 disk group errors name conflict 52 new host ID 50 disk IDs fixing duplicate 65 disks causes of failure 1 cleaning up configuration 35 failures 7 fixing duplicated IDs 65 reattaching 5 DMP fixing duplicated disk IDs 65 Symbols /etc/ioconfig file 25 /stand/ioconfig file 25 /var/adm/configd.log file 37 /var/adm/syslog/syslog.
kernel 46 Cannot get all disks from the kernel 46 Cannot get kernel transaction state 46 Cannot get private storage from the kernel 46 Cannot get private storage size from the kernel 46 Cannot get record from the kernel 47 Cannot kill existing daemon 47 Cannot make directory 47 cannot open /dev/vx/config 47 Cannot open /etc/ioconfig with specified flags 25 Cannot recover operation in progress 48 Cannot recover temp database 50 Cannot remove last disk group configuration copy 66 Cannot reset VxVM kernel 48 C
Plex plex not attached 72 Plexes do not belong to the same snapshot volume 72 RAID-5 plex does not map entire volume length 13 Read of directory failed 56 Record already exists in disk group 67, 68 Record is associated 67 Record volume is in disk group diskgroup1 plex in in group diskgroup2 72 Reimport of disk group failed 53 Request crosses disk group boundary 68 Retry rolling upgrade 76 Return from cluster_establish is Configuration daemon error 76 Rootdg cannot be imported during boot 24 Segmentation fau
Detached disk 63 Detached log for volume 64 Detached plex in volume 64 Detached subdisk in volume 64 Detached volume 64 disabled controller connected to disk array 70 disabled dmpnode 70 disabled path belonging to dmpnode 70 enabled controller connected to disk array 70 enabled dmpnode 70 enabled path belonging to dmpnode 70 ktcvm_check sent to slave node 75 Offlining config copy 64 Path failure 71 read error on object 44 Reason found for abort 75 removed disk array 71 Rootdisk has just one enabled path 69
detached subdisks 7 failures 6 hot-relocation 9 importance of log plexes 6 parity resynchronization 10 recovering log plexes 11 recovering stale subdisk 11 recovering volumes 9 recovery process 8 stale parity 6 starting forcibly 14 starting volumes 12 startup recovery process 8 subdisk move recovery 12 unstartable volumes 12 reattaching disks 5 reconstructing-read mode, stale subdisks 7 recovery after booting from recovery media 20 after VxVM emergency startup 20 disk 5 reinstalling entire system 27 REPLAY
vxdisksetup used to initialize a replacement root disk 19 vxdmp NOTICE messages 69 vxdmpadm ERROR messages 71 vxinfo command 4 vxmend command 4 vxplex ERROR messages 72 vxplex command 11 vxreattach command 5 VxVM emergency startup 20 maintenance mode 21 obtaining system information x RAID-5 recovery process 8 recovering configuration of 30 reinstalling 30 starting after booting from recovery media 20 vxvmboot used to write VxVM information to LABEL file 23 vxvol recover command 11 vxvol resync command 10 vx
Uncorrectable read error 42 Uncorrectable write error 42 vold_turnclient failed 63 volume already has at least one snapshot Index plex 45 volume is detached 40 Volume remapped 63 write error on mirror plex of volume 43 85
