When Good Disks Go Bad: Dealing with Disk Failures under LVM Abstract ..............................................................................................................................................3 Background .........................................................................................................................................3 1. Preparing for Disk Recovery ..........................................................................................................
Scenario 1: Best Case .................................................................................................................19 Scenario 2: No Mirroring and No LVM Online Replacement ...........................................................20 Scenario 3: No Hot-Swappable Disk.............................................................................................21 Disk Replacement Flowchart......................................................................................................
Abstract This white paper discusses how to deal with disk failures under the HP-UX Logical Volume Manager (LVM). It is intended for system administrators or operators who have experience with LVM. It includes strategies for preparing for disk failure, means for recognizing that a disk has failed, and steps for removing or replacing a failed disk. Background Whether managing a workstation or server, your goals include minimizing system downtime and maximizing data availability.
1. Preparing for Disk Recovery Forewarned is forearmed. Knowing that hard disks will fail eventually, you can take some precautionary measures to minimize your downtime, maximize your data availability, and simplify the recovery process. Consider the following guidelines before you experience a disk failure. Defining a Recovery Strategy As you create logical volumes, choose one of the following recovery strategies. Each choice strikes a balance between cost, data availability, and speed of data recovery.
# swlist -l fileset | grep -i mirror LVM.LVM-MIRROR-RUN B.11.23 LVM Mirror The process of mirroring is usually straightforward, and can be easily accomplished using the system administration manager SAM, or with a single lvextend command. These processes are documented in Managing Systems and Workgroups. The only mirroring setup task that takes several steps is mirroring the root disk; for the recommended procedure for adding a root disk mirror, refer to Appendix A: Procedures.
reinstall a minimal system, restore a backup, and be back online within three hours of diagnosis and replacement of hardware. Three disks in the root volume group are better than two, because of quorum restrictions. With a two-disk root volume group, a loss of one disk can require you to override quorum to activate the volume group; if you must reboot to replace the disk, you must interrupt the boot process and use the –lq boot option.
2. Recognizing a Failing Disk The guidelines in the previous section will not prevent disk failure on your system. Assuming you follow all the recommendations, how can you tell when a disk has failed? This section explains how to look for signs that one of your disks is having problems, and how to determine which disk it is. I/O Errors in the System Log Often an error message in the system log file is your first indication of a disk problem. In /var/adm/syslog/syslog.
LVM Command Errors Sometimes LVM commands, such as vgdisplay, return an error suggesting that a disk has problems. For example: # vgdisplay –v | more … --- Physical volumes --PV Name PV Status Total PE Free PE … /dev/dsk/c0t3d0 unavailable 1023 173 The physical volume status of unavailable indicates that LVM is having problems with the disk. You can get the same status information from pvdisplay.
3. Confirming Disk Failure Once you suspect a disk has failed or is failing, make certain that the suspect disk is indeed failing. Replacing or removing the incorrect disk makes the recovery process take longer. It can even cause data loss. For example, in a mirrored configuration, if you were to replace the wrong disk—the one holding the current good copy rather than the failing disk—the mirrored data on the good disk is lost. It is also possible that the suspect disk is not failing.
For example, the following shows a successful read of the first 64 megabytes of the disk: # dd if=/dev/rdsk/c0t5d0 of=/dev/null bs=1024k count=64 64+0 records in 64+0 records out The following shows an unsuccessful read of the whole disk: # dd if=/dev/rdsk/c1t3d0 of=/dev/null bs=1024k dd read error: I/O error 0+0 records in 0+0 records out 10
4. Gathering Information about a Failing Disk Once you know which disk is failing, you can decide how to deal with it. You can choose to remove the disk if your system does not need it, or you can choose to replace it. Before deciding on your course of action, you must gather some information to help guide you through the recovery process. Is the Questionable Disk Hot-Swappable? This determines whether you must power down your system to replace the disk.
extents are mapped to an unavailable physical volume. The lvdisplay command shows ’???’ for the physical volume if it is unavailable. The problem with this approach is that it is not precise if more than one disk is unavailable; to ensure that multiple simultaneous disk failures have not occurred, run vgdisplay to see if the active and current number of physical volumes differs by exactly one. A third option for determining which logical volumes are on the disk is to use the vgcfgdisplay command.
5. Removing the Disk If you have a copy of the data on the failing disk, or you can move the data to another disk, you can choose to remove the disk from the system instead of replacing it. Removing a Mirror Copy from a Disk If you have a mirror copy of the data already, you can stop LVM from using the copy on the failing disk by reducing the number of mirrors. To remove the mirror copy from a specific disk, use lvreduce, and specify the disk from which to remove the mirror copy.
physical extents on that disk to any other disks in the volume group, subject to any mirroring allocation policies. For example: # pvmove pvname You can select a particular target disk or disks, if desired. For example, to move all the physical extents from c0t5d0 to the physical volume c0t2d0, enter the following command: # pvmove /dev/dsk/c0t5d0 /dev/dsk/c0t2d0 You can choose to move only the extents belonging to a particular logical volume.
# vgreduce -f vgname # mv /etc/lvmtab /etc/lvmtab.save # vgscan –v This completes the procedure for removing the disk from your LVM configuration. If the disk hardware allows it, you can remove it physically from the system. Otherwise, physically remove it at the next scheduled system reboot.
6. Replacing the Disk If you decide to replace the disk, you must perform a five-step procedure. How you perform each step depends on the information you gathered earlier (hot-swap information, logical volume names, and recovery strategy), so this procedure varies. This section also includes several common scenarios for disk replacement, and a flowchart summarizing the disk replacement procedure. The five steps are: 1. 2. 3. 4. 5. Temporarily halt LVM attempts to access the disk.
3. To kill processes using the logical volume, enter the following command: # fuser –ku /dev/vgname/lvname 4. Then try to unmount the file system again as follows: # umount /dev/vgname/lvname o If the logical volume is being accessed as a raw device, you can use fuser to find out which applications are using it. Then you can halt those applications.
Step 2: Replacing the Faulty Disk If the disk is hot-swappable, you can replace it without powering down the system. Otherwise, power down the system before replacing the disk. For the hardware details on how to replace the disk, refer to the hardware administrator’s guide for the system or disk array. If you powered down the system, reboot it normally. The only exception is if you replaced a disk in the root volume group.
Step 5: Restoring Lost Data to the Disk This final step can be a straightforward resynchronization for mirrored configurations, or a recovery of data from backup media. • If a mirror of the root disk was replaced, initialize its boot information: • For an Integrity server, follow steps 5, 6, and 8 in Appendix A: Mirroring the Root Volume on Integrity Servers. • For a PA-RISC server, follow steps 4, 5, and 7 in Appendix A: Mirroring the Root Volume on PA-RISC Servers.
Root: lvol3 on: /dev/dsk/c0t5d0 Swap: lvol2 on: /dev/dsk/c0t5d0 Dump: lvol2 on: /dev/dsk/c0t5d0, 0 # pvdisplay –v /dev/dsk/c2t15d0 | more … --- Distribution of physical volume --LV Name LE of LV PE for LV /dev/vg01/lvol1 4340 4340 … # lvdisplay –v /dev/vg01/lvol1 | grep “Mirror copies” Mirror copies 1 # lvdisplay -v /dev/vg01/lvol1 | grep –e /dev/dsk/c2t15d0 –e ’???’ | more 00000 /dev/dsk/c2t15d0 00000 current /dev/dsk/c5t15d0 00000 current 00001 /dev/dsk/c2t15d0 00001 current /dev/dsk/c5t15d0 00001 current
# fuser -u /dev/vg01/lvol1 /dev/vg01/lvol1: 27815c(root) 27184c(root) # ps -fp27815 -p27184 UID PID PPID C STIME TTY TIME COMMAND root 27815 27184 0 09:04:05 pts/0 0:00 vi test.c root 27184 27182 0 08:26:24 pts/0 0:00 -sh # fuser -ku /dev/vg01/lvol1 /dev/vg01/lvol1: 27815c(root) 27184c(root) # umount /dev/vg01/lvol1 For this example, it is assumed that you are permitted to halt access to the entire volume group while you recover the disk.
Disk Replacement Flowchart The following flowchart summarizes the disk replacement process.
Conclusion In your role as system manager, you will encounter disk failures. LVM can lessen the impact of those disk failures, enabling you to configure your data storage to make a disk failure transparent to users, and to keep your system and data available during the recovery process. By making use of hardware features such as hot-swappable disks and software features such as mirroring and online disk replacement, you can maximize your system availability and minimize data loss due to disk failure.
Appendix A: Procedures This section contains details on some of the procedures described earlier. Mirroring the Root Volume on PA-RISC Servers To set up a mirrored root configuration, you must add a disk to the root volume group, mirror all the root logical volumes onto it, and make it bootable. For this example, the disk is at path 2/0/7.15.0 and has device special files named /dev/rdsk/c2t15d0 and /dev/dsk/c2t15d0. 1. Use the insf command with the -e option to make sure the device files are in place.
# # # # lvextend lvextend lvextend lvextend –m –m –m –m 1 1 1 1 /dev/vg00/lvol5 /dev/vg00/lvol6 /dev/vg00/lvol7 /dev/vg00/lvol8 /dev/dsk/c2t15d0 /dev/dsk/c2t15d0 /dev/dsk/c2t15d0 /dev/dsk/c2t15d0 6. Update the root volume group information: # lvlnboot -R /dev/vg00 7. Verify that the mirrored disk is displayed as a boot disk and that the boot, root, and swap logical volumes appear to be on both disks: # lvlnboot –v 8.
Mirroring the Root Volume on Integrity Servers The procedure to mirror the root disk on Integrity servers is similar to the procedure for PA-RISC servers. The difference is that Integrity server boot disks are partitioned; you must set up the partitions, copy utilities to the EFI partition, and use the HP-UX partition device files for LVM commands. For this example, the disk is at hardware path 0/1/1/0.1.0, with a device special file named /dev/rdsk/c2t1d0. 1.
# mkboot –e –l /dev/rdsk/c2t1d0 6. Update the autoboot file in the EFI partition. a. Create an AUTO file in the current directory. If you expect to boot from this disk only when you have lost quorum, you can use the alternate string “boot vmunix –lq” to disable quorum checking: # echo “boot vmunix” > ./AUTO b. Copy the file from the current directory into the new disk EFI partition. Make sure to use the device file with the s1 suffix: # efi_cp -d /dev/rdsk/c2t1d0s1 ./AUTO /efi/hpux/auto 7.
11. Add a line to /stand/bootconf for the new boot disk using vi or another text editor: # vi /stand/bootconf l /dev/dsk/c2t1d0s2 where l denotes LVM.
Appendix B: LVM Error Messages This appendix lists some of the warning and error messages reported by LVM. For each message, the cause is listed, and an administrator action is recommended. The appendix is divided into two sections, one for LVM command errors, and one for the system log file /var/adm/syslog/syslog.log error messages.
Act PV 1 Max PE per PV 4350 VGDA 2 PE Size (Mbytes) 4 Total PE 4340 Alloc PE 3740 Free PE 600 Total PVG 0 Total Spare PVs 0 Total Spare PVs in use 0 In this example, the total free space is 600 * 4 MB, or 2400 MB. b. The logical volume is mirrored with a strict allocation policy, and there are not enough extents on a separate disk to comply with the allocation policy.
Cause: LVM OLR is not completely installed. Both the LVM command and kernel components are required to enable LVM OLR. In this case, the command patch is installed and the kernel patch is not. Recommended Action: Install the appropriate kernel patch to enable LVM OLR, or use an alternate replacement procedure. Unable to detach the path or physical volume via the pathname provided.
Detach the physical volume or deactivate the volume group before attempting to restore the physical volume. If there is reason to believe that the data on the disk is corrupted, the disk can be detached and marked using vgcfgrestore then attached again without replacing the disk. This causes LVM to reinitialize the disk and synchronize any mirrored user data mapped there.
vgchange: Couldn't set the unique id for volume group "/dev/vgname" Cause: There are multiple LVM group files with the same minor number. Recommended Action: List the LVM group files. If there are any duplicate minor numbers, export one of the affected volume groups, create a new group file with a unique minor number, and re-import the volume group. If you are not familiar with this procedure, contact your HP support representative for assistance. # ll /dev/*/group # vgexport -m vgname.map -v -f vgname.
Warning: couldn't query physical volume "pvname": The specified path does not correspond to physical volume attached to this volume group Warning: couldn't query all of the physical volumes. Cause: This error has several possible causes. They are described earlier under the vgchange error messages. Recommended Action: Refer to the recommended actions under the vgchange error messages. vgextend vgextend: Not enough physical extents per physical volume. Need: #, Have: #.
Syslog Error Messages LVM: lost only from VG 64 0x010000: Data in one or more logical volumes on PV 188 0x072000 was when the disk was replaced. This occurred because the disk contained the copy of the data. Prior to using these logical volumes, restore the data backup. Cause: LVM cannot synchronize the data on a replaced disk automatically, as when LVM discovers an unmirrored logical volume residing on a disk that was just replaced.
For more information To learn more about LVM and HP-UX system administration, refer to the following documents on the HP documentation website http://docs.hp.com/: • • LVM Online Disk Replacement (LVM OLR) http://docs.hp.com/en/7161/LVM_OLR_whitepaper.pdf Managing Systems and Workgroups http://docs.hp.com/en/B2355-90912/B2355-90912.pdf Call to action HP welcomes your input.
