HP-UX 11i v3 Native Multi-Pathing for Mass Storage Technical white paper Table of contents Audience ............................................................................................................................................ 2 Introduction ......................................................................................................................................... 2 Overview...........................................................................................................
Audience This document describes the native multi-pathing feature offered by the mass storage subsystem on HPUX 11i v3. It is intended for system administrators or operators. For a comprehensive description of all the features of the HP-UX 11i v3 mass storage subsystem refer to The Next Generation Mass Storage Stack HP-UX 11i v3 white paper.
Overview Features Multi-pathing is the ability to manage the various paths to a LUN device. A path to a LUN is also known as an I_T_L nexus. It corresponds to the route taken by an I/O request issued from an HP-UX host to a SCSI device. The following is an overview of the native multi-pathing features. For more information, see Native Multi-Pathing Features.
New SAN resources are readily made available to the host because of the automated discovery of SCSI devices. You can delete stale SAN resources to free up system resources. Scalability The mass storage subsystem can theoretically address up to 224 (16 million) LUNs. There is no architectural limit on the number of lunpaths allowed to a LUN. For information on the maximum number of LUNs and lunpaths that were tested, see the HP-UX 11i v3 Release Notes.
Native multi-pathing features LUN WWID and DSF The various lunpaths to LUN devices are correlated based on a world-wide unique identifier (WWID) that is associated with each LUN device. Indeed, the host can determine that different lunpaths lead to the same LUN only if a LUN can be uniquely identified by the operating system. The LUN WWID is based on SPC-3 commands Inquiry and VPD Inquiry Page 83h data.
• Round-robin (round_robin) – This policy distributes the I/O load equally across all active lunpaths irrespective of the current load on each lunpath. It is suitable when lunpaths have similar I/O operation turnaround characteristics. • Least command load (least_cmd_load) – This policy selects the lunpath with the least number of pending I/O requests for the next I/O operation. It is suitable when lunpaths have asymmetric performance characteristics.
Note For information on determining or configuring the optimized/unoptimized controller or target port, see the disk device’s user documentation. Based on this information you can set the preferred_path and the pref_tport attributes appropriately on the host system. • Weighted round-robin (weighted_rr) – This policy distributes the I/O load across all active lunpaths in a round-robin manner and according to the weight assigned to each lunpath.
messages in the system log, then the SCSI subsystem resorts to path failover and recovery mechanisms to provide applications with continuous access to the LUN end devices; only when certain critical errors happen, is administrator intervention required. In summary, after detecting a SCSI component error, the operating system reports the error to system administrators and offers a palette of pro-active recovery actions: automatic path failover or dynamic replacement of the failing component.
Path recovery policies When a disk device no longer responds to I/O operations sent through a lunpath for a configurable period of time (path_fail_secs attribute), this lunpath is considered offline, and is no longer used for I/O transfer. Prior to the March 2008 release of HP-UX 11i v3, the system automatically monitors the lunpath by periodically sending inquiry requests to the device through it, and declares the lunpath back online when the inquiry requests succeed.
Media based errors If the SCSI stack detects the LUN disk device could not commit a write from its cache to the media, then it freezes the I/O traffic to the disk to prevent data corruption. When the error condition on the LUN has been cleared, the system administrator must run scsimgr enable –D lun_DSF to resume I/O operations on the LUN.
• Newly discovered active lunpaths are automatically used in the lunpath selection scheme of I/O load balancing. Therefore, applications sending I/O operations through existing LUN DSFs are not affected by the discovery of new lunpaths. • New LUN DSFs corresponding to newly discovered LUNs are seemlessly created. SAN component deletion Throughout the life of a system, SAN components may be replaced or removed.
For more information, see the LVM Migration from Legacy to Agile Naming Model HP-UX 11i v3 white paper. File systems In HP-UX 11i v3, HFS supports both legacy and persistent LUN DSFs. Therefore, HFS takes advantage of multi-pathing by default. For support of VxFS on HP-UX 11iv3, see the release notes from Symantec. Commands Existing commands such as ioscan and sar have evolved to use multi-pathing. New options have been introduced.
Figure 1 HP-UX 11i v3 host /dev/[r]disk/disk13 /dev/[r]disk/disk14 /dev/[r]disk/disk15 /dev/[r]disk/disk16 /dev/[r]disk/disk17 FC adapter 0/2/1/0 FC adapter 0/4/1/0 FC interconnect FC target port (wwn : FC target port (wwn : 0x500508b3009097f9) 0x500508b3009097f1) Lun id Lun wwid 0x4014000000000000 0x600508b3009097f0421745217a6c0056 Disk17 Lun id Lun wwid 0x4015000000000000 0x600508b3009097f0a9d6537f5385005 Disk16 Lun id Lun wwid 0x4016000000000000 0x600508b3009097f01499645fe5250058 Disk13
• /dev/[r]disk/disk17 For each LUN device, the esdisk driver selects by default the round-robin dynamic I/O load balancing policy. Viewing native multi-pathing related information Hardware paths to lunpaths, targetpaths, and LUNs Run ioscan to see the hardware paths for SCSI HBA, target paths, lunpaths, and LUNs in the agile view.
lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path 2 for 21 for 22 for 23 for 26 for 24 for 25 for 8 for 13 for 11 for 12 for 10 for 9 for 0/2/1/0.0x500508b3009097f9.0x4018000000000000 disk14 0/4/1/0.0x500508b3009097f1.0x0 ctl2 0/4/1/0.0x500508b3009097f1.0x4014000000000000 disk17 0/4/1/0.0x500508b3009097f1.
The ioscan command can query the mapping combinations between LUN and lunpaths using DSF or hardware paths.
Last Open or Close state = ACTIVE LUN path : lunpath13 Class Instance Hardware path SCSI transport protocol State Last Open or Close state = = = = = = lunpath 13 0/4/1/0.0x500508b3009097f9.0x4014000000000000 fibre_channel UNOPEN ACTIVE LUN path : lunpath16 Class Instance Hardware path SCSI transport protocol State Last Open or Close state = = = = = = lunpath 16 0/2/1/0.0x500508b3009097f1.
Busy status Queue full status Reservation conflicts Invalid Request status Select Timeout status Incomplete status No Resource status Target Path Offline status IO Timeout status IO Aborted status Reset Occured status Unrecognized CDB status Bad Sense data status Deferred errors Recovered errors NOT READY sense status Medium errors Hardware errors Illegal request sense status Unit Attentions = = = = = = = = = = = = = = = = = = = = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 1 Lunpath and LUN health The health
For an example on how the health property changes when a target port becomes unavailable, see Native Multi-Pathing: A Concrete Example. LUN and lunpath attributes The scsimgr command reads and sets the multi-pathing related attributes of LUN components. All the LUN and lunpath attributes are described in details in the scsimgr SCSI Management and Diagnostics Utility white paper.
# ioscan -m hwpath -H 64000/0xfa00/0x5 Lun H/W Path Lunpath H/W Path Legacy H/W Path ==================================================================== 64000/0xfa00/0x5 0/2/1/0.0x500508b3009097f9.0x4014000000000000 0/2/1/0.3.8.0.0.2.4 0/4/1/0.0x500508b3009097f9.0x4014000000000000 0/4/1/0.3.8.0.0.2.4 0/2/1/0.0x500508b3009097f1.0x4014000000000000 0/2/1/0.3.14.0.0.2.4 0/4/1/0.0x500508b3009097f1.0x4014000000000000 0/4/1/0.3.14.0.0.2.
Value of attribute load_bal_policy saved successfully Value of attribute pref_tport saved successfully Value of attribute preferred_path saved successfully # scsimgr get_attr –D /dev/rdisk/disk17 –a load_bal_policy –a pref_tport –a preferred_path SCSI ATTRIBUTES FOR LUN : /dev/rdisk/disk17 name = load_bal_policy current = round_robin default = round_robin saved = name = pref_tport current = default = saved = name = preferred_path current = default = saved = To change the load_bal_policy attribute to least_c
name = lpt_to_lockdown current = 0/4/1/0.0x0x500508b3009097f1.0x0 default = saved = Setting lunpath failure detection tunables The mass storage subsystem monitors the I/O error statistics on lunpaths to detect failing lunpaths to LUNs. If the lunpath continues to see I/O errors with no successful completions for a certain time interval, the SCSI stack takes the broken lunpath offline.
how multi-pathing works, depending on the setting of leg_mpath_enable, when an application accesses disk17 through the legacy DSF /dev/rdsk/c5t2d4. All lunpaths of disk17 are used when /dev/rdsk/c5t2d4 is opened while leg_mpath_enable is set to true.
lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path 16 for 5 for 22 for 13 for 0/2/1/0.0x500508b3009097f1.0x4014000000000000 disk17 0/2/1/0.0x500508b3009097f9.0x4014000000000000 disk17 0/4/1/0.0x500508b3009097f1.0x4014000000000000 disk17 0/4/1/0.0x500508b3009097f9.
Class … lunpath LUN path lunpath LUN path lunpath LUN path lunpath LUN path I 16 for 5 for 22 for 13 for H/W Path Driver S/W State H/W Type Description 0/2/1/0.0x500508b3009097f1.0x4014000000000000 disk17 0/2/1/0.0x500508b3009097f9.0x4014000000000000 disk17 0/4/1/0.0x500508b3009097f1.0x4014000000000000 disk17 0/4/1/0.0x500508b3009097f9.
Total I/Os processed Total I/Os processed = 73880 = 73780 Configuring automatic path recovery By default, failed lunpath automatic recovery uses the basic ping and immediate path recovery policies. This section gives examples of how you can change the default configuration to meet requirements for different use cases. The following settable attributes control how automatic path recovery is performed. For more information, see scsimgr_esdisk(7).
# scsimgr set_attr -D /dev/rdisk/disk17 –a ping_recovery=count_based -a ping_count_threshold=20 Value of attribute ping_recovery set successfully Value of attribute ping_count_threshold set successfully Setting a time-based deferred path recovery policy The following example shows how to consider failed lunpaths of disk17 to be back online only if pings continually succeed for 10 seconds. The ping_type attribute is already set to basic or extended for disk17.
As a result of the target port 0x500508b3009097f9 being disconnected, all the lunpaths using this SAN component become unavailable. The ioscan command shows these lunpaths with a NO_HW state: # ioscan -kfnNC lunpath Class I H/W Path Driver S/W State H/W Type Description ================================================================== lunpath 14 0/0/2/0.0.0x0.0x0 eslpt LUN path for disk20 lunpath 7 0/1/1/0.0x0.0x0 eslpt LUN path for disk19 lunpath 6 0/1/1/0.0x1.
tgtpath lunpath lunpath lunpath lunpath lunpath lunpath ba slot fc fc ba slot fc tgtpath lunpath lunpath lunpath lunpath lunpath lunpath tgtpath lunpath lunpath lunpath lunpath lunpath lunpath … disk disk disk disk disk 3 0 5 4 1 3 2 3 1 1 2 4 2 3 8 21 22 23 26 24 25 6 8 13 11 12 10 9 0/2/1/0.0x500508b3009097f9 0/2/1/0.0x500508b3009097f9.0x0 0/2/1/0.0x500508b3009097f9.0x4014000000000000 0/2/1/0.0x500508b3009097f9.0x4015000000000000 0/2/1/0.0x500508b3009097f9.0x4016000000000000 0/2/1/0.0x500508b3009097f9.
The following messages are immediately logged in /var/adm/syslog/syslog.log: Jan Jan The Jan Jan The 31 02:41:48 31 02:41:48 target path 31 02:41:48 31 02:41:48 target path hpfcs835 hpfcs835 h/w path hpfcs835 hpfcs835 h/w path vmunix: class : tgtpath, instance 3 vmunix: Target path (class=tgtpath, instance=3) has gone online. is 0/2/1/0.0x500508b3009097f9 vmunix: class : tgtpath, instance 6 vmunix: Target path (class=tgtpath, instance=6) has gone online. is 0/4/1/0.
tgtpath lunpath lunpath lunpath lunpath lunpath lunpath … disk disk disk disk disk 6 8 13 11 12 10 9 0/4/1/0.0x500508b3009097f9 0/4/1/0.0x500508b3009097f9.0x0 0/4/1/0.0x500508b3009097f9.0x4014000000000000 0/4/1/0.0x500508b3009097f9.0x4015000000000000 0/4/1/0.0x500508b3009097f9.0x4016000000000000 0/4/1/0.0x500508b3009097f9.0x4017000000000000 0/4/1/0.0x500508b3009097f9.
Glossary Agile Addressing The ability to address a logical unit with the same DSF regardless of the location of the LUN, that is, the device file for a LUN remains the same even if the LUN is moved from one Host Bus Adapter (HBA) to another, from one switch or hub port to another, or presented via a different target port to the host. The LUN device file stays the same even if the N-Port ID of the target port changes in the case of Fibre Channel LUNs.
H/W Path in the legacy format (pre HP-UX 11i v3). Used for all components, including mass storage. Legacy Naming Model / Legacy Format DSF format convention used prior to HP-UX 11i v3. This model is maintained in HP-UX 11i v3 for backward compatibility purpose. Legacy or agile I/O Tree The I/O Tree is the HP-UX representation of physical and virtual devices discovered by the operating system. The pre-11i v3 I/O tree view is present on HP-UX 11i v3 for backward compatibility reasons.
SPC-3 T10 “SCSI Primary Commands – 3” (T10 SPC-3) standard (available at http://www.t10.org). Target Id Target port identifier as defined in SCSI transport protocols. Target Path Path to a target port. It is also known as an I_T nexus. Target Path Hardware Path The hardware path of a Target Path. In the agile view of the I/O tree, the target path hardware path has the following format: hba_path.Target_port_WWN or Target_Id VPD Vital Product Data. A term defined by the T10 SPC-3 standard (section 7.
For more information To read more about the HP-UX mass storage subsystem, see the following: • White papers The Next Generation Mass Storage Stack white paper scsimgr SCSI Management and Diagnostics Utility on HP-UX 11i v3 white paper HP-UX 11i v3 Crash Dump Improvements white paper HP-UX 11i v3 Mass Storage Device Naming white paper HP-UX 11i v3 Mass Storage I/O Performance Improvements white paper HP-UX 11i v3 Persistent DSF Migration Guide white paper HP-UX 11i v3 Writing Active/Passive Switch Plug-Ins w
