HP ENTERPRISE VIRTUAL ARRAY FAMILY WITH VMWARE VSPHERE 4.0 , 4.1 AND 5.0 CONFIGURATION BEST PRACTICES Technical white paper Table of contents Executive summary............................................................................................................................... 3 The challenges .................................................................................................................................... 3 Key concepts and features ..............................................
Monitoring EVA performance in order to balance throughput .............................................................. 42 Optimizing I/O size ....................................................................................................................... 44 Summary of best practices .................................................................................................................. 45 Summary .............................................................................................
Executive summary The HP Enterprise Virtual Array (EVA) family has been designed for mid-range and enterprise customers with critical requirements to improve storage utilization and scalability. EVA arrays can fulfill application-specific demands for transactional I/O performance, while supporting easy capacity expansion, instantaneous replication, and simplified storage administration.
Successfully addressing these challenges is imperative if you wish to maximize the return on investment (ROI) for your SAN while continuing to meet the changing needs of the business. To help you achieve these goals, this paper presents best practices for configuring, tuning, and deploying a vSphere SAN environment. Key concepts and features This section introduces key concepts and features associated with the successful configuration, tuning, and deployment of a vSphere SAN environment.
Explicit ALUA mode (explicit transition) – A host driver can set or change the managing controller for the Vdisk EVA arrays also support the following ALUA access types: Active-Optimized (AO) – The path to the Vdisk is through the managing controller Active-Non-Optimized (ANO) – The path to the Vdisk is through the non-managing controller ALUA compliance in vSphere 4.x and 5.0 ALUA compliance was one of the major features added to the vSphere 4 SCSI architecture and remains standard in vSphere 4.
Configuring EVA arrays HP provides tools to help you configure and maintain EVA arrays. For example, intuitive Command View EVA can be used to simplify day-to-day storage administration, allowing you to create or delete Vdisks, create data replication groups, monitor the health of system components, and much more.
Running Command View EVA within a VM Your ability to deploy Command View EVA within a VM may be impacted by the following: EVA model Command View EVA version being used Availability of a host bus adapter (HBA) that can be dedicated to the VM Table 1 compares the options for deploying Command View EVA in a VM. Table 1.
Best practices for deploying Command View EVA in a VM with VMDirectPath I/O Deploy Command View EVA on the local datastore of the particular vSphere server. If a SAN-based Command View EVA deployment is required, then deploy instances on multiple arrays within the infrastructure to ensure the management interface remains available in the event the array hosting the VM for the primary Command View EVA instance becomes inaccessible.
Figure 1 shows the overview screen for the HP Insight Control Storage Module for vCenter plug-in. Figure 1. Typical overview screen for the vCenter plug-in, viewed via the HP Insight Software tab The Storage Module for vCenter can enhance VMware functionality by providing detailed views of the relationships between the virtual and physical environments.
The Storage Module for vCenter also provides automated provisioning for datastores and VMs. After the storage administrator has configured the plug-in to support automated provisioning for specific EVA disk groups, the VMware administrator can then perform automated storage provisioning operations – quickly, without requiring the assistance of the storage administrator.
The resulting topology should be similar to that presented in Figure 3, which shows a vSphere 4.x server attached to an EVA4400 array through a redundant fabric. Figure 3. Highly-available EVA/vSphere 4.x SAN topology The benefits of this topology include the following: The topology provides increased fault tolerance, with protection against the failure of an HBA, a single fabric, or a controller port or controller.
In a direct-connect environment, the same principles can be achieved with two more HBA or HBA ports; however, the configuration is slightly different, as shown in Figure 4. Figure 4. EVA/vSphere 4.x and 5.0 direct-connect topology If the direct-connect configuration were to use two rather than four HBA ports, there would be a oneto-one relationship between every HBA and controller. Thus, a controller failover would result in an HBA failover and vice versa, creating a configuration that is not ideal.
Note When configuring VMware Consolidated Backup (VCB) with an EVA array, all vSphere hosts must be set to VMware. However, the VCB proxy host, which is a Microsoft® Windows® server attached to the EVA, must be set to Microsoft Windows (Windows Server 2003) or Microsoft Windows 2008 (Windows Server 2008). Disk group provisioning An EVA disk group is the largest storage object within the EVA storage virtualization scheme and is made up of a minimum of eight physical disks (FC or Near-Line drives).
The overhead created by sparing is calculated as follows: "Sparing capacity = " ("Size of largest disk in disk group * 2" )" * (Protection level)" In this formula, the value for disk drive failure protection level (Protection Level) may be as follows: None Single – The disk group survives the failure of a single disk Double – The disk group survives the failure of two disks You can use Command View EVA to set a disk drive failure protection level in the properties for the particular disk group, as sho
Sizing storage for any application that is being virtualized begins with understanding the characteristics of the workload. In the white paper, “Best Practices for the HP EVA Array using VMware vCenter Site Recovery Manager,” HP describes how to determine the number of disks needed to support a particular application.
Optimizing for availability When optimizing for availability, your goal is to accommodate particular levels of failures in the array.
Vdisk provisioning All EVA active-active arrays are asymmetrical and comply with the SCSI ALUA standard.
Vdisks should be created with their controller failover/failback preference alternating between Controller A and B. The above recommendations provide additional benefits in a multi-pathing configuration, as described in Configuring multi-pathing. Best practice controller ownership Controller ownership for EVA Vdisks should alternate between Controller A and Controller B, using the Path-A-Failover/Failback or Path-B-Failover/Failback setting. Implementing multi-pathing in vSphere 4.x and 5.
Figure 7 shows a typical multi-pathing implementation using ESX 3.5 or earlier. Figure 7. EVA connectivity with ESX 3.5 or earlier Here, Vdisks 1 – 4 are managed through Controller 1; Vdisks 5 – 7 are managed through Controller 2. Whether you were to use MRU or Fixed I/O path policies, only the first I/O path discovered would be used to queue I/Os, causing all I/Os from the ESX server to access a single port through a single controller.
Multi-pathing in vSphere 4.x and 5.0 vSphere 4 introduced the concept of path selection plug-ins (PSPs), which are essentially I/O multipathing options. These plug-ins are described in more detail in Configuring multi-pathing. Table 2 outlines multi-pathing options in vSphere 4.x and 5. Table 2. Multi-pathing options I/O path policy PSP vSphere 4 vSphere 4.
Fixed_AP Introduced in vSphere 4.1, Fixed_AP I/O path policy extends the functionality of Fixed I/O path policy to active-passive and ALUA-compliant arrays. Fixed_AP can also identify the preferred controller for a Vdisk. Despite being ALUA-aware, the primary path selection attribute for Fixed_AP is the preferred controller for a Vdisk and not just its access state.
Figure 8 shows a typical multi-pathing implementation using vSphere 4.x/5. Figure 8. EVA connectivity with vSphere 4.x/5 All I/Os to Vdisks 1 – 4 are routed through one or more ports on Controller 1 and through Paths and/or , regardless of the HBA that originates the I/O. Similarly, all I/Os to Vdisks 5 – 7 are routed to Controller 2 through Paths and/, regardless of the originating HBA. The vSphere 4.
Note that the preferred controller for accessing a Vdisk in an ALUA-capable array is defined in SCSI by the PREF bit, which is found in byte 0, bit 7 of the target port descriptor format5. If the PREF bit is set to one, this indicates that the Vdisk is preferred to the controller the target port group request was sent to; if it is set to zero, this indicates the Vdisk is not preferred to the controller the target port group request was sent to.
Summary In vSphere 4.x/5, ALUA compliance and support for round robin I/O path policy have eliminated the intricate configuration required to implement multi-pathing in ESX 3.5 or earlier. These new features also help provide much better balance than you could achieve with MRU; furthermore, round robin policy allows I/Os to be queued to multiple controller ports on the EVA, helping create an instant performance boost.
Figure 10 outlines key components of the multi-pathing stack. Figure 10. vSphere 4.x and 5 multi-pathing stack The key features of the multi-pathing plug-ins are as follows: SATP The SATP is an array-specific plug-in that handles specific operations such as device discovery, the management of array-specific error codes, and failover.
NMP The NMP ties together the functionality delivered by the SATP and PSP by handling many non-array specific activities, including: – Periodical path probing and monitoring – Building the multi-pathing configuration When a path failure occurs, the NMP communicates with the SATP and PSP and then takes the appropriate action. For example, the NMP would update its list of available paths and communicate with the PSP to determine how I/O should be re-routed based on the specified path selection policy.
Table 3. vSphere 4.
Connecting to an active-active EVA array in vSphere 4 When connecting a vSphere 4 host to an active-active EVA array, you should use the VMW_SATP_ALUA SATP as suggested above. This SATP is, by default, associated with VMW_PSP_MRU, a PSP that uses MRU I/O path policy. There are two steps for connecting a vSphere 4.
for i in `esxcli nmp device list | grep naa.600` ; do esxcli nmp roundrobin setconfig -t iops –I 1 -d $i; done For ESXi5 for i in `esxcli storage nmp device list | grep naa.600` ; do esxcli storage nmp psp roundrobin deviceconfig set -t iops –I 1 device $i; done For environments with multiple array models, merely change grep naa.600 so that it matches the pattern to devices on the desired arrays only.
Connecting to an active-active EVA array in vSphere 4.1/5 In vSphere 4.1, VMware introduced more granular SATP and PSP configuration options. As in vSphere 4, each SATP has a default PSP. However, vSphere 4.1/5 also gives you the ability to configure a particular PSP based on the storage array model. This is a significant improvement, allowing multiple arrays to use the same SATP but, by default, utilize different PSPs, which provides a tremendous savings in configuration time.
– Create a new rule in the SATP rule table for the array specified with –vendor and –model – Set the default SATP for this array to VMW_SATP_ALUA – Set the default PSP to VMW_PSP_RR – Set the round robin option to IOPS=1 Repeat this command line for each ALUA-compliant array model to be shared by your vSphere 4.1/5 cluster. With this single command line, you can achieve the same results as the two-stage configuration process required for vSphere 4. Thus, regardless of the deployment type in vSphere 4.
Deleting a manually-added rule To delete a manually-added rule, use the esxcli nmp satp deleterule command; specify the same options used to create the rule.
Arrays from two or more vendors are ALUA-compliant There are different default recommendations for PSPs vSphere 4.1/5 deployment If the multi-vendor SAN is being shared by a vSphere 4.1 cluster, then create a new SATP rule entry for each array, setting the configuration parameters as recommended by the particular vendor.
Figure 11. Relationship between Vdisks and the DR group Disk Group DR Group Vdisk Just like a Vdisk, a DR group is managed through one controller or the other; in turn, this controller must manage all Vdisks within the particular DR group. Thus, when a Vdisk is added to an existing DR group, its optimal controller is the one currently managing the DR group.
Upgrading EVA microcode An online upgrade of EVA microcode is supported with vSphere 4.x. When performing such upgrades it is critical to follow the general EVA Online Firmware Upgrade (OLFU) guidelines defined in the OLFU best practices guide7. From a vSphere 4.x perspective, VMs using RDM Vdisks are more susceptible to issues resulting from an OLFU. It is important to ensure that the SCSI disk timeout for all VMs is set to a minimum of 60 seconds, higher (60 – 90 seconds) in a larger environment.
Using VMFS VMFS is a high-performance cluster file system designed to eliminate single points of failure, while balancing storage resources. This file system allows multiple vSphere 4.x hosts to concurrently access a single VMDK (Virtual Machine Disk Format), as shown in Figure 12. VMFS supports Fibre Channel SAN, iSCSI SAN, and NAS storage arrays. Figure 12. VMFS datastore Best practices for deploying VMFS To avoid spanning VMFS volumes, configure one VMFS volume per logical unit.
– pRDM requires the guest to use the virtual LSI Logic SAS controller – pRDM is most commonly used when configuring MSCS clustering There are some limitations when using RDM in conjunction with MSCS or VMware snapshots – for example, when configuring an MSCS cluster with both physical and virtual machines. You should utilize pRDM since it is not possible to use Vdisks or RDMs as shared storage in vRDM.
naming convention because the number for a Vdisk in Datacenter A may not be maintained when the Vdisk is replicated and presented to a host in Datacenter B. Similarly, you should not include Vdisk size in a naming convention because it is probable that this size will change. To avoid confusion, you need detailed documentation on each array, datastore, Worldwide Name (WWN), and host name.
Table 5 outlines the various components of this naming convention. Table 5. Sample naming convention Component Description Example Geographical location of the team, group or division for which the storage is being allocated The location could be a city, state, country, or a combination thereof.
Best practice for naming datastores When naming a datastore, utilize the same name used in Command View when the Vdisk was created Use simple, consistent names for your datastores – in the future, you may need to add vSphere hosts to the cluster Sizing the vSphere cluster Knowing how many vSphere hosts can be supported per Vdisk will enhance your ability to manage the cluster9. Monitor resource utilization to recognize VMs that are I/O-intensive.
Best practices for aligning the file system No alignment is required with Windows Vista, Windows 7, or Windows Server 2008. Use the vSphere client when creating your datastore, which correctly aligns the file system. Verify that VMDKs used by the guest operating system are correctly aligned. Enhancing storage performance A range of vSphere 4.x/5 features and tuning options provide opportunities for enhancing storage performance.
vSphere administrators often enable adaptive queuing as a means to address storage congestion issues. However, while this approach can temporarily help reduce storage congestion, it does not address the root cause of the congestion. Moreover, although adaptive queuing can enhance performance during times when storage is congested, overall I/O performance is superior in a welltuned, balanced configuration.
single controller despite the environment being balanced from the perspective of Vdisk access. Each port on Controller 2 is processing 300 MB/s of I/O throughput, while ports on Controller 1 are only processing 20 MB/s each. Figure 14. Unbalanced I/O access in a vSphere 4.x environment Better balanced throughput could be achieved in this example by moving one or more of the Vdisks on Controller 2 (VDISK5, 6 or 7) to Controller 1.
Figure 15 shows a better-balanced environment, achieved by moving the controller ownerships of VDISK 5 and 6 to Controller 1 and of VDISK1 and 2 to Controller 2. Figure 15. Balanced I/O access in a vSphere 4.x environment after changing controller ownership for certain Vdisks This type of tuning can be useful in most environments, helping you achieve the optimal configuration.
Note VMware makes a similar recommendation in their knowledge base article 1003469. Best practice for improving the performance of VMs that generate large I/Os Consider setting the vSphere advanced parameter Disk.DiskMaxIOSize to 128 KB to enhance storage performance.
In an exclusively EVA environment, change the default PSP option for VMW_SATP_ALUA to VMW_PSP_RR. Round robin I/O path policy is recommended for EVA active-active arrays. MRU is also suitable if round robin is not desired in a particular environment. Configure MSCS cluster Vdisks to use MRU I/O path policy. Since the recommended default setting for all EVA Vdisks is round robin, you must manually configure MSCS Vdisks to MRU.
Summary In most environments, the best practices highlighted in this document can help you reduce configuration time and improve storage performance. However, as with all best practices, you must carefully evaluate the pros and cons of the recommendations presented herein and assess their value in your particular environment. In addition to serving as a reference guide for anyone configuring an EVA-based SAN in conjunction with vSphere 4.
Glossary Array In the context of this document, an array is a group of disks that is housed in one or more disk enclosures. The disks are connected to two controllers running software that presents disk storage capacity as one or more virtual disks. The term “array” is synonymous with storage array, storage system, and virtual array. Controller firmware The firmware running on each controller within the array manages all aspects of array operation, including communications with Command View EVA.
Management server A management server runs management software such as HP Command View EVA and HP Replication Solutions Manager. RDM VMware Raw Device Mapping (RDM) technology allows an LU to be mapped directly from an array a VM. Server-based management The term “server-based management” implies management from a server.
Appendix A – Using SSSU to configure the EVA The sample SSSU script provided in this appendix creates and present multiple Vdisks to vSphere hosts.
ADD VDISK "\Virtual Disks\DATA_DISKS\Vdisk004" DISK_GROUP="\Disk Groups\DG1" SIZE=180 REDUNDANCY=VRAID5 WRITECACHE=WRITEBACK MIRRORCACHE=MIRRORED READ_CACHE NOWRITE_PROTECT OS_UNIT_ID=0 PREFERRED_PATH=PATH_B_BOTH WAIT_FOR_COMPLETION ADD Vdisk 4 VDISK="\Virtual Disks\DATA_DISKS\Vdisk004\ACTIVE" HOST="\Hosts\ESX1" ADD Vdisk 4 VDISK="\Virtual Disks\DATA_DISKS\Vdisk004\ACTIVE" HOST="\Hosts\ESX2" More information For more information on the SSSU command set, refer to the SSSU user guide, which can be found in t
Appendix B – Miscellaneous scripts/commands This appendix provides scripts/utilities/commands for the following actions: Set I/O path policy to round robin Change the default PSP for VMW_SATP_ALUA Configure the disk SCSI timeout for Windows and Linux guests Setting I/O path policy This script automatically sets the I/O path policy to round robin for Vdisks connected to ESX 4.x servers: Note This script should only be used for environments with EVA Vdisks connected to vSphere 4.x/5 servers. For ESX4.
Linux guest Use one of the following commands to verify that the SCSI disk timeout has been set to a minimum of 60 seconds: cat /sys/bus/scsi/devices/W:X:Y:Z/timeout or cat /sys/block/sdX/device/timeout If required, set the value to 60 using one of the following commands: echo 60 > /sys/bus/scsi/devices/W:X:Y:Z or echo 60 | cat /sys/block/sdX/device/timeout where W:X:Y:Z or sdX is the desired device. No reboot is required for these changes to take effect.
Appendix C – Balancing I/O throughput between controllers The example described in this appendix is based on an environment (shown in Figure C-1) with balanced Vdisk access but imbalanced I/O access. The appendix explores the steps taken to balance I/O access. Figure C-1. Sample vSphere 4.
Figure C-2. I/O routes In this example, even though the EVA array has a total of eight controller ports (four on each controller), all I/O seems to be routed through just two ports on Controller 1. Note that SAN zoning is only allowing each HBA to see ports 1 and 2 of each controller, explaining why no I/O is seen on ports 3 and 4 even though round robin I/O path policy is being used.
Figure C-4. Path information for VDISK5 Alternatively, you can review Vdisk properties in Command View EVA to determine controller ownership, as shown in Figure C-5 (VDISK9) and C-6 (VDISK5). Figure C-5.
Figure C-6. Vdisk properties for VDISK5 For a more granular view of throughput distribution, use the following command: evaperf vd –sz –cont X –dur Y This command displays statistics at the EVA Vdisk level, making it easier for you to choose the appropriate Vdisk(s) to move from one controller to the other in order to better balance controller throughputs.
After a rescan or vCenter refresh, you can verify that the change has been implemented, as shown in Figure C-8. Figure C-8. Confirming that ownership has changed I/O is now round robin on FP1 and FP2 of Controller B. Validating the better balanced configuration You can review the output of EVAperf (as shown in Figure C-9) to verify that controller throughput is now better balanced. Run the following command: evaperf hps –sz -cont X –dur Y Figure C-9.
Appendix D – Caveat for data-in-place upgrades and Continuous Access EVA The vSphere datastore may become invisible after one of the following actions: Performing a data-in-place upgrade from one EVA controller model to another Using Continuous Access EVA to replicate from one EVA model to another Following these actions, ESX treats the new datastore as being a snapshot and, by default, does not display it.
Note A similar mismatch would occur if you attempted to use Continuous Access EVA to replicate from the EVA4400 to the EVA8400. When such a mismatch occurs, datastores are treated as snapshots and are not exposed to ESX. However, vSphere 4.x allows you to force-mount or re-signature these snapshots to make them accessible. For more information, refer to the following VMware Knowledge Base (KB) articles: 1011385 and 1011387.
Appendix E – Configuring VMDirectPath I/O for Command View EVA in a VM This appendix describes how to configure VMDirectPath I/O in a vSphere 4.x environment for use with Command View EVA. An example is presented. Note that Command View EVA in a VM is not supported with VMDirectPath I/O with ESXi5 Note The configuration described in this appendix is only provided for the purposes of this example.
Figure E-1. Storage Adapters view, available under the Configuration tab of vSphere Client This appendix shows how to assign HBA3 to VM2 in vSphere 4.x. EVA configuration This example uses four ports on an EVA8100 array (Ports 1 and 2 on each controller). A single EVA disk group was created. The EVA configuration is summarized in Table E-2.
Table E-2.
Fibre Channel configuration This example uses two HP 4/64 SAN switches, with a zone created on each. The Fibre Channel configuration is summarized in Table E-3. Table E-3.
The procedure is as follows: 1. Identify which HBAs are present on the vSphere server by issuing the following command: [root@lx100 ~]# lspci | grep "Fibre Channel" This command provides a quick view of the HBAs in your system and their respective PCI hardware IDs. Alternatively, you can view HBAs via the vSphere Client; however, PCI hardware IDs would not be shown. The output to the above command is similar to that shown in Figure E-2. Figure E-2.
However, if your server hardware does not support Intel® Virtualization Technology for Directed I/O (VT-d) or AMD Extended Page Tables (EPT), Nested Page Tables (NPT), and Rapid Virtualization Indexing (RVI), it cannot support VMDirectPath. In this case, the Advanced Settings screen would be similar to that shown in Figure E-4, which indicates that the host does not support VMDirectPath. Figure E-4.
3. If your server has compatible hardware, click on the Configure Passthrough… link to move to the Mark devices for passthrough page, as shown in Figure E-5. Review the device icons: – Green: Indicates that the device is passthrough–capable but not currently running in passthrough mode – Orange arrow: Indicates that the state of the device has changed and that the server needs to be rebooted for change to take effect Figure E-5.
4. Select the desired devices for VMDirectPath; select and accept the passthrough device dependency check shown in Figure E-6. IMPORTANT If you select OK, the dependent device is also configured for VMDirectPath, regardless of whether or not it was being used by ESX. If your server is booting from SAN, be careful not to select the incorrect HBA; your server may subsequently fail to reboot. Figure E-6.
As shown in Figure E-7, the VMDirectPath Configuration screen reflects the changes you have made. Device icons indicate that the changes will only take effect when the server is rebooted. Figure E-7. Indicating that four HBA ports have been enabled for VMDirectPath but that these changes will not take effect until a server reboot 5. Reboot the server through the vSphere client or the command line.
6. After the reboot, confirm that device icons are green, as shown in Figure E-8, indicating that the VMDirectPath-enabled HBA ports are ready to use. Figure E-8. The HBA ports have been enabled for VMDirectPath and are ready for use 7. Issue the following command to validate the VMDirectPath-enabled HBA ports: [root@lx100 ~]# vmkchdev -l | grep vmhba Review the resulting output, which is shown in Figure E-9. Figure E-9. Validating that four HBA ports have indeed been enabled for VMDirectPath 000:31.
Note The changes you have just made are stored in file /etc/vmware/esx.conf. Configuring the array Use Command View EVA to perform the following steps: 1. Create the Vdisks. 2. Add the hosts: – vSphere server: Set the Command View EVA host mode to VMware – VM2: Set the Command View EVA host mode to Windows Server 2008 3. Add Vdisks presentation. Configuring the VM Caveats HBA ports are assigned to the VM one at a time, while the VM is powered off.
Procedure Carry out the following steps to add VMDirectPath devices to a selected VM: 1. From the vSphere client, select VM2 from the inventory, ensuring that it is powered off. 2. Right-click on the VM and select Edit Settings. 3. Select the Hardware tab and then click on Add. 4. Select PCI Device and then click on Next, as shown in Figure E-11. Figure E-11.
5. From the list of VMDirectPath devices, select the desired device to assign to the VM, as shown in Figure E-12. In the example, select Port 1 of HBA3 (that is, device 21:00.0). For more information on selecting devices, refer to Caveats. Figure E-12. Selecting VMDirectPath devices to be added to the VM 6. Repeat Step 5 to assign Port 2 of HBA2 (that is, device 21:00.1) to the VM. 7. Use vSphere Client open a console window to the Windows Server 2008 VM. 8.
For more information Data storage from HP http://welcome.hp.com/country/us/en/prodser v/storage.html HP virtualization with VMware http://h18004.www1.hp.com/products/servers /vmware/index.html VMware storage solutions from HP http://www.hp.com/go/storage/vmware Documentation for a specific EVA array (such as the “EVA OnLine Firmware Upgrade (OLFU) Best Practices Guide”) – select the appropriate EVA model http://h20000.www2.hp.com/bizsupport/TechSuppo rt/Product.
