EMC CLARiiON Integration with VMware ESX Server Applied Technology Abstract This white paper provides an overview of how VMware ESX Server integrates with EMC® CLARiiON® storage systems. It introduces the Navisphere® VM-aware feature, a feature that automatically discovers virtual machines managed under VMware vCenter Server and provides end-to-end, virtual-to-physical mapping information.
Copyright © 2006, 2007, 2008, 2009, 2010 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Table of Contents Executive summary ............................................................................................ 5 Introduction ......................................................................................................... 5 Audience ...................................................................................................................................... 5 CLARiiON terminology..............................................................................................
CLARiiON and virtual machine clustering...................................................... 38 In-the-box cluster ....................................................................................................................... 38 Out-of-the-box cluster ................................................................................................................ 38 Virtual-to-virtual clustering......................................................................................................
Executive summary EMC is aggressively expanding product sets from high-end to midtier markets. Through VMware—the industry leader of x86 server-virtualization software—and EMC® CLARiiON®, which offers the best performance in midtier storage, EMC is integrating cutting-edge virtualization technology into its core storage business. Our latest enhancement, available on the CX4 series, is the Navisphere® VM-aware feature.
ESX Server — VMware’s high-end server product that installs directly on the physical hardware and therefore offers the best performance. ESX Server supports more virtual machines per physical CPU than its other virtualization products such as VMware Server (previously called GSX server). Farm or Data Center— The primary organizational structure used in VMware vCenter, which contains hosts and virtual machines. The term Farm is used with VMware vCenter 1.x while the term Data Center is used with vCenter.4.
ESX overview ESX consists of virtualization software that provides server consolidation by allowing several instances of similar and dissimilar operating systems to run as virtual machines on one physical machine. This costeffective, highly scalable virtual machine platform offers advanced resource management capabilities. ESX minimizes the total cost of ownership (TCO) of computing infrastructure by: • Increasing resource utilization. • Decreasing the number of servers and all associated costs.
Features ESX has several features that work with the CLARiiON storage system. The features discussed in this paper are VMotion, Distributed Resource Scheduling and VMware HA, VMware Clustering, and Consolidated Backup technology. The Distributed Resource Scheduling, VMware HA, and Consolidated Backup features are enhanced in ESX 4.0/ESX 4i. VMware VMotion ESX Server version 2.0.1 was the first platform to support VMotion.
and events from the primary VM and sends them to the secondary VM, so the VMs are identical. The secondary VM can take over execution from the primary VM, thus providing another level of fault tolerance. All hosts must have access to the primary VMs datastores and networks through a distributed switch. When a VM is configured to be fault tolerant, the DRS feature is automatically disabled. For more details, please see the vSphere Availability Guide available on www.vmware.com.
CLARiiON storage systems address a wide range of storage requirements by providing flexible levels of capacity, functionality, and performance. The AX4-5 is an entry-level system that consists of singlecontroller and dual-controller models. It supports both Serial Attached SCSI (SAS) and SATA drives and connectivity for up to 64 high availability (HA) connected hosts. The CX4 Model 120 supports up to 120 drives and connectivity for up to 128 HA hosts.
MirrorView/S Yes N/A MirrorView/A Yes N/A *Support for 10 hosts with the base pack and 64 hosts with the expansion enabler Why use CLARiiON with VMware ESX Server? CLARiiON and VMware complement each other with the features that they provide. Some of the reasons CLARiiON is an ideal fit for VMware in the midrange storage market include: • CLARiiON provides a family of storage systems of varying specifications with Fibre Channel and iSCSI connectivity.
VMware vCenter server or the Management User Interface assigns a LUN or part of a LUN to the individual virtual machines. The VMware ESX/ESXi server allows the user to add virtual disks to a virtual machine without powering the virtual machine down. This functionality is provided in the Add dialog box in VMware vCenter 2.0 and later, which is shown in Figure 2. When connecting a CLARiiON (CX4 or AX4) Fibre Channel storage system to ESX both direct and FCSW connections are supported.
Booting from a CLARiiON storage system This section discusses the procedure for booting the VMware ESX server and guest operating systems— Windows, Linux, NetWare, and Solaris—from CLARiiON LUNs. Booting ESX 4.0, 3.x, and ESX 2.5.x from CLARiiON LUNs with ESX In ESX 4.0, 3.x, and 2.5 containing Fibre Channel HBAs zoned to a CLARiiON storage system, the service console can boot from a CLARiiON LUN through the HBAs. To boot an ESX 2.5.
VMware does not support booting ESX Server over iSCSI storage using the software initiator; however, it does support booting VMs residing on iSCSI LUNs, which is a requirement for VMotion. In addition, a virtual machine can install an iSCSI software initiator and connect directly to iSCSI ports on the CLARiiON storage system. Thus, a VMware ESX server can be connected (via Fibre Channel or iSCSI) to a CLARiiON, and a VM on the ESX server can also be connected (via iSCSI) to the same CLARiiON.
Figure 3. Executing the lunmapinfo command issued on a Windows virtual machine and sent to the Navisphere Agent on the ESX service console You can use the symm inq utility to get device mapping information from the virtual machine to the CLARiiON LUN level. The virtual disks assigned to the virtual machine must be configured as raw mapped LUN for this to work correctly. Figure 4 shows output for the inq –clar_wwn command. Figure 4.
Figure 5. Using the Navisphere Server Utility with a VMware ESX 3.x/3i server Figure 6, Figure 7, and Figure 8 are examples for reports generated by the server utility for ESX 4.0 and 3.5. In Figure 7, the server utility reports the policies configured for the LUNs. Figure 6. Report generated by the Navisphere Server Utility showing configuration information for ESX 4.
Figure 7. Report generated by the Navisphere Server Utility showing configuration information for ESX 3.x Figure 8. Report generated by the Navisphere Server Utility showing guest OS and VMFS volume information Multipathing and failover with ESX on CLARiiON Multipathing and load balancing increase the level of availability for applications running on ESX servers. CLARiiON storage systems also support the nondisruptive upgrade (NDU) operation for VMware’s native failover software and EMC PowerPath®.
of auto assign) controls ownership of the LUN in a storage system with two SPs. For more information about the auto-assign LUN, please see Knowledgebase case emc165941. VMware native multipathing and failover on ESX 4.0 with CLARiiON VMware ESX 4.0 contains its own native multipathing software that is built into its kernel. This failover software, called Native Multipathing Plugin (NMP), has three policies: • FIXED policy • Round Robin policy • Most Recently Used (MRU) policy On VMware 4.
Figure 9. VMware’s native multipathing software on ESX Server 4.0 configured with the FIXED policy setting for CLARiiON storage systems EMC PowerPath multipathing and failover on ESX 4.0 with CLARiiON EMC PowerPath software is supported on the ESX 4.0 server and is installed using RemoteCLI. RemoteCLI is a software package available for remotely managing the ESX server.
Figure 10. EMC PowerPath software configured on ESX 4.0 connected to a CLARiiON storage system iSCSI configurations and multipathing with ESX 4.0 Figure 11 shows how to configure the iSCSI software on a CLARiiON storage system. Note that the iSCSI hardware-initiator configuration is similar to the Fibre Channel HBA configuration, and is not covered in this section. Two virtual switches (vSwitches), each containing one or more NICs, can be configured on ESX 4.0 as shown in Figure 12.
vSwitch 1 SPA port 0: 10.14.19.45 VMkernel port -> NIC 1 SPB port 0: 10.14.19.46 IP address: 10.14.19.22 vSwitch 2 SPA port 1: 10.14.17.54 VMkernel port -> NIC 2 IP address: 10.14.17.80 SPB port 1: 10.14.17.55 Figure 11. Dual virtual switch iSCSI configuration With port binding enabled, a single vSwitch with two NICs can be configured so that each NIC is bound to one vmkernel port. The two NICs, or vmkernel ports, should be on different subnets.
VMware ESXi, you can use VMware vCenter or the RemoteCLI package to see the number of paths to a given LUN. If you are using an ESX version with a service console, type: # esxcfg-mpath -l Figure 13. VMware ESX Server 3.0 path information for Fibre Channel devices Figure 13 shows the seven devices attached to the CLARiiON storage system. The vmhba0:x:x devices are Fibre Channel devices. All Fibre Channel devices have paths to both SP A and SP B.
Figure 14. VMware ESX Server 3.0 path information for iSCSI devices The vmkmultipath command, when issued on an ESX 2.x server, provides details about the devices and the number of paths attached to each device. At the service console of the VMware ESX server, type: # vmkmultipath -q Figure 15. VMware ESX Server 2.x path information through the native failover software The most recently used MRU policy is the default policy for active/passive storage devices in ESX 2.x and 3.0.
active path for some LUNs; however, this configuration is not persistent across reboots. After a reboot, the LUNs will be on a single HBA. The advantage of this configuration is it prevents unnecessary trespasses of LUNs in the case of failure. The failover time can be adjusted at the HBA, ESX, and virtual machine levels. The Fibre Channel SAN Configuration Guide and iSCSI SAN Configuration Guide found on www.vmware.
Raw Disk Virtual Disks Raw Device Mapping (RDM) Open () Virtual Machine Level VM 2- 4 GB Read (), Write () VM 1- 5 GB Virtualization ESX Service Console Level .vmdk 5G .vmdk 4G Raw Disk VMFS Partition – 10 GB .vmdk file Raw Disk Figure 16. Partitioning a CLARiiON LUN Raw disks For raw disks, an entire CLARiiON LUN is presented to a single virtual machine without being partitioned at the ESX service console level.
Raw device mapping (RDM) VMware ESX 2.5 introduced a new technology called raw device mapping (RDM); this is also called a mapped raw LUN when assigned to a virtual machine. This technology has a SCSI pass-through mode that allows virtual machines to pass SCSI commands directly to the physical hardware. Utilities like admsnap and admhost, when installed on virtual machines, can directly access the virtual disk when the virtual disk is in physical compatibility mode.
VMFS-3 volume. Ensure that the VMFS-3 volume has enough space to accommodate the swap files. With ESX 2.x, a single swap file is used for all virtual machines. Application data disks residing on virtual machines should be aligned with the CLARiiON disk stripe, just as they are on physical servers. When aligning RDMs, align them at the virtual machine level. For Windows virtual machines, use diskpart from Windows 2003 SP1 to perform the alignment.
VMware ESX 2.x to ensure the ESX service console and VMkernel see the additional space. Since the LUN is presented to the virtual machine, expansion must take place at the virtual machine level. Use the native tools available on the virtual machine to perform the file system expansion at the virtual machine level. CLARiiON LUN migration conducted on VMFS or RDM volumes is transparent to the guest OS.
Figure 17. VMFS datastore “Properties” dialog for LUN expansion on ESX 4.0 With ESX 3.5, after expanding the VMFS volume, you can expand the individual virtual disk given to the virtual machine by using the vmkfstools –extendvirtualdisk option, but first you must power off the virtual machine that uses the virtual disk With ESX 4.0, hot virtual disk (.
When CLARiiON LUN migration is used to migrate to a larger LUN, after the migration completes, and a rescan is performed on the VMware ESX server, additional space for the LUN is visible. Use the procedure for expanding the VMFS volume for CLARiiON metaLUNs discussed previously. CLARiiON Virtual Provisioning with VMFS and RDM volumes A CLARiiON thin pool can contain multiple thin LUNs that can be assigned to multiple hosts. The space assigned to these thin LUNs is the space that the VMware ESX server sees.
operating file system initializes all blocks, the virtual disk will need all the space to be allocated up front. Note that when the first write is triggered on a zeroedthick virtual disk, it will write zeroes on the region defined by the VMFS block size and not just the block that was written to by the application. This behavior will impact performance of array-based replication software since more data needs to be copied based on the VMFS block size than needed.
FC storage systems. Please note that remote replication software (MirrorView and SAN Copy) is supported on CLARiiON iSCSI storage systems: CLARiiON replication considerations with VMware ESX Server Please note that: • Use of RDM volumes for replication is not supported when an ESX 2.5.x server is booted from a SAN LUN. In other words, when the Fibre Channel HBAs are shared between the service console and the virtual machines, RDM cannot be configured on an ESX 2.5.x server.
system. Use Navisphere Manager or Navisphere CLI instead. The only commands that will work are admsnap flush and admhost flush. • VMFS volumes are not supported when replicating application data images from a physical (native) server to an ESX server. • For ESX 2.5.x, ensure that a CLARiiON snapshot, clone, or mirror of a VMFS volume is not in a device not ready state (snapshot not activated, session not started, clone not fractured, or secondary mirror not promoted) when it is assigned to an ESX server.
consistent data-disk replica of the RDM volume to a virtual machine connected directly (via iSCSI) to a CLARiiON storage system or a physical server. If the application data disk replica needs to be assigned to the VMware ESX server, the LUN must be manually placed in the VMware ESX server storage group, and the VMware administrator must then assign the replica (application data disk) to an existing virtual machine.
ESX hosts) for VMotion with VMFS volumes create a single storage group for all ESX servers in a farm or cluster since LUNs may be assigned different Host LUN numbers if separate storage groups are created. For additional information, see EMC Knowledgebase cases emc151686 and emc153719. Starting with the VMware ESX Server 3.5 latest patch version, LUNs are identified by their LUN NAA (WWNs), thus the requirement that the Host LUN numbers (HLU) must match across storage groups for VMotion is no longer valid.
VMware ESX Server (Initiator) Virtual Switch (vSwitch) Virtual Machine (to be migrated) NIC VMware ESX Server (Target) NIC Gigabit NIC NIC NIC NIC dedicated for VMotion/VMkernel NIC dedicated for VMotion/VMkernel NIC dedicated to the Service Console Gigabit NIC NIC dedicated to VMkernel NIC dedicated to VMkernel NIC dedicated to the Service Console Figure 20.
Figure 21. VMware vCenter 4.0 management screen showing how VMotion migration is initiated and completed CLARiiON with VMware Distributed Resource Scheduling and High Availability Both VMware Distributed Resource Scheduling (DRS) and VMware High Availability (HA), when used with VMotion technology, provide load balancing and automatic failover for virtual machines with VMware ESX 4.0/3.x/ESXi. To use VMware DRS and HA, a cluster definition must be created using VMware vCenter 2.0.
CLARiiON and virtual machine clustering Clustering refers to providing services through a group of servers to achieve high availability and/or scalability. Clustering with VMware ESX Server is supported at the virtual machine level. Refer to the ELab Navigator for the cluster software products that are supported on virtual machines. To implement clustering at the virtual machine level, the virtual machines must boot from local disks and not CLARiiON disks.
Virtual-to-virtual clustering IN A CLUSTER VMware ESX Server 1 Virtual Machine 1 (Boot from Local Disks) Nonshared CLARiiON LUNs VMware ESX Server 2 Virtual Machine 2 (Boot from Local Disks) Virtual Machine 1 (Boot from Local Disks) Nonshared CLARiiON LUNs Nonshared CLARiiON LUNs Storage-group: ESX 1 Nonshared LUNs (LUN 1, 2, 3) Virtual Machine 2 (Boot from Local Disks) Nonshared CLARiiON LUNs Storage-group: ESX 2 Shared LUNs (Quorum) (LUN 7) Nonshared LUNs (LUN 4, 5, 6) Figure 23.
Physical-to-virtual clustering IN A CLUSTER VMware ESX Server 1 Physical Server Virtual Machine 1 (on Local disks) Virtual Machine 2 (on Local disks) Nonshared CLARiiON LUNs Nonshared CLARiiON LUNs Application OS Hardware Storage-group: ESX 1 Nonshared LUNs (LUN 1, 2, 3) Storage-group: Physical server Shared LUNs (Quorum) (LUN 7) Nonshared LUNs (LUN 4, 5, 6) Figure 24.
MV/CE and guest clustering using RDMs In this type of guest clustering, RDM volumes are assigned to the virtual machines and failover-cluster software, and MV/CE software is installed and configured on these virtual machines. The following restrictions apply when working with this configuration: • Only physical compatibility mode RDMs can be used for cluster shared storage. • Virtual-to-physical clustering (for example, standby host clustering) is not supported. • MV/CE 3.1.0.
Figure 25. Enable NPIV for a virtual machine after adding a RDM volume For some switches, the virtual WWNs’ names must be entered manually within the switch interface and then zoned to the storage system. The CLARiiON storage system can then see the initiator records for the virtual machine (virtual WWNs). These initiator records need to be manually registered as shown in Figure 26.
The following points summarize the steps required to configure NPIV: 1. Ensure the HBA, switch, and ESX version support NPIV. 2. Assign a RDM volume to the ESX server and then to the virtual machine. 3. Enable NPIV for that virtual machine to create virtual WWNs. 4. Manually enter the virtual WWNs within the switch interface. Some switches might be able to view the WWNs without manual entry of the virtual machine virtual WWNs. 5.
The following steps outline the process for initializing an SRM environment using Navisphere Manager and/or Navisphere SecureCLI. The commands must be issued from a management host that is network connected to the production CLARiiON storage array. Note that all of these commands can be performed in the Navisphere Manager GUI or in CLI. Using Navisphere Manager to configure MirrorView To configure sync or async mirrors, open the wizard and follow the instructions in the wizard. Figure 27.
naviseccli –h SP ipaddress mirror –sync –addimage –name -arrayhost -lun 4. Even if there is only a single LUN being replicated to the secondary site, you still need to create a consistency group for SRM. The following commands show how to create a consistency group and add existing mirrors to the consistency group.
5. If for some reason the mirrors are fractured, the syncgroup option (shown next), can be used to resynchronize the primary and secondary images: naviseccli –h SP ipaddress mirror –async –syncgroup –name 6.
Configuring SnapView snapshots for SRM testing purposes via NaviSecCli 1. Add the LUNs bound for SnapView Sessions into the Reserved LUN Pool. naviseccli –h SP ipaddress reserved –lunpool –addlun 2. Create a snapshot for each LUN at the recovery site, and add the SnapShot to ESX Server’s CLARiiON Storage Group at the recovery site.
Table 6. Maximum number of async mirrors and consistency groups Parameter CX4-120 CX4-240 CX4-480 CX4-960 Total mirrors per storage system Total mirrors per consistency group Total consistency groups per storage system 256 256 256 256 32 32 64 64 64 64 64 64 Note: The maximum allowed number of consistency groups per storage system is 64. Both MirrorView/A and MirrorView/S consistency groups count toward the total.
Figure 29. SRM recovery plan During this test, you would see the following events occur: 1. 2. 3. 4. Production VMs are still up and running CLARiiON SnapView sessions are created and activated against the snapshots created above All resources created within the SRM Protection Group carry over to the recovery site VMs power on in the order defined within the recovery plan Once all the VMs are powered on according to the recovery plan, SRM will wait for the user to verify that the test works correctly.
You should execute a SRM recovery plan only in the event of a declared disaster, to resume operation at the recovery site. Failback scenarios The nature of the disaster, and which components of the data center infrastructure are affected, will dictate what steps are necessary to restore the original production data center. For details on how to address different failback scenarios for MirrorView, please see the white paper MirrorView Knowledgebook on Powerlink.
Figure 31. Virtual Machine tab available under ESX server Clicking on one of the virtual machines in Figure 32 opens the Virtual Machine Properties dialog box shown in Figure 33. You can see which LUNs have been assigned to this virtual machine and how they have been configured. In this example, the VM configuration (.vmx) file location, virtual disk properties (thick or thin), and raw mapped volume information are listed for the virtual machine.
In addition, a new report called the Virtual Machine report can now be generated with the Reporting wizard on the Navisphere Task Bar. This report gives you an overall picture of the LUN-to-VM device mapping. Figure 33. Virtual Machine report The search function in FLARE release 29 allows you to search for a given virtual machine.
Figure 34. Search for a virtual machine using the “Advanced Search” function in Navisphere Use Cases Issue with VM performance -- Find out which LUN a virtual machine is using. Before VM-aware Navisphere After VM-aware Navisphere If using VMFS: In the VMware Client GUI, open the VM “Edit Setting” dialog box, and find the datastore used by the VM hard disk. Find the ESX device name for that datastore. In Navisphere, search for the VM by name or IP.
Validate changes made to the VMware environment -- Find and record all VM to all LUNs mapping. Before VM-aware Navisphere After VM-aware Navisphere Generate an ESX device to CLARiiON mapping report. Then the VMware admin must perform multiple steps in VCenter to calculate which VM disks uses which LUN, and put the two reports together side by side. They could also maintain updated Excel spreadsheets for this mapping information. Generate an end-to-end mapping of VM disk to CLARiiON LUN report.
Authenticate your CLARiiON storage system using the symcfg command Discover your CLARiiON storage system using the symcfg discover -clariion • Install the Storage Viewer plug-in • Navisphere CLI (recommended) • VMware vCenter 2.5 or later is needed Once the software stack above has been installed and the CLARiiON storage system has been discovered by Solutions Enabler, enable the Storage Viewer plug-in using the Managed Plugin tab within vCenter.
Figure 37. Virtual machine and corresponding CLARiiON detailed LUN information visible within VMware vCenter For more detailed information on installing and configuring EMC Storage Viewer, see the Using the EMC Storage Viewer for Virtual Infrastructure Client white paper available on Powerlink. Conclusion EMC CLARiiON and VMware technologies provide the complete Information Lifecycle Management solutions that customers need to consolidate their storage and servers at a low cost.
• iSCSI SAN Configuration Guide http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf Appendix A: Copying data from a VMFS to RDM volume The export/import CLI command can be used at the ESX Server level to copy data from a VMFS volume to a raw device mapping volume. Consider an example; The virtual machine resides on a VMFS volume and has the virtual disk name test.vmdk. The data on this virtual machine needs to be copied to a RDM volume.
Appendix B: Using vm-support on VMware ESX Server VM support is the command tool used to aid in diagnostics and/or troubleshooting of the ESX server. This service tool is supported on ESX 4.0, 3.x, and 2.x. For VMware ESXi, use VI Client’s Export Diagnostics Data option to get vm-support files. The following procedure outlines the steps executed on the ESX 4.0/3.x service console. Enter the vmsupport command on the ESX service console. This script generates a .tgz file in the current directory.
