Dell Fluid Cache for SAN Version 2.0.
Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your computer. CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem. WARNING: A WARNING indicates a potential for property damage, personal injury, or death. Copyright © 2016 Dell Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws.
Contents 1 Dell Fluid Cache for SAN Product Overview.................................................... 6 Key Features...........................................................................................................................................6 Connectivity...........................................................................................................................................6 2 Requirements for Fluid Cache....................................................................
Removing Volume Mappings from a Server.................................................................................27 Removing Volume Mappings from a Subcluster......................................................................... 28 Removing Volumes............................................................................................................................. 28 Removing a Server from a Fluid Cache Cluster.................................................................................
F Related Documentation.................................................................................... 49 Accessing Enterprise Manager and Storage Center Documentation...............................................
Dell Fluid Cache for SAN Product Overview 1 Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store. Key Features Fluid Cache supports the following features: • Write-back caching.
Figure 1. Fluid Cache Connectivity 1. Fluid Cache nodes communicate with each other over a private network using RDMA. 2. Enterprise Manager creates, manages, and monitors the Fluid Cache clusters. 3. Fluid Cache nodes communicate with the Management IP (VIP) of the Storage Controllers. 4. Enterprise Manager manages the Dell Compellent array. The interfaces illustrated in the figure are associated with specific ports (see Checking Network Connections).
Requirements for Fluid Cache 2 The following are prerequisites for deploying Fluid Cache for SAN. Refer to the Release Notes for the most recent requirements. The supported versions of firmware, drivers, and software are required to run Fluid Cache. These are listed in the Release Notes and are available at dell.com/support. For download instructions, see Downloading Firmware, Drivers, and Software.
Element Operating Systems Requirement • • • • Dell PowerEdge R730XD Dell PowerEdge R820 Dell PowerEdge R920 Dell PowerEdge T620 • • • • • Red Hat Enterprise Linux (RHEL) 6.4 (64-bit) Red Hat Enterprise Linux (RHEL) 6.5 (64-bit) SUSE Linux Enterprise Server (SLES) 11 SP3 (64-bit) Oracle Enterprise Linux (OEL) 6.5 (64–bit) Oracle Enterprise Linux (OEL ) 6.5 (64–bit) For Oracle Linux 6.4 (64-bit), use the RHEL 6.4 Fluid Cache RPM package. NOTE: The Mellanox ConnectX-3 Driver, version 2.0.2.6.
Element Requirement SAN Software • • • • • • • Dell Compellent Enterprise Manager 2014 R1 Dell Compellent Enterprise Manager 2014 R2 Dell Compellent Enterprise Manager 2015 R1 Dell Compellent Enterprise Manager 2015 R2 Dell Compellent Storage Center SCOS 6.5.2 Dell Compellent Storage Center SCOS 6.6.5 Dell Compellent Storage Center SCOS 6.7.
Fluid Cache Deployment Overview 3 Complete the following tasks to deploy Fluid Cache for SAN. The sections referenced in each task give detailed instructions. 1. Download the required software, firmware, and drivers. See Downloading Firmware, Drivers, and Software. 2. Identify and prepare the Fluid Cache servers and cache network switch. See Preparing the Fluid Cache Servers and Cache Network Switch. 3. Install the Fluid Cache software on each server in the cluster.
Downloading Firmware, Drivers, and Software 4 NOTE: It is recommended that you update servers using iDRAC. For more information, go to en.community.dell.com/techcenter/b/techcenter/archive/2013/04/17/idrac7-now-supportsupdating-server-components-using-racadm-and-web-gui.aspx. 1. Determine which firmware, drivers, and BIOS are installed for your system components by following the procedures in Checking System Configuration. 2. Go to dell.com/support. 3.
Preparing the Fluid Cache Components 5 Before installing Fluid Cache, you must prepare the components of the Fluid Cache network: the servers, cache devices, network cards, and switches. The instructions that follow assume that you have an existing SAN configured and managed. All nodes in the cache cluster must be connected to the SAN and visible on the Dell Compellent array. Instructions for racking and cabling a SAN solution are beyond the scope of this document.
Interface Port number Port type Fluid Cache to Storage Center 3033 TCP NOTE: Fluid Cache cannot be configured and does not function properly if any of these ports are unavailable. To diagnose network connection problems and reassign port numbers, see the troubleshooting topic Cannot Configure Fluid Cache. RDMA Functionality To make sure that RDMA is working correctly between the cluster nodes, use the utility ib_send_bw: 1. On one node, run the command: ib_send_bw 2.
1. Enter the system BIOS and set System Profile Settings to Performance. 2. Click Back, and then click OK to return to the main BIOS page. 3. Restart the server to activate the BIOS changes. 4. Repeat this process for each server in the Fluid Cache cluster. Preparing the Cache Devices • Make sure that all cache devices used in the Fluid Cache cluster have the minimum firmware and driver versions specified in Requirements for Fluid Cache.
NOTE: It is recommended to use an MTU of 1500 for the cache network. NOTE: Do not use a network adapter driver downloaded from the manufacturer’s site. Fluid Cache uses a custom driver available only through Dell. 3. Create and configure an IP address on all RoCE network interfaces that are connected: a. Open the interface configuration file by running the commands: cd /etc/sysconfig/network-scripts vi ifcfg- b.
ONBOOT="yes" IPADDR="" NETMASK="" NAME="bond0" BONDING_OPTS="mode=1 fail_over_mac=1 miimon=100 downdelay=300 updelay=300" NOTE: In the BONDING_OPTS line, the values for mode and fail_over_mac must be set exactly as shown in the example. 5.
Preparing the Cache Network Switch • For a schematic of an example Fluid Cache installation, see Example Cabling Diagram. • For configuration tasks for a supported Dell Networking switch, see Configuring a Dell Networking Switch. For all other switches, see the manufacturer’s documentation. • For a list of supported switches, see Requirements for Fluid Cache. 1. Connect the ports on the network adapter to properly configured ports on a network switch.
Installing and Setting up Fluid Cache 6 Before completing the tasks in this section, install the required Linux dependencies. See Checking Linux Dependencies. Installing the Fluid Cache Software 1. Copy to the server the Fluid Cache tar.gz package that you downloaded earlier (see Downloading Firmware, Drivers, and Software). 2. Expand the tar.gz package. A new Fluid Cache directory is created, which contains an RPM file. 3.
An example of the required information is a device with an IP address of 172.18.1.2, whose network address is 172.18.1.0, and netmask is 255.255.255.0. 1. Change to the following directory:/opt/dell/fluidcache/bin/ 2. Start the Host Cache Node (HCN) Setup tool by running the following command: ./hcn_setup.py HCN Setup sets up a server for use as a Fluid Cache cluster node, and starts an agent on the server that allows it to be discovered by Enterprise Manager. 3.
Configuring and Managing Fluid Cache Clusters 7 Make sure that your system meets the following prerequisites: • The Fluid Cache software must be installed and running on all servers to be used in the Fluid Cache cluster. • All servers in the Fluid Cache cluster must be connected to the SAN and be visible on the Compellent array. • Enterprise Manager must be running and configured to manage the Storage Center to be used with Fluid Cache.
The Cluster Settings page of the configuration wizard is displayed. 8. Type or select appropriate data in the Cluster Settings window. a. In the Name box, enter a name for the cluster. b. Click Browse next to the License File box. The Select Fluid Cache License File dialog box is displayed. c. Browse to the location of the license file, select the file, and then click Save. d. Verify that the license file and path displayed are correct and click Next.
To map a volume to a server: 1. In the Storage view, expand Storage Centers if necessary, and then select the Storage Center that contains the appropriate volume. 2. In the Storage tab, expand Volumes if necessary, and then locate the volume you want to map. 3. Right-click the volume and select Map Volume to Server. 4. In the Map Volume to Server window, select the server. 5. Click Next. 6. Select Enable Fluid Cache. 7.
3. Right-click the Fluid Cache cluster (not Fluid Cache Clusters itself) and click Tasks → Add Servers to Cluster. The Add Servers to Cluster dialog box is displayed. By default, all servers are selected that have Fluid Cache installed. 4. Clear the check boxes next to unwanted servers or select the Unselect All option, and then select the servers to be added. (To use all available servers again, select the Select All option.) 5. Click OK.
6. Click OK.
A Maintaining Fluid Cache Installations This chapter describes tasks that must occasionally be performed after initial Fluid Cache deployment. WARNING: When performing any operation to a server platform that hosts Fluid Cache software, you must perform either one of the following methods: • Shut down a single Fluid Cache Cluster node at a time and perform the host-based server updates. After the updates to host server are complete, reboot the host and bring the Fluid Cache Cluster node online.
NOTE: The Fluid Cache mappings for the server are listed in the Volumes area at the bottom of the page. Removing Volume Mappings from a Subcluster 1. In Enterprise Manager’s Storage view, expand Storage Centers if necessary and select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.) 2. In the Storage tab, expand Servers, and then the Fluid Cache clusters if necessary and select the subcluster whose mappings you want to remove. 3.
b. In the Summary tab, locate the server in the Servers section. c. Right-click the server and select Remove Server from Cluster. d. When asked to confirm the action, click OK. After the system processes for some time, the server reappears outside of the cluster in the Servers area. You may need to manually refresh the display.
3. Remove all Fluid Cache mappings from all volumes in the cluster (see Removing Volume Mappings). Note that you do not have to remove non-Fluid Cache mappings. 4. In Enterprise Manager’s Storage view, right-click the Fluid Cache cluster and click Delete. 5. When prompted to confirm the deletion, click OK. There may be a delay while the deletion is processed. 6. Shut down the Fluid Cache hosts. Removing a Cluster 1. Stop all I/O on cached volumes. 2. Unmount cached volumes. 3.
1. Make sure the SAN and Storage Center are fully operational, and the cache network is configured and running. 2. In Enterprise Manager, click the Storage view. 3. In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster. 4. Click Restart. Recreating a Fluid Cache Cluster Although it is not part of standard operations or maintenance, in some instances you may want to recreate a Fluid Cache cluster. 1. Make sure that no cached LUNs on the servers are in use.
B Troubleshooting Fluid Cache Installations If you have issues running Fluid Cache after a successful completion of the installation procedure, contact your Compellent Copilot. Troubleshooting the Compellent array and SAN architecture is beyond the scope of this document. For additional troubleshooting information, refer to the Enterprise Manager Administrator’s Guide and the documentation for other hardware and software components. See Related Documentation.
daemon start. To ensure that the Avahi daemon starts when the server is restarted, run the command: chkconfig avahi-daemon on. If your avahidaemon.conf file does not use the default settings, make sure that Disable Publishing is set to no, that Deny Interfaces does not list Fluid Cache interfaces, and that Allow Interfaces is either commented out or lists all Fluid Cache interfaces.
Solution From the server, run the command: service avahi-daemon start. See also Checking Linux Dependencies. Cache Device Does Not Appear in List of Cache Devices Possible Cause The cache device you are trying to add is not supported by Fluid Cache. Solution Refer to the list of supported devices in Requirements for Fluid Cache. Possible Cause The driver or firmware for the cache device is not a supported version.
Solution Remove the volume mapping and create a new mapping. When you create the new mapping, you can then select a different cache mode (“cache policy”). See Removing Volume Mappings and Mapping Volumes. Possible Cause The cluster is in maintenance mode. Solution Take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and clearing the Maintenance Mode option.
Fluid Cache License Is Expired Possible Cause System settings such as changes to the system date cause the current Fluid Cache license to expire. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active. Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs.
ib_send_bw that prevents connections to another server and thus blocks Fluid Cache network traffic. Possible Cause One of the ports required by Fluid Cache is in use by another process. Solution Refer to the required ports listed in Checking Network Connections and reassign ports as needed.
Cannot Assign or Remove a Storage Center Possible Cause The Storage Center is already assigned to another Fluid Cache cluster. Solution In Enterprise Manager, see whether or not a Storage Center is listed for the Fluid Cache cluster. Possible Cause Network connectivity issues are preventing Enterprise Manager from communicating with Storage Center. Solution Make sure the network is functioning properly.
Solution Make sure the application has an I/O profile that can leverage Fluid Cache, which accelerates reads, writes, and read-after writes. If the application does, check the running cache mode for that volume in Enterprise Manager. If necessary, change cache modes by deleting the volume mapping and creating a new one with a different cache mode. See Mapping Volumes. Possible Cause The active data set greatly exceeds the size of the cache pool.
C Example Cabling Diagram Figure 2. Example Cabling Diagram 1. Enterprise Manager console 2. Management interface switch 3. Servers (4) 4. Cache network switches (2) 5. SAN switch 6. Storage Center controllers (2) 7.
• Fluid Cache is installed on four servers, which conforms to the minimum of three servers per each cache cluster and maximum of nine servers required. • Each of the servers has a network adapter connected to a port on each of the two cache network switches. The switches are uplinked to each other. • The servers access the Storage Center using SAN connectivity. • Enterprise Manager is used to configure and monitor Fluid Cache.
D Configuring a Dell Networking Switch The following procedure is for one of the supported Dell Networking switches listed in the table in Requirements For Fluid Cache. For all other switches, see the manufacturer’s documentation. To configure a switch containing physical ports 0, 1, 2, and 3 for a cache network with four nodes connected to the switch’s 10-gigabit Ethernet ports: 1. Telnet into the switch and enter the login name and password to enter Exec mode.
E Checking the System Configuration If any of the system checks below show that the firmware or drivers are earlier than the required versions, update the components. See Requirements for Fluid Cache and Downloading Firmware, Drivers, and Software. Checking the Cache Device Firmware and Driver NVMe Cache Devices • Firmware— Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD User’s Guide for instructions on checking your current firmware version.
The following example illustrates potential Fluid Cache interfaces on Mellanox network adapters: p6p2: mtu 1500 qdisc noop state DOWN qlen 1000 link/ ether a0:36:9f:14:12:a2 brd ff:ff:ff:ff:ff:ff 3. Check that you have the required OFED package installed by running the command: ofed_info 4.
/opt/dell/fluidcache/bin/fldcsan --user=: --list –hcn The results should be similar to the following, and should list all the servers you intend to include in the Fluid Cache cluster: ------------------------------------------------------hcnId: 75fd0b3d-8626-4c1c-85da-3c2a2e518932 hccId: b8ddf6de-ba87-4ce9-9ab7-6c5b39dbf7af hostname: server1.company.
F Related Documentation For PowerEdge server documentation, go to dell.com/support/manuals and enter your Service Tag. For cache device documentation, go to dell.com/storagecontrollermanuals and click Dell Power Edge Express Flash PCIeSSD. The following table lists documents you may want to refer to while installing Fluid Cache. Component Document Content Server PowerEdge Owner’s Manual Describes how to install, remove, configure, and troubleshoot server components.
Accessing Enterprise Manager and Storage Center Documentation Documentation for Dell Compellent products is not available at dell.com/support/manuals. To download Enterprise Manager and Storage Center documentation: 1. Go to portal.compellent.com. 2. Enter your user name and password and click Login. If you do not have a registration, send an email to customer.portal@compellent.com. 3. In the portal page, click Knowledge Center. 4.
