White Papers
Table Of Contents
- Executive Summary (updated May 2011)
- 1. Introduction
- 2. Dell NFS Storage Solution Technical Overview
- 3. NFS Storage Solution with High Availability
- 4. Evaluation
- 5. Performance Benchmark Results (updated May 2011)
- 6. Comparison of the NSS Solution Offerings
- 7. Conclusion
- 8. References
- Appendix A: NSS-HA Recipe (updated May 2011)
- A.1. Pre-install preparation
- A.2. Server side hardware set-up
- A.3. Initial software configuration on each PowerEdge R710
- A.4. Performance tuning on the server
- A.5. Storage hardware set-up
- A.6. Storage Configuration
- A.7. NSS HA Cluster setup
- A.8. Quick test of HA set-up
- A.9. Useful commands and references
- A.10. Performance tuning on clients (updated May 2011)
- A.11. Example scripts and configuration files
- Appendix B: Medium to Large Configuration Upgrade
- Appendix C: Benchmarks and Test Tools
Dell HPC NFS Storage Solution - High Availability Configurations
Page 20
Once both servers are manually power cycled, they rejoin the cluster and one server takes
ownership of the HA service to provide the file system to the clients.
5) Fence device failure - simulated by disconnecting the iDRAC cable from the server.
If the iDRAC on a server fails, the server will be fenced via the network PDUs which are defined
as secondary fence devices in the cluster configuration files.
At the time of fencing, one server will attempt to fence the other server via the iDRAC. When
that fails, it will attempt to fence via the network PDUs. The server will rotate through these
fence devices until fencing is successful.
6) One SAS link failure - simulated by disconnecting one SAS link between the PowerEdge R710
server and the PowerVault MD3200 storage.
In the case where only one SAS link fails, the cluster service is not interrupted. Since there are
multiple paths from the server to the storage, a single SAS link failure does not break the data
path from the clients to the storage and thus does not trigger a cluster service failover.
Note that for all cases (1) through (6) it was observed that the HA service failover takes in the
range of one to two minutes. Thus in a healthy cluster, any failure event should be noted by
the Red Hat cluster management daemon and acted upon within minutes.
7) Multiple SAS link failures - simulated by disconnecting all SAS links between one PowerEdge
R710 server and the PowerVault MD3200 storage.
When all SAS links on the active server fail, the multipath daemon on the active server retries
the path to the storage based on the parameters configured in the multipath.conf file. This
is set to 150 seconds by default. After this process times out, the HA service will attempt to
failover to the passive server.
If the cluster service is unable to cleanly stop the LVM and the file system because of the
broken path, a watchdog script will reboot the active server after five minutes. At this point
the passive server will fence the active server, restart the HA service and provide the data path
to the clients. This failover can therefore take anywhere in the range of three to eight
minutes.
When the active server reboots, it will rejoin the cluster. It will, however, not be a candidate
to run the HA service until at least one SAS link is established between the server and the
storage.
Impact to clients
Clients mount the NFS file system exported by the server using the HA service IP. This IP is
associated with either an InfiniBand or a 10 Gigabit Ethernet network interface on the server. To
measure any impact on the client, the dd utility and the iozone benchmark were used to read and
write large files between the client and the file system. Component failures were introduced on
the server while the client was actively reading and writing data from the file system.
In all scenarios it was observed that the client processes complete the read and write operations
successfully. As expected, the client processes take longer to complete if the process is actively
accessing data during a failover event. In the InfiniBand case, the client process takes 15-35%