Open Source Object Storage for Unstructured Data: Ceph on HP ProLiant SL4540 Gen8 Servers

Reference Architecture| Ceph on HP ProLiant SL4540 Gen8 Servers
Upgrading Ubuntu
Update kernel and packages to latest; as of this writing Ubuntu12.04.3 is running with 3.8.0-37 on the sample reference
configuration.
If you need to upgrade the kernel before installing Ceph, use ‘apt-get dist-upgrade’ to do intelligent package upgrade.
Despite the name, the ‘apt-get dist-upgrade’ allows upgrade of the kernel packages, not the distribution.
sudo apt-get update && sudo apt-get dist-upgrade
Useful Additional Package Configuration
Install NTP. Since cluster coherency requires minimal time skew at least one of the monitors should be an NTP reference. A
real time reference is optimal, but a synchronizing time reference will work for testing.
For runtime, some additional useful packages (none are absolutely necessary for cluster setup)
• lsscsi for initial OSD creation, supplementing ceph-deploy disk list to map unassigned drives to OSDs.
• collectl for checking performance during runtime to tune/evaluate cluster load.
Command Line Logical Drive Configuration
Install the HP CLI package hpssacli to configure new logical volumes and other array parameters. This will help reduce
having to reboot the system to get to the HP SSA GUI and manually creating dozens of single drive RAID 0 volumes per
SL4540. The recommended configuration for boxes tested is all spinning media as single drive RAID 0; here’s the CLI
syntax:
sudo /usr/sbin/hpssacli controller slot=1 create type=arrayr0 drives=allunassigned
The hpssacli binary is not distributed in .deb format in sync with .rpm releases. To get the latest HP Smart Storage
Administrator CLI for Linux, download the .rpm and extract the files.
Go to http://www8.hp.com/us/en/products/server-software/product-detail.html?oid=5409020#!tab=features
Download the 64-bit Linux RPM. At the time of this paper it was hpssacli-1.50-4.0.x86_64.rpm. Copy to a directory where
the extraction can be run.
If not already present, download rpm2cpio to convert and ‘install’ the files from the extraction directory.
sudo apt-get install rpm2cpio
rpm2cpio hpssacli-1.50-4.0.x86_64.rpm | cpio –idmv
sudo cp –R opt/* /opt
sudo cp –R usr/* /usr
The CLI can now be run from /usr/sbin with sudo permissions.
Updating the Driver for the 10GbE Mellanox NIC
HP recommends that you upgrade to the v2.1 or later mlnx_en driver for use with the HP SL4540 10GbE IO Module. This
driver is available from http://www.mellanox.com/page/products_dyn?product_family=27
. For this reference architecture,
we used the mlnx_en_2.1-1.0.0.tgz package. Copy this file to a directory where the extraction can be run. These steps
extract the driver source, build it and install the driver so that they will load persistently. If you haven’t already installed
them, install make and gcc.
tar xzvf mlnx-en-2.1-1.0.0.tgz
cd mlnx-en-2.1-1.0.0/SOURCES
gunzip mlnx-en_2.1.orig.tar.gz
tar –xvf mlnx-en_2.1.orig.tar
cd mlnx-en-2.1
./scripts/mlnx_en_patch.sh
make
sudo modprobe –r mlx4_en
edit /etc/depmod.d/ubuntu.conf
Add extras to the search list, resulting in the line:
search extra updates ubuntu built-in
sudo make install
sudo modprobe mlx4_en
sudo vi /usr/share/initramfs-tools/modules.d/mlx4_en
Contents of the file should be these lines:
mlx4_en
mlx4_core
Save and quit back to command line.
sudo update-initramfs –u –k all
38