Recovering Segment Server after HDD Failure

Solution ID: 45

Created: 8/5/2008

DETAILS:

This document walks through Segment Server Recovery after a Server experiences HDD

failure.

Background:

Host ibrix2 had a failed system disk; the customer has since replaced it and installed RHEL4u4

again. While on the FM your current state of the two servers in the cluster are

#/usr/local/ibrix_host -l -S

ibrix1 Up

Ibrix2 Down, Infailover

Issue Notes:

So the failover never worked due to ibrix2 system disk crash so we will have to install ibrix

RPM on ibrix2, recover ibrix2 from the FM then fix the HA failover status.

1. Power up ibrix2, Install Ibrix RPM on ibrix2. scp the tarball from ibrix1 or the FM to ibrix2's

/tmp or / dir then untar and install ibrix on ibrix2 with the ibrixinit command.

Note: The Segment Server software installation will fail. This is expected behavior.

ibrixinit -ts -C eth0 -i xxx.xxx.xxx.xxx

2. Continue with the procedure to recover the Segment Server despite the error returned from

step 1.

Regenerate the /etc/ibrix/iadconf.xml file for ibrix2:

#/usr/local/ibrix/bin/register_server -p FusionManagerName -c ClusterInterFace -s -x

3. Start IBRIX services on ibrix2:

#/etc/init.d/ibrix_server start

4. Execute the following command on the FusionManager to recover the Segment Server:

#/usr/local/ibrix/bin/ibrix_fm -r -h SegmentServerName

PAGE 1
Recovering Segment Server after HDD Failure Solution ID: 45 Created: 8/5/2008 DETAILS: This document walks through Segment Server Recovery after a Server experiences HDD failure. Background: Host ibrix2 had a failed system disk; the customer has since replaced it and installed RHEL4u4 again.
PAGE 2
5. Now fix the Up, infailover status for ibrix2 by forcing the failover for ibrix2 from the FM [with p pwr off is recommended] #/usr/local/ibrix/bin/ibrix_host -f -p -F -M -h ServerName ..starting failover ..migrating segments ..operation successful 6. Now ibrix_host -l -S status should be ibrix1 Up ibrix2 Down, FailedOver 7. Power on ibrix2 after fail over is succesful #/usr/local/ibrix/bin/ibrix_host -P on -h ServerName 8. Failback ibrix2 now #/usr/local/ibrix/bin/ibrix_host -f -U -h ServerName 9.