Segment Full Alert: Using Ibrix_Rebalance to Resolve a Segment 90% Full Alert

Title: Segment Full Alert : Using ibrix_rebalance to resolve a Segment 90% full Alert

Problem:

Segment 5 reporting Segment Full Alert at 90% and Segment 3 is not far behind. As

such we will attack this in 2 rebalance jobs. The first will mitigate the primary concern of

filling up Segment 5 in a short amount of time therefore we need a faster result from

running a source to destination type ibrix_rebalance command. THe second will then be a

general ibrix_rebalance job for all segments.

ibrix_fs -i fs_name prior to any reblanacne:

SEGMENT OWNER LVNAME BLOCKSIZE BLOCKS(1k) BFREE(1k)

BAVAIL(1k) FILES FFREE USED% BACKUP O NBACKUP? INFSCK?

TYPE

1 n1mwpseg01 ilv1 4096 257902212 34536408 33930156 16768000

16548631 86 n1mwpseg02 No No MIXED No No

2 n1mwpseg02 ilv2 4096 257902212 38927984 38321732 16768000

16547208 84 n1mwpseg01 No No MIXED No No

3 n1mwpseg03 ilv3 4096 257902212 28319968 27713716 16768000

16547670 89 n1mwpseg04 No No MIXED No No

4 n1mwpseg04 ilv4 4096 257902212 46977332 46371080 16768000

16546347 81 n1mwpseg03 No No MIXED No No

5 n1mwpseg01 ilv5 4096 257902212 24938288 24332036 16768000

16548161 90 n1mwpseg02 No No MIXED No No

6 n1mwpseg02 ilv6 4096 253902212 221439196 220801692 32768000

32689585 12 n1mwpseg01 No No MIXED No No

7 n1mwpseg03 ilv21 4096 257902212 186182052 185575800 16768000

16677096 27 n1mwpseg04 No No MIXED No No

8 n1mwpseg04 ilv22 4096 257902212 197863448 197257196 16768000

16675081 23 n1mwpseg03 No No MIXED No No

Currently Segment 6 stands at 12%, segments 7 and 8 are reporting a USED% of 27 and

23% respectively. It looks segments 7 and 8 were added recently for extending the

namespace as such the FS is in need of a rebalance. As long as the FS is an Ibrix 3.1

version cluster and as such carries the ability to execute a rebalance job to prevent from

filling up segment 5 and also 3 which is now at 89%. Where the FS is close to the

Segment full watermark on 2 Segments we would suggest a source to destination type

rebalance job to yield a quicker result than a standard full File System rebalance job

which may take longer than desired. The rebalance job is run from your Fusion Manager

PAGE 1
Title: Segment Full Alert : Using ibrix_rebalance to resolve a Segment 90% full Alert Problem: Segment 5 reporting Segment Full Alert at 90% and Segment 3 is not far behind. As such we will attack this in 2 rebalance jobs. The first will mitigate the primary concern of filling up Segment 5 in a short amount of time therefore we need a faster result from running a source to destination type ibrix_rebalance command. THe second will then be a general ibrix_rebalance job for all segments.
PAGE 2
but the source to destination rebalance requires that you monitor the job itself every 15 minutes or so for a few hours or until the source segments' USED% is close or equal to that of the destination segments' USED% . Note: This is an online operation which runs in the background, however You must monitor the job by issuing the ibrix_fs -i -f ibhome command from your FM every 15 minutes or so and keep an eye on the USED% field for the segments called out in the command below.
PAGE 3
5 n1mwpseg01 ilv5 4096 257902212 42455516 41849264 16768000 16557536 83 n1mwpseg02 No No MIXED No 6 n1mwpseg02 ilv6 4096 253902212 207194060 206556556 32768000 32677672 18 n1mwpseg01 No No MIXED No 7 n1mwpseg03 ilv21 4096 257902212 168471596 167865344 16768000 16665277 34 n1mwpseg04 No No MIXED No 8 n1mwpseg04 ilv22 4096 257902212 167436520 166830268 16768000 16663241 35 n1mwpseg03 No No MIXED No Rebalance Job#1 results summary: Ok that looks good for segments 5 , 1 and 3.