HP StorageWorks 9100 Extreme Data Storage System administration guide V1.0.2 (AN540-96018, February 2010)

3. Before rebooting any servers, determine whether the system is otherwise operating normally. If
the mx alert status command is not reporting problems with other virtual hosts or other
filesystems, then the system can continue operation as the /var/mxso filesystem and the database
are not needed for file I/O operations, NFS file serving, or HTTP serving. If these are working
normally, you can postpone any action to correct the database access until a later time.
4. Run the exds_stdiag command to check the physical connectivity and state of storage blocks.
The database is located on /var/mxso which is normally hosted by the first array. See The
exds_stdiag utility section on page 149 for information on how to interpret the output from
exds_stdiag. If necessary, shutdown all servers and power cycle the affected storage compon-
ents.
5. Run the df -h command to see if any filesystems (including local filesystems such as /var) are
full. If so, clear out logs and other large files.
6. If /var/mxso is mounted and not full, reboot the server that is currently serving the exdsadmin
virtual host alias (this is usually the first server).
7. If the source of the problem is not immediately obvious from the mx alert status command
and the above actions, shut down all servers and boot one server only. The database should
become operational again and you can use the exdsmgr command to boot the remaining
servers.
8. If the varmxso (/var/mxso utility filesystem) appears to be corrupt, you can switch to a backup
filesystem. Instructions for doing this are described in Recovering from corrupt or lost utility
filesystem (/var/mxso) on page 132.
Matrix and files ystem information seems out-of-date
The exdsmgr show matrix and exdsmgr show filesystem commands display data which
is collected by the somon_matrix_state daemon. This daemon (/opt/hp/mxso/bin/
somon_matrix_state) runs on all servers but performs most of its work on the first member of the
cluster (typically, the first 'up' server listed by mx server status).
If you suspect that the exdsmgr show matrix and exdsmgr show filesystem command
output is problematic or not up-to-date, it could be because the /opt/hp/mxso/bin/
somon_matrix_state daemon has hung or stopped running on one or more servers. The solution
is to restart the somon_matrix_state daemon.
To restart the somon_matrix_state daemon:
1. Restart the daemon on the first member of the cluster using following command:
# /etc/init.d/somon_scripts restart
2. If the show matrix or show filesystem command still shows wrong or incomplete data,
then restart the daemon on all servers.
3. If the command to restart the daemon does not run cleanly, you might need to kill the daemon
by hand:
a. Search for the process by entering the following command:
ps auwx|grep somon_matrix
pmxs 14359 1.0 0.0 103752 6128 ? Ssl 10:43 0:00 /usr/bin/so-
mon_matrix_state monitor neo1 verbose
b. Execute the kill command:
# kill -9 14359
c. Run the # /etc/init.d/somon_scripts restart, which should now execute cleanly.
9100 Extreme Data Storage System administration guide 143