Distributed Systems Administration Utilities User's Guide HP Part Number: T2786-90327 Published: March 2009 Edition: 1.
© Copyright 2009 Hewlett-Packard Development Company, L.P. Legal Notices Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents About this Document..........................................................................................................7 Intended Audience.................................................................................................................................7 Typographic Conventions......................................................................................................................7 Related Information.........................................................
3.3.1.2 Configuring a Serviceguard Cluster as a Log Consolidation Server with clog_wizard................................................................................................................................49 3.3.1.3 Cluster Configuration Notes for clog..............................................................................52 3.3.1.4 Serviceguard Automation Features................................................................................52 3.3.1.
List of Figures 2-1 3-1 3-2 4-1 cfengine Overview.........................................................................................................................15 syslog-ng Log-Forwarding Configuration....................................................................................44 syslog-ng Log Consolidator Configuration...................................................................................45 pdsh Architecture .....................................................................
List of Tables 1 2 1-1 1-2 1-3 1-4 1-5 1-6 1-7 1-8 2-1 3-1 3-2 3-3 4-1 4-2 4-3 6 Conventions.....................................................................................................................................7 DSAU Publishing History...............................................................................................................7 Configuration Synchronization Command...................................................................................
About this Document Distributed Systems Administration Utilities provide tools to simplify the management of groups of systems and of Serviceguard clusters. This document provides information on each component tool in separate chapters. Intended Audience This document is written for system administrators to assist them in learning about Distributed Systems Administration Utilities and how to use them. Typographic Conventions Table 1 Conventions Book Title Title of a book or other document.
For specific versions of HP-UX , Serviceguard, and open source components, see the Distributed Systems Administration Utilities V2.4 Release Notes for HP-UX 11i v3 March 2009 available on the HP Technical Documentation web site at http://www.docs.hp.com. Product Support For product support, contact your HP Support Representative, your HP Services Representative, or your authorized HP reseller. For more information about support services, see the HP Support web site at http://www.hp.com/go/support.
1 Introduction The Distributed Systems Administration Utilities provide several tools for simplifying the management of groups of systems and of Serviceguard clusters. There are three utilities: • Configuration Synchronization: - with this utility, based on the open source tool cfengine or “configuration engine,” the administrator can centrally define management actions to be applied to a set of managed systems. cfengine is a client/server based tool.
file and directory copies to be performed in parallel to a set of remote systems. The dshbak filter allows the output from multiple systems to be formatted and consolidated for better on-screen presentation. The cexec, ccp, ckill, cps, and cuptime tools are wrappers around the pdsh and pdcp commands optimized for use in a Serviceguard cluster. They default to executing commands cluster-wide.
Table 1-3 Command Fanout Commands (continued) Command What it Does When to Use it cps Distributes a ps(1) command to multiple To collect process information from groups hosts in parallel. In a Serviceguard cluster, of systems simultaneously. issues command cluster-wide by default. cuptime Reports uptime(1) information for multiple systems. In a Serviceguard cluster, issues command cluster-wide by default.
Table 1-7 Open Source syslog-ng Command Command What it Does syslog-ng Tool that performs consolidated logging. 1.
2 Configuration Synchronization Managing the configuration and configuration drift of a set of distributed systems is a constant challenge for system administrators. There are a variety of tools available to help manage various aspects of multi-system configuration management. For example, for account management, standard solutions include the Network Information System (NIS) and Lightweight Directory Access Protocol (LDAP).
appropriate for each group of managed clients. For example, every five minutes, once an hour, or once a day. The administrator can also invoke cfagent directly for on-demand synchronization runs. 2.1.1 cfengine Daemons and Commands cfengine employs several daemons and commands to perform configuration synchronization operations. The following list describes the primary cfengine components. • cfagent -- the cfagent command is cfengine’s workhorse.
Figure 2-1 cfengine Overview cfrun 2 cfservd 3 cfagent 1 cron cfexecd 5 + /var/opt/dsau/cfengine/inputs -update.conf -cfagent.conf -cfservd.conf -cfrun.hosts 2. 3. 4. 5. cfron cfagent cfexecd 4 Master Policy Files: +/dir/cfengine_master/master_files/ - +/dir/cfengine_master/inputs/ -update.conf -cfagent.conf -cfservd.conf -cfrun.hosts + /var/opt/dsau/cfengine/inputs -update.conf -cfagent.conf -cfservd.conf -cfrun.hosts Client Master Server 1.
synchronization service to groups of remote client systems. Those clients can be standalone systems or Serviceguard clusters. The cluster providing the cfengine service can be a client of itself and also take advantage of cfengine’s configuration synchronization features. A possible but somewhat unusual configuration is to have a fixed member of a Serviceguard cluster act as the master server but no package is configured so cfservd will not be highly available.
Table 2-1 Configuration Data for csync_wizard Configuration Data Example LVM volume group /dev/vgcsync Logical volume /dev/vcgsync/lvol1 Filesystem mount point /csync Mount options -o rw, largefiles Filesystem type vxfs Package IP Address (a registered DNS name) 192.10.25.12 Package subnet 192.10.25.0 Your Value NOTE: If you have used the wizard previously to configure a cfengine master server and rerun it to reconfigure the master server, stop the currently running configuration first.
Press Enter to continue; choose item 1 from the menu below to configure a master server: Configuration Synchronization Wizard Menu ========================================= (1) Set up a cfengine master server (2) Add a client (3) Remove a client (4) Manage keys for cfengine clients (5) Display current configuration (9) Exit Enter choice: 1 The cfengine master server is being configured on: local_hostname The wizard then asks if you would like to additionally configure managed clients immediately after conf
synchronizing critical files such as /etc/hosts, package scripts, etc. All the actions in the template are disabled by default (commented out). Uncomment the lines corresponding to the desired synchronization actions for the cluster. See the cfengine reference documentation for a description of additional cfengine features: /opt/dsau/doc/cfengine/ Press “Enter” to continue...
cfengine “master”. The master contains the configuration description and configuration files that will be used by all the clients. Clients copy the configuration description from the master and apply it to themselves. The configuration description supports a rich set of management actions such as copying configuration files from the master to the client, performing edits to files, checking file ownerships, permissions, and checksums, executing shell commands, checking for processes, etc.
NOTE: The wizard only supports creating packages based on LVM volume groups. When using CFS or VxVM, manual configuration is required. See the section on “Manually Configuring a Serviceguard Cluster Synchronization Server” (page 29) for details on manually configuring the csync package.
Would you like to manage clients? [N]: The wizard now has all the data it needs to configure the cluster and proceeds to do so: ******* WARNING!!!! ******** To protect against possible corruption of sensitive configuration files, control-c has been disabled for the remainder of this configuration. Configuring the “csync” Serviceguard package. Applying the “csync” Serviceguard package configuration file. This will take a moment. Starting the “csync” Serviceguard package. This will take a few moments...
Press “Enter” to continue... The cfengine environment consists of: Master server (policy host): package_hostname Master clients: cluster_member_1, cluster_member_2, ... A file containing the answers for this run of the Configuration Synchronization Wizard is stored here: /var/opt/dsau/cfengine/tmpdir/csync_wizard_input.txt This configuration can be reestablished by issuing the following command: /opt/dsau/sbin/csync_wizard \ -f /var/opt/dsau/cfengine/tmpdir/csync_wizard_input.txt 2.3.1.
3. 4. 5. 6. 7. for this member are also distributed to the /var/opt/dsau/cfengine/ppkeys directories on the other cluster members. The new member’s /var/opt/dsau/cfengine/inputs directory is populated. cfservd is started on the new member. The package files are copied to /etc/cmcluster/csync/ on the new member. A cfagent synchronization run is performed on the master to populate the master’s /var/opt/dsau/cfengine/inputs directory. A cfagent synchronization run is performed on the newly added member.
NOTE: When adding members to a cluster, consider the following: • When adding a member to a cluster that is configured as a highly available master server, the csync package must be running when the member is added. The add member processing task copies the configuration data from the package’s mounted filesystem to the new member’s /var/opt/dsau/cfengine directories. If the package is not running, the filesystem will not be accessible and the new member will not be properly configured.
A remote Serviceguard cluster can be configured as a managed client. However, each member must be configured individually. Repeat the configuration tasks described below for each member of the cluster. Start by logging in as root on the master server and configure ssh access to the remote system: # csshsetup hostname_of_managed_client csshsetup first tests ssh access to the remote system. If it is not configured, ssh prompts for the managed client’s password.
NOTE: You can use csshsetup to configure a trust relationship between the master server and the managed clients. This will allow you to use command fanout commands such as cexec and ccp (see cexec(1) and ccp(1)). Using these commands can simplify the configuration steps described below when files need to be distributed to managed clients. 2.3.2.1 Manually Configuring a Standalone Synchronization Server Perform the following one-time steps to configure a standalone system as a cfengine master server: 1.
Use the cfagent -p (or --parse-only) flag to verify the syntax of update.conf. 4. 5. Distribute the master update.conf to each managed client. This step is described in “Configuring a Synchronization Managed Client” (page 35). Create the master server’s security keys. cfengine uses a public/private key exchange to authenticate remote clients. A public/private key pair is generated on the master server and all managed clients.
9. The file /var/opt/dsau/cfengine_master/inputs/cfservd.conf controls which managed clients have access to the files served by cfservd on the master. Make the following edits to cfservd.conf: • Replace the “<%CFSERVD_DOMAIN_LISTS%>“token with a comma-separated list of wildcard DNS domains or hostnames for the systems that are allowed to access this server. For example: domain_list = ( “*.abc.xyz.com,*.cde.xyz.com“ ) This statement allows all hosts in the abc.xyz.com and cde.xyz.
Serviceguard package and the mechanism used to distribute cfengine’s security keys. Follow all the steps described below. • Initial Serviceguard Package Preparation 1. 2. Start by obtaining an IP address for the package. This address is typically registered in DNS to simplify management of remote clients. If you are using cfengine for intra-cluster use only, it is sufficient to make sure the address is added to each member’s /etc/ hosts file.
It is critical to keep this file very simple and avoid errors. Errors in this file will require manually copying a new version to each managed client. The file contains tokens in the form <%token-name%> which are replaced by the csync_wizard with the administrator’s answers to questions. Replace the tokens as follows: — Replace the <%POLICYHOST_NAME%> token with the fully qualified domain name of the Serviceguard csync package. For example: policyhost = ( “csync.abc.xyz.
• Edit the cfservd.conf File The file /var/opt/dsau/cfengine_master/inputs/cfservd.conf controls which managed clients have access to the files served by cfservd on the master. Make the following edits to cfservd.conf: — Replace the “<%CFSERVD_DOMAIN_LIST%>” token with a comma-separated list of wildcard DNS domains or hostnames for the systems that are allowed to access this server. For example: domain_list = ( “*.abc.xyz.com,*.cde.xyz.com” ) This statement allows all hosts in the abc.xyz.com and cde.
This will create keys named localhost.priv and localhost.pub in the directory /var/opt/dsau/cfengine/ppkeys. 2. The public key, localhost.pub is then copied to root-package IP address.pub. For example, # cp localhost.pub root-192.10.25.12.pub where 192.10.25.12 is the relocatable IP address of the csync package. 3. This member’s localhost.pub is then used to create the member-specific keys for each member: # cp localhost.pub root-member1 IP address.pub # cp localhost.pub root-member2 IP address.
3. Edit the csync.conf package ASCII configuration file to replace the placeholder tokens with the appropriate values. The tokens are in the form <%token-name%>. Find the line "SUBNET <%SG_PKG_SUBNET%>" 4. and replace the token with the subnet for the csync package. Use netstat -i to help identify the subnet. Edit the package control script, and substitute appropriate values for the placeholder tokens. NOTE: The default script template assumes you are using an LVM-based storage configuration.
1. On a managed client, use the command: # cfagent --no-lock --verbose --no-splay The verbose output will display the client, checking for updated copies of the master policy files, copying them into /var/opt/dsau/cfengine/inputs if needed, and then executing the contents of cfagent.conf/cf.main. 2. On the master server, test the cfrun command: # cfrun -- --inform --inform instructs the remote cfagent to use the --inform flag which will produce messages for all changes cfengine performs on the system.
# mkdir -p /var/opt/dsau/cfengine/inputs # cd /var/opt/dsau/cfengine/inputs # scp master_server:/var/opt/dsau/cfengine/inputs/update.conf ./update.conf To allow this client to accept cfrun requests, do the following: 1. 2. Edit /etc/rc.config.d/cfservd and set the CSYNC_CONFIGURED variable to "1" -this will start cfservd at system boot time. Start cfservd: # /sbin/init.d/cfservd start 3.
• • Encryption Checksum alerts 2.4.1 Key Exchange All the key exchange examples shown thus far have used scp to securely transfer the master server public key to the managed client and the managed client’s public key to the master server. This scheme provides the highest level of security but can be inconvenient in certain situations.
2.4.4 Checksum Alerts cfengine has a checksum alert feature. To monitor changes to a file’s checksum, do the following: • Add the following stanza to /var/opt/dsau/cfengine_master/inputs/ cfagent.conf: ChecksumUpdates = ( “on” ) • In cfagent.conf’s "files" actionsequence, add checksum = md5 or checksum = sha options for the files to monitor.
2.7 cfengine Troubleshooting The following are some troubleshooting hints for working with cfengine. 1. 2. Run cfservd on the master server using the --no-fork (-F) and the --verbose (-v) options. This will provide useful information for any troubleshooting efforts. You may see authentication errors. When performing a "cfagent -K" operation, the following messages are displayed: cfengine:: BAD: key could not be accepted on trust cfengine:: Authentication dialogue with master-server. abc.xyz.
Check your master file repository and ensure that the file in question is available and has the right permissions. 7.
3 Consolidated Logging Distributed Systems Administration Utilities offers consolidated logging features, including the standard logging features offered by syslogd, and syslog Next Generation (syslog-ng) features in standalone and cluster log consolidation environments. The next sections of this document describe their use. 3.1 Introduction to syslog syslogd is a ubiquitous component of UNIX systems that performs system logging activities.
Table 3-2 syslog Facilities Messages (continued) Message Description LOG_MAIL Mail subsystem. LOG_NEWS USENET news subsystem. LOG_SYSLOG Messages generated internally by syslogd. LOG_USER (default) Generic user-level messages. LOG_UUCP UUCP subsystem. 3.1.2 Message Filtering Using /etc/syslog.conf, messages can be filtered based on their priority level and facility. Messages can be directed to: • • • • • Specific log files The console A specified user.
• • Improved filtering functionality. In addition to syslog's facility/priority level filtering, syslog-ng can perform regular expression filtering against the program name, hostname, text of the message itself, the sender's IP address, and so on. TCP transport - In addition to syslogd’s UDP transport, syslog-ng supports a TCP transport which offers better delivery guarantees. NOTE: syslog-ng's support for a TCP transport does not imply that it safeguards against all message loss.
Figure 3-1 syslog-ng Log-Forwarding Configuration 1. 2. 3. 4. 5. The grey area represents standard syslogd operation. Applications such as Serviceguard’s cmcld daemon call syslog (see syslog(3C)) to send messages to syslogd. syslog writes messages to the local system’s /var/adm/syslog/syslog.log and related files. Applications also frequently have application-specific log files. In this example, Serviceguard maintains a log of package operations in /etc/cmcluster/package-name/package-name.log.
Figure 3-2 syslog-ng Log Consolidator Configuration 1. 2. 3. The syslog-ng server reads the incoming log data from the UDP or TCP connected clients. Note: gray arrows indicate a read operation; black arrows, a write. The grey area is identical to the client configuration in Figure 3-1: “syslog-ng Log-Forwarding Configuration”. In terms of the local system, syslog-ng acts as a client and processes locally forwarded syslog messages and clog_tail messages.
3.3.1 Using the Log Consolidation Wizard The Log Consolidation Wizard is installed as /opt/dsau/sbin/clog_wizard.
Do you want to configure log consolidation? (y/n) [y]: Answer yes (y) or press Enter. The next question is: You can configure this system hostname as either a: - Consolidation server - Client that forwards logs to a remote consolidation server Do you want to configure hostname as a Consolidation Server? (y/n) [y]: Answer yes (y).
Next, the wizard prompts for which local logs should be consolidated: Log files that reside on this system can be consolidated. Would you like to consolidate this system's syslogs? (y/n) [y]: Answering yes places this log consolidation system’s own local syslog data in the consolidated log along with the client system's syslog data. To preserve the priority and facility of syslog entries, UDP local loopback is used, and syslog is configured to also forward entries to its local UDP port 514.
Starting syslog-ng. Successfully configured hostname as a log consolidation server. 3.3.1.2 Configuring a Serviceguard Cluster as a Log Consolidation Server with clog_wizard When running the clog_wizard (see clog_wizard(1M)) in a cluster, first make sure that all the cluster members are up and available. The wizard needs to perform configuration operations on each member. It only needs to be run once, from any member of the cluster. If you run the wizard more than once, additional prompts may appear.
proceeding. - An IP address and subnet address pair for the package. IPv4 or IPv6 addresses can be used. The IP address should be registered in DNS, if this cluster will consolidate logs from remote clients. This should be appropriately configured on each cluster member before proceeding with the consolidation server configuration. Answer yes (y). In a cluster, the wizard configures syslog-ng to be highly available using a Serviceguard package. For consolidated logging, the package name is clog.
Would you like to consolidate this cluster's syslogs? (y/n) [y]: Would you like to consolidate this cluster's package logs? (y/n) [y]: In a Serviceguard cluster, you can consolidate all the member-specific syslog files into a single consolidated syslog containing syslog.log, mail.log and syslog-ng.log. Each member-specific package log can also be consolidated.
Creating /etc/rc.config.d/syslog-ng, the log consolidation configuration file. Updating the syslog configuration: Updating the /etc/rc.config.d/syslogd file to add -N to SYSLOGD_OPTS. This stops syslogd from listening to UDP port 514. Updating the /etc/syslog.conf file for UDP local loopback. Starting syslogd for the configuration changes to take effect. Registering the log consolidation ports in the /etc/services file. Starting syslog-ng. Setting up the log consolidation server to be highly available.
added and deleted, the DSAU consolidated logging tools will automatically take the appropriate configuration actions. Specifically: • When adding a member to the cluster, the new member is automatically configured to participate in log consolidation according to the cluster’s configuration. The following files are automatically configured on the added member: — /etc/rc.config.d/syslog-ng — /etc/rc.config.d/syslogd — /etc/syslog.conf — /etc/syslog-ng.conf.client, /etc/syslog-ng.conf.
3.3.1.6 Configuring a Log Forwarding Client Using clog_wizard There are two ways to configure a log forwarding client: as a standalone machine or as a Serviceguard cluster. When configuring a cluster as a log forwarding client, all the members of the cluster will be configured identically as clients. The wizard asks the same questions and performs the same configuration actions for single systems and for clusters. The examples below show use of the clog wizard on a Serviceguard cluster.
Shell Authentication set up with the consolidator. You can use the tool /opt/dsau/bin/csshsetup to configure non interactive Secure Shell Authentication. Do you want to configure Secure Shell port forwarding? (y/n) [y]: Choose yes in order to use ssh port forwarding. This will encrypt all the traffic sent from this local log forwarding client to the log consolidator. NOTE: A special ssh security configuration is required on the server when a Serviceguard cluster is the log consolidation server.
Creating a symbolic link from /etc/syslog-ng.conf to the /etc/syslog-ng.conf.client configuration file. Creating /etc/rc.config.d/syslog-ng, the log consolidation configuration file. Updating the syslog configuration: Updating the /etc/rc.config.d/syslogd file to add -N to SYSLOGD_OPTS. This stops syslogd from listening to UDP port 514. Updating the /etc/syslog.conf file for UDP local loopback. Starting syslogd for the configuration changes to take effect.
where log-consolidation-server is the fully qualified domain name of the consolidation server. The name must be fully qualified or syslogd will not forward the messages properly. NOTE: There must be a before each @ sign. If you have customized syslog.conf, make sure to add the forwarding lines for your customizations as well. syslogd must be stopped and restarted for these changes to take effect, using the following commands: # /sbin/init.d/syslogd stop # /sbin/init.
where the <%IP%> is replaced by the server’s IP address or local hostname and the <%PORT%> is replaced by the selected TCP port number. For UDP: destination d_syslog_udp { udp(“local_hostname” port(514)); } where <%IP%> is replaced by the server’s IP address or local hostname and the <%PORT%> token is replaced by 514, the standard syslog UDP port. • Replace the<%FS%> token with the filesystem or directory where the consolidated logs will be kept.
1. 2. 3. Run /opt/dsau/sbin/syslog-ng with the -s or --syntax-only option to verify the syntax of the /etc/syslog-ng.conf file. This should be a symbolic link to /etc/ syslog-ng.conf.server as described previously. Start syslog-ng using /sbin/init.d/syslog-ng start. If consolidating the local syslogs, use logger test-message and make sure this message is in the consolidated syslog.log. If you are not consolidating the local logs, use the logger command from a log forwarding client.
1. If you want the local syslog messages for the cluster itself to be part of the consolidated syslog, complete the following tasks: a. Start by configuring the standard syslogd to co-exist with a syslog-ng consolidator. By default, syslogd listens for incoming log messages on UDP port 514. To use the UDP protocol or consolidate this server’s local syslogs, syslog-ng must listen on UDP port 514. Edit/etc/rc.config.
a. When using the TCP protocol and configuring the consolidation server to consolidate its own syslogs, replace the <%UDP_LOOPBACK_SOURCE%> token with: source s_syslog_udp { udp(port(514)); }; Replace the <%UDP_LOOPBACK_LOG%> token with: log { source(s_syslog_udp); destination(d_syslog_tcp); }; This causes the syslog-ng consolidator to read the local syslogd’s UDP messages and send them to syslog-ng on the local TCP port.
e. Replace the <%FS%> token with the filesystem or directory where the consolidated logs will be kept. This filesystem/directory is the one managed by the Serviceguard package. For example: destination d_syslog { file(“<%FS%>/syslog/syslog.log”); }; becomes: destination d_syslog { file(“/clog/syslog/syslog.log”); }; Make sure that this filesystem mount point exists cluster-wide and that the storage fails over correctly cluster-wide.
4. The syslog-ng startup procedure, /sbin/init.d/syslog-ng, relies on several configuration variables. Edit /etc/rc.config.d/syslog-ng as follows: a. Change the CLOG_CONFIGURED line to: CLOG_CONFIGURED=1 b.
1. Find the line “VG[0]=“<%SG_PKG_VOL_GRP%>”” and replace the token with the name of the VM volume group for the package. For example: VG[0]=“vgclog” If using VxVM, comment out the LVM Volume Group line VG[0]=”<%SG_PKG_VOL_GRP%>”. Uncomment the line “VXVM_DG[0]=” and put in the VxVM Disk Group. 2. Find the line “LV[0]=“<%SG_PKG_LOG_VOL%>”” and replace the token with the full name of the logical volume. For example: LV[0]=“/dev/vgclog/lvol1” 3.
CLOG_PKG_FS=filesystem mount point where the consolidated logs are stored CLOG_PKG_MNT_OPT=file systems mount options such as -o rw,largefiles CLOG_PKG_FS_TYPE=file system type CLOG_PKG_IP=IP address of the clog package CLOG_PKG_SUBNET=subnet of the clog package’s IP address CLOG_SYSTEM_LOG_CONSOLIDATION_DIR=file system mount point/syslog CLOG_SERVICEGUARD_PACKAGE_LOG_CONSOLIDATION_DIR=file system mount point/packages CLOG_PKG_RETRY_TIMES=1 CLOG_PKG_MONITOR_INTERVAL=5 4.
5. Validate that log forwarding is working properly. If consolidating the cluster’s local syslogs, use “logger test-message” and make sure this message is in the consolidated syslog.log. If you are not consolidating local logs, use the logger command from a log forwarding client. Note that logger messages are first sent to the local syslogd, which forwards them to syslog-ng. By default, syslogd suppresses duplicate messages. If you issue multiple logger test messages, make sure each is unique.
a. If configuring the system to forward its syslogs to the consolidation server, replace the <%UDP_LOOPBACK_SOURCE%> token with: source s_syslog_udp { udp(port(514)); }; Replace the <%UDP_LOOPBACK_LOG%> token with: log { source(s_syslog_udp); destination(d_syslog_type); }; where type is either tcp or udp depending on the desired log transport. This causes syslog-ng to read the local syslogd’s UDP messages and send them to the log consolidation server.
3. The syslog-ng startup procedure, /sbin/init.d/syslog-ng, relies on several configuration variables. Edit /etc/rc.config.d/syslog-ng as follows: a. Change the CLOG_CONFIGURED line to: CLOG_CONFIGURED=1 b. Add the following lines: CLOG_CONSOLIDATOR=0 CLOG_CONS_IP=IP address of the log consolidator c.
1. If you want the syslog messages for the cluster to be forwarded to the log consolidator, do the following: a. Start by configuring the standard syslogd to co-exist with a syslog-ng forwarder. By default, syslogd listens for incoming log messages on UDP port 514. To forward this cluster’s syslogs, syslog-ng must listen on UDP port 514. Edit /etc/rc.config.d/syslogd and change SYSLOGD_OPTS to add the-N switch; this prevents syslogd from listening on port 514. For example, SYSLOGD_OPTS=“-D -N” b.
2. To configure syslog-ng, start with the same syslog-ng.conf templates used by the clog_wizard. On one cluster member, copy the /opt/dsau/share/clog/templates/ syslog-ng.conf.client.template to /etc/syslog-ng.conf.client. This file contains tokens named <%token-name%> which are replaced by the wizard based on the administrator’s answers to the wizard’s questions. Manually replace the tokens in /etc/syslog-ng.conf.client as follows: a.
3. The syslog-ng startup procedure, /sbin/init.d/syslog-ng, relies on several configuration variables. Edit /etc/rc.config.d/syslog-ng as follows: a. Change the CLOG_CONFIGURED line to: CLOG_CONFIGURED=1 b. Add the following lines: CLOG_CONSOLIDATOR=0 CLOG_CONS_IP=IP address of the log consolidator c.
7. Test the configuration by performing the following steps: a. Run /opt/dsau/sbin/syslog-ng with the -s or --syntax-only option to verify the syntax of the/etc/syslog-ng.conf file. This should be a symbolic link to /etc/ syslog-ng.conf.client as described above. b. Start syslog-ng on all cluster members using # cexec “/sbin/init.d/syslog-ng start” c. If consolidating the local syslogs, use “logger test-message” and make sure this message is in the consolidated syslog.log on the log consolidation server.
If the text file is already formatted using the syslog-compatible format shown above, then add the corresponding CLOG_TEXT_FORMAT[n]entry with a value of “syslog”. For example, CLOG_TEXT_LOG[2]=/var/adm/app/logs/app.log CLOG_TEXT_FORMAT[2]="syslog" If no CLOG_TEXT_FORMAT[]entry is made for a corresponding CLOG_TEXT_LOG[]entry, the default is “custom”. For an example of a file in syslogformat, see the actual system log file /var/adm/ syslog/syslog.log. 3.
For the log line: log { source(s_syslog_type); filter (f_node1_text1);destination(d_node1_text1); flags(final);}; where text1 is the text logfile name, node1 is the relocatable IP address (for a Serviceguard cluster) or hostname (for a non-Serviceguard cluster) that is forwarding this text log, fs is the filesystem on the log consolidator where the consolidated logs will be stored, type is the “s_source” definition, either _tcp or _udp, depending on the log transport selected, and textdir is the name of th
1. For each package that will be forwarded from a cluster client, add the following destination, filter and log lines to the syslog-ng.conf.server file, after the HP_AUTOMATED_LOG_FILE_CONSOLIDATION section. destination d__ { file(“/packages/_.log”); }; filter f__ { program(_.
#/sbin/init.d/syslogd start 4. Stop syslog-ng: # /sbin/init.d/syslog-ng stop 5. Edit the /etc/rc.config.d/syslog-ng file, as follows: a. Change the CLOG_CONFIGURED line to CLOG_CONFIGURED=0. b. Remove all other CLOG lines except for the following: CLOG_LAYOUTS_DIR=/var/opt/dsau/layouts CLOG_ADDITIONAL_LOG_DIRS[0]=/var/adm/syslog 6. If the TCP protocol was used, remove the following line from /etc/services: clog_tcp port/tcp # Consolidated logging with syslog-ng 3.4.
1. If syslog messages were being forwarded to the log consolidator, edit /etc/rc.config.d/syslogd and change SYSLOGD_OPTS to remove the -N switch. For example, SYSLOGD_OPTS=“-D” 2. Edit the systems /etc/syslog.conf file to remove the following lines: mail.debug *.info;mail.none @fully qualified hostname @fully qualified hostname where fully qualified hostname is the fully qualified hostname of this system. NOTE: 3. A precedes each @ sign.
5. Edit the /etc/rc.config.d/syslog-ng file and change the CLOG_CONFIGURED line to CLOG_CONFIGURED=0. Remove all other CLOG lines except for the following: CLOG_LAYOUTS_DIR=/var/opt/dsau/layouts CLOG_ADDITIONAL_LOG_DIRS[0]=/var/adm/syslog 6. If ssh port forwarding had been configured, remove the following line from /etc/ services: clog_ssh port/tcp # Consolidated logging with ssh port forwarding 3.
In general, using ssh port forwarding requires that the log consolidation server perform a key exchange with the log forwarding client. Specifically, the ssh public key for the remote log forwarding client must be added to the consolidation server’s authorized keys file. Also, the fingerprint for the log consolidation server is added to the log forwarding client’s /.ssh/ known_hosts file.
allowing the administrator to choose which security features to enable or disable from hardening/lockdown checklists. Bastille can be used to harden a log consolidation server by enabling security tools such as IP filtering. If IP filtering is enabled, the ports described in “clog Network Port Usage” (page 79) must not be blocked.
additional information on using the System and Consolidated Log Viewer, use the help available from within the application. 3.
4 Command Fanout Command fanout utilities allow the system administrator to replicate shell commands across multiple systems. Traditionally, administrators have created wrappers around tools such as remote shell (see remsh(1)) and secure shell (see ssh(1)) to provide command fanout functions. 4.1 Parallel Distributed Shell The Distributed Systems Administration Utilities (DSAU) include the open source tool Parallel Distributed Shell (pdsh).
Figure 4-1 pdsh Architecture For more information on pdsh and dshbak, refer to their reference manpages. 4.2 pdsh Utility Wrappers Administrators can build wrapper commands around pdsh for commands that are frequently used across multiple systems and Serviceguard clusters. Several such wrapper commands are provided with DSAU. These wrappers are Serviceguard cluster-aware and default to fanning out cluster-wide when used in a Serviceguard environment.
4.3 Security Configuration The command fanout tools support both remote shell (rsh or rcmd) and ssh transports. Each requires specific security setup steps in order to authorize the user initiating the command fanout operation to execute a command on the remote target systems. The command fanout tools require that the remote system not prompt for a password. Both rsh and ssh transports must be preconfigured on each remote system to allow non-interactive access.
4.4 Command Fanout Troubleshooting This section contains troubleshooting tips for common error messages produced by pdsh and the wrapper commands. You may see the following error messages when using command fanout: Table 4-1 ssh Command Messages Message Cause To Correct pdsh@local_hostname: target_hostname:ssh exited with exit code 1 The target system is unreachable. Ensure that the target system is up and connected.
A HP-Supported Open Source pdsh Options This release of DSAU includes open source pdsh code that was compiled with the following options: • readline support. • The secure shell (ssh) remote command module (pdsh —R ssh). • The remote shell (rsh) remote command module (pdsh —R rsh). Because of security concerns, this option is disabled by default at installation time. For information on Parallel Distributed Shell, see the Chapter 4 (page 83) chapter. • The machines module (pdsh —a option).
Index A adoptive node, 25, 52 alert checksum, 38 authentication errors, 39 B backup file, 22 Bastille, 79 Berkeley commands, 85 C can’t stat messages, 39 ccp, 33 central management server, 15 cf.main, 18, 36 cfagent, 14, 15, 36 cfagent.conf, 15, 27, 30 CFANOUT_HOSTS environment variable, 84 cfengine, 13, 15, 22, 25 configuring, 16 disabling, 38 error, 40 security keys, 26 security setup, 39 troubleshooting, 39 cfengine_master, 15 cfexecd, 36 cfrun, 16, 36 cfrun command, 13 cfrun.
K key exchange, 37 integrity, 35 public/private, 23, 32 security, 26, 35 known host fingerprint, 79 L lockdown tool Bastille, 79 log consolidation, 52 configuration, 45 disabling, 75 overview, 42 log forwarding, 52 client, 54 disabling, 76 log traffic intra-cluster, 78 logs package, 71 loss message, 43 LVM storage configuration, 20 volume groups, 21 M managed client, 25, 35 marker tokens, 63 master policy files, 35 server, 25, 28 update.
message format, 41 syslog-ng, 43 consolidator, 56 syslogd, 41, 43 System Management Homepage, 80 Systems Insight Manager, 15 T TCP port, 37 port errors, 40 protocol, 54 transport, 78 TCP port, 47 timestamped subdirectories, 22 tokens marker, 63 transport TCP or UDP, 78 troubleshooting cfengine, 39 trust relationships, 85 U UDP port, 48, 56, 66 protocol, 54 unconfigure, 38 update.
