Using Distributed Systems Administration Utilities

cfengine employs several daemons and commands to perform configuration synchronization operations. The following list describes the primary cfengine components.

Figure 3-1 “cfengine Overview” illustrates the relationship of the cfengine commands and daemons, and shows an example of the administrator using cfrun. The dashed lines in the diagram indicate calling sequences (for example, A calls B). Solid lines indicate that data is being read from configuration files.

Figure 3-1 cfengine Overview

To configure a synchronization server for a standalone system, run the csync_wizard(1) on the standalone system you wish to configure as the master synchronization server:

# /opt/dsau/sbin/csync_wizard

The wizard displays the following introductory screen:

Querying the system <local hostname> for current status, one moment...
 
This Configuration Synchronization Wizard helps you set up the Configuration
Engine (cfengine) environment. Cfengine is a powerful tool for performing
policy-based management for groups of systems and cluster environments.
 
It is a client/server based utility. A standalone system or Serviceguard 
cluster can be configured as the cfengine ‘master’. The master contains 
the configuration description and configuration files that will be used by 
all the clients. Clients copy the configuration description from the master 
and apply it to themselves. The configuration description supports a rich set 
of management actions such as copying configuration files from the master to
the client, performing edits to files, checking file ownerships, permissions, 
and checksums, executing shell commands, checking for processes, etc. 
 
For a detailed description of the cfengine management actions, please refer to
cfengine man page.
 
This wizard will help you set up this system as a cfengine master or to add or
remove a cfengine client, and to perform the required security setup.
 
Press ‘Return’ to continue...

Press return to continue and choose item 1 from the menu below to configure a master server:

 Configuration Synchronization Wizard Menu
 =========================================
 
 (1) Set up a cfengine master server
 
 (2) Add a client
 
 (3) Remove a client
 
 (4) Key management for cfengine users
 
 (9) Exit
 
Enter choice: 1
 
The cfengine master server is being configured on:
 
<local hostname>

The wizard then asks if you would like to additionally configure managed clients immediately after configuring the server. For this example, answer no. Managed clients will be added later.

You can optionally specify additional remote clients to manage at this time. 
If you are running in an HA environment, you do not need to specify the 
cluster members.
 
Would you like to manage clients? [N]: n

The wizard proceeds to configure the system as a master server:

******* WARNING!!!! ******** 
 
To protect against possible corruption of sensitive configuration files, 
control-c has been disabled for the remainder of this configuration.
 
Configuration of the cfengine master server is starting.
 
Verifying the master has an entry in the /etc/hosts file on each client...
 
Keys are being created...
 
Keys have been created, now distributing....
 
Starting cfengine on the master and any pertinent client machines. This may 
take a few minutes....

When the configuration is completed, the wizard displays the following summary screens which direct the administrator to the main policy file, /var/opt/dsau/cfengine_master/inputs/cf.main, and the recorded answer file for this run of the wizard.

The Configuration Synchronization Wizard has completed the configuration of cfengine:
 
- The master configuration description template is here:
 
</var/opt/dsau/cfengine_master/inputs/cf.main>
 
This default template has examples of typical configuration synchronization 
actions performed in cfengine. For example, synchronizing critical files such
as /etc/hosts, package scripts, etc.
 
All the actions in the template are disabled by default (commented out).
Uncomment the lines corresponding to the desired synchronization actions for 
this configuration. See the cfengine reference documentation for a description 
of additional cfengine features: /opt/dsau/doc/cfengine/
 
Press ‘Return’ to continue...
 
The cfengine environment has been created with Master Host: <local hostname> 
and members:

Note that when configuring a master server but not adding any managed clients during the server configuration, the members (list of managed clients), will be empty as shown in the above example.

A file recording the answers for this run of the Configuration Synchronization
Wizard is stored here...
 
/var/opt/dsau/cfengine/tmpdir/csync_wizard_input.txt
 
This configuration can be reestablished by issuing the command:
 
/opt/dsau/sbin/csync_wizard \ 
-f /var/opt/dsau/cfengine/tmpdir/csync_wizard_input.txt

To configure a synchronization server for a Serviceguard cluster, there are two configuration choices:

This section will focus on using the wizard to configure a highly available configuration synchronization service.

	NOTE: Make sure that this cluster’s MAX_CONFIGURED_PACKAGES value can accommodate the additional package. For more information on this setting, please refer to the Managing Serviceguard manual that is part of the Serviceguard documentation set. Also, before starting the wizard, all the current cluster members should up and running in the cluster.

Start by running the Configuration Synchronization Wizard, csync_wizard(1):

# /opt/dsau/sbin/csync_wizard
 
Querying the system <local hostname> for current status, one moment...
 
This Configuration Synchronization Wizard helps you set up the Configuration
Engine (cfengine) environment. Cfengine is a powerful tool for performing
policy-based management for groups of systems and cluster environments.
 
It is a client/server based utility. A standalone system or Serviceguard cluster
can be configured as the cfengine ‘master’. The master contains the configuration
description and configuration files that will be used by all the clients. Clients
copy the configuration description from the master and apply it to themselves.
The configuration description supports a rich set of management actions such as
copying configuration files from the master to the client, performing edits to
files, checking file ownerships, permissions, and checksums, executing shell
commands, checking for processes, etc.
 
For a detailed description of the cfengine management actions, please refer to
cfengine man page.
 
This wizard will help you set up this system as a cfengine master or to add or
remove a cfengine client, and to perform the required security setup.
Press ‘Return’ to continue...

Press return to continue and choose item 1 from the menu below to configure a master server:

 Configuration Synchronization Wizard Menu
=========================================
 
(1) Set up a cfengine master server
 
(2) Add a client
 
(3) Remove a client
 
(4) Key management for cfengine users
 
(9) Exit
 
Enter choice: 1

After choosing 1 and pressing return, the wizard displays the following text:

This system is a member of a Serviceguard cluster. The cfengine configuration
will be defined as a package for high availability unless you answer no to the
below question. If you answer no, for the purposes of cfengine control, this
machine will be treated as a single machine without failover capability for
cfengine.
 
If you accept the default answer of ‘HA’ to the below question, cfengine will be
configured as a highly available Serviceguard package. This will ensure that your
cfengine master server is available as long as one of the cluster members that
can run the package is also available.
 
You will need a free IP address for this package and you need to configure
storage for the package before proceeding. For details on creating highly
available file systems, please refer to ‘Creating a Storage Infrastructure’
chapters of the Managing Serviceguard documentation.
 
Will this master server be Highly Available (HA) [Y]:

The wizard names the package name “csync” for configuration synchronization. This specific package name is required. The LVM storage configuration and network configuration for the package must be set up before continuing or before running the wizard. All the cluster members should also be up and available. For details, refer to the Managing Serviceguard book, under “Building an HA Cluster Configuration”, “Creating a Storage Infrastructure with LVM.”

	NOTE: The wizard only supports creating packages based on LVM volume groups. When using CFS or VxVM, manual configuration is required. See the section on manually configuring a Serviceguard cluster for details on manually configuring the csync package.

The wizard prompts for the following, all of which should have already been configured:

Once the storage infrastructure is configured and the IP address obtained, press return to access the default answer of ‘yes’ and proceed with creating the package. The wizard now prompts for the package information:

Configuring the csync Serviceguard package for a highly available cfengine
master.
 
The cfengine master server is being configured as a HA Serviceguard Package on
this cluster.
 
Please provide the following information for the package:
 
Enter the Volume group? []: vgcsync
 
You need to enter a fully qualified Logical Volume, for example:
/dev/vgname/lvol1 
 
Enter the fully qualified Logical Volume? []:
/dev/vgcsync/lvol1
 
Enter the Filesystem (Mount Point)? []: /csync
 
Enter the Mount Options? [-o rw,largefiles]:<return>
 
Enter the Filesystem Type? [vxfs]: <return>
 
Enter the IP address? []: 12.345.6.78
 
Enter the Subnet? []: 12.345.7.8

The wizard now asks if you would like to manage additional remote clients, that is, systems outside the cluster. The wizard automatically configures the current members of the cluster. This is why all members must be up and accessible when running the wizard. In the example shown below, the administrator chose ‘no’ so only cluster members are configured as clients initially.

Note that additional remote clients can easily be added later using the wizard. It is not necessary to use the wizard to add new clients when additional members are added to the cluster. Refer to the section on Serviceguard Automation features for details.

You can optionally specify additional remote clients to manage at this time. If
you are running in an HA environment, you do not need to specify the cluster
members.
 
Would you like to manage clients? [N]: <return>

The wizard now has all the data it needs to configure the cluster and proceeds to do so:

******* WARNING!!!! ******** 
 
To protect against possible corruption of sensitive configuration files,
control-c has been disabled for the remainder of this configuration.
 
Configuring the ‘csync’ Serviceguard package.
 
Applying the ‘csync’ Serviceguard package configuration file, this will take a
moment.
 
Starting the ‘csync’ Serviceguard package, this will take a few moments...
 
The ‘csync’ Serviceguard package has been started on <local hostname>.
 
Configuration of the cfengine master server is starting.
 
Verifying the master has an entry in the /etc/hosts file on each client...
 
Keys are being created...
 
Keys have been created, now distributing....
 
Starting cfengine on the master and any pertinent client machines.
This may take a few minutes....

When the configuration is done, the wizard displays the following summary screens which direct the administrator to the main policy file, mount_point/cfengine_master/inputs/cf.main, and the recorded answer file for this run of the wizard. Note that the policy file is located on the newly configured filesystem associated with the package. In our example, the administrator chose to mount the filesystem for the package as /csync.If the administrator had previously configured cfengine, before overwriting any existing configuration files, the wizard creates backups in the directory:

/var/opt/dsau/cfengine/backups

The top level files in this directory are the most recent backup files. Any configurations before that are saved in timestamped subdirectories named v_timestamp.

The Configuration Synchronization Wizard has completed the configuration of
cfengine:
 
- The master configuration description template is here:
 
</csync/dsau/cfengine_master/inputs/cf.main>
 
This default template has examples of typical configuration synchronization
actions performed in a cluster. For example, synchronizing critical files such as
/etc/hosts, package scripts, etc.
 
All the actions in the template are disabled by default (commented out).
Uncomment the lines corresponding to the desired synchronization actions for this
cluster. See the cfengine reference documentation for a description of additional
cfengine features: /opt/dsau/doc/cfengine/
 
Press ‘Return’ to continue...
 
The cfengine environment has been created with Master Host: <package hostname>
 
and members: <cluster member1>, <cluster member2>, ...
 
A file recording the answers for this run of the Configuration Synchronization
Wizard is stored here...
 
/var/opt/dsau/cfengine/tmpdir/csync_wizard_input.txt
 
This configuration can be reestablished by issuing the command:
 
/opt/dsau/sbin/csync_wizard \ 
-f /var/opt/dsau/cfengine/tmpdir/csync_wizard_input.txt

This section describes the details of a high availability configuration of cfengine in a Serviceguard cluster. For more information on the role of the various cfengine daemons and commands, refer to“cfengine Daemons and Commands”. The Serviceguard package ensures that cfengine’s cfservd daemon remains highly available. The cfengine configuration files update.conf and cfagent.conf define the master configuration synchronization server to be the registered DNS name for the relocatable IP address of the package. When managed clients run cfagent (see cfagent(8)), cfagent connects to cfservd on the package’s adoptive node. Thus the cluster members themselves are all managed clients. The member hosting the package additionally acts as the master server for the policy files.

When booting the cluster, each member will start a client cfservd. This is the cfservd that responds to cfrun requests. When the package starts on a member, that cfservd now has access to the filesystem of the package and becomes the master cfservd that serves the policy files to all managed clients. This cfservd is monitored by the package. If cfservd fails, the package will attempt to restart on another member. That member’s cfservd will then become the master cfservd.

Note that halting the package does not stop the cfservd daemon on the adoptive member since the expectation is that the daemon is present to respond to future cfrun requests. Also, unlike some other HA services, if the csync package is down or unavailable, remote clients are not adversely impacted. The clients continue to run with their currently defined configurations. The administrator would need to make sure the package is up and running in order to distribute any new configuration instructions to the managed clients.

The wizard automates cfengine key distribution to all cluster members. For a detailed description of key distribution steps performed, refer to “Security Notes”.

The Distributed Systems Administration Utilities require Serviceguard 11.17 or later. With Serviceguard 11.17 or later, when members are added to or deleted from the cluster the configuration synchronization tools will automatically take the appropriate configuration actions. Specifically:

Note that the administrator can define cfengine groups or classes that enumerate all the members of a particular Serviceguard cluster. These class definitions will not be updated automatically and the administrator must manually update the cfagent.conf and related files for cluster membership changes.

You can use the Configuration Synchronization Wizard to add managed clients to an existing cfengine configuration. Run the wizard on the master server, not the client system. When a Serviceguard cluster is the master server, run the wizard on the adoptive node for the csync package. Note that adding a new cluster member in a High Availability setup as a client is done automatically. For more information, see the “Serviceguard Automation Features”.

Also, in order to securely distribute cfengine keys, the client must be configured for non-interactive ssh access by the root account of the master server. The csshsetup tool (see csshsetup(1)) makes it easy to configure ssh access to a remote system. That tool is used in the examples below.

The wizard can currently only manage clients when the clients are in the same DNS domain as the master server. For multi-domain configurations, refer to “Manual Configuration ” for instructions on adding clients manually.

Note that if adding a Serviceguard cluster as a managed client, each cluster member must be added individually.

Start by logging in as root on the master server and configure ssh access to the remote system:

# csshsetup <hostname of managed client>

csshsetup first tests ssh access to the remote system. If it is not configured, ssh prompts for the managed client’s password.

Run the Configuration Synchronization Wizard and choose option 2 to add a new client:

 Configuration Synchronization Wizard Menu
 =========================================
 
 (1) Set up a cfengine master server
 
 (2) Add a client
 
 (3) Remove a client
 
 (4) Key management for cfengine users
 
 (9) Exit
 
Enter choice: 2

When prompted, enter the name of the client to add:

This option will configure additional clients to the cfengine domain.
 
Enter the name of the client to add: <new client>

The wizard then proceeds to configure the client and report on its progress:

Verifying the master has an entry in the /etc/hosts file on each client...
 
Keys are being created...
 
Keys have been created, now distributing....
 
The client <new client> has been added to the cfengine domain

The wizard configures each new client to run cfservd so it can respond to cfrun requests and adds the client to master’s cfrun.hosts file.

Perform the following one-time steps to configure a standalone system as a cfengine master server:

Configuring cfengine for high availability in a Serviceguard cluster is similar to configuring it for a standalone machine, which is described in section “Using the Wizard to Configure a Standalone Synchronization Server”. The primary differences are the creation of the Serviceguard package and the mechanism used to distribute cfengine’s security keys. Follow all the steps described below.

When manually configuring managed clients, the basic steps are:

To configure a Serviceguard cluster as a client of an existing cfengine master server, then each cluster member is treated as if it were a standalone system and configured individually. If you add new members to the cluster, each must be configured appropriately.If you are adding a new member to a Serviceguard cluster and that cluster is running the csync package, then the members of the cluster are automatically configured as cfengine-managed clients. In this scenario, the csync package acts as the master server. New members will be automatically configured as a managed clients and also able to handle failover for the package. The package’s storage infrastructure and filesystem should be configured before adding the member to the cluster. To add a new managed client, start by configuring the trust relationship between the client and the master server. The two systems exchange security keys to authenticate each other. The master server’s public key needs to be copied to the client and the client’s public key is copied to the master server:

To allow this client to accept cfrun requests, do the following:

For additional troubleshooting information, refer to the section “cfengine Troubleshooting”.

As the administrator, you can push changes out to managed clients by using the cfrun command (see cfrun(8)). cfrun contacts the cfservd daemon on each managed client and cfservd invokes cfagent does the actual synchronization work. You can also choose to have cfagent run at intervals on the client. There are two approaches:

For a standalone system, the wizard first displays introductory paragraphs explaining log consolidation and then asks:

Do you want to configure log consolidation? (y/n) [y]:

Answer yes or simply press Enter. The next question is:

You can configure this system hostname as either:

- Consolidation server

- Client that forwards logs to a remote consolidation server

Do you want to configure hostname as a Consolidation Server?
(y/n) [y]:

Answer yes. The wizard then prompts:

Enter the fully qualified directory where the consolidated
logs should be stored? []:

It is typically best to select a dedicated filesystem for the consolidated logs. Since consolidated logs like syslog can grow rapidly, HP also recommends that the filesystem be configured for “largefiles.” For this example, assume that a filesystem named “/clog” is used.

Next, the wizard prompts for the client’s transport:

You can choose to have the clients forward logs to this
consolidation server via the UDP protocol or the TCP
protocol (recommended).

Do you want to use the TCP protocol? (y/n) [y]:

Note that selecting TCP does not necessarily preclude the use of UDP forwarded log messages by clients. Whether the log consolidator allows TCP log messages exclusively, depends on whether the system is consolidating its own local syslog file. See below for details.

You need to choose a free port on this system for log
receiving.

Note: When configuring log consolidation on the clients,
this port will need to be specified.

Enter the TCP port to be used for log receiving? []:

There is no reserved port for the TCP transport of syslog-ng. Any port that is not in use can be chosen. HP recommends that the administrator choose a port from the reserved range, that is, ports below 1024. Only privileged processes on a remote system can connect to privileged ports. Note that this provides only a weak security guarantee because it implies that the administrator trusts the remote system. See the ssh section in the log forwarding client section for establishing stronger security guarantees.

The /etc/services file documents the well-known reserved ports. When choosing a reserved port, the wizard will check both /etc/services and use “netstat -an” to check that the port is not in use.

Note that syslogd uses UDP port 514. TCP port 514 is reserved for use by remsh. remsh is not a secure protocol and is disabled at many sites. If remsh has been disabled on the consolidator, you could use TCP port 514. This has the advantage that it is a privileged port and it is the same as the UDP port number so it is easy to remember and manage. However, if the administrator changes the system to re-enable the use of remsh, syslog-ng would have to be reconfigured to use a new port and all the client systems that forward to this system would have to be updated.

Unlike UDP, TCP is a connection-oriented protocol. Each log forwarding client using TCP will have a connection to the log consolidation server. In order to avoid denial of service attacks, the default number of TCP connections accepted by syslog-ng is limited to 10 connections. For larger numbers of clients, edit the consolidation server’s /etc/syslog-ng.conf.server file. Find the TCP source line in the file:

source s_syslog_tcp { tcp(port(<TCP port>)
keep-alive(yes));};

and add a max-connections() attribute as follows:

source s_syslog_tcp { tcp(port(<TCP port>) keep-alive(yes)
max-connections(N)); };

where N is the expected number of clients.

Next, the wizard prompts for which local logs should be consolidated:

Log files that reside on this system can be consolidated.

Would you like to consolidate this system's syslogs? (y/n) [y]:

Answering yes places this log consolidation system’s own local syslog data in the consolidated log along with the client system syslog data. To preserve the priority and facility of syslog entries, UDP local loopback is used, and syslog is configured to also forward entries to its local UDP port 514. syslog-ng is configured to read from this port. Thus, consolidating this system’s syslogs allows clients to also connect to this log consolidation server via UDP port 514, even if TCP transport is specified earlier. If you choose not to consolidate this system’s syslogs, then choosing a TCP transport earlier will require that all log forwarding clients be configured to use the TCP transport.The wizard displays a summary of all the configuration choices made by the administrator:

Summary of Log Consolidation Configuration:

       You have chosen to configure hostname as a Log
       Consolidation Server.
       Logs will be forwarded from the remote consolidation
       clients to local port 1776 using the TCP protocol.

       The consolidated logs will be stored under directory:
               /clog

       The following logs from the local system will be
       consolidated:
              Syslog

If these choices are correct, continue:

Do you want to continue? (y/n) [y]: y

The wizard displays its progress by describing which files are being modified. For a complete description of the modified files, please refer to “Manually Configuring Log Consolidation”.

Copying files that will be modified by the wizard to
/var/opt/dsau/root_tmp/clog. These files will be used to
restore the system to its current log consolidation
configuration, in the event of a failure.
 
Configuring hostname as a log consolidation server.
 
Creating the /etc/syslog-ng.conf.server configuration
file.
 
Creating a symbolic link from /etc/syslog-ng.conf to the 
/etc/syslog-ng.conf.server configuration file.
Creating /etc/rc.config.d/syslog-ng, the log
consolidation configuration file.
 
Updating the syslog configuration:
       Updating the /etc/rc.config.d/syslogd file to add
       -N SYSLOGD_OPTS. This stops syslogd from
       listening to UDP port 514.
 
       Updating the /etc/syslog.conf file for UDP local
       loopback.
 
       Starting syslogd for the configuration changes to
       take effect.
 
Registering the log consolidation ports in the
/etc/services file.
 
Starting syslog-ng.
 
Successfully configured hostname as a log consolidation
server.

When running the clog_wizard (see clog_wizard(1M)) in a cluster, first make sure that all the cluster members are up and available. The wizard needs to perform configuration operations on each member. It only needs to be run once, from any member of the cluster.

The wizard will set up and create a Serviceguard package for the consolidated logging service. Make sure that this cluster’s MAX_CONFIGURED_PACKAGES value can accommodate the additional package. For more information on this setting, please refer to the Managing Serviceguard manual which is part of the Serviceguard documentation set.

The wizard first displays introductory paragraphs explaining log consolidation:

Log file consolidation allows you combine the log entries from
multiple remote systems into a single file. The standard use of
this feature is to consolidate syslog data from several
systems. Each remote system continues to write entries to its
local syslog.log and additionally forwards the entries to the
consolidating host. The system's forwarding log entries are
consolidation clients. The system they are sending the entries
to is the consolidation server. In addition to syslog data,
arbitrary textual log files can also be consolidated.
 
In a Serviceguard cluster, this tool can help you automate package
log file consolidation. Log consolidation is especially useful in a
Serviceguard cluster, since it allows you to look at a single
consolidated file instead of the per-member logs. This tool needs
to be run only once in the cluster and not on each cluster member.
All cluster members should be up when running this wizard.
 
clog_wizard will prompt you for information to configure log
file consolidation. Some of these questions have a default
answer contained within square brackets. If you press Return,
the default answer is assumed.
 
Do you want to configure log consolidation? (y/n) [y]:

Answer yes or simply press Enter. The next question is

You can configure this cluster clustername as either:
- Consolidation server
- Client that forwards logs to a remote consolidation server
 
Do you want to configure clustername as a Consolidation Server?  (y/n) [y]:

Answer yes.

In a cluster, the wizard configures syslog-ng to be highly available using a Serviceguard package. The package name is “clog” for consolidated logging. The LVM storage configuration and network configuration for the package must be set up before continuing or before running the wizard. For additional details, refer to Managing Serviceguard, in Chapter 5, “Building an HA Cluster Configuration,” in the section “Creating a Storage Infrastructure with LVM.”

	NOTE: The wizard only supports creating packages based on LVM volume groups. When using CFS or VxVM, manual configuration is required. See the section “Manually Configuring Log Consolidation” for details.

The wizard prompts for the following, all of which should have already been configured:

Next, the wizard prompts for the client’s transport.

You can choose to have the clients forward logs to this
consolidation server via the UDP protocol or the TCP protocol
(recommended).
 
Do you want to use the TCP protocol? (y/n) [y]:

Note that selecting TCP does not necessarily preclude the use of UDP forwarded log messages by clients. Whether the log consolidator allows TCP log messages exclusively, depends on whether the system is consolidating its own local syslog file. See below for details.

When answering Yes to TCP, you must select a free TCP port. This port must be free on all cluster members. See the section “Configuring a Log Forwarding Client Using clog_wizard” using the clog_wizard for details on choosing a TCP port.

Next the wizard prompts for which local logs should be consolidated:

Log files that reside on this cluster can be consolidated.
 
Would you like to consolidate this cluster's syslogs? (y/n) [y]:
 
Would you like to consolidate this cluster's package logs? (y/n) [y]:

In a Serviceguard cluster, you can consolidate all the member-specific syslog files into a single consolidated syslog containing syslog.log, mail.log and syslog-ng.log. Each member-specific package log can also be consolidated.

Note that if you choose to consolidate the cluster’s syslogs, the remote clients can also forward UDP syslog messages to the cluster, regardless of the answer to the “Do you want to use the TCP protocol” question. If you choose not to consolidate this cluster’s syslogs, then choosing a TCP transport earlier will require that all log forwarding clients be configured to use the TCP transport.

The consolidated logs are placed in the filesystem associated with the package. Using the example above, the consolidated syslog.log would be located here:

/clog/syslog/syslog.log,mail.log,syslog-ng.log

The consolidated packages logs would be located here:

/clog/package/package1.log, package2.log, ..., packageN.log

The wizard now has all the data it needs to configure the consolidated logging package. It displays a summary confirmation screen and then performs the configuration:

Summary of Log Consolidation Configuration:
       You have chosen to configure clustername as a Log Consolidation
       Server.
       Logs will be forwarded from the remote consolidation clients to local 
       port port-number using the TCP protocol.
 
       For high availability the Serviceguard package "clog" will be 
       configured with the following attributes:
             Volume Group:     vgclog
             Logical Volume:   /dev/vgclog/lvol1
             Filesystem:       /clog
             Mount Options:    -o rw,largefiles
             Filesystem Type:  vxfs
             IP Address:       123.456.789.10
             Subnet:           123.456.788.0
 
       The following logs on this cluster will be consolidated:
             Syslog
             Serviceguard package logs Do you want to continue? (y/n) [y]:
 
Do you want to continue? (y/n) [y]:
 
Copying files that will be modified by the wizard to /var/opt/dsau/root_tmp/clog on each cluster node. These files will be used to restore the cluster to its current log consolidation configuration, in the event of a failure.
 
Configuring clustername as a log consolidation server.
 
The configuration will be done on all cluster nodes.
It will take a few minutes....
 
Creating the /etc/syslog-ng.conf.server configuration file.
 
Creating the /etc/syslog-ng.conf.client configuration file.
 
Creating a symbolic link from /etc/syslog-ng.conf to the
/etc/syslog-ng.conf.client configuration file.
 
Halting the "clog" Serviceguard package if it is up.
 
Creating /etc/rc.config.d/syslog-ng, the log consolidation configuration file.
Updating the syslog configuration:
       Updating the /etc/rc.config.d/syslogd file to add -N to SYSLOGD_OPTS.
       This stops syslogd from listening to UDP port 514.
 
       Updating the /etc/syslog.conf file for UDP local loopback.
 
       Starting syslogd for the configuration changes to take effect.
 
Registering the log consolidation ports in the /etc/services file.
 
Starting syslog-ng.
 
Setting up the log consolidation server to be highly available.
 
Configuring the "clog" Serviceguard package.
 
Applying the "clog" Serviceguard package configuration file, this will take a moment.
 
Starting the "clog" Serviceguard package, this will take a few moments...
 
The "clog" Serviceguard package has been started on cluster-member.
 
Successfully created the "clog" Serviceguard package.
 
Successfully configured clustername as a log consolidation server.

In a Serviceguard cluster, the adoptive node for the clog package performs the log consolidation functions. All the other cluster members participate as log forwarding clients and send log messages to the relocatable IP address of the clog package. DSAU maintains two configuration files that control whether the instance of syslog-ng on a particular cluster member operates as a consolidation server or a log forwarding client: /etc/syslog-ng.conf.server and /etc/syslog-ng.conf.client. The symbolic link /etc/syslog-ng.conf points to one of the configuration files. When the cluster is booted, all the members start as log forwarding clients with syslog-ng running on each member. The /sbin/init.d/syslog-ng startup script sets the symbolic link /etc/syslog-ng.conf to point to /etc/syslog-ng.conf.client. When the clog package starts, the adoptive node restarts that instance of syslog-ng as a log consolidation server instance. The package script resets the /etc/syslog-ng.conf symbolic link to point to /etc/syslog-ng.conf.server. If the clog package is halted, the symlink is changed to point to /etc/syslog-ng.conf.client and the syslog-ng instance on that member restarted. Note that when a cluster is a log consolidation server, and the package is down, no log consolidation occurs and forwarded log messages are lost.

Note that cluster members can forward log messages to the consolidator using either UDP or TCP. Within a Serviceguard cluster, ssh port forwarding is not used. ssh port forwarding can be used to secure the log traffic forwarded by remote clients outside the cluster. For additional information, refer to “ssh Port Forwarding”.

The Distributed Systems Administration Utilities require Serviceguard 11.17 or later. With Serviceguard 11.17 or later, when members are added to or deleted from cluster or packages are added and deleted, the DSAU consolidated logging tools will automatically take the appropriate configuration actions. Specifically:

When there is a failure on the adoptive node, it takes a finite amount of time for the clog package to fail over to another cluster member. The longer this failover time, the more likely that messages could be lost from the consolidated log. Use the following guidelines to minimize message loss during failover.

There are two ways to configure a log forwarding client: as a standalone machine or as a Serviceguard cluster. When configuring a cluster as a log forwarding client, all the members of the cluster will be configured identically as clients. The wizard asks the same questions and performs the same configuration actions for single systems and for clusters. The examples below show use of the clog wizard on a Serviceguard cluster.After starting clog_wizard, answer “yes” to the following question:

Do you want to configure log consolidation? (y/n) [y]:

or simply press Enter. The next question is:

You can configure this cluster clustername as either:
- Consolidation server
- Client that forwards logs to a remote consolidation server
 
Do you want to configure clustername as a Consolidation Server?  (y/n) [y]: n

Answer “No” here. At this point you are configuring a log forwarding client. The wizard displays the following:

You now need to specify what system will be the
consolidator. If the consolidator is a Serviceguard
cluster, you should specify the IP address of the "clog"
Serviceguard package for this question. The "clog"
package is used to make log consolidation highly
available on the consolidator.
 
The consolidation server must already be configured.
 
Enter the hostname or IP address of the consolidator?
[]: clog.usa.xyz.com

After entering the hostname or IP address of the log consolidation server, the wizard asks if you want to use the TCP transport when forwarding log messages:

You can choose to forward logs to the consolidator via
the UDP protocol or the TCP protocol (recommended).
 
Do you want to use the TCP protocol? (y/n) [y]:

Standard syslogd forwards messages using the UDP protocol. UDP is a high-performance, broadcast-oriented protocol with no flow control or message delivery verification. syslog-ng supports syslogd’s UDP protocol and a TCP protocol. The TCP transport offers both flow control and message delivery checks. However, since TCP is a connection-oriented protocol, it requires additional resources on the log consolidation server. The consolidation server’s max-connections() attribute must be set according to the maximum number of expected clients. Refer to section “Configuring a Log Consolidation Standalone Server with clog_wizard” for a discussion of the max-connections() setting. If you answer “yes” to using TCP, the next question asks for the TCP port to forward messages to:

You need to find out from the administrator of the
consolidation server the TCP port that was configured
for log receiving.
 
Enter the TCP port configured on the CONSOLIDATOR for
log receiving? []: 1776

You must use the TCP port selected by the system administrator of the log consolidation server. If the clog_wizard was used to configure the server, the port number is saved in /etc/rc.config.d/syslog-ng as the variable CLOG_TCP_PORT. In this example, TCP port 1776 was used. If you answer “yes” to the TCP question, the following question is displayed:

The TCP protocol can be used in conjunction with Secure
Shell port forwarding to enhance security. Each member
of this cluster must already have non interactive Secure
Shell Authentication set up with the consolidator. You
can use the tool /opt/dsau/bin/csshsetup to configure
non interactive Secure Shell Authentication.
 
Do you want to configure Secure Shell port forwarding? (y/n) [y]:

Choose yes in order to use ssh port forwarding. This will encrypt all the traffic sent from this local log forwarding client to the log consolidator.

	NOTE: A special ssh security configuration is required on the server when a Serviceguard cluster is the log consolidation server. For details, refer to “ssh Port Forwarding”. ssh port forwarding requires an additional free TCP port on the local client system:

You need to choose a free port on this cluster for ssh port forwarding. The port chosen should be free on all cluster nodes.

Enter the ssh port to be used for port forwarding? []: 1175

The same guidelines for choosing a free syslog-ng TCP port apply to this port. For details, refer to “Configuring a Log Consolidation Standalone Server with clog_wizard”. In this example, the local port 1775 was used. For a Serviceguard cluster log forwarding client, the cluster’s syslogs and package logs can be forwarded to the log consolidation server. For a standalone system, the wizard asks only about forwarding syslog messages:

Log files that reside on this cluster can be forwarded to the consolidator.
 
Would you like to forward this cluster's syslogs? (y/n) [y]:
 
Would you like to forward this cluster's package logs? (y/n) [y]:

Note that when forwarding a cluster’s package logs, manual configuration is required on the consolidation server in order to add the syslog-ng filtering lines to cause these package logs to be consolidated into their own unique files. See “Manually Configuring Log Forwarding Clients” for details.

After all the questions have been answered, the clog_wizard displays the following summary screen:

Summary of Log Consolidation Configuration:
 
       You have chosen to configure clustername as a Log Consolidation Client.
       Logs will be forwarded to the remote consolidation server
       clog.usa.xyz.com on port 1776 using the TCP protocol.
 
       The TCP protocol will be used in conjunction with Secure Shell 
       Port Forwarding using port 1775, for added security.
 
       The following logs will be forwarded for consolidation:
            Syslog
            Serviceguard package logs
 
Do you want to continue? (y/n) [y]:

Confirm your answers with a “yes” response and the wizard summarizes the configuration steps that it performs:

Copying files that will be modified by the wizard to /var/opt/dsau/root_tmp/clog on each cluster node.
These files will be used to restore the cluster to itscurrent log consolidation configuration, in the eventof a failure.
 
Configuring clustername as a log consolidation client.
 
The configuration will be done on all cluster nodes.
It will take a few minutes....
 
Creating the /etc/syslog-ng.conf.client configuration file.
 
Creating a symbolic link from /etc/syslog-ng.conf to the /etc/syslog-ng.conf.client configuration file.
 
Creating /etc/rc.config.d/syslog-ng, the log consolidation configuration file.
Updating the syslog configuration:
       Updating the /etc/rc.config.d/syslogd file to add -N to SYSLOGD_OPTS.
       This stops syslogd from listening to UDP port 514.
 
       Updating the /etc/syslog.conf file for UDP local loopback.
 
       Starting syslogd for the configuration changes to take effect.
 
Registering the log consolidation ports in the /etc/services file.
 
Starting syslog-ng.
 
Successfully configured clustername as a log consolidation client.

For additional information on the configuration actions performed by the clog_wizard, refer to “Manually Configuring a Serviceguard Cluster as a Log Consolidation Server”.

The following sections describe the detailed steps required to configure a standalone system or a Serviceguard cluster as a log consolidation server manually.

Manually Configuring a Standalone Log Forwarding Client

Start by configuring the standard syslogd to co-exist with a syslog-ng consolidator. By default, syslogd listens for incoming log messages on UDP port 514. If you want to accept UDP syslog messages from remote clients or consolidate this server’s local syslogs, syslog-ng must listen on UDP port 514. Edit /etc/rc.config.d/syslogd and change SYSLOGD_OPTS to add the -N switch, which prevents syslogd from listening on port 514. For example:

SYSLOGD_OPTS=“-D -N”

If you want the local syslog messages from the log consolidation server itself to be part of the consolidated syslog, edit the consolidator’s /etc/syslog.conf file to forward log messages to port 514 on the local host where they will be read by syslog-ng. Using the HP-UX default /etc/syslog.conf as an example, add the following lines:

mail.debug                @<log consolidation server>
*.info;mail.none          @<log consolidation server>

where <log consolidation server> is the fully qualified domain name of the consolidation server. The name must be fully qualified or syslogd will not forward the messages properly. Note that there must be a <tab> before each @ sign.

If you have customized syslog.conf, make sure to add the forwarding lines for your customizations as well.

syslogd must be stopped and restarted for these changes to take effect, using the following commands:

# /sbin/init.d/syslogd stop

# /sbin/init.d/syslogd start

With syslogd appropriately configured, now configure syslog-ng. Start with the same syslog-ng.conf templates used by the clog_wizard. Copy /opt/dsau/share/clog/templates/syslog-ng.conf.server.template to /etc/syslog-ng.conf.server. This file has tokens named <%token-name%> that are replaced by the wizard based on the administrator’s answers to the wizard’s questions.

Replace the tokens as follows:

Manually Configuring a Serviceguard Cluster as a Log Consolidation Server

Configuring a Serviceguard cluster as a log consolidation server is similar to the steps for a single system. All cluster members must be up and accessible before proceeding.

Create the configuration files described below on every cluster member. The simplest approach is to configure one member completely and then copy each configuration file cluster-wide. The cexec and ccp tools can simplify replicating changes cluster-wide.

For a cluster configuration, syslog-ng is configured as a package so the log consolidation service is highly available. The package must be named clog and the package configuration files require the following information:

Complete IP address registration and storage/filesystem configuration before continuing. For additional information on creating the Serviceguard storage/filesystem configuration for a package, refer to the Managing Serviceguard manual.

Refer to section “Cluster Configuration Notes” for an overview of how to configure consolidated logging in a cluster.

Creating the clog Package

To create the consolidated logging or clog package, start by copying the package templates:

# mkdir /etc/cmcluster/clog
# cd /etc/cmcluster/clog
# cp /opt/dsau/share/serviceguard/templates/clog.conf.template clog.conf
# cp /opt/dsau/share/serviceguard/templates/clog.script.template clog
# chmod +x clog

Both the clog.conf package configuration file and the clog package control script need to be edited to replace marker tokens with the correct values.

For clog.conf, there is only one token to replace, <%SG_PKG_SUBNET%>. This identifies the package’s subnet. It is the same as the subnet value in the package control script. Use netstat -i to help identify the proper subnet for the package’s IP address.

For the package control script, clog, make the changes described below.

The default script template assumes you are using an LVM-based storage configuration. Replace the volume group/filesystem related tokens as appropriate for the package’s storage configuration as follows:

Now, you need to distribute the package files clusterwide. To do this, do the following steps:

Testing and Starting the clog Package

Before starting the package, test the configuration thus far:

Using VxVM Instead of LVM

The default clog package script template assumes that you are using LVM based storage. To use VxVM storage instead, you must edit the clog package script under /etc/cmcluster/clog/clog. Comment out the LVM Volume Group line “VG[0]=“xxx””, uncomment the line “VXVM_DG[0]=”, and enter the VxVM Disk Group.