Configuring Hardware Monitoring

This section describes how to configure hardware monitoring using the Hardware Monitoring Request Manager (monconfig) utility. This section adresses the following topics:

Running the Monitoring Request Manager

To run the monconfig utility, complete the following steps:

Log in as superuser.
Enter the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig

The following message is displayed in a new window:

========================================================
==========      Event Monitoring Service     ===========
==========     Monitoring Request Manager    ===========
========================================================
                                                INDICATES
EVENT MONITORING IS CURRENTLY ENABLED      <== MONITORING
                                                STATUS
========================================================
======   Monitoring Request Manager Main Menu   ========
========================================================
Select:
 (S)how current monitoring requests configured via monconfig
 (C)heck detailed monitoring status
 (L)ist descriptions of available monitor
 (A)dd a monitoring request               <== MAIN MENU
 (D)elete a monitoring request                SELECTION
 (M)odify an existing monitoring request      OPTIONS
 (E)nable Monitoring
 (K)ill (disable) monitoring
 (H)elp
 (Q)uit
Enter selection: [s]




	NOTE: By default, event monitoring is enabled when you install the OnlineDiag bundle.

Enabling Monitoring

Monitors are automatically enabled when you install the OnlineDiag bundle. This section describes how to enable monitoring if the monitors are disabled. When you enable monitoring, all monitors and their associated monitoring requests become functional.

To enable monitoring, complete the following steps:

Log in as superuser.
Enter the following command at the HP-UX prompt to run the monconfig utility:
# /etc/opt/resmon/lbin/monconfig
Enter Eat the main menu selection prompt.
Monitoring is now enabled.

Default Monitoring Requests

A set of default monitoring requests are available for each monitor. These default requests monitor and protect the hardware resources that are controlled by the monitor. Table 3-6 lists the default monitoring requests that are common to all monitors.

Table 3-7 Default Monitoring Requests for Each Monitor

Severity Levels	Notification Method
All	textlog: `/var/opt/resmon/log/event.log`
MAJOR WARNING, SERIOUS, CRITICAL	syslog
MAJOR WARNING, SERIOUS, CRITICAL	console Note: Messages are not sent to the console by default.
MAJOR WARNING, SERIOUS, CRITICAL	E-MAIL address: root




	NOTE: To gain greater control over the notification methods, you must modify or add new monitoring requests. You can add new notification methods or remove those notification requests that are not required. Creating custom monitoring requests also enables you to specify events you want to subscribe, including the severity levels.

Listing Monitor Descriptions

To view descriptions of the available monitors and the hardware resource each monitor supports, complete the following steps:

Run the monconfig utility by entering the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig
Enter Lat the main menu selection prompt.

A complete list of the available monitors and the hardware each monitor supports is displayed. Identify the name of the desired monitor and then proceed with the monitoring request task.

For a list of the specific products each monitor supports, see “Software and Hardware Requirements”. You can also view the manpage for the particular monitor, for example, disk_em(1m).

Viewing Current Monitoring Requests

Default monitoring requests are created during system startup. Before adding or modifying monitoring requests, you must examine the current monitoring requests. By examining the current requests, you can determine what additional requests are required to implement your monitoring and notification strategy.

The Show Monitoring Requests option displays all the monitoring requests including requests that are inactive. For information on viewing only the active monitoring requests, see “Checking Detailed Monitoring Status ”.

To view the current monitoring requests, complete the following steps:

Run the monconfig tool by entering the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig
Enter Sfrom the main menu selection prompt.

A list of all the current monitoring requests configured for the monitors is displayed, as follows:

============================================================
==========    Current Monitoring Requests    ===============
============================================================

EVENT MONITORING IS CURRENTLY ENABLED

The current monitor configuration is:
1) Send events generated by all monitors
   with severity >= INFORMATION to TEXTLOG / var/opt/resmon/log/event.log
2) Send events generated by all monitors
   with severity >= MAJOR WARNING to SYSLOG
3) Send events generated by all monitors
   with severity >= MAJOR WARNING to EMAIL root
Hit <enter> to continue...

Adding a Monitoring Request

Each new notification method requires a separate monitoring request. You can add monitoring requests only at the monitor level, which creates an identical request for all instances of the hardware resources supported by the monitor. You cannot add monitoring requests for a specific hardware instance.

To add a monitoring request, complete the following steps:

Run the monconfig utility by entering the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig
Enter A at the main menu selection prompt.
The numbers for the monitors are displayed.

Enter the number assigned to the monitor at the Monitors selection prompt. You can enter multiple numbers separated by a comma, or you can enter a to create a request for all monitors.




	NOTE: Creating a request for all the monitors enables you to apply the request to a new class of supported hardware resource that you add to your system. This ensures that the new hardware is automatically included in your monitoring strategy.

Enter the number of the preferred severity level at the Criteria Threshold prompt.
Enter the number of the preferred operator at the Criteria Operator prompt.
Enter the number of the preferred notification method at the Notification Method prompt.
Enter any comments about the monitoring request at the User Comment prompt. This text is sent with events that match this monitoring request.
Enter one of the following at the Client Configuration File prompt:
- (C)lear — to use the default Client Configuration File
- A(dd) — to specify the name of a specific client configuration file for this request
  Unless you have a specific client that requires a Client Configuration File, choose (C)lear (the default).
The Client Configuration File permits you to enable or disable events, set thresholding criteria, and set severity levels for events on a client basis, for example, for HP Support Applications. Adding a Client Configuration File at the Client Configuration File prompt does not create or edit the file, but only sets up the monitoring request to use the file. For information on configuration settings, see Table 3-2.
Save the request when prompted.
Repeat steps 3 to 9 for each new monitoring request.




	NOTE: A new hardware resource added to the system inherits the same monitoring requests assigned to other hardware of the same type because monitoring requests are created at the monitor level and not at the hardware instance level. This ensures that new hardware is automatically added to the monitoring configuration. When you restart the system or execute the IOSCAN utility, the new hardware is automatically monitored. If you add a new class of supported hardware resource to your system, monitoring requests that apply to all the monitors are used for the new hardware. This ensures that your hardware is protected immediately from failure.

The following example illustrates the process of adding a monitoring request. In this example, a request which is added that sends all CRITICAL events detected by the AutoRAID Disk Array monitor to admin@hp.com.

===================================================================
===========      Monitoring Configuration Main Menu   =============
===================================================================
Select:
   (S)how current monitoring requests configured via monconfig
   (C)heck detailed monitoring status
   (L)ist descriptions of available monitors
   (A)dd a monitoring request
   (D)elete a monitoring request
   (M)odify an existing monitoring request
   (E)nable Monitoring
   (K)ill (disable) monitoring
   (H)elp
   (Q)uit
Enter selection: [s] a                         <== SELECT ADD OPTION

===========================================================================
=====================     Add Monitoring Request      =====================
===========================================================================

Start of edit configuration:

A monitoring request consists of:
  A list of monitors to which it applies
  A severity range (A relational expression and a severity. For example,
  %< "MAJOR WARNING" means events with a severity "INFORMATION" and
  "MINOR WARNING")
  A notification method
Please answer the following questions to specify a monitoring request.

Monitors to which this configuration can apply:
   1) /storage/events/disk_arrays/AutoRAID
   2) /storage/events/disks/default
   3) /adapters/events/FC_adapter
   4) /connectivity/events/multiplexors/FC_SCSI_mux
   5) /storage/events/enclosures/ses_enclosure
   6) /storage/events/tapes/SCSI_tape
   7) /storage/events/disk_arrays/FW_SCSI
   8) /storage/events/disk_arrays/High_Availability
   Enter monitor numbers separated by commas
   {or (A)ll monitors, (Q)uit, (H)elp} [a] 1   <== SELECT AUTORAID MONITOR

Criteria Thresholds:
   1) Informational     2) Minor Warning     3) Major Warning
   4) Serious     5) Critical
   Enter selection {or (Q)uit,(H)elp} [4] 5           <==  SELECT ONLY
                                                           CRITICAL EVENTS
Criteria Operator:                                        
   1) %<     2) %<=    3) >    4) >=    5) =    6) !       
   Enter selection {or (Q)uit,(H)elp} [4] 5           <==  (=CRITICAL)

Notification Method:
   1) UDP       2) TCP       3) OPC      4) SNMP
   5) TEXTLOG   6) SYSLOG    7) EMAIL    8) CONSOLE
   Enter selection {or (Q)uit,(H)elp} [7]             <== SELECT EMAIL
                                                          ADDRESS FOR
   Enter Email Address: [root] admin@hp.com               admin@hp.com

User Comment:
   (C)lear   (A)dd
   Enter selection {or (Q)uit,(H)elp} [c] a           <== ADD COMMENT
   Enter comment:  []   This is a test message.           IF DESIRED

Client Configuration File:
   (C)lear   (A)dd
   Use Clear to use the default file.
   Enter selection {or (Q)uit,(H)elp} [c] c           <== SPECIFY CLCFG FILE
                                                          IF DESIRED (USUALLY
                                                          CHOOSE DEFAULT)

New entry:
      Send events generated by all monitors
        /storage/events/disk_arrays/AutoRAID          <== NEW MONITORING
      with severity = CRITICAL to EMAIL admin@hp.com      REQUEST
      with comment: 
        This is a test message
     Are you sure you want to keep these changes?
    {(Y)es,(N)o,(H)elp} [n] y

Modifying a Monitoring Request

Modifying an existing monitoring request enables you to alter one of the settings used in the request. You must select a monitoring request and change the preferred setting. All other aspects of the request remain unchanged.

To modify a monitoring request, complete the following steps:

Run the monconfig utility by entering the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig
Enter Mat the main menu selection prompt.
All current monitoring requests are displayed.
From the list of current monitoring requests, enter the number of the request you want to modify.
Change the settings to achieve the desired results.
Save the request when prompted.

Verifying Hardware Event Monitoring

After creating the monitoring requests for your system, you must verify hardware event monitoring. The most effective way of verifying hardware event monitoring is to simulate a hardware failure or event. Depending on the hardware, you can do this by removing a disk from an array, unplugging a cable, turning off the hardware resource, using known defective media, and so on.

The simulated fault must generate event messages using all the notification methods you have specified. If it does not, check the monitoring requests and ensure that they are configured properly.

Checking Detailed Monitoring Status

The Detailed Monitoring Status option enables you to view detailed information about all active monitoring requests. This information is organised by the resource instance. It lists all the monitoring requests currently applied to each instance.

Unlike the Show Monitoring Requests option, which displays all the monitoring requests that are created using the monconfig utility, the detailed status displays only the requests that are currently active. For example, you can create a monitoring request for a monitor that is inactive. However, it is not displayed in the list.

An inactive monitor is identified with a NOT MONITORING status. A monitor that does not have resources to monitor is inactive.




	NOTE: The PSM automatically creates the TCP monitoring request. This enables the TCP monitoring request to gather event information from each monitor.

Following is a sample entry displayed for detailed monitoring status:

For /storage/events/disks/default/10_12_5.2.0:
Events >= 1 (INFORMATION) Goto TEXTLOG; file=/var/opt/resmon/log/event.log
Events >= 4 (MAJOR WARNING) Goto SYSLOG
Events >= 4 (MAJOR WARNING) Goto EMAIL; addr=root
Events =  5 (CRITICAL) Goto TCP; host=hpbs1266.boi.hp.com port=53327

For /adapters/events/FC_adapter/8_12.8:
Events >= 1 (INFORMATION) Goto TEXTLOG; file=/var/opt/resmon/log/event.log
Events >= 4 (MAJOR WARNING) Goto SYSLOG
Events >= 4 (MAJOR WARNING) Goto EMAIL; addr=root

>/connectivity/events/multiplexors/FC_SCSI_mux ... NOT MONITORING.
(Possibly there is no hardware to monitor.)

>/system/events/memory ... OK.
For /system/events/memory/49:
Events >= 1 (INFORMATION) Goto TEXTLOG; file=/var/opt/resmon/log/event.log
Events >= 4 (MAJOR WARNING) Goto SYSLOG
Events >= 4 (MAJOR WARNING) Goto EMAIL; addr=root
Events >= 4 (MAJOR WARNING) Goto TCP; host=hpbs1266.boi.hp.com port=53327

Retrieving and Interpreting Event Messages

Event messages generated by the monitors can be delivered using a variety of notification methods. To simplify the process of receiving event messages, you can use e-mail or textlog notification methods. Both methods, which are included in the default monitoring, receive the entire content of the message.

Methods such as console, syslog, and SNMP alert you about the occurrence of an event, but they do not deliver the entire message. You must retrieve the entire message using the resdata utility. For these methods, the event includes a message similar to the following:

Execute the following command to obtain event details: /opt/resmon/bin/resdata
-R 392036357 -r /storage/events/tapes/SCSI_tape/10_12_5.0.0 -n 392036353 -a

You must execute the command exactly as described, specifying the two critical number fields that are indices for the resdata entries.

Following is a portion of a sample event message:

> Event Monitoring Service Event Notification %<

Notification Time: Wed Sep  9 10:48:30 1998

hpbs8684 sent Event Monitor notification information:

/storage/events/disks/default/10_4_4.0.0 is >= 1.
Its current value is CRITICAL(5).

Event data from monitor:

Event Time : Wed Sep  9 10:48:30 1998
Hostname : hpbs8684.boi.hp.com      IP Address  : 15.62.120.25
Event Id : 0x0035f6b15e00000000      Monitor     : disk_em
Event #  : 100037                    Event Class : I/O
Severity : CRITICAL

Disk at hardware path 10/4/4.0.0 : Media failure
Associated OS error log entry id(s):
      000000000000000000

Description of Error:

     The device was unsuccessful in reading data for the current I/O request
     due to an error on the medium. The data could not be recovered. The
     request was likely processed in a way which could cause damage to or loss
     of data.

Probable Cause / Recommended Action:

     The medium in the device is flawed. If the medium is removable, replace
     the medium with a fresh one. Alternatively, if the medium is not
     removable, the device has experienced a hardware failure.
     Repair or replace the device, as necessary.

=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=

Deleting Monitoring Requests

You can delete monitoring requests for a hardware resource that is removed from your system. However, you must delete only requests that are created exclusively for the removed resource.




	CAUTION: Deleting monitoring requests makes your system vulnerable to undetected hardware failures. This is particularly true for the default monitoring requests, which provide protection for all the supported hardware resources on your system.

To delete a monitoring request, complete the following steps:

Log in as superuser.
Run the monconfig utility by entering the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig
Enter Dat the main menu selection prompt.
All the current monitoring requests are displayed.
From the list of the current monitoring requests, enter the number assigned to the request you want to delete.
Delete the request when prompted.

Disabling Monitoring

You can disable hardware monitoring, if necessary. However, you must disable all EMS Hardware Monitors, because you cannot disable a specific monitor. While monitoring is disabled, all monitoring requests are disabled. The monitoring requests are retained and become operational when you re-enable monitoring.




	CAUTION: When monitoring is disabled, your hardware resources are vulnerable to undetected failures. Disabling monitoring impacts HP Serviceguard if package dependencies are created for the monitors.

To disable monitoring, complete the following steps:

Run the monconfig utility by entering the following command at the HP-UX prompt:
# /etc/opt/resmon/lbin/monconfig
Enter Kat the main menu selection prompt.
Confirm disabling when prompted. For information on how to re-enable hardware monitoring, see “Enabling Monitoring ”.