The Logtool Utility allows you to monitor recoverable errors detected by the computer, such as single-bit memory errors, and I/O device errors. By analyzing this information, you can troubleshoot failures before they become acute. (For a quick start, see Viewing a Log.)
Error information is contained in system log files. Since the raw files are hard for humans to read, Logtool allows you to format the files and to filter them for desired entries (for example, for all entries related to SCSI disks).
Unlike other support tools, this utility program is not device specific and does not require selection of a device to run. A license is not required to run the Logtool Utility.
The Logging Daemon
When STM is installed on a computer, a logging daemon is activated to monitor recoverable errors that have been detected by the system, for example:
For more information, see Log Entry Types.
Logging
When the logging daemon detects an event, it logs an entry in a system log file located (by default) in the directory "/var/stm/logs/os". The raw log file names have the format "logN.raw," where N is a number that starts at 1 and is incremented each time a new log file is created with the command Switch Log. The only exception to the raw log file naming convention is the log file to which the logging daemon is currently logging data; this file is called "logN.raw.cur."
Memory errors, when detected by the memory logging daemon, are logged into the memory log file called memlog.
The chassis code logging daemon logs information to the chassis code error log file and boot log files called ccerrlog and ccbootlog respectively. These log records are generated by various system hardware, software and firmware modules. They can contain error indications as well as informational messages. Chassis code logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
Logging Control
To change the file to which the daemon is logging data, select the Switch Log command from the File Menu.
You can modify the behavior of the logging daemon by editing the configuration file "diaglogd.cfg" located in the directory "/var/stm/config/sys". The file itself contains documentation regarding its format.
Logtool and the Logging Process
The role of Logtool in the logging process is shown in the diagram and is explained in more detail below.
errors
|
v logger
STM logger <- config
| file
v +-------------+
FILE: log1.raw -> | Raw Display | -> DISPLAY
| | Filter | of raw
v +-------------+ data
+---------------+
| Format Filter |
+---------------+
|
v +-------------------+
FILE: log1.fmt1 -> | Formatted Display | -> DISPLAY of
| Filter | formatted
+-------------------+ data
Raw Log Files
The first step in analyzing the log files is usually to enter the Select Raw command from the File Menu. A summary of the contents of the log file(s) will automatically be displayed. In addition the summary information can be obtained independently with the Raw Summary command in the View Menu. To view the entries in the log file, you use the Raw Log command in the View Menu.
Looking at all the entries in the raw log file can be burdensome:
To solve these dilemmas, Logtool offers the ability to format and filter the entries in the log files.
Formatting the Data
To translate the raw log file into a more readable format, select the Format Raw command from the File Menu. The formatted data is put into a file with the name "logN.fmtN" where the first number (N) matches that of the raw log file and the second number (N) starts at 1 and is incremented for each new formatted file generated from the same raw log file. If multiple files have been chosen, the file name will have a plus sign ("+") in it. For example, log1.raw and log2.raw will be formatted into log1+.fmt1.
To choose a particular formatted log file(s) for viewing or filtering, use the Select Formatted command from the File Menu. A summary of the contents of the formatted log file(s) will automatically be displayed. In addition the summary information can be obtained independently with the Formatted Summary command in the View Menu. To view the entries in the formatted log file(s), you use the Formatted Log command in the View Menu.
Filtering the Data
Even if the data is formatted, there is generally a large number of entries contained in a typical log. When you are troubleshooting a problem, only certain entries may be relevant. For example, you may want to see only I/O errors that occurred during a certain time period on a certain hardware path.
Logtool offers three different filters to select the entries relevant to your task.
Memory Log
The memory log is handled somewhat differently than the system logs described above. The commands to view the memory report log or to clear the memory log are found in the Memory Menu. You can also use the Memory->Set Timer command to change the memory polling interval.
The memory report log can be viewed in two separate ways. The entire log file can be viewed (View Detail) or information pertaining only to memory deallocation can be viewed (View Dealloc).
Chassis Logs
The commands to view the Chassis error log file or Chassis boot log files are found in the Chassis Menu. The user can select which file they want to view from the displayed list of the current files.
The chassis error logs are created by the logging daemon (diaglogd). Chassis Boot Log entries can come from various system hardware, software and firmware modules. The Chassis Error log contains entries for chassis codes reported with an alert level indicating that an error has occurred. For instance, chassis codes are logged if problems occur with the power supplies. By examining the chassis error log, you could see a record of these power supply problems.
The entry types currently logged by Logtool include:
Logged by I/O system drivers, I/O error entries typically contain information concerning exception conditions that the driver encountered.
Logged by the system when a Low Priority Machine Check (LPMC) condition has occurred. An LPMC is a recoverable error detected by hardware; the hardware is able to recover from the error and continue operation, though perhaps in a degraded fashion. In contrast, a High Priority Machine Check is an error detected by hardware from which it cannot recover.
Logged by the system when hardware detects a recoverable over-temperature condition (over-temp warning).
Logged by the Logical Volume Manager, typically containing information concerning exception conditions the manager encountered.
Logged by the logger process. The type of events that are logged include suspension and resumption of logging.
To run the Logtool Utility:
For more information, see online help for STM as a whole (use the Help menu in the STM main window).
Logtool offers many different ways to select log entries and view them. A typical method for viewing logs involves these steps:
For example, you may have used a format filter that resulted in all entries from the SCSI disk driver being formatted. After viewing a summary of the formatted entries, you may decide that only entries for a particular device are of interest or, perhaps, only entries after a certain date and time. You could then set the formatted display filter to specify a specific device path or date and time and re-display the formatted log file.
To troubleshoot a disk problem (sample procedure):
Follow the general procedure described in Viewing a Log, but specify disk-related parameters in any of the filters commands.
For example, de-activate the selections for all of the entry types except I/O Errors, such as "Low Priority Machine Check", "LVM Entries", etc., and active the "Filter I/O Errors" radio button under "I/O Error Entries". Under "Products:", select only the relevant products (for example select "SCSI Disk" and not "LAN Interface").
To troubleshoot a suspected problem with memory (sample procedure):
The method for viewing the memory error log file involves these steps:
If most of the errors in the file indicate the same slot for the memory controller, then the SIMM module in that particular slot is probably bad.
If occasional errors report the same slot for the memory controller, then the SIMM module in that particular slot is suspect.
If the same bit number is indicated across many slots then the memory controller is bad.
Consult the tool's Activity Log, accessible from the STM main window.
The Logtool Utility may be waiting for a response to a query message it is displaying. Look underneath the windows displayed on the monitor for a query message. Also, de-iconize any windows related to stm or the tool.
Entries in the raw system log are hard to understand; format the log to make the entries more readable (Format Raw command in the File Menu).
Specify that only certain entries in the log file be formatted or displayed; use one of the commands in the Filters Menu.
Start a new current system log (Switch Log command in the File Menu).
At present, Logtool only supports the formatting of log files that originated on the host system. Format the log files on the computer on which they were generated. Once a log has been formatted, it can be moved to other computers for display and filtering.
The Logtool Utility offers the following commands and buttons:
The Select Raw command displays the names of the raw log files in the default log directory and allows you to select one or more for viewing, filtering or formatting. If multiple files are selected, all of the entries are combined together and treated as one file. Once the file(s) is selected, the summary information is automatically displayed.
The current raw log file has the extension ".cur" and is the default selected file. Each time a new log file is generated, its file name is incremented by one. For example, the initial file will be "log1.raw.cur". When a new file is generated, "log1.raw.cur" will become log1.raw, and the current raw log file will be "log2.raw.cur".
If the current raw log file is selected, the output of subsequent commands such as Raw Log or Format Raw may reflect additional records/and or counts that weren't displayed initially. This is because more errors may have been logged to the currently active file since it was last read.
NOTE: Users can optionally specify a file of their own choosing but are cautioned that only log files generated on the host system should be formatted on that system. Attempts to format log files from other systems may result in formatting problems or incorrect formatted output. Once a formatted log file is generated, it can be moved to other systems and displayed without any problems.
The Format Raw command formats entries from the currently selected raw log file or files chosen via the Select Raw command under the File menu. If a file has not been selected, the current raw log file, if there is one, is used as the default. Formatting makes the log entries much easier for humans to understand.
NOTE: Only log files generated on the host system should be formatted on that system. Attempts to format log files from other systems may result in formatting problems or incorrect formatted output. Once a formatted log file is generated, it can be moved to other systems and displayed without any problems.
The entries in the raw log file(s) are filtered according to the Format filter settings (Format command). Entries which do not match the filter are not formatted. If you wish to format entries which were previously filtered out, change the filter and reformat the log file. If no filters have been selected all entries will be formatted.
The data portion of each entry is formatted into text by the appropriate log decoding routine. If an appropriate log decoding program cannot be located for a particular entry, the data portion of the entry is displayed in hex.
Formatted log file names have the form "logN.fmtN" where the first number (N) matches that of the raw log file from which it was generated and the second number (N) starts at 1 and is incremented for each new formatted file that is generated from the same raw log file. For example, "log1.raw" becomes "log1.fmt1" the first time it is formatted. If you then format the same raw log file a second time, the newly formatted file is named "log1.fmt2".
After a raw file is formatted, the formatted log summary is automatically displayed. The newly created formatted log file becomes the selected formatted file to be displayed or summarized by the Formatted Summary and Formatted Log commands under the View menu.
The Select Formatted command displays the names of the formatted log files that are currently in the default log directory on the system and allows you to select one or more of them for viewing or filtering. Once the file(s) is selected, the summary information is automatically displayed.
The Switch Log command moves the existing current raw log file "logN.raw.cur" to "logN.raw." The next time the logger process needs to write an error it opens a new "logN.raw.cur" file, incrementing the "N" by one.
For example, the Switch Log command moves "log5.raw.cur" to "log5.raw," causing the logger process to open a new raw log file called "log6.raw.cur."
Individual log files can be kept from becoming too large by issuing this command occasionally. Alternatively, a size threshold can be set in the diaglogd.cfg file.
Use the Save As command to save the contents of Logtool's main window. (Only the most recent 200 lines can be saved.)
Use the Print command to print the contents of Logtool's main window. (Only the most recent 200 lines can be printed.)
Use the Exit command to leave the Logtool Utility and return to the main STM window.
The Raw Summary command from the View menu can be selected independently but is also run automatically when file(s) are selected. It displays a summary of the entries in the currently selected raw log file(s). If no files were selected, the current raw log file, if there is one, is used as the default.
The summary reports the total number of each type of error entry that occurs in the file(s) (for example, "Number of LPMC entries: 6"). If multiple files are selected the entries are combined together and treated as one file.
The summary reports additional information on I/O error entries, listing the specific device paths, products, product qualifiers, and logger events that occur in the file(s), and the number of times they occur.
For more information, see Log Entry Types.
The Raw Log command from the View menu displays the entries from the currently selected raw log file(s). The header portion is displayed in text and the data portion is displayed in hex.
The raw log file(s) could have been selected via the Select Raw command from the File menu. If a raw log file was not selected, then the current raw log file, if there is one, is used as the default. Filters may have been selected and applied via the Raw command under the Filters->Display Menu. Entries which do not match the filter are not displayed. If you wish to display entries which were previously filtered out, change the filter and redisplay the log file using the Raw command in the Filters->Display Menu. If multiple files are selected the entries are combined together and treated as one file.
The top of the display in the "Display Filters: area" lists the filter criteria specified at the time the raw log file(s) was filtered.
The Formatted Summary command from the View menu can be selected independently but is also run automatically when file(s) are selected. It displays a summary of the entries in the currently selected formatted log file(s). If no files were selected, via the Select Formatted command under the File menu, then the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file. If multiple files have been selected the entries are combined together and treated as one file.
The summary reports the number of each type of error entry that occurs in the file(s) (for example, "Number of LPMC entries: 6").
The summary reports additional information on I/O error entries, listing the specific device paths, products, product qualifiers, loggers, device types and device qualifiers that occur in the file(s), and the number of times they occur.
For more information, see Log Entry Types.
The Formatted Log command from the View menu displays the formatted log data that was generated by the Format Raw command under the File menu.
The formatted log file(s) could have been selected via the Select Formatted command from the File menu. If a file was not selected, the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file. Filters may have been selected and applied via the Formatted command under the Filters->Display Menu. Entries which do not match the filter are not displayed. If you wish to display entries which were previously filtered out, change the filter and redisplay the log file.
If multiple files have been selected the entries are combined together and treated as one file.
The top of the display in the "Display Filters: area" lists the filter criteria specified at the time the raw log file(s) was filtered.
The Format command from the Filters menu allows you to select filtering criteria to be applied to the entries in the raw log file(s) before formatting.
In order to set the filter, you should have first selected a raw log file(s) via the Select Raw command under the File menu. If a file was not selected, then the current raw log file, if there is one, is used as the default.
Items which occur in the raw log file(s) are displayed as candidates for filtering. These items are displayed in a dialog screen and you can select as many items as desired.
To select more than one item in a box, hold down the control key CTRL and click on the item with the left mouse button.
Once one or more items have been selected, subsequent Format Raw commands (under the File menu) only format entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display Entries of the following types" has a checkbox beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by clicking on the appropriate entry type that you're not interested in seeing.
The I/O Error Entry types have a separate section which is controlled by radio buttons. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries you can select the radio button labeled "No I/O Errors". If you wish to filter on a specific qualifier you can select the radio button labeled "Filter I/O Errors" and then make your selections from the choices below by clicking on the items you wish to use for filter criteria.
To clear all selected items you can press the Defaults button. To clear a specific highlighted item, hold down the control key CTRL and click on the item with the left mouse button.
To select more than one item in a box, hold down the control key CTRL and click on the item with the left mouse button.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
Pressing the Defaults button resets the selections to all entries in the file.
The Raw command from the Filters->Display menu is used to select which entries in the raw log file(s) to display.
In order to set the filter, you should have first selected a raw log file(s) via the Select Raw command under the File menu. Otherwise, the current raw log file, if there is one, is used as the default. Items which occur in the raw log file(s) are displayed as candidates for filtering. These items are displayed in a dialog screen and you can select as many items as desired.
To select more than one item in a box, hold down the control key CTRL and click on the item with the left mouse button.
Once one or more items have been selected, subsequent Raw Log commands (under the View menu) only display entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display Entries of the following types" has a checkbox beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by clicking on the appropriate entry type that you're not interested in seeing.
The I/O Error Entry types have a separate section which is controlled by radio buttons. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries you can select the radio button labeled "No I/O Errors". If you wish to filter on a specific qualifier you can select the radio button labeled "Filter I/O Errors" and then make your selections from the choices below by clicking on the items you wish to use for filter criteria.
To clear all selected items you can press the Defaults button. To clear a specific highlighted item, hold down the control key CTRL and click on the item with the left mouse button.
To select more than one item in a box, hold down the control key CTRL and click on the item with the left mouse button.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
In addition to specifying criteria for each of the header fields, you can enter a hex pattern to search for in the data for each entry. If a search pattern is specified, it is applied in addition to any other fields already specified.
NOTE: Specifying a search pattern may result in a noticeable delay because each entry's data must be searched. When a search pattern is not used, only header contents are examined.
Pressing the Defaults button resets the selections to all entries in the file.
The Formatted command from the Filters->Display menu is used to select which entries in the formatted log file(s) to display.
In order to set the filter, you should first select one or more formatted log files via the Select Formatted command under the File menu. If a file was not selected in this way, then the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file.
Items which occur in the log file(s) are displayed as candidates for filtering. These items are displayed in a dialog screen and you can select as many items as desired.
To select more than one item in a box, hold down the control key CTRL and click on the item with the left mouse button.
Once one or more items have been selected, subsequent Formatted Log commands (under the View menu) only display entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display Entries of the following types" has a checkbox beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by clicking on the appropriate entry type that you're not interested in seeing.
The I/O Error Entry types have a separate section which is controlled by radio buttons. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries you can select the radio button labeled "No I/O Errors". If you wish to filter on a specific qualifier you can select the radio button labeled "Filter I/O Errors" and then make your selections from the choices below by clicking on the items you wish to use for filter criteria.
To clear all selected items you can press the Defaults button. To clear a specific highlighted item, hold down the control key CTRL and click on the item with the left mouse button.
To select more than one item in a box, hold down the control key CTRL and click on the item with the left mouse button.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
Pressing the Defaults button resets the selections to all entries in the file.
The Clear Log command will notify the memory logging daemon to create a new memory log file.
The Set Timer command allows you to change the interval at which the memory logging daemon polls the memory for errors.
The View Dealloc command allows you to display the deallocated page information from the memory log file.
The View Detail command allows you to display the entire contents of the memory log file.
The Chassis View Boot Log command allows you to display the entire contents of a chassis code boot log file (ccbootlog.[1-10]).
The boot log contains mostly informational log entries output by the system as it boots. If the system is experiencing problems booting, this log may contain useful information about the problem.
The boot log is created by the logging daemon (diaglogd). Log entries can come from various system hardware, software and firmware modules.
Chassis logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
The Chassis View Error Log command allows you to display the entire contents of the chassis code error log file, ccerrlog.
The chassis code error log contains entries for chassis codes reported with an alert level indicating that an error has occurred.
For instance, chassis codes are logged if problems occur with the power supplies. By examining the error log, you could see a record of these power supply problems.
The error log is created by the logging daemon (diaglogd). Log entries can come from various system hardware, software and firmware modules.
Chassis logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
The General Help command displays online help for the Logtool Utility (the help system currently being viewed).
For information on using the help system, select the "Using Help..." command in the Help menu in this Help window.
For online help on STM as a whole, use the Help menu in the STM main window.
The Version command displays version and copyright information for the Logtool Utility.
The Abort button allows you to abort the Logtool Utility, halting its execution and returning to the STM main window.
The Interrupt button allows you to interrupt the tool.
When the tool receives the interrupt, it stops whatever it is doing and prompts you to respond if you want to continue with the operation. If the reply is "Continue", the tool resumes what it was doing at the time of the interrupt. If the reply is "Abort", it terminates the operation and returns to the Logtool Utility main window.
The tool may not immediately receive the interrupt as it could be in the middle of an operation that should not or could not be interrupted. The interrupt is received as soon as the tool completes the operation.
The tool will receive and ignore user interrupts when it is already in the process of returning to the main window of the tool.
The shortcut commands described below were implemented beginning with version 2.0 of Logtool.
When entering Logtool, there are several "shortcut" commands that can be invoked. The shortcut commands automatically select the current raw log file (*raw.cur) as the basis for all of their operations. If there is not a current raw log file to use, the user is informed that they need to select a file.
In addition, the Select Raw and Select Formatted commands automatically display the file's summary information once a file has been selected.
Once a file has been selected, the raw log summary is automatically displayed.
Defaults to the current raw log file, if there is one, and displays the raw log summary.
Defaults to the current raw log file, if there is one, and displays the raw log file. The header portion of the file entries is displayed in text; the data portion is displayed in hex.
Defaults to the current raw log file, if there is one, and presents the dialog to the user.
Defaults to the current raw log file, if there is one, and presents the dialog to the user.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log summary.
Once a file has been selected, the formatted log summary is automatically displayed.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log summary.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log file.
Defaults to the current raw log file, if there is one, automatically formats the file, and presents the dialog to the user.
Throughout this document there are references to multiple file selection capabilities and the behaviors when multiple files have been selected.
The handling of multiple files was implemented beginning with version 3.0 of Logtool.
See the specific commands for information regarding multiple file handling. In general, multiple files may be selected in Select Raw or Select Formatted. Once selected, all entries from all the selected files are combined together and treated as one file for viewing, filtering and formatting.