The Logtool Utility allows you to monitor recoverable errors detected by the computer, such as single-bit memory errors and I/O device errors. By analyzing this information, you can troubleshoot failures before they become acute. (For a quick start, see Viewing a Log.)
Error information is contained in system log files. Since the raw files are hard for humans to read, Logtool allows you to format the files and to filter them for desired entries (for example, for all entries related to SCSI disks).
Unlike other support tools, this utility program is not device specific and does not require selection of a device to run. A license is not required to run the Logtool Utility.
In mstm, commands are selected from the pulldown menus. Use the tab key or the "Menubar on/off" function key to move the cursor to the menu bar. The cursor arrow keys can then be used to select a specific menu, and the "Return" key can be used to pull down the highlighted menu or select the highlighted command from a menu.
Summary of Logtool Menu Commands
--- File Pulldown Menu Keys ---
Select Raw - Select a raw log file or files to view, filter
or format
Format Raw - Format the currently selected raw log file(s)
Select Formatted - Select a formatted log file or files to view
or filter
Switch Log - Switch to (create) a new "current" raw log file
Save As - Save Logtool's main window
Print - Print Logtool's main window
Exit - Exit Logtool and return to the main STM screen
--- View Pulldown Menu Keys ---
Raw Summary - View the summary of the currently selected
raw log file(s)
Raw Log - View the currently selected raw log file
Formatted Summary - View the summary of the currently selected
formatted log file(s)
Formatted Log - View the currently selected formatted log
file(s)
--- Filters Pulldown Menu Keys ---
Format - Select format filters for the currently
selected raw log file(s)
Display Menu Keys
Raw - Select display filters for the currently
selected raw log file(s)
Formatted - Select display filters for the currently
selected formatted log file(s)
--- Memory Pulldown Menu Keys ---
Clear Log - Clear the memory log file
Set Timer - Change the polling interval for logging
memory errors
Report Log Menu Keys
View Dealloc - View errors pertaining to deallocated
memory errors
View Detail - View the entire memory log file
--- Chassis Pulldown Menu Keys ---
Chas Bootlog - View the chassis code boot log file
Chas Errorlog - View the chassis code error log file
--- Help Menu Keys ---
General help - Display online help for Logtool
Version - Display Logtool version and copyright
information
Control-C - Interrupt or abort execution of logtool;
escape to the user interface
The Logging Daemon
When STM is installed on a computer, a logging daemon is activated to monitor recoverable errors that have been detected by the system, for example:
For more information, see Log Entry Types.
Logging
When the logging daemon detects an event, it logs an entry in a system log file located (by default) in the directory "/var/stm/logs/os". The raw log file names have the format "logN.raw," where N is a number that starts at 1 and is incremented each time a new log file is created with the command Switch Log. The only exception to the raw log file naming convention is the log file to which the logging daemon is currently logging data; this file is called "logN.raw.cur."
Memory errors, when detected by the memory logging daemon, are logged into the memory log file called memlog.
The chassis code logging daemon logs information to the chassis code error log file and boot log files called ccerrlog and ccbootlog respectively. These log records are generated by various system hardware, software and firmware modules. They can contain error indications as well as informational messages. Chassis code logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
Logging Control
To change the file to which the daemon is logging data, select the Switch Log command in the File pulldown menu.
You can modify the behavior of the logging daemon by editing the configuration file "diaglogd.cfg" located in the directory "/var/stm/config/sys". The file itself contains documentation regarding its format.
Logtool and the Logging Process
The role of Logtool in the logging process is shown in the diagram and is explained in more detail below.
errors
|
v logger
STM logger <- config
| file
v +-------------+
FILE: log1.raw -> | Raw Display | -> DISPLAY
| | Filter | of raw
v +-------------+ data
+---------------+
| Format Filter |
+---------------+
|
v +-------------------+
FILE: log1.fmt1 -> | Formatted Display | -> DISPLAY of
| Filter | formatted
+-------------------+ data
Raw Log Files
The first step in analyzing the log files is usually to enter the Select Raw command in the File pulldown menu. A summary of the contents of the log file will automatically be displayed once the file is selected. In addition the summary information can be obtained using the Raw Summary command in the View pulldown menu. To view the entries in the log file, you use the Raw Log command in the View pulldown menu.
Looking at all the entries in the raw log file can be burdensome:
To solve these dilemmas, Logtool offers the ability to format and filter the entries in the log files.
Formatting the Data
To translate the raw log file into a more readable format, use the Format Raw command from the File pulldown menu. The formatted data is put into a file with the name "logN.fmtN" where the first number (N) matches that of the raw log file and the second number (N) starts at 1 and is incremented for each new formatted file generated from the same raw log file. If multiple files have been chosen, the file name will have a plus sign ("+") in it. For example, log1.raw and log2.raw will be formatted into log1+.fmt1.
To choose a particular formatted log file(s) for viewing or filtering, use the Select Formatted command from the File pulldown menu. A summary of the contents of a formatted log file can be obtained independently with the Formatted Summary command in the View pulldown menu. To view the entries in the formatted log file, you use the Formatted Log in the View pulldown menu.
Filtering the Data
Even if the data is formatted, there is generally a large number of entries contained in a typical log. When you are troubleshooting a problem, only certain entries may be relevant. For example, you may want to see only I/O errors that occurred during a certain time period on a certain hardware path.
Logtool offers three different filters to select the entries relevant to your task.
Memory Log
The memory log is handled somewhat differently than the system logs described above. The commands to view the memory report log or to clear the memory log are found in the Memory pulldown menu. You can also use the Set Timer command to change the memory polling interval.
The memory report log can be viewed in two separate ways. The entire log file can be viewed (View Detail) or information pertaining only to memory deallocation can be viewed (View Dealloc).
Chassis Code Logs
The chassis code logs can be viewed via the Chassis pulldown menu. The two types of chassis code logs are Boot Log (View Boot Log) and Error Log (View Error Log).
The entry types currently logged by Logtool include:
Logged by I/O system drivers, I/O error entries typically contain information concerning exception conditions that the driver encountered.
Logged by the system when a Low Priority Machine Check (LPMC) condition has occurred. An LPMC is a recoverable error detected by hardware; the hardware is able to recover from the error and continue operation, though perhaps in a degraded fashion. In contrast, a High Priority Machine Check is an error detected by hardware from which it cannot recover.
Logged by the system when hardware detects a recoverable over-temperature condition (over-temp warning).
Logged by the Logical Volume Manager, typically containing information concerning exception conditions the manager encountered.
Logged by the logger process. The type of events that are logged include suspension and resumption of logging.
To run the Logtool Utility:
For more information, see online help for STM as a whole.
Logtool offers many different ways to select log entries and view them. A typical method for viewing logs involves these steps:
For example, you may have used a format filter that resulted in all entries from the SCSI disk driver being formatted. After viewing a summary of the formatted entries, you may decide that only entries for a particular device are of interest or, perhaps, only entries after a certain date and time. You could then set the formatted re-display filter to specify a specific device path or date and time and display the formatted log file.
To troubleshoot a disk problem (sample procedure):
Follow the general procedure described in Viewing a Log, but specify disk-related parameters in any of the filters commands.
For example, de-activate the selections for all of the entry types except I/O Errors, such as "Low Priority Machine Check", "LVM Entries", etc., and activate the selection "Filter I/O Errors" under "I/O Error Entries". Under "Products:", select only the relevant products (for example select "SCSI Disk" and not "LAN Interface").
The method for viewing the memory error log file involves these steps:
If most of the errors in the file indicate the same slot for the memory controller, then the SIMM module in that particular slot is probably bad.
If occasional errors report the same slot for the memory controller, then the SIMM module in that particular slot is suspect.
If the same bit number is indicated across many slots then the memory controller is bad.
Consult the tool's Activity Log, accessible from the STM main screen.
The Logtool Utility may be waiting for a response to a query message it is displaying. Answer the query and continue.
Entries in the raw system log are hard to understand; format the log to make the entries more readable (Format Raw command in the File pulldown menu).
Specify that only certain entries in the log file be formatted or displayed; use one of the commands in the Filters pulldown menu.
Start a new current system log (Switch Log command in the File pulldown menu).
At present, Logtool only supports the formatting of log files that originated on the host system. Format the log files on the computer on which they were generated. Once a log has been formatted, it can be moved to other computers for display and filtering.
The Logtool Utility offers the following commands:
The Select Raw command in the File pulldown menu displays the names of the raw log files in the default log directory and allows you to select one or more for viewing, filtering or formatting. If multiple files are selected, all of the entries are combined together and treated as one file. Once the file is selected, the summary information is automatically displayed.
The current raw log file has the extension ".cur". Each time a new log file is generated, its file name is incremented by one. For example, the initial file will be "log1.raw.cur". When a new file is generated, "log1.raw.cur" will become log1.raw, and the current raw log file will be "log2.raw.cur".
If the current raw log file is selected, the output of subsequent commands such as Raw Log or Format Raw may reflect additional records/and or counts that weren't displayed initially. This is because more errors may have been logged to the currently active file since it was last read.
NOTE: Users can optionally specify a file of their own choosing but are cautioned that only log files generated on the host system should be formatted on that system. Attempts to format log files from other systems may result in formatting problems or incorrect formatted output. Once a formatted log file is generated, it can be moved to other systems and displayed without any problems.
The Format Raw command in the File pulldown menu formats entries from the currently selected raw log file or files chosen via the Select Raw command from the File pulldown menu. If a file has not been selected, the current raw log file, if there is one, is used as the default. Formatting makes the log entries much easier for humans to understand.
NOTE: Only log files generated on the host system should be formatted on that system. Attempts to format log files from other systems may result in formatting problems or incorrect formatted output. Once a formatted log file is generated, it can be moved to other systems and displayed without any problems.
The entries in the raw log file(s) are filtered according to the format filter settings (Format command in the Filters pulldown menu). Entries which do not match the filter are not formatted. If you wish to format entries which were previously filtered out, change the filter and reformat the log file. If no filters have been selected all entries will be formatted.
The data portion of each entry is formatted into text by the appropriate log decoding routine. If an appropriate log decoding program cannot be located for a particular entry, the data portion of the entry is displayed in hex.
Formatted log file names have the form "logN.fmtN" where the first number (N) matches that of the raw log file from which it was generated and the second number (N) starts at 1 and is incremented for each new formatted file that is generated from the same raw log file. For example, "log1.raw" becomes "log1.fmt1" the first time it is formatted. If you then format the same raw log file a second time, the newly formatted file is named "log1.fmt2".
After a raw file is formatted, the formatted log summary is automatically displayed. The newly created formatted log file becomes the selected formatted file to be displayed or summarized by the Formatted Summary and Formatted Log commands from the View pulldown menu.
The Select Formatted command in the File pulldown menu displays the names of the formatted log files that are currently in the default log directory on the system and allows you to select one or more of them for viewing or filtering. Once the file(s) is selected, the summary information is automatically displayed.
The Switch Log command in the File pulldown menu moves the existing current raw log file "logN.raw.cur" to "logN.raw." The next time the logger process needs to write an error it opens a new "logN.raw.cur" file, incrementing the "N" by one.
For example, the Switch Log command moves "log5.raw.cur" to "log5.raw," causing the logger process to open a new raw log file called "log6.raw.cur."
Individual log files can be kept from becoming too large by issuing this command occasionally. Alternatively, a size threshold can be set in the diaglogd.cfg file.
The Save As command in the File pulldown menu saves the contents of Logtool's main window. (Only the most recent 200 lines can be saved.)
The Print command in the File pulldown menu prints the contents of Logtool's main window. (Only the most recent 200 lines can be printed.)
The Exit command in the File pulldown menu allows the user to leave the Logtool Utility and return to the main STM screen.
The Raw Summary command in the View pulldown menu can be selected independently but is also run automatically when file(s) are selected. It displays a summary of the entries in the currently selected raw log file(s). If no files were selected, the current raw log file, if there is one, is used as the default.
The summary reports the number of each type of error entry that occurs in the file(s) (for example, "Number of LPMC entries: 6"). If multiple files are selected the entries are combined together and treated as one file.
The summary reports additional information on I/O error entries, listing the specific device paths, products, product qualifiers, and logger events that occur in the file(s), and the number of times they occur.
For more information, see Log Entry Types.
The Raw Log command in the View pulldown menu displays the entries from the currently selected raw log file(s). The header portion is displayed in text and the data portion is displayed in hex.
The raw log file(s) could have been selected via the Select Raw command from the File pulldown menu. If a raw log file was not selected, then the current raw log file, if there is one, is used as the default. Filters may have been selected and applied via the Raw command in the Filters->Display pulldown menu. Entries which do not match the filter are not displayed. If you wish to display entries which were previously filtered out, change the filter and redisplay the log file using the Raw command in the Filters->Display pulldown menu. If multiple files are selected the entries are combined together and treated as one file.
The top of the display in the "Display Filters: area" lists the filter criteria specified at the time the raw log file(s) was filtered.
The Formatted Summary command in the View pulldown menu can be selected independently but is also run automatically when file(s) are selected. It displays a summary of the entries in the currently selected formatted log file(s). If no files were selected via the Select Formatted command in the File pulldown menu, then the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file. If multiple files have been selected the entries are combined together and treated as one file.
The summary reports the number of each type of error entry that occurs in the file(s) (for example, "Number of LPMC entries: 6").
The summary reports additional information on I/O error entries, listing the specific device paths, products, product qualifiers, loggers, device types and device qualifiers that occur in the file(s), and the number of times they occur.
For more information, see Log Entry Types.
The Formatted Log command in the View pulldown menu displays the formatted log data that was generated by the Format Raw command in the File pulldown menu.
The formatted log file(s) could have been selected via the Select Formatted command in the File pulldown menu. If a file was not selected, the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file. Filters may have been selected and applied via the Formatted command in the Filters->Display pulldown menu. Entries which do not match the filter are not displayed. If you wish to display entries which were previously filtered out, change the filter and redisplay the log file.
If multiple files have been selected the entries are combined together and treated as one file.
The top of the display in the "Display Filters: area" lists the filter criteria specified at the time the raw log was filtered.
The Format command in the Filters pulldown menu allows you to select filtering criteria to be applied to the entries in the raw log file(s) before formatting.
In order to set the filter, you should have first selected a raw log file(s) via the Select Raw command in the File pulldown menu. If a file was not selected, then the current raw log file, if there is one, is used as the default.
Items which occur in the raw log file(s) are displayed as candidates for filtering. These items are displayed on the screen and you can select as many items as desired. To select an item place the cursor at the beginning of the entry and hit the RETURN key. To select more than one item, position the cursor on the entry and hit the SPACE BAR.
Once one or more items have been selected, subsequent Format Raw commands (in the File pulldown menu) only format entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display entries of the following types" has a checkbox beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by putting the cursor on the appropriate "X" and hitting the SPACE BAR.
The I/O Error Entry types have a separate section which is controlled by radio buttons. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries, select the radio button labeled "No I/O Errors" by placing the cursor in the angle brackets and hitting the SPACE BAR. If you wish to filter on a specific qualifier you can select the radio button labeled "Filter I/O Errors" and then make your selections from the choices below by highlighting the items you wish to use for filter criteria.
To clear all selected items you can hit the Defaults function key. To clear a specific highlighted item, press the SPACE BAR.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
Pressing the Defaults function key resets the selections to all entries in the file.
The Raw command in the Filters->Display pulldown menu is used to select which entries in the raw log file(s) to display.
In order to set the filter, you should have first selected a raw log file(s) via the Select Raw command in the File pulldown menu. Otherwise, the current raw log file, if there is one, is used as the default. Items which occur in the raw log file(s) are displayed as candidates for filtering. These items are displayed on the screen and you can select as many items as desired. To select an item place the cursor at the beginning of the entry and hit the RETURN key. To select more than one item, position the cursor on the entry and hit the SPACE BAR.
Once one or more items have been selected, subsequent Raw Log commands (in the View pulldown menu) only display entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display entries of the following types" has a checkbox beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by putting the cursor on the appropriate "X" and hitting the SPACE BAR.
The I/O Error Entry types have a separate section which is controlled by radio buttons. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries, select the radio button labeled "No I/O Errors" by placing the cursor in the angle brackets and hitting the SPACE BAR. If you wish to filter on a specific qualifier you can select the radio button labeled "Filter I/O Errors" and then make your selections from the choices below by highlighting the items you wish to use for filter criteria. To clear all selected items you can hit the Defaults function key. To clear a specific highlighted item, press the SPACE BAR.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
In addition to specifying criteria for each of the header fields, you can enter a hex pattern to search for in the data for each entry. If a search pattern is specified, it is applied in addition to any other fields already specified.
NOTE: Specifying a search pattern may result in a noticeable delay because each entry's data must be searched. When a search pattern is not used, only header contents are examined.
Pressing the Defaults function key resets the selections to all entries in the file.
The Formatted command in the Filters->Display pulldown menu is used to select which entries in the formatted log file(s) to display.
In order to set the filter, you should first select one or more formatted log files via the Select Formatted command in the File pulldown menu. If a file was not selected, then the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file.
Items which occur in the log file(s) are displayed as candidates for filtering. These items are displayed on the screen and you can select as many items as desired. To select an item place the cursor at the beginning of the entry and hit the RETURN key. To select more than one item, position the cursor on the entry and hit the SPACE BAR.
Once one or more items have been selected, subsequent Formatted Log commands (in the View pulldown menu) only display entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display entries of the following types" has a checkbox beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by putting the cursor on the appropriate "X" and hitting the SPACE BAR.
The I/O Error Entry types have a separate section which is controlled by radio buttons. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries, select the radio button labeled "No I/O Errors" by placing the cursor in the angle brackets and hitting the SPACE BAR. If you wish to filter on a specific qualifier you can select the radio button labeled "Filter I/O Errors" and then make your selections from the choices below by highlighting the items you wish to use for filter criteria.
To clear all selected items you can hit the Defaults function key. To clear a specific highlighted item, press the SPACE BAR.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
Pressing the Defaults function key resets the selections to all entries in the file.
The Clear Log command in the Memory pulldown menu will notify the memory logging daemon to create a new memory log file.
The Set Timer command in the Memory pulldown menu allows you to change the interval at which the memory logging daemon polls the memory for errors.
The View Dealloc command in the Memory->Report Log pulldown menu allows you to display the deallocated page information from the memory log file.
The View Detail command in the Memory->Report Log pulldown menu allows you to display the entire contents of the memory log file.
The Chassis BootLog command allows you to display the entire contents of a chassis code boot log file (ccbootlog.[1-10]).
The boot log contains mostly informational log entries output by the system as it boots. If the system is experiencing problems booting, this log may contain useful information about the problem.
The boot log is created by the logging daemon (diaglogd). Log entries can come from various system hardware, software and firmware modules.
Chassis logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
The Chassis ErrorLog command allows you to display the entire contents of the chassis code error log file, ccerrlog.
The chassis code error log contains entries for chassis codes reported with an alert level indicating that an error has occurred.
For instance, chassis codes are logged if problems occur with the power supplies. By examining the error log, you could see a record of these power supply problems.
The error log is created by the logging daemon (diaglogd). Log entries can come from various system hardware, software and firmware modules.
Chassis logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
The General help command in the Help pulldown menu displays online help for the Logtool Utility (the help system currently being viewed).
For online help on STM as a whole, use the Help pulldown menu in the STM main screen.
The Version command in the Help pulldown menu displays version and copyright information for the Logtool Utility.
Pressing Control-C (CNTL-C) allows you to interrupt or abort execution of tool, or to escape to the user interface. After you press Control-C, the following function keys are displayed:
The ESCAPE TO UI function key allows you to disconnect from Logtool and return to the main UI (User Interface) command level. To reconnect to Logtool, select the Attach to Tool menu item in the Tools->Tool Management pulldown menu.
The INTERUPT function key allows you to interrupt Logtool.
When Logtool receives the interrupt, it stops whatever it is doing and asks you if you want to continue with the operation. If the reply is "Continue", Logtool resumes what it was doing at the time of the interrupt. If the reply is "Abort", it terminates the operation and returns to the Logtool Utility main menu.
Logtool may not immediately receive the interrupt as it could be in the middle of an operation that should not or could not be interrupted. The interrupt is received as soon as Logtool completes the operation.
Logtool will receive and ignore user interrupts when it is already in the process of returning to its main menu.
The ABORT function key allows you to abort Logtool.
Logtool will be requested to abort and the state of Logtool's execution in the system map will be set to ABORT_PENDING. When Logtool actually exits, its state will be set to ABORTED.
The CONTINUE function key allows you to continue with Logtool execution. Logtool will continue execution just as if the Control-C keys had not been pressed.
The shortcut commands described below were implemented beginning with version 2.0 of Logtool.
When entering Logtool, there are several "shortcut" commands that can be invoked. The shortcut commands automatically select the current raw log file (*raw.cur) as the basis for all of their operations. If there is not a current raw log file to use, the user is informed that they need to select a file.
In addition, the Select Raw and Select Formatted functions automatically display the file's summary information once a file has been selected.
Once a file has been selected, the raw log summary is automatically displayed.
Defaults to the current raw log file, if there is one, and displays the raw log summary.
Defaults to the current raw log file, if there is one, and displays the raw log file. The header portion of the file entries is displayed in text; the data portion is displayed in hex.
Defaults to the current raw log file, if there is one, and presents the dialog to the user.
Defaults to the current raw log file, if there is one, and presents the dialog to the user.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log summary.
Once a file has been selected, the formatted log summary is automatically displayed.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log summary.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log.
Defaults to the current raw log file, if there is one, automatically formats the file, and presents the dialog to the user.
Throughout this document there are references to multiple file selection capabilities, and the behaviors when multiple files have been selected.
The handling of multiple files was implemented beginning with version 3.0 of Logtool.
See the specific commands for information regarding multiple file handling. In general, multiple files may be selected in Select Raw or Select Formatted in the File pulldown menu. Once selected, all entries from all the selected files are combined together and treated as one file for viewing, filtering and formatting.