Instrumenting Kernel Subsystems for Kernel Logging

Procedure 1-1 Ascertain That an Appropriate ID is Defined for Your Subsystem

A subsystem ID is a unique string that identifies your subsystem to the KL infrastructure. It must be unique among the subsystem IDs, and evaluate to an integer in the range [513-1023]. When it is defined, as described in this procedure, your subsystem will be assigned the next available integer.

Several subsystem IDs have already been defined: they are listed, under the comment /* KL supported subsystems */, in the file/ux/core/kern/common/sys/subsys_id.h. For example:

    /* KL supported subsystems */
    #define KL_FORMATTER            512    /* The values for KL subsystems must be within 513 and 1023 */
    /* since 512 value is reserved for the KL formatter */
    #define KL_VM                   513
    #define KL_PKM                  514
    #define KL_DLKM                 515
    #define KL_PM                   516
    #define KL_VFS                  517
    #define KL_VXFS                 518
    

1. Send mail to the KL team (KL_team_NJ@fpk.hp.com) describing the code to be instrumented and discuss with them what the ID for your subsystem should be.

2. If you and the KL team decide to use a pre-existing subsystem ID for your instrumentation work, you are done with this task.

3. If you and the KL team determine that a new subsystem ID must be defined-- because there is no pre-existing subsystem ID suitable for your code, or because the pre-existing subsystem IDs are too general (for example, in a huge subsystem like VM, a subsystem ID that further subdivides it might be very useful)--then the following source files must be modified to support the new subsystem ID:

   • /ux/core/kern/common/sys/subsys_id.h -- this file must include a line for your new subsystem like the lines shown in the example in the introduction to this procedure.

   • /ux/core/lan/src/NETTRACELOG/conf/NETTL-MIN/nettlgen.conf -- this file must include a line for your new subsystem like the following sample line taken from the section of the file captioned # Subsystems supported by the Kernel Logging facility:

     SS:513:KL_VM:8:k:NULL:klfmt:NULL:NULL:Kernel Logging

     Clear explanations of the fields in this entry are given in the section of notes captioned # SUBSYSTEM RECORD, earlier in the file.

4. Copy the modified nettlgen.conf file to the /etc directory on your target machine.

5. Before you submit these modified files to the final product, notify the KL team so a final review can be done.

Procedure 1-2 Create a Local Header File for Kernel Events Specific to Your Subsystem

You will be adding instrumentation points to your kernel subsystem code wherever you want to identify an event to be logged. Each such point must be given an "event ID," which is a string that identifies that event to the KL infrastructure. Each must be unique among the event IDs for your subsystem, and is assigned an integer value in the range [1-65535]. You, not the KL team, assign integer values to event IDs in your subsystem.

1. Add the following line to both your local header file (if you have one) and your .c file:

   #include <sys/net_diag.h>

2. Construct an event ID for each instrumentation point in your subsystem. Event IDs are constructed according to the following convention:

   subsystemID_eventName

   where subsystemID is the subsystem identifier you and the KL team previously agreed upon, and eventName is an identifier for the particular event you are instrumenting. For example, the Dynamic Tuning subsystem has the ID KL_DYNTUNE. In that subsystem, one of the kernel events is setting a tunable parameter. The kernel developer has assigned the name SETTUNE_EVENT to this event. Thus, the event ID for this kernel event becomes

   KL_DYNTUNE_SETTUNE_EVENT

3. Add your event IDs to your local header file, or to your .c file, and assign each an integer value. It is good practice to keep all #define statements of a particular subsystem in one file, typically in a local header file. However, if you do not have a local header file, you can define event IDs in your .c file.

   Use the following format:

   eventID=n

   where eventID is constructed as described in Step 2, and n is the unique value assigned to this event.

   In addition to being unique to this event ID in this subsystem, the assigned integer value n also serves to specify the "event class" of the event.Four event classes for Kernel Logging are defined in the subsys_id.h file. The following table lists and defines them, and identifies the range of integer values that can be assigned in each class:

   Table 1-1 Title not available (Create a Local Header File for Kernel Events Specific to Your Subsystem)

   Event Class	Definition	Recommended Integer Values
   DISASTER	The DISASTER class signals an event or condition which affects the operation of an entire subsystem, or the kernel, causing several programs to fail or the machine to panic	1000 - 1999
   ERROR	The ERROR class signals an event or condition which does not affect the overall operation of an entire subsystem, but may cause an application to fail.	2000 - 2999
   WARNING	The WARNING class indicates an abnormal event, possibly caused by problems in an individual subsystem.	3000 - 4999
   INFORMATIVE	The INFORMATIVE class describes important routine operations and current system values.	5000 and up

    

   Continuing the example of the Dynamic Tuning subsystem, and assuming that the developer had decided the event is an INFORMATIVE event, and that it is the first event of that class in the subsystem, the entry in the local header file for this event would be as follows:

   KL_DYNTUNE_SETTUNE_EVENT=5000

Procedure 1-3 Add KL Instrumentation Points to Your Subsystem

The three macros listed in the table below are the entry points to the Kernel Logging infrastructure. (These macros represent Kernel Logging Instrumentation Points.) Use them to record the dynamic parts of the user-friendly messages that will be logged from your subsystem. User-friendly messages are discussed in the next section of this document.

1. Select the point in your subsystem code to be instrumented for Kernel Logging.

   	IMPORTANT: Do not instrument MALLOC and FREE macros: They are used in the KL infrastructure itself, and recursion will occur if you instrument them.

2. Select the KL macro you want to call, based upon the specific type of information you want to pass to the kernel log. (See the manual page at the end of this document for complete information on the use of the macros.)

3. Construct a user-friendly message for this instrumentation point, and add it to the message catalog file. (See the following procedure for details on doing this.)

Procedure 1-4 Construct a User-Friendly Message to be Associated with an Instrumentation Point

The data passed to the KL infrastructure can be combined with a user-friendly message describing the event and its implications. These user-friendly messages are composed, stored, and built by you, in a message catalog file. The steps to complete this task are given below. But first, a couple of examples.

Suppose a kernel developer instruments some part of the VFS subsystem (subsystem ID KL_VFS) and wants to log a message when there is something wrong with a file whose name is specified, for example, through an open() system call. The developer wants the full message about the event to be "Could not open foo.bar file due to I/O related error." Such a message has a static part: "Could not open ... file due to I/O related error"; and a dynamic part: filename, in this case, foo.bar. To avoid consuming memory, the static part of the message is not logged in the kernel. The dynamic part, filename, is logged, however, via the KL_LOG_STR macro with a string pointer pointing to the foo.bar string. At the same time, the following string is added to the local header file as the static part of the message associated with this instrumentation point: Could not open %s file due to I/O related error.

Or, suppose a kernel developer wants to log the following message: "I/O related error while accessing file, inode is 505, erno is 22." In this case, the instrumentation point uses the KL_LOG_INFO macro and the user-friendly message is: I/O related error while accessing file, inode is %lld, errno is %lld.

	NOTE: Because the Kernel Logging infrastructure accepts 64-bit integers and the decoder of the KL files is a 32-bit command, use %lld (or %llx, %llu, etc.) templates to display the integers.

1. Compose the user-friendly message that will be logged when this instrumentation point is reached by a running system.

2. Edit the message catalog file, /us/core/lan/src/NETTRACELOG/format/klfmt.msg, on the view server and add your message(s). The format of this file is as follows:

   $setN subsystem_ID
   n message
   .
   .
   .
   

   where N is the number and subsystem_ID is the ID assigned to your kernel subsystem by the KL team; n is the number you have assigned to an event ID, and message is the user-friendly message you have composed for that event.

   The $setn ... line signals the beginning a new set of messages, and only needs to be added if subsystem_ID is a new one.

3. Add a new n message line for each new instrumentation point you have introduced. The following example messages are from the Machine Check (KL_MC) subsystem;

   $set525 KL_MC
   1000 Machine Check Abort (MCA occurred)
   2000 Corrected Machine Check (CMC) occurred
   3000 System Abstraction Layer (SAL) returned unexpected Error Record (%s)
   3001 Memory SBE address FIFO is full. Address 0x%01611x cannot be queued
   5000 Error Record obtained from System Abstraction Layer (SAL)
   6000 Bus check occurred. check info: 0x%01611x req: 0x1611x res: 0x%1611x tgt: 0x%01611x \
   ip: 0x%01611x

4. Be in the /ux/core/lan/src/NETTRACELOG/format directory, and issue the following command to build the message catalog:

   clearmake all

   When the compilation is complete, the KL message catalog file, /ux/core/lan/obj/locales/NETTRACE/format/klfmt.cat, will include your messages.

5. Copy the klfmt.cat file to the following directory on the target machine:

   /usr/lib/nls/C