The choice of tracing tools for kernel-mode programming is more limited. Kernel-mode debuggers, such as SoftIce, let you log breakpoints instead of pausing, but this feature is often too slow to be useful. Win9x provides a documented method for hooking VxD API functions, as Russinovich and Cogswell showed in their article “Examining VxD Service Hooking.”[1] But NT provides no documented method for intercepting kernel-mode system calls. You can use a filter driver to monitor file system activity, as the same authors showed in “Examining the Windows NT Filesystem”[2]. In “Windows NT System-Call Hooking”[3], they described another technique for monitoring some NT system calls, by replacing entries in the NT service table. This table keeps some 200 pointers to kernel-mode API functions, which NT makes accessible from user mode. Some of these functions are also available for drivers (the ZwXxx() API). The authors presented a utility, REGMON, which illustrated this technique to monitor some system calls. It is available at their site at www.sysinternals.com.

Filter drivers and service table interception let you monitor file I/O and registry activity, but there are many more kernel-mode API functions which can be interesting to intercept (ntoskrnl.exe exports about 1,000 functions). For example, a device driver developer may need to intercept and trace calls to IoAllocateIrp() and IoCallDriver(). Intercepting ExAllocatePool() and related functions would let you monitor memory allocation. Watching for HalBeginSystemInterrupt() and READ_PORT_x/WRITE_PORT_x calls might shed light on hardware activity. Fortunately, NT uses the same PE file format for kernel-mode modules as for Win32 executables and DLLs. Therefore, it’s possible to apply the same technique of API spying described by Matt Pietrek in his article “Learn System-Level Win32 Coding Techniques by Writing an API Spy Program”[4] and in his book Windows 95 System Programming Secrets[5].

In this article, I will present a kernel-mode API tracing tool for NT 4.0 running on a Pentium-based PC. It also works on Windows 2000 Beta 3. This tool can intercept any function exported from a kernel-mode module, such as ntoskrnl.exe or hal.dll. It accepts a list of functions to intercept and their parameters in a configuration file, and prints trace messages to a memory buffer. At the end, it writes the buffer to a file. The purpose of this tool is to assist in debugging and troubleshooting software under NT. The presented technique may not be suitable for commercial drivers or applications.

Kernel-Mode Modules vs. Win32 Executables

NT uses the same Portable Executable (PE) file format for kernel-mode system components, device drivers, and Win32 modules. Usually drivers have the extension .sys and user-mode modules use .exe and .dll. There are a few exceptions, such as the kernel-mode system components ntoskrnl.exe and hal.dll.

Like Win32 modules, drivers use dynamic linking. Most of the drivers import API functions from NTOSKRNL and HAL. These two system modules also import functions from each other. Some drivers export functions that other drivers may import and use. For example, ndis.sys exports functions for network miniport and protocol drivers. Kernel-mode dynamic linking uses the same import and export tables inside the PE file as user-mode DLL linking. NTOSKRNL has routines for loading and unloading the kernel modules (also called images). Unfortunately, none of these functions is documented. There are no documented kernel-mode equivalents to functions such as LoadLibrary(), FreeLibrary(), GetModuleHandle(), and GetProcAddress() — the crucial functions for user-mode dynamic linking.

The documented way to load and unload drivers is using user-mode functions in advapi32.dll, such as OpenService(), StartService(), and ControlService(). In the kernel world, I do not know any documented way to obtain a list of loaded modules, but the undocumented ZwQuerySystemInformation() can provide it. There is no easy way to be notified about loading and unloading of new modules in NT 4.0. (Windows 2000 has a new API function PsSetLoadImageNotifyRoutine().) NT 4.0 has debugger support using interrupt 0x2D. As Jose Flores showed in his article “Monitoring NT Debug Services”[6], you can hook this interrupt to get notifications of kernel-mode events that include image loading and unloading, as well as debug string output.

You can use Visual C++ to produce drivers, not just user-mode executables and DLLs. You can use other tools, such as dumpbin.exe, editbin.exe and depends.exe, to examine and modify any of these modules. To build kernel-mode components, you’ll need the NT DDK; you can download it from www.microsoft.com/ddk.

There are some differences between the structure of a driver and a DLL. To build a driver, you must use the “/subsystem:native” linker option. Another option, “/driver”, invokes some optimizations. The most important effect of this option (from the point of view of this article) is the addition of a new section called “INIT”. This section combines several “.idata” sections, which contain a list of IMAGE_IMPORT_DESCRIPTOR structures and the names of imported functions and modules. This “INIT” section is marked as discardable to let the NT image loader throw it away after locating the imported function addresses. This optimization may save a few bytes of valuable non-paged memory, where the “.idata” sections will end up by default without the “/driver” switch. Unfortunately, it creates a significant problem in the design of the API tracer, because the data necessary to locate the import tables is not in memory.

Kernel-Mode Tracer Requirements

The tracer should be able to intercept and monitor functions calls between NT kernel-mode modules. It will intercept only calls made using import and export tables. Other methods, such as direct calls using pointers or calls though the service table (discussed earlier) will not be covered.

The tracer should accept a configuration file supplied by the user. This file will describe API functions to be intercepted: the name of the function, the name of the module that exports it, and parameters.

When the tracer detects a call to the intercepted function it should print the name of the function and the values of its parameters. At user request, it may print some additional information, such as the current IRQL, the current thread ID, etc. The tracer should be able to intercept the function’s return and print the return value and output parameters.

The output can be directed to a memory buffer and/or to a debug monitor. The tracer should dump the buffer to a file at the end of the session. For simplicity and efficiency, the tracer will not try to display the output in a GUI window.

The tracer will not differentiate between calls made by different drivers. The tracer will print all calls made by various drivers in the context of different processes and threads to the same output stream. It should be able to work at interrupt time, too.

The Tracer Design

The technique of API interception for Win32 modules was originally published by Matt Pietrek, and I used it in my COM tracer article in the July 1999 WDJ[7]. The NT kernel-mode tracer presented here will have a similar structure with necessary modifications to adapt to the new environment. As with other API spies, the tracer will have two components: an application and an interceptor module. Unlike user-mode tracers, the interceptor component is packaged as a kernel-mode driver, not as a user-mode DLL.

The application will read and parse the configuration file and store the results in binary form in another file. That binary file will contain the current settings of the program and an array of stubs for intercepting individual functions. Each function will have a correspondent stub. This stub should contain information about the function name, module, parameters, and a small piece of code, which should transfer control to a single interceptor function. After processing the configuration file, the application will register the driver using the service API. It will let the user start the driver immediately or later, using the control panel’s Drivers applet. If the user decides to proceed immediately, the application will start the driver and display a message box. At that time, the driver will perform actual tracing of the API calls. When the message box is dismissed by the user, the application will terminate the driver and, optionally, unregister it.

The major difference between this kernel tracer application and Pietrek’s user-mode API spy is the absence of a Win32 debugger component. Pietrek’s user-mode spy used the Win32 debugging API to launch an application and receive notifications about important events, such as module load and unload. The kernel-mode tracer does not launch any application. It simply starts the driver, which performs tracing in the context of all applications. The lack of convenient notifications about loading and unloading of drivers causes some shortcomings in the tracer functionality.

The interceptor component, packaged as a kernel-mode driver, reads the binary configuration file, prepared by the application. Then it enumerates all loaded kernel-mode modules. For each module, it looks through the import and export table and intercept functions specified in the configuration file.

The first obstacle to developing the kernel tracer was the lack of a documented way to enumerate modules. User-mode Win32 programs could use VirtualQuery(), the ToolHelp32 API, or PSAPI. From kernel mode, however, the only solution I know is the undocumented ZwQuerySystemInformation(). Some examples of using this function can be found on the Web. This function returns an array of structures, each of which contains the name and base address of a kernel-mode module, allowing you to enumerate all loaded drivers and system components.

The second problem is getting notified when NT loads or unloads a module. Since I want the code to run under NT 4.0, I can’t rely on the new W2K routine PsSetLoadImageNotifyRoutine(). There are several possible alternatives. First, I could take over interrupt 0x2D, as demonstrated in Jose Flores’ “Monitoring NT Debug Services”[6]. Second, I could intercept DbgLoadImageSymbols(), an undocumented function that NTOSKRNL exports. Unfortunately, the regular method of interception, patching the import table, doesn’t work here, because NTOSKRNL calls this function from itself using a direct call instruction. It may be possible to scan the code sections of NTOSKRNL looking for this instruction. Another way is to insert a breakpoint instruction (INT 3) at the beginning of DbgLoadImageSymbols() and install a handler for this interrupt. The final way is to use the technique presented by Galen Hunt and Doug Brubacher[8] (available at www.research.microsoft.com/research/sn/detours/). Their idea is to place a jump instruction at the beginning of the target function. This instruction will redirect control to a “detour” function, which will later call a “trampoline” function. The trampoline contains the original few instructions of the target and jumps to the remainder of the target. This method requires a simple disassembler to recognize the instructions.

While writing a prototype of the tracer, I implemented several of these techniques for detecting a new module load. They all worked, but I decided to drop them from the final version, because some of them appeared to be dangerous, unreliable, or too complex to implement. Instead I decided to replace function addresses in the export tables of already-loaded modules. When NT loads a new driver, it copies addresses of imported functions from export tables of other modules. If the tracer replaces the exported addresses, then all new drivers will receive the substituted addresses automatically. This lets me trace calls to any module loaded before the tracer itself even if the call is made by a driver that starts later. But if the new driver provides its own API, it will not be intercepted.

This means you have to load the tracer after any modules you want to spy on. Unfortunately, some calls may be missed. If this is a problem, you can implement one of the other methods of detecting module load. Another problem with export table patching is the danger that some drivers may obtain and preserve some of the modified addresses and use them after the tracer terminates, causing a system crash. This is not normally a problem, because there is no kernel-mode equivalent to GetProcAddress(). Unless the driver uses some tricks to get the address of the function from another module, the tracer will be able to restore all replaced addresses. If this leads to crashes, the simplest solution is to prevent the tracer from unloading.

The third challenge in the tracer design is getting control after the original function returns. Win32 API spies allocate private stacks for each thread and store them in thread local storage. To receive control after the target function returns, the spy replaces the return address with its own interceptor routine and stores the original address in its own stack. The interceptor also uses these private stacks to store pointers to output parameters of the function, to print them after the function returns. Another benefit is an ability to indent function names in the output trace according to the depth of recursion.

Unfortunately, there’s no kernel-mode API that provides thread local storage. Even if such an API existed, it would be useless in many situations when the current thread is undefined. Therefore, I decided to allocate a simple array of return stubs, instead of using per-thread stacks. Such a stub keeps the original return address, pointers to the return parameters, and a call instruction to the common return entry point. To intercept the function’s return address, the interception function needs to search the array of stubs for the empty slot. After that, the interceptor stores the call information in the stub, changes the return address on the stack to point to the stub, and calls the target function. After the target function returns, the stub transfers control to the common return entry point, which prints the return values and releases the stub. I use the InterlockedXxx() functions to ensure that access to the array of stubs is both thread- and interrupt-safe.

The fourth obstacle to intercepting imported functions is the absence of a key data structure, IMAGE_IMPORT_DESCRIPTOR, from the memory image of kernel-mode modules. These modules are compiled with “/driver” linker options, causing this structure to be placed in a discardable “INIT” section. The import table itself is not discardable, because it is used all the time to make calls to the imported functions. The problem is to locate the table. The only solution I have found is to read the IMAGE_IMPORT_DESCRIPTOR from the module’s executable file on disk. This means that the tracer has to locate, open and read several bytes from the file for all loaded kernel modules and drivers. This solution may slow down the process of loading the tracer. In some rare cases it may fail if the tracer cannot find the executable file or the file was modified after the driver was loaded.

Implementation

The kernel tracer is structured like the Win32 COM tracer I presented in my previous article[7]. The launcher application remains similar, but the former interceptor DLL is now repackaged as a driver. I continued to use C++, which is usually not recommended for drivers. I used it as a “better C” and avoided some features, like virtual functions and new and delete operators. Drivers do not include standard C or C++ libraries, but NTOSKRNL provides replacements for many C library routines, such as sprintf() and _strnicmp(), and even supports exceptions. It is a great improvement in comparison with my Win32 interceptor DLL, which did not use the C library at all.

Another frequently mentioned reason to avoid C++ in drivers is the limited amount of stack size. In my case, the limited use of C++ does not introduce a significant increase in stack consumption. The biggest stack consumption comes from local text buffers and related structures, which the tracer creates in order to print the function name and its parameters. Another offender was _snprintf(), used to format the trace messages. The stack consumption dropped to about 400 bytes after I moved text buffers from the stack to the return stub structure and eliminated calls to _snprintf() in the most common cases. In some situations, for example to print an error message, the program can still use _snprintf().

The complete source code is available in this month’s code archive, but I will describe here how the work is divided. Some modules responsible for parsing the configuration file and preparing the output binary file (parsecfg.cpp, intermap.cpp, parmtype.cpp, and cfgmgr.cpp) remain practically unchanged from my previous article[7]. Two new parameter types, “UNICODE_STRING” and “ANSI_STRING”, were added. I made some enhancements in several modules while dropping COM-specific functionality. One of them is trace.cpp, which generates different kinds of output messages. Some of them are useful for debugging, and others produce the main output of the API interceptor. The Win32 tracer supported trace to a debug monitor, to a file, and to a message box. The kernel tracer does not support output to a file. Instead, I added the ability to dump trace messages to a memory buffer and to the NT error log. The memory buffer is the fastest way to generate the trace. The program can use it at any time (even at interrupt time). The tracer can use the error log for error messages. The program can also print traces to the debug monitor. Formatting of names and parameters of intercepted functions is done in log.cpp, which also contains the main interceptor entry point. return.cpp contains ReturnMgr, the class that maintains an array of return stubs and provides an entry point that gets control after the intercepted function return. krnltrcd.cpp contains SpyGlobalData, the class that initializes and manages the global settings in the driver, including the configuration file.

The following modules participate in the actual business of interception. pehelper.cpp includes routines for dealing with the internal structures of a PE executable. It provides such useful functions as bGetModuleFileName() and pGetProcAddress() (see Figure 1). The Interceptor class in Intrcpt.cpp is responsible for enumerating modules, import and export tables, and actual replacement of function addresses. One function, iInterceptImportedFunctionsInModule(), is shown in Figure 2. This function reads import descriptors from executable files and replaces function addresses in the import tables. The StubManager class in stubmgr.cpp manages the configuration data and interception stubs.

The following modules and classes are new to the kernel tracer project. drvinit.cpp contains standard routines for the NT device driver, such as DriverEntry(), TracerUnload(), and TracerDispatch(). The EnumModules class in enummods.cpp is a wrapper around ZwQuerySystemInformation(). It provides a facility to enumerate the loaded kernel modules. One of its methods, GetModuleHandle(), is an analog of the Win32 function of the same name. filesupp.cpp contains MyFileImpl, a class that wraps some file access API functions under Win32 and kernel mode. krnltrc.cpp is the main module of the application. InstallDriver in instdrv.cpp is the class the application uses to register, start, and stop the driver.

Usage

You have to create a configuration file to instruct the program which functions to intercept and how to print function parameters. The syntax for describing a function is:

EXP:ModuleName:FunctionName
VOID

or

EXP:ModuleName:FunctionName
Param1
...
ParamN

Each parameter may be one of the types shown in Table 1.

A line containing a parameter type may also be followed with one or more modifiers that affect how the parameter is interpreted. If the parameter is a pointer, place an “*” after the parameter type. An “OUT” indicates that the parameter is passed from the function to the caller. Such a parameter should be a pointer (e.g., DWORD * or LPWSTR). An “IN” indicates that the parameter is passed from the caller to the function. This is the default type. You can use both “IN” and “OUT” to declare that the parameter passes data both directions.

Besides the configuration file, you also need a file called krnltrc.ini. It should contain only one section, called “[Options]”. Table 2 shows the types of entries it can contain.

When the configuration file and the .ini file are ready, you can start the tracer. It is important to remember that it will operate in kernel mode and perform some unusual activities, such as using undocumented functions and replacing function pointers in the NT kernel. While blue screens are frequent guests for most NT programmers (even if they do only user-mode stuff), use of the tracer can increase the possibility of a system crash. Therefore, it is reasonable to make an emergency repair disk and reboot the computer before starting the tracer — use this software at your own risk!

After you start krnltrc.exe, if it can parse the configuration file, it will display a message box asking whether you want to start the tracer. If you click “Yes”, the program starts the driver and displays another message box with the message “Working”.

To stop the trace, just click the “OK” button. The program will ask whether to unregister the driver. Click “Yes” and the program will unload and unregister the interceptor driver. After that the program will launch notepad.exe with the trace file and terminate.

A Tracing Example

As an example of using the tracer, I prepared a small configuration file called hardware.cfg, a portion of which appears in Figure 3. It intercepts an internal NT routine, HalBeginSystemInterrupt(), which appears to be always called by an internal interrupt handler before calling a driver’s registered interrupt service routine. The interrupt vector number is passed as a parameter. hardware.cfg also intercepts the READ_PORT_x and WRITE_PORT_x family of functions, which are documented and exported from hal.dll. This lets you trace most interrupts and port accesses on a computer.

Figure 4 shows an excerpt from actual output. It contains traces produced when I typed “WDJ” on the keyboard (after deleting a lot of unrelated lines). The second parameter of HalBeginSystemInterrupt() function is the interrupt number. On my computer it is equal to 0x61 for the keyboard IRQ 1. In response to every interrupt, the keyboard driver reads a status byte from port 0x64 and a data byte from port 0x60. The data byte is the scan code of the pressed key. When the key is released, another interrupt arrives and the driver reads the new scan code, which is equal to the original one plus 0x80. The scan codes are: ‘W’ - down = 0x11, up = 0x91, ‘D’ - down = 0x20, up = 0xA0, ‘J’ - down = 0x24, up = 0xA4.

Overhead

It is important to check the overhead of the tracer because intercepting key system routines, such as an interrupt routines shown in the previous example, could slow down the entire system. I used the RDTSC instruction to obtain the processor’s 64-bit internal time stamp counter. In stat.h (Listing 1) I implemented a simple class that can store the time stamp and later determine the elapsed time. It accumulates the total time spent, as well as minimum and maximum times, in static structures. I used these measurements inside the interceptor routines as well as from a separate test driver. According to my calculations, the tracing of one function without parameter printing and without return interception takes about 1,500 clock cycles (between 3 and 4 microseconds) on a Pentium III 450. The tracing with one DWORD parameter and return interception takes about 4,600 cycles (11 microseconds). It may be possible to reduce the overhead by implementing a custom interception entry point for tracing specific functions and by using custom logging instead of the generic tracing mechanism.

Shortcomings

The tracer is not able to print structures passed as function parameters. This feature is needed to print such important structures as DRIVER_OBJECT, DEVICE_OBJECT, IRP, etc. It would not be hard to extend the configuration file parser and parameter printing module to support structures.

The tracer needs to read the binary configuration file and executable files for all loaded drivers. This makes it impossible to use the tracer during the early stages of the NT boot process. The tracer can start only after the file system is loaded. It may be possible to solve this problem by reading all necessary information in advance and storing it in the registry.

There is no information about callers of the functions in the output log. It might be easy to store a global instance of the EnumModules class with the list of loaded modules. In the interceptor, it would be necessary to search the list to locate the name of the calling module. Unfortunately, this mechanism will fail to display names of drivers loaded later. In the design section, I described several schemes for receiving notifications about loading new modules which can be implemented to update the module list.

The tracer uses the method of interception of imported functions, which works only for calls made from one module to another. It will not see many other important calls, which are made inside the modules. For example, the tracer cannot be used to spy for registry access, which system call hooking program can intercept.

Conclusion

The kernel tracer is a valuable addition to my set of debugging tools. It lets me observe the dynamics of processes going on in the heart of NT, such as interrupts, port accesses, calls between drivers, etc. Still, there is some hidden activity happening inside the system components and drivers, which the tracer cannot intercept. It may be possible to combine the kernel tracer presented here with the system-call hooking published by Mark Russinovich and Bryce Cogswell and the detours technology designed by Galen Hunt and Doug Brubacher. Such a combined tool would eliminate the last remaining blind spots from the field of view of an NT developer.

References

[1] Mark Russinovich and Bryce Cogswell. “Examining VxD Service Hooking,” Dr. Dobbs Journal, May 1996.

[2] Mark Russinovich and Bryce Cogswell. “Examining the Windows NT Filesystem,” Dr. Dobbs Journal, February 1997.

[3] Mark Russinovich and Bryce Cogswell. “Windows NT System-Call Hooking,” Dr. Dobbs Journal, January 1997.

[4] Matt Pietrek. “Learn System-Level Win32 Coding Techniques by Writing an API Spy Program,” Microsoft Systems Journal, December 1994.

[5] Matt Pietrek. Windows 95 System Programming Secrets, IDG Books, 1995.

[6] Jose Flores. “Monitoring NT Debug Services,” Windows Developer’s Journal, February 2000.

[7] Dmitri Leman. “Spying on COM Objects,” Windows Developer’s Journal, July 1999.

[8] Galen Hunt and Doug Brubacher. “Detours: Binary Interception of Win32 Functions,” Proceedings of the 3rd USENIX Windows NT Symposium. Seattle, WA, July 1999.

About the Author

Dmitri Leman is a software engineer in Silicon Valley. He has been developing Windows, Windows NT, and DOS applications and device drivers for eight years. He can be contacted at DmitriL@Pacbell.net.