22.4. On-Line Kernel Debugging Using DDB

While gdb -k as an off-line debugger provides a very high level of user interface, there are some things it cannot do. The most important ones being breakpointing and single-stepping kernel code.

If you need to do low-level debugging on your kernel, there is an on-line debugger available called DDB. It allows to setting breakpoints, single-stepping kernel functions, examining and changing kernel variables, etc. However, it cannot access kernel source files, and only has access to the global and static symbols, not to the full debug information like gdb.

To configure your kernel to include DDB, add the option line

    options DDB

to your config file, and rebuild. (See Kernel Configuration for details on configuring the FreeBSD kernel.

Once your DDB kernel is running, there are several ways to enter DDB. The first, and earliest way is to type the boot flag -d right at the boot prompt. The kernel will start up in debug mode and enter DDB prior to any device probing. Hence you can even debug the device probe/attach functions.

The second scenario is a hot-key on the keyboard, usually Ctrl-Alt-ESC. For syscons, this can be remapped; some of the distributed maps do this, so watch out. There is an option available for serial consoles that allows the use of a serial line BREAK on the console line to enter DDB (options BREAK_TO_DEBUGGER in the kernel config file). It is not the default since there are a lot of crappy serial adapters around that gratuitously generate a BREAK condition, for example when pulling the cable.

The third way is that any panic condition will branch to DDB if the kernel is configured to use it. For this reason, it is not wise to configure a kernel with DDB for a machine running unattended.

The DDB commands roughly resemble some gdb commands. The first thing you probably need to do is to set a breakpoint:

    b function-name
    b address

Numbers are taken hexadecimal by default, but to make them distinct from symbol names; hexadecimal numbers starting with the letters a-f need to be preceded with 0x (this is optional for other numbers). Simple expressions are allowed, for example: function-name + 0x103.

To continue the operation of an interrupted kernel, simply type:

    c

To get a stack trace, use:

    trace

If you want to remove a breakpoint, use

    del
    del address-expression

The first form will be accepted immediately after a breakpoint hit, and deletes the current breakpoint. The second form can remove any breakpoint, but you need to specify the exact address; this can be obtained from:

    show b

To single-step the kernel, try:

    s

This will step into functions, but you can make DDB trace them until the matching return statement is reached by:

    n

To examine data from memory, use (for example):

    x/wx 0xf0133fe0,40
    x/hd db_symtab_space
    x/bc termbuf,10
    x/s stringbuf

for word/halfword/byte access, and hexadecimal/decimal/character/ string display. The number after the comma is the object count. To display the next 0x10 items, simply use:

    x ,10

Similarly, use

    x/ia foofunc,10

to disassemble the first 0x10 instructions of foofunc, and display them along with their offset from the beginning of foofunc.

To modify memory, use the write command:

    w/b termbuf 0xa 0xb 0
    w/w 0xf0010030 0 0

The command modifier (b/h/w) specifies the size of the data to be written, the first following expression is the address to write to and the remainder is interpreted as data to write to successive memory locations.

If you need to know the current registers, use:

    show reg

Alternatively, you can display a single register value by e.g.

    p $eax

and modify it by:

    set $eax new-value

Should you need to call some kernel functions from DDB, simply say:

    call func(arg1, arg2, ...)

The return value will be printed.

For a ps(1) style summary of all running processes, use:

    ps

Now you have now examined why your kernel failed, and you wish to reboot. Remember that, depending on the severity of previous malfunctioning, not all parts of the kernel might still be working as expected. Perform one of the following actions to shut down and reboot your system:

    panic

This will cause your kernel to dump core and reboot, so you can later analyze the core on a higher level with gdb. This command usually must be followed by another continue statement.

    call boot(0)

Which might be a good way to cleanly shut down the running system, sync() all disks, and finally reboot. As long as the disk and file system interfaces of the kernel are not damaged, this might be a good way for an almost clean shutdown.

    call cpu_reset()

is the final way out of disaster and almost the same as hitting the Big Red Button.

If you need a short command summary, simply type:

    help

However, it is highly recommended to have a printed copy of the ddb(4) manual page ready for a debugging session. Remember that it is hard to read the on-line manual while single-stepping the kernel.