Linux Kernel Programming

Debugging ARM kernel with kgdb in QEMU

If you came here from Google and are only interested in the kgdb setup for arm64 qemu, you can skip here or read the more technical article.

Otherwise, let’s start from the beginning.

What is kgdb?

According to Wikipedia:

KGDB is a debugger for the Linux kernel […]. It requires two machines that are connected via a serial connection. […] The target machine (the one being debugged) runs the patched kernel and the other (host) machine runs gdb. The GDB remote protocol is used between the two machines.

The highlights tell us exactly what we need to do.

Setup

Preparing the kernel

First, we need a kernel with kgdb enabled.

For this go to your kernel source directory and run:

make menuconfig

Then navigate to Kernel hacking and enable Compile the kernel with debug info and KGDB: kernel debugging with remote gdb. _{(Those settings might be in a different place depending on the kernel version.)}

Then rebuild your kernel.

Installing gdb on the host

The good news is that there is gdb build for macOS, the bad news is that it only works on Intel-based Macs. And it won’t change until someone patches it for aarch64.

Luckily, we don’t have to care about that, because Rosetta 2 exists. This means we can just run the Intel version of gdb on our Apple Silicon Mac.

If you don’t have it yet, install Rosetta:

/usr/sbin/softwareupdate --install-rosetta

Now we can start an x86 terminal to run our Intel programs. In your terminal run:

arch -x86_64 zsh

To install the gdb we need a package manager (your arm brew won’t work here):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Yes, it’s the same command as for the arm version, but because the environment is x86 it will install the Intel version of Homebrew.

if [ "$(arch)" = "arm64" ]; then
    eval "$(/opt/homebrew/bin/brew shellenv)"
else
    eval "$(/usr/local/bin/brew shellenv)"
fi

Just to be sure check you have the right brew active:

❯ which brew
/usr/local/bin/brew

Now you can install gdb:

brew install gdb

Serial connection to the target machine

After the first part, our script to start the VM looked like this:

qemu-system-aarch64 
    -machine virt  
    -cpu max  
    -m 1024  
    -drive file=arch_aarch64.qcow2,format=qcow2 
    -serial stdio 
    -kernel "<path to linux directory>/arch/arm64/boot/Image.gz" 
    -append "root=/dev/vda2"

One could think, adding another serial connection to the VM should be this easy:

qemu-system-aarch64 \
    -machine virt  \
    -cpu max  \
    -m 1024  \
    -drive file=arch_aarch64.qcow2,format=qcow2 \
    -serial stdio \
+    -serial tcp::1234,server,nowait \
    -kernel "<path to linux directory>/arch/arm64/boot/Image.gz" \
    -append "root=/dev/vda2"

The problem is that the generic qemu arm64 machine has only one serial port: ttyAMA0. Which is already used for the console. Luckily, it also supports PCI devices like a serial card, so we can add another serial port to the VM:

qemu-system-aarch64 \
    -machine virt  \
    -cpu max  \
    -m 1024  \
    -drive file=arch_aarch64.qcow2,format=qcow2 \
-   -serial stdio \
-   -serial tcp::1234,server,nowait \
+   -chardev stdio,mux=on,id=char0 \
+   -chardev socket,path=/tmp/qemu_socket.sock,server=on,wait=off,id=gnc0 \
+   -mon chardev=char0,mode=readline \
+   -serial chardev:char0 \
+   -device pci-serial,id=serial0,chardev=gnc0 \
    -kernel "<path to linux directory>/arch/arm64/boot/Image.gz" \
-    -append "root=/dev/vda2"
+    -append "root=/dev/vda2 console=ttyAMA0 kgdboc=ttyS0 kgdbwait"

Of course, we need to tell the kernel to use the new serial port for kgdb. And we’ve also added kgdbwait to tell the kernel to wait for the debugger to connect when booting.

You can find the full startup script here and more technical write-up here.

Debugging

Now we are ready to test our debugging setup.

In one terminal window start the VM:

❯ arch
arm64
❯ ./qemu-run.sh
...
<VM output>
...

In another navigate to your kernel source directory, start gdb, connect to the VM and continue the boot process:

❯ cd <path to linux directory>/linux
❯ arch -x86_64 zsh
❯ gdb ./vmlinux
(gdb) target remote /tmp/qemu_socket.sock
Remote debugging using /tmp/qemu_socket.sock
warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread
[Switching to Thread 4294967294]
arch_kgdb_breakpoint () at ./arch/arm64/include/asm/kgdb.h:21
21              asm ("brk %0" : : "I" (KGDB_COMPILED_DBG_BRK_IMM));
(gdb) c
Continuing.

If you don’t have vmlinux you have to build the kernel first.

Now the VM should boot as usual and you can stop the execution at any time with

echo 0 > /proc/sys/kernel/hung_task_timeout_secs

After pausing the execution, you can do stuff in gdb, e.g., see the system info:

(gdb) print init_uts_ns.name.release
$4 = "6.6.0-mastermakrela-g90b0c2b2edd1-dirty", '\000' <repeats 25 times>

or even change it:

(gdb) set var init_uts_ns.name.release="hello, world!"
(gdb) c

And see the change in the VM:

uname -a
Linux alarm hello, world! #3 SMP PREEMPT Sat Mar  2 00:19:49 CET 2024 aarch64 GNU/Linux