Debugging ARM kernel with kgdb in QEMU
During the development of the kernel or its modules,
it’s often necessary to debug it/them while running.
The kernel provides a built-in debugger for it called kgdb.
The problem is that the generic qemu arm64 machine has only one serial port: ttyAMA0,
which is usually already used for the console.
The solution is to add another serial port to the VM. Below I explain how to easily do it.
Starting point
Your startup script probably looks similar to mine:
HDA="-drive file=arch_aarch64.qcow2,format=qcow2"
SHARED="./share"
VIRTFS+=" --virtfs local,path=${SHARED},mount_tag=share,security_model=passthrough,id=share "
KERNEL=./linux/arch/arm64/boot/Image
CMDLINE='root=/dev/vda2 rw console=ttyAMA0'
qemu-system-aarch64
-nodefaults
-nographic
-vga none
-cpu max
-machine virt
-m 1024
${HDA}
${VIRTFS}
-serial stdio
-kernel "${KERNEL}"
-append "${CMDLINE} "It does the basic setup (disks, shared folder, use my kernel, etc.) and starts the VM in terminal mode — useful when working in a VM over ssh.
Kernel debugging
I’ll assume you know how to configure the kernel for debugging.
If not, you might want to read my article about Debugging ARM kernel with kgdb in QEMU on macOS.
We can’t reuse the serial connection we’re already using for the console, so we need to add another one through qemu.
HDA="-drive file=arch_aarch64.qcow2,format=qcow2"
SHARED="./share"
VIRTFS+=" --virtfs local,path=${SHARED},mount_tag=share,security_model=passthrough,id=share "
KERNEL=./linux/arch/arm64/boot/Image
- CMDLINE='root=/dev/vda2 rw console=ttyAMA0'
+ CMDLINE='root=/dev/vda2 rw console=ttyAMA0 kgdboc=ttyS0 kgdbwait'
qemu-system-aarch64 -nodefaults -nographic -vga none -cpu max -machine virt -m 1024 ${HDA} ${VIRTFS} -serial stdio + -serial tcp::1234,server,nowait -kernel "${KERNEL}" -append "${CMDLINE} "Now we should be able to connect to the VM with gdb:
cd ./linux
gdb ./vmlinux
(gdb) target remote :1234
:1234: Connection timed out.The Problem
With the above setup, the gdb will always time out,
as if there was no server listening on the other end.
In fact, that’s exactly what happens.
The kernel starts the kgdb server and connects it to ttyS0.
There is only one small problem: ttyS0 doesn’t exist.
According to QEMU’s documentation, the ARM virt machine has only one serial port: ttyAMA0.
So now that we know the problem, how do we fix it?
Luckily virt machine supports a PCI serial card that can be added to the VM:
qemu-system-aarch64 \
-nodefaults \
-nographic \
-vga none \
-cpu max \
-machine virt \
-m 1024 \
${HDA} \
${VIRTFS} \
- -serial stdio \
- -serial tcp::1234,server,nowait \
+ -chardev stdio,mux=on,id=char0 \
+ -chardev socket,path=/tmp/qemu_socket.sock,server=on,wait=off,id=gnc0 \
+ -mon chardev=char0,mode=readline \
+ -serial chardev:char0 \
+ -device pci-serial,id=serial0,chardev=gnc0 \
-kernel "${KERNEL}" \
-append "${CMDLINE} "Because chardev doesn’t support tcp, we switched to a unix socket.
The good news is that gdb doesn’t care if the remote is a tcp or unix socket,
so it just works:
gdb ./vmlinux
(gdb) target remote /tmp/qemu_socket.sock
Remote debugging using /tmp/qemu_socket.sock
warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread
[Switching to Thread 4294967294]
arch_kgdb_breakpoint () at ./arch/arm64/include/asm/kgdb.h:21
21 asm ("brk %0" : : "I" (KGDB_COMPILED_DBG_BRK_IMM));
(gdb) c
Continuing.You can find the full script here with the whole development setup I’m using.
The solution seems obvious once it’s found, but to come to this elegant solution I had to look for hours through different mailing lists, forums, and documentation.
So if you found this page by googling “kgdb aarch64 qemu” or something similar, you’re welcome. :D
If you are here, you might be interested in my other articles about Linux kernel development on macOS.
Big thank you to the sources that helped me find the solution: