https://coscup.org/2021/en/session/39M73K https://www.youtube.com/watch?v=L_Gyvdl_d_k Engineers have plenty of debug tools for user space programs development, code tracing, debugging and analyzing. Except “printk”, do we have any other debug tools for Linux kernel development? The “KGDB” mentioned in Linux kernel document provides another possibility. Will share how to experiment with the KGDB in a virtual machine. And, use GDB + OpenOCD + JTAG + Raspberry Pi in the real environment as the demo in this talk. 開發 user space 軟體時，工程師們有方便的 debug 工具進行查找、分析、除錯。但在 Linux kernel 的開發，除了 printk 外，還可以有哪些工具可以使用呢？從 Linux kernel document 可以看到 KGDB 相關的資訊，提供了在 kernel 除錯時的另一個可能性。 本次將分享，從建立最簡單環境的虛擬機機開始，到實際使用 GDB + OpenOCD + JTAG + Raspberry Pi 當作展示範例。

1. Let's trace Linux Kernel
with KGDB
潘建宏 Jian-Hong Pan (StarNight)
@ COSCUP 2021

2. Who am I
潘建宏 / Jian-Hong Pan (StarNight)
Endless OS Foundation
You can find me at
● http://www.slideshare.net/chienhungpan/
● GitHub: starnight
● Email:
jhp [AT] endlessos.org
chienhung.pan [AT] gmail.com

3. Outline
● How to debug Linux kernel
● KGDB
○ The Linux kernel build config
● KGDB with VM (QEMU)
● KGDB with real system (Raspberry Pi 4B)
○ How does Raspberry Pi 4B boot kernel
○ Raspberry Pi 4B’s JTAG
○ OpenOCD & the JTAG adapter
○ Add U-Boot to catch up kernel boot
○ Real debug
● Reference

4. How to Debug Linux kernel???
● Read the kernel message log, then grep the key words in kernel
source codes.
● Add some more debug messages with printk like functions in
interesting parts.
● Kprobes, ftrace, eBPF ....
● Export the signals to GPIO pins!!!???
Basic idea is “trace the code path to understand why it goes here, there.”
Can kernel be debugged with GDB like the user space programs?

5. KGDB
The kernel debugger kgdb, hypervisors like QEMU or JTAG-based
hardware interfaces allow to debug the Linux kernel and its modules
during runtime using gdb.
- from Debugging kernel and modules via gdb
Host Target Machine
Debug Line
The debug line could be professional like
JTAG, or simple one like serial.

6. Build Linux Kernel on Host
Build the kernel with following configuration (at least)
● # CONFIG_RANDOMIZE_BASE is not set
● CONFIG_DEBUG_INFO=y
● # CONFIG_DEBUG_INFO_REDUCED is not set
● CONFIG_GDB_SCRIPTS=y
● CONFIG_FRAME_POINTER=y

7. Start with Virtual Environment - QEMU
Host Target Machine
(QEMU VM)
Debug Line
built kernel
Install

8. Install the Built Kernel into the Target Machine (VM)
Have a QEMU guest VM with an OS (for example, Debian) as the target
machine
1. Start the VM with 9p shared folder (on Host)
$ qemu-system-x86_64 -m 2048 -smp 2 qemu-images/debian.qcow2 -enable-kvm -virtfs
local,path=linux-stable/,mount_tag=host0,security_model=mapped-xattr,id=host_share
2. Create a folder for mounting the shared folder from host (on guest
VM)
$ mkdir linux-stable && sudo mount -t 9p -o trans=virtio host0 linux-stable/
3. Installed the built kernel into the guest VM (on guest VM)
$ cd linux-stable && sudo make install modules_install

9. Debug the Kernel on Target Machine (VM)
Host Target Machine
(QEMU VM)
Debug Line
built kernel gdb connection

10. Enable the gdb stub of QEMU/KVM
● Start the guest VM with enabled gdb stub (on Host)
$ qemu-system-x86_64 -m 2048 -smp 2 qemu-images/debian.qcow2 -s -S
● Must disable kvm
● Manual of qemu-system-x86_64:
-s
...
freeze CPU at startup (use 'c' to start execution)
-gdb dev accept gdb connection on 'dev'. (QEMU defaults to
starting the guest without waiting for gdb to connect;
use -S too if you want it to not start execution.)
-S shorthand for -gdb tcp::1234

11. Use GDB on Host
1. Start gdb in the linux project folder
$ cd /path/to/linux-stable && gdb vmlinux
2. Attach to the booted geust VM
(gdb) target remote localhost:1234
3. Use gdb: set break points, continue, debug …
# Add a break point for test
(gdb) break drm_core_init
Breakpoint 1 at 0xffffffff82a7ebf0: file drivers/gpu/drm/drm_drv.c, line 1116.
# Continue the kernel
(gdb) continue
Continuing.
Thread 1 hit Breakpoint 1, drm_core_init () at drivers/gpu/drm/drm_drv.c:1116
1116 drm_connector_ida_init();
(gdb) backtrace
#0 drm_core_init () at drivers/gpu/drm/drm_drv.c:1116
#1 0xffffffff81000d61 in do_one_initcall (fn=0xffffffff82a7ebf0 <drm_core_init>) at init/main.c:1205

13. Let’s debug in the Real World!
Raspberry Pi as the target machine

14. Debug the Kernel on Target Machine (Raspberry Pi)
Target Machine
(Raspberry Pi)
Debug Line
JTAG
Host
built kernel
OpenOCD

15. How Debian Boots on Raspberry Pi 4B
● Download Debain images for Raspberry Pi 4B and install it by
following RaspberryPiImages
● The image has 2 partitions. First one is the boot partition.
● Here are the files in the boot partition
$ ls
bcm2711-rpi-4-b.dtb cmdline.txt fixup_cd.dat start4db.elf start_x.elf
bcm2837-rpi-3-a-plus.dtb config.txt fixup.dat start4.elf sysconf.txt
bcm2837-rpi-3-b.dtb fixup4cd.dat fixup_db.dat start4x.elf vmlinuz-5.10.0-7-arm64
bcm2837-rpi-3-b-plus.dtb fixup4.dat fixup_x.dat start_cd.elf
bcm2837-rpi-cm3-io3.dtb fixup4db.dat initrd.img-5.10.0-7-arm64 start_db.elf
bootcode.bin fixup4x.dat start4cd.elf start.elf
Device Boot Start End Sectors Size Id Type
/dev/sda1 8192 614399 606208 296M c W95 FAT32 (LBA)
/dev/sda2 614400 124735487 124121088 59.2G 83 Linux

16. Debian’s config.txt & cmdline.txt
$ cat config.txt
# Switch the CPU from ARMv7 into ARMv8 (aarch64) mode
arm_64bit=1
enable_uart=1
upstream_kernel=1
kernel=vmlinuz-5.10.0-7-arm64
# For details on the initramfs directive, see
# https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=10532
initramfs initrd.img-5.10.0-7-arm64
$ cat cmdline.txt
console=tty0 console=ttyS1,115200 root=/dev/mmcblk1p2 rw fsck.repair=yes net.ifnames=0 rootwait

17. Enable Raspberry Pi 4B’s JTAG
● The chip is BCM2711
● Raspberry Pi 4B maps JTAG pins to GPIO
○ Setting enable_jtag_gpio=1 selects Alt4 mode for GPIO pins 22-27, and
sets up some internal SoC connections, thus enabling the JTAG interface
for the ARM CPU.
Pin # Function
GPIO22 ARM_TRST
GPIO23 ARM_RTCK
GPIO24 ARM_TDO
GPIO25 ARM_TCK
GPIO26 ARM_TDI
GPIO27 ARM_TMS
Reference: Raspberry Pi Documentation's GPIO control in config.txt

18. Table 94. GPIO Pins Alternative
Function Assignment of
BCM2711 ARM Peripherals
JTAG interface

19. Open On-Chip Debugger (OpenOCD)
● It does so with the assistance of a debug adapter, which is a small
hardware module which helps provide the right kind of electrical
signaling to the target being debugged. These are required since
the debug host (on which OpenOCD runs) won’t usually have native
support for such signaling, or the connector needed to hook up to
the target.
● A JTAG Adapter supports JTAG signaling, and is used to
communicate with JTAG (IEEE 1149.1) compliant TAPs on your target
board. A TAP is a “Test Access Port”, a module which processes
special instructions and data. TAPs are daisy-chained within and
between chips and boards. JTAG supports debugging and boundary
scan operations.
● List of Debug Adapter Hardware, or search the list of tcl/interface in
OpenOCD repository directly
Reference: OpenOCD User's Guid - About

20. Try to Buy a JTAG Adatper
● I do not have much money :( $$$$$
● It is really hard to buy and get the electronic parts, due to the
pandemic.
● I have no idea why all of the JTAG adapters come from overseas.
● The shipment becomes extremely expensive and much longer
waiting time.
So, I can only suspend this study. Sad … T^T
Until ...

21. Thanks to Mat lend me a
FT2232H-56Q Mini Module
as a JTAG interface!!!
Hacking Thursday! Hurray!!!

22. JTAG interface
Table 3.13 FT2232H Pin
Configurations for 56-pin
VQFN package of FT2232H
Dual High Speed USB to
Multipurpose UART/FIFO IC
Datasheet

23. JTAG interface
Table 3.1 FT2232H-56Q Mini
Module Connection – CN2 of
FT2232H-56Q Mini Module
Datasheet

24. Prepare JTAG Interface cfg for OpenOCD
# Refer to tcl/interface/ftdi/minimodule.cfg
#
# FTDI MiniModule
#
# http://www.ftdichip.com/Support/Documents/DataSheets/Modules/DS_FT2232H_Mini_Module.pdf
#
adapter driver ftdi
#ftdi_device_desc "FT2232H-56Q MiniModule"
ftdi_vid_pid 0x0403 0x6010
# Every pin set as high impedance except TCK, TDI, TDO and TMS
ftdi_layout_init 0x0008 0x000b
transport select jtag
# nTRST defined on pin CN2-13 of the MiniModule (pin ADBUS5 [AD5] on the FT2232H chip)
# This choice is arbitrary. Use other GPIO pin if desired.
ftdi_layout_signal nTRST -data 0x0020 -oe 0x0020

25. Execute OpenOCD with the cfgs
$ openocd -f minimodule.cfg -c "set USE_SMP 1" -f bcm2711.cfg -c "reset_config trst_only"
Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
1
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : JTAG tap: bcm2711.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x4)
Info : bcm2711.cpu0: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu1: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu2: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu3: hardware has 6 breakpoints, 4 watchpoints
Info : starting gdb server for bcm2711.cpu0 on 3333
Info : Listening on port 3333 for gdb connections
Note: the patch tcl/board: Add Raspberry Pi 4 model B board

26. But the debug connection,
including gdb and OpenOCD is too
late to catch up the kernel boot.
And, Raspberry Pi does not expose
JTAG System Reset (SRST) pin.

27. The RUN/GLOBAL_EN pin on
Raspberry Pi boards might be
implemented as the
System Reset (SRST) pin.
Re: openocd, jtag, fyi
But, OpenOCD still cannot catch up
and halt target system on time.

28. How about intercept boot with
U-Boot ...

29. 3th stage
Kernel boots
Boot Linux kernel via U-Boot on Raspberry Pi 4B
Raspberry Pi OS’ firmwares
3th stage
U-Boot loads kernel, initramfs and
device tree, then boots kernel with
boot script
1st stage
Boot from GPU with firmware in
ROM loads start4x.elf ... in FAT
2rd stage
Start4.elf loads DT,
kernel …
and starts CPU
U-Boot

30. Build U-Boot
1. Build U-Boot & install
$ make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- rpi_4_defconfig
$ make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-
$ cp u-boot.bin <RPi image’s root partition path>/boot/kernel8.img
2. Prepare the boot script for U-Boot
3. Prepare the boot script and following configuration file for U-Boot
4. Get RPi 4B’s device tree blob from Raspberry Pi OS for U-Boot
5. Modify config.txt to make boot firmware load and execute the kernel8.img

31. Prepare U-Boot script
$ cat boot.src
echo "Load u-Boot environment from uEnv.txt"
load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} uEnv.txt
env import -t ${scriptaddr} ${filesize}
echo "Load device tree from "${fdtdir}/${fdtfile}
load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${fdtdir}/${fdtfile}
echo "Load kernel from "${kernel_image}
load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${kernel_image}
echo "Load RAM disk from "${ramdisk_image}
load ${devtype} ${devnum}:${distro_bootpart} ${ramdisk_addr_r} ${ramdisk_image}
echo "Boot ..."
setenv bootargs ${bootargs}
booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}
$ mkimage -T script -C none -d boot.src <Path to image’s boot partition>/boot.scr

32. And, the uEnv.txt for U-Boot script
$ cat uEnv.txt
kernel_image=vmlinux-5.12.10
ramdisk_image=initrd.img-5.12.10
fdtdir=dtbs/5.12.10
bootargs=console=tty0 console=ttyS1,115200 root=/dev/mmcblk1p2 rw
fsck.repair=yes net.ifnames=0 rootwait

33. Modify RPi’s config.txt for Executing kernel8.img
$ cat config.txt
arm_64bit=1
enable_uart=1
enable_jtag_gpio=1

34. The New Status of the Boot Partition
$ ls
bcm2711-rpi-4-b.dtb fixup4db.dat start4x.elf
bcm2837-rpi-3-a-plus.dtb fixup4x.dat start_cd.elf
bcm2837-rpi-3-b.dtb fixup_cd.dat start_db.elf
bcm2837-rpi-3-b-plus.dtb fixup.dat start.elf
bcm2837-rpi-cm3-io3.dtb fixup_db.dat start_x.elf
bootcode.bin fixup_x.dat sysconf.txt
boot.scr initrd.img-5.10.0-7-arm64 System.map-5.12.10
cmdline.txt initrd.img-5.12.10 uEnv.txt
config.txt kernel8.img vmlinux-5.12.10
dtbs start4cd.elf vmlinuz-5.10.0-7-arm64
fixup4cd.dat start4db.elf
fixup4.dat start4.elf

36. Press any key to stop autoboot in U-Boot
U-Boot 2021.04 (Jun 20 2021 - 17:36:34 +0800)
...
scanning bus xhci_pci for devices... 3 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
U-Boot> boot
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
724 bytes read in 13 ms (53.7 KiB/s)
…
Starting kernel ...
Press a key to make U-boot paused,
then connect the GDB via OpenOCD + JTAG.
Finally, continue boot

37. Launch OpenOCD to Start GDB Server with JTAG
$ openocd -f minimodule.cfg -c "set USE_SMP 1" -f bcm2711.cfg -c "reset_config trst_only"
Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
1
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : JTAG tap: bcm2711.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver:
0x4)
Info : bcm2711.cpu0: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu1: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu2: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu3: hardware has 6 breakpoints, 4 watchpoints
Info : starting gdb server for bcm2711.cpu0 on 3333
Info : Listening on port 3333 for gdb connections

38. Debug with GDB via the OpenOCD + JTAG
$ aarch64-linux-gnu-gdb vmlinux
...
Reading symbols from vmlinux...
(gdb) target extended-remote localhost:3333
Remote debugging using localhost:3333
0x000000003b39133c in ?? ()
(gdb) continue
Continuing.
^C
...
Program received signal SIGINT, Interrupt.
mem_serial_in (p=<optimized out>, offset=<optimized out>)
at drivers/tty/serial/8250/8250_port.c:399
399 return readb(p->membase + offset);
(gdb) hbreak drm_core_init
Hardware assisted breakpoint 1 at 0xffff80001115775c: file drivers/gpu/drm/drm_drv.c, line 1044.
(gdb) continue
Continuing.
...
Continue U-Boot to boot the kernal image
Intercept/Interrupt the kernal boot at a proper point
hbreak in Setting Breakpoints

39. Debug with GDB via the OpenOCD + JTAG (cont.)
Breakpoint 1, drm_core_init () at drivers/gpu/drm/drm_drv.c:1044
1044 drm_connector_ida_init();
(gdb) backtrace
#0 drm_core_init () at drivers/gpu/drm/drm_drv.c:1044
#1 0xffff800010012e80 in do_one_initcall (
fn=0xffff80001115775c <drm_core_init>) at init/main.c:1226
#2 0xffff8000111212b4 in do_initcall_level (
command_line=0xffff000040264500 "console", level=6) at init/main.c:1299
#3 do_initcalls () at init/main.c:1315
#4 do_basic_setup () at init/main.c:1335
#5 kernel_init_freeable () at init/main.c:1537
#6 0xffff800010c5419c in kernel_init (unused=<optimized out>)
at init/main.c:1424
#7 0xffff80001001448c in ret_from_fork () at arch/arm64/kernel/entry.S:955
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

40. Summary
● Some key items for KGDB in Linux kernel building config
● Tried KGDB with QEMU easily
○ Learned steps to debug the Lnux kernel on QEMU guest OS
● Tried KGDB with real physical system Raspberry Pi 4B
○ The JTAG interface, the adapter and corresponding hardware feature
○ BCM2711 on Raspberry Pi 4B’s JTAG feature and configuration
○ Debugged the Linux kernel on Raspberry Pi 4B with KGDB via OpenOCD + JTAG
adapter
● If you have some money, please buy a professional JTAG adapter. It
has less barriers and saves your time.
● KGDB is not the only way to debug Linux kernel. The printk series
might be good enough for most of cases. Especially, timing related
issues.

41. Reference
● Debugging kernel and modules via gdb
● JTAG
● OpenOCD
● Bare Metal Raspberry Pi 3B+: JTAG
● Baremetal Raspberry Pi 4 with FT2232H
● FT2232H Dual High Speed USB to Multipurpose UART/FIFO IC datasheet
● FT2232H-56Q Mini Module Datasheet
● Raspberry Pi → Documentation → The boot folder
● Device Tree
● RPi U-Boot
● GDB → Setting Breakpoints
● JTAG Reset on Raspberry Pi forum
● A Journey to Boot Linux on Raspberry Pi