The usage of kdump

Notes about Linux kernel kdump.

1. motivation

有时候Oops发生的同时系统也会宕机，此时这些出错信息是来不及存入文件中的，关掉电源后就无法再看到了。我们只能通过其他的方式来记录：手抄或者拍照。

还有更坏的情况，如果Oops信息过多的话，一页屏幕显示不全，我们怎么来查看完整的内容呢？第一种方法，在grub里用vga参数指定更高的分辨率以使屏幕可以显示更多的内容。很明显，这个方法其实解决不了太多的问题；第二种方法，使用两台机器，把调试机的Oops信息通过串口打印到宿主机的屏幕上。但现在大部分的笔记本电脑是没有串口的，这个解决方法也有很大的局限性；第三种方法，使用内核转储工具kdump把发生Oops时的内存和CPU寄存器的内容dump到一个文件里，之后我们再用工具来分析问题。

2. Overview

Kdump uses kexec to quickly boot to a dump-capture kernel whenever a dump of the system kernel’s memory needs to be taken (for example, when the system panics). The system kernel’s memory image is preserved across the reboot and is accessible to the dump-capture kernel.

You can use common commands, such as cp, scp or makedumpfile to copy the memory image to a dump file on the local disk, or across the network to a remote system.

When the system kernel boots, it reserves a small section of memory for the dump-capture kernel. This ensures that ongoing Direct Memory Access (DMA) from the system kernel does not corrupt the dump-capture kernel. The kexec -p command loads the dump-capture kernel into this reserved memory.

With the dump-capture kernel, you can access the memory image through /proc/vmcore. This exports the dump as an ELF-format file that you can write out using file copy commands such as cp or scp. You can also use makedumpfile utility to analyze and write out filtered contents with options, e.g with ‘-d 31’ it will only write out kernel data. Further, you can use analysis tools such as the GDB and the Crash tool to debug the dump file.

3. System kernel config options

There are two possible methods of using Kdump.

Build a separate custom dump-capture kernel for capturing the kernel core dump.
Or use the system kernel binary itself as dump-capture kernel and there is no need to build a separate dump-capture kernel. This is possible only with the architectures which support a relocatable kernel. As of today, i386, x86_64, ppc64, ia64, arm and arm64 architectures support relocatable kernel.

本文选择的是第二种方法。

system kernel config options配置如下：

CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_KEXEC_CORE=y
CONFIG_CRASH_CORE=y
CONFIG_SYSFS=y
CONFIG_DEBUG_INFO=y

4. Setup and Installation

4.1 Ubuntu

参考: ubuntu Kernel Crash Dump

1	sudo apt install linux-crashdump -y

wget https://github.com/crash-utility/crash/archive/refs/tags/7.3.1.tar.gz
tar xzvf 7.3.1.tar.gz
cd crash-7.3.1

sudo apt install bison libz-dev libncurses5-dev libncursesw5-dev texinfo -y

make
sudo make install

sudo reboot

5. 强制内核崩溃

1 2	echo 1 > /proc/sys/kernel/sysrq echo c > /proc/sysrq-trigger

重启后可以在/var/crash/目录下看到vmcore日志文件。

$ cd /var/crash/
$ ls
202112241159  kdump_lock  kexec_cmd
$ cd 202112241159
$ ls
vmcore.202112241159
$ file vmcore.202112241159
vmcore.202112241159: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style

6. Kdump analysis using crash

1	crash vmcore.202112241159 /usr/lib/debug/lib/modules/5.15.0-rc6-virt-ui+/vmlinux

crash> bt
PID: 1579   TASK: ffff8d6087084f80  CPU: 0   COMMAND: "bash"
 #0 [ffffae2c80fc7cc0] machine_kexec at ffffffffa9061348
 #1 [ffffae2c80fc7d20] __crash_kexec at ffffffffa91bc670
 #2 [ffffae2c80fc7de8] panic at ffffffffa9f301f0
 #3 [ffffae2c80fc7e68] sysrq_handle_crash at ffffffffa98327a6
 #4 [ffffae2c80fc7e70] __handle_sysrq.cold at ffffffffa9f62e96
 #5 [ffffae2c80fc7ea0] write_sysrq_trigger at ffffffffa98331d4
 #6 [ffffae2c80fc7eb0] proc_reg_write at ffffffffa93e83c3
 #7 [ffffae2c80fc7ec8] vfs_write at ffffffffa934dc2d
 #8 [ffffae2c80fc7f00] ksys_write at ffffffffa934df45
 #9 [ffffae2c80fc7f38] do_syscall_64 at ffffffffa9fa9553
#10 [ffffae2c80fc7f50] entry_SYSCALL_64_after_hwframe at ffffffffaa00007c
    RIP: 00007f5ab67f5224  RSP: 00007ffe5b9e7038  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f5ab67f5224
    RDX: 0000000000000002  RSI: 0000561b7a365e00  RDI: 0000000000000001
    RBP: 0000561b7a365e00   R8: 000000000000000a   R9: 0000000000000001
    R10: 000000000000000a  R11: 0000000000000246  R12: 00007f5ab6ad1760
    R13: 0000000000000002  R14: 00007f5ab6acd2a0  R15: 00007f5ab6acc760
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

7. MISC

crashkernel syntax
Configure Dump Location
Configure Core Collector

Configure Dump Location和Configure Core Collector可以参考How to use kdump for Linux Kernel Crash Analysis。

参考资料:

文章目录