Recently Google updated their bounty program for KVM (kvmCTF), so I decided to do some research on KVM.

First thing we need to do is finding a way to debug KVM. It is related to Linux kernel debug, and most steps are really same.

Due to the previous research on KVM by Project Zero (https://googleprojectzero.blogspot.com/2021/06/an-epyc-escape-case-study-of-kvm.html), we can find nested virtualization is a vulnerable attack surface. So we need to build a debug environment with 3 levels: L0 (host), L1 (guest), L2 (nested guest).

As mentioned in the rules of kvmCTF (https://github.com/google/security-research/blob/master/kvmctf/rules.md), the host runs v6.1.74 Linux kernel on an Intel CPU. So let’s download the Linux kernel source code first.

L0

Here is the config for compile, wich is different compared to kvmctf provide but work for me.

Then we can get the bzImage and vmlinux.

The next step is to build a system image, I chose Debian Bullseye Linux image.

Except these, kvm also has kernel module component (kvm.ko and kvm-intel.ko), we need to create a dictory and run this command:

make modules_install INSTALL_MOD_PATH=xxx/modules

Copy 6.1.74/ in modules/ to bullseye/lib/modules/

Use following script to build a new image

dd if=/dev/zero of=bullseye.img bs=1M seek=6144 count=1
sudo mkfs.ext4 -F bullseye.img
sudo mkdir -p /mnt/bullseye
sudo mount -o loop bullseye.img /mnt/bullseye
sudo cp -a bullseye/. /mnt/bullseye/.
sudo umount /mnt/bullseye

Now we have all the materials, we can use these two scripts to run L0 and debug it.

run.sh

#!/bin/sh
qemu-system-x86_64 \
	-m 2G \
	-smp 2 \
	-kernel ./linux-6.1.74/arch/x86/boot/bzImage \
	-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
	-drive file=./image/bullseye.img,format=raw \
	-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
	-net nic,model=e1000 \
	-enable-kvm \
	-cpu Cascadelake-Server,+vmx \
	-nographic \
	-pidfile vm.pid \
	-fsdev local,security_model=passthrough,id=fsdev0,path=./shared \
	-device virtio-9p-pci,id=fsdev0,fsdev=fsdev0,mount_tag=hostshare \
	-gdb tcp::1234 \
	-S \
	2>&1 | tee vm.log

debug.sh

#!/bin/sh
gdb \
-ex "target remote localhost:1234" \
-ex "continue" \
-ex "disconnect" \
-ex "file ./linux-6.1.74/vmlinux" \
-ex "set architecture i386:x86-64:intel" \
-ex "target remote localhost:1234" \
-ex "add-symbol-file xxx/modules/lib/modules/6.1.74/kernel/arch/x86/kvm/kvm-intel.ko 0xffffffffc010f000" \
-ex "add-symbol-file xxx/modules/lib/modules/6.1.74/kernel/arch/x86/kvm/kvm.ko 0xffffffffc0005000"

If you are not sure kvm is running, use kvm-ok to check.

L1 and L2

Now we only have L0, we need to prepare the qemu-kvm environment on L0.

To test KVM, I suggest to run and modify kvm-unit-tests.

The kvm-unit-tests will be running as L1, if you run vmx.flat, the guest_main code in vmx_tests will be running on L2, for testing nested virtualization.