本文将介绍在QEMU KVM 环境下，cpu hot(un)plug的使用。

1. Kernel Configuration

To use the cpu hotplug feature, need to select the following items:

  • CONFIG_SMP
  • CONFIG_HOTPLUG_CPU
  • CONFIG_ACPI_HOTPLUG_CPU

2. QEMU launch script setting

1
qemu [...] -smp 1,maxcpus=4 -qmp unix:/tmp/qmp-sock,server=on,wait=off

note that the “maxcpus” is mandatory to allow vCPU hotplug.

3. Run ‘qmp-shell’

Run ‘qmp-shell’ (located in the source tree, under: “scripts/qmp/“) to connect to the just-launched QEMU:

1
2
3
4
5
$ ./qmp-shell -p -v /tmp/qmp-sock
Welcome to the QMP low-level shell!
Connected to QEMU 5.1.0

(QEMU)

4. Find out which CPU types could be plugged, and into which sockets:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
(QEMU) query-hotpluggable-cpus
{
    "execute": "query-hotpluggable-cpus",
    "arguments": {}
}
{
    "return": [
        {
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 3
            },
            "vcpus-count": 1,
            "type": "host-x86_64-cpu"
        },
        {
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 2
            },
            "vcpus-count": 1,
            "type": "host-x86_64-cpu"
        },
        {
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 1
            },
            "vcpus-count": 1,
            "type": "host-x86_64-cpu"
        },
        {
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 0
            },
            "vcpus-count": 1,
            "qom-path": "/machine/unattached/device[0]",
            "type": "host-x86_64-cpu"
        }
    ]
}
(QEMU)

The query-hotpluggable-cpus command returns an object for CPUs that are present (containing a “qom-path” member) or which may be hot-plugged (no “qom-path” member). From the output, we can see that host-x86_64-cpu is present in socket 0, while hot-plugging a CPU into socket 1 requires passing the listed properties to QMP device_add.

5. Hotplug

Before running device_add, run the following command lines.

1
2
3
4
$ cd /sys/devices/system/cpu
$ ls
cpu0
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(QEMU) query-cpus-fast
{
    "execute": "query-cpus-fast",
    "arguments": {}
}
{
    "return": [
        {
            "arch": "x86",
            "thread-id": 20077,
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 0
            },
            "qom-path": "/machine/unattached/device[0]",
            "cpu-index": 0,
            "target": "x86_64"
        }
    ]
}
(QEMU)

CPU hot-add:
{ ‘command’: ‘device_add’, ‘data’: {‘driver’: ‘str’, ‘id’: ‘str’, … }}

  • mandatory properties for every CPU
    • driver: cpu model type name
    • id: unique device name
  • target/configuration dependent properties
    • socket-id: socket number in range [0..max sockets)
    • core-id: core number in range [0..max cores)
    • thread-id: thread-id in range [..max threads)
    • node-id: NUMA node ID the CPU belongs to
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(QEMU) device_add driver=host-x86_64-cpu socket-id=1 core-id=0 thread-id=0 id=cpu2
{
    "execute": "device_add",
    "arguments": {
        "driver": "host-x86_64-cpu",
        "socket-id": 1,
        "core-id": 0,
        "thread-id": 0,
        "id": "cpu2"
    }
}
{
    "return": {}
}
(QEMU)

After running device_add, run the following command lines.

1
2
3
4
5
$ cd /sys/devices/system/cpu
$ ls
cpu0
cpu1
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
(QEMU) query-cpus-fast
{
    "execute": "query-cpus-fast",
    "arguments": {}
}
{
    "return": [
        {
            "arch": "x86",
            "thread-id": 20077,
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 0
            },
            "qom-path": "/machine/unattached/device[0]",
            "cpu-index": 0,
            "target": "x86_64"
        },
        {
            "arch": "x86",
            "thread-id": 20117,
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 1
            },
            "qom-path": "/machine/peripheral/cpu2",
            "cpu-index": 1,
            "target": "x86_64"
        }
    ]
}
(QEMU)

Optionally online newly added CPU inside guest
Linux kernel doesn’t online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

1
echo 1 > /sys/devices/system/cpu/cpu1/online

6. Hotunplug

1
2
3
4
5
6
7
8
9
10
11
(QEMU) device_del id=cpu2
{
    "execute": "device_del",
    "arguments": {
        "id": "cpu2"
    }
}
{
    "return": {}
}
(QEMU)

After running device_del, run the following command lines.

1
2
3
4
$ cd /sys/devices/system/cpu
$ ls
cpu0
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(QEMU) query-cpus-fast
{
    "execute": "query-cpus-fast",
    "arguments": {}
}
{
    "return": [
        {
            "arch": "x86",
            "thread-id": 20077,
            "props": {
                "core-id": 0,
                "thread-id": 0,
                "socket-id": 0
            },
            "qom-path": "/machine/unattached/device[0]",
            "cpu-index": 0,
            "target": "x86_64"
        }
    ]
}
(QEMU)

参考资料:

  1. Virtual CPU hotplug
  2. Features/CPUHotplug
  3. QEMU CPU Hotplug
  4. cpu-hotplug.txt