Numa CPU Pinning with KVM/Virsh

According to Wikipedia, Numa is — “a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can access its own local memory faster than non-local memory (memory local to another processor or memory shared between processors). The benefits of NUMA are limited to particular workloads, notably on servers where the data are often associated strongly with certain tasks or users.“

So what does this mean for a Virtual Machine optimization under KVM/Libvirt? It means that for best performance, you want to configure your multi-vcpu VMs to use only cores from the same physical CPU (or numa node).

So how do we do this? See the example below from one of my homelab servers. This machine has two hyperthreaded quad core Xeons (x5550) — for a total of 16 cores.

First we use the “lspcu” command to determine which cpu cores are tied to which CPU. This is in bold below.

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Model name: Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
Stepping: 5
CPU MHz: 2668.000
BogoMIPS: 5319.11
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3,8-11
NUMA node1 CPU(s): 4-7,12-15

Using the virsh command, we can inspect the CPU pinning for my test VM called “mytestvm“.

# virsh vcpupin mytestvm
VCPU: CPU Affinity
———————————-
0: 0-15
1: 0-15
2: 0-15
3: 0-15

As you can see from the output above my VM has 4 vcpus that can run as a thread on any of my 16 cores. This is not optimal. Let’s pin vcpu0 on this test VM to NUMA node0, AKA physical CPU 1.

#virsh vcpupin mytestvm 0 0,1,2,3,8,9,10,11

Now lets rinse and repeat for vcpus 1-3.

#virsh vcpupin mytestvm 1 0,1,2,3,8,9,10,11
#virsh vcpupin mytestvm 2 0,1,2,3,8,9,10,11
#virsh vcpupin mytestvm 3 0,1,2,3,8,9,10,11

Now we validate our changes.

# virsh vcpupin mytestvm
VCPU: CPU Affinity
———————————-
0: 0-3,8-11
1: 0-3,8-11
2: 0-3,8-11
3: 0-3,8-11