Vfio

From Linux - Help
Jump to navigation Jump to search
Vfio small.png

A Windows 10 VM running on top of QEMU/KVM & VFIO ... Pentium 2 FTW !

The goal of this document is to provide an up-to-date way to setup PCI passthrough (GPU, audio, ...) on QEMU/KVM with the help of VFIO on any Linux host.

This guide focus on Intel CPU and Nvidia GPU as I don't own another type of hardware, that being said, some parts of the tutorial are hardware agnostic.

This guide use the most "recent" KVM VGA passthrough method which is OVMF + vfio-pci (compared to Seabios + pci-stub) and requires a GPU with EFI ROM.

Hardware prerequisites

  • VT-x & VT-d enabled hardware (CPU + motherboard), see https://ark.intel.com for compatibles CPU and your MB specsheet. i.e Intel Core i5-3470 and ASRock B75 Pro3
  • 2 GPU's (One discrete and one integrated is fine) i.e Intel HD and GeForce GTX 750 Ti
  • 1 screen with two inputs or two screens
  • (optional) coffee or cold beers

Software

   Gentoo amd64, Linux kernel >= 4.6.x
   QEMU >= 2.5.1 + KVM
   virt-manager >= 1.3.2 & libvirt >= 1.3.5
   OVMF firmware
   Windows 10 Pro VM

Host system setup

Kernel configuration

   CONFIG_VFIO_IOMMU_TYPE1=m
   CONFIG_VFIO_VIRQFD=m
   CONFIG_VFIO=m
   CONFIG_VFIO_PCI=m
   CONFIG_VFIO_PCI_VGA=y
   CONFIG_VFIO_PCI_MMAP=y
   CONFIG_VFIO_PCI_INTX=y
   CONFIG_VFIO_PCI_IGD=y
   CONFIG_KVM_VFIO=y
   CONFIG_IOMMU_HELPER=y
   CONFIG_VFIO_IOMMU_TYPE1=m
   CONFIG_IOMMU_API=y
   CONFIG_IOMMU_SUPPORT=y
   CONFIG_INTEL_IOMMU=y
   CONFIG_INTEL_IOMMU_SVM=y
   CONFIG_INTEL_IOMMU_DEFAULT_ON=y
   CONFIG_INTEL_IOMMU_FLOPPY_WA=y

Kernel boot options & GRUB2

edit /etc/default/grub and add to GRUB_CMDLINE_LINUX

   intel_iommu=on

Then regenerate the GRUB menu with

   grub-mkconfig -o /boot/grub/grub.cfg

or add the kernel option at boot time

Reboot.

Verify that Intel IOMMU is working

   dmesg | grep -e DMAR

You should see this amongst others lines :

   DMAR: Intel(R) Virtualization Technology for Directed I/O

IOMMU groups

 #!/bin/bash
 for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);
   do printf "\nIOMMU group $(basename "$iommu_group") \n";
   for device in $(ls -1 "$iommu_group"/devices/);
     do echo -n $'--> '; lspci -nns "$device";
 done;
 done

This command should report the various IOMMU groups from your machine (An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine)

If there's nothing it means IOMMU is not properly enabled/working

Isolating the GPU with vfio-pci

Get your vendor-id :

   lspci | grep -i vga

note down the first number, it is the slot number i.e 01:00.0

   lspci -nns 01:00.0

Note down the last number between "[]", this is the vendor-id, i.e 10de:1380

   lspci -nnk -d vendor-id

Edit /etc/modprobe.d/vfio.conf with the vendor-id from your GPU you want to isolate gathered from the previous command, in this example, the vendor-id is 10de:13c2 for the GPU and 10de:0fbb for the audio

   options vfio-pci ids=10de:13c2,10de:0fbb

Add these modules to /etc/conf.d/modules (Gentoo/OpenRC specific)

   modules="vfio vfio-pci vfio_iommu_type1 vfio_virqfd"

module loading at boot is enabled by

   rc-update add modules boot

Reboot

Check that your GPU is correctly isolated

   $ dmesg | grep -i vfio
   [    0.329224] VFIO - User Level meta-driver version: 0.3
   [    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000 <-- Good
   $ lspci -nnk -d <VENDOR_ID>
   Kernel driver in use: vfio-pci <-- Good

Same check for your sound card if compiles as a module

If lspci -nnk -d reports a "Kernel driver in use" which is not vfio-pci, there's something wrong.

You can blacklist module loading by settings up the blacklist in /etc/modprobe.d/blacklist.conf

   blacklist snd_hda_intel

Alternatively, building snd_hda_intel in-kernel [*] (as opposed to a module [M] ) works ok !

Kernel modules

Make sure these modules are present when you run lsmod

   $ lsmod
   vfio
   kvm
   kvm_intel
   vfio-pci

QEMU / KVM / libvirt & OVMF

Below example is for Gentoo & OpenRC, adjust for your distro.

   # emerge --ask qemu virt-manager libvirt git
   # rc-update add libvirt-guests
   # service libvirt-guests start
   # service libvirtd start

Get OVMF via https://www.kraxel.org/repos/jenkins/edk2/ and choose the

edk2.git-ovmf-x64-[_some_date_] RPM.

Unpack it and copy OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd to /usr/share/ovmf/x64/, creating the directory if needed.

Add this line to /etc/libvirt/qemu.conf :

   nvram = [
           "/usr/share/ovmf/x64/ovmf_x64.bin:/usr/share/ovmf/x64/ovmf_vars_x64.bin"
   ]

Create a new machine with virt-manager and check "edit settings" before finishing the setup, change firmware from BIOS to UEFI (or to "Custom: /usr/share/ovmf/x64/ovmf_x64.bin") then boot the VM.

If you're dropped to an EFI shell, make sure the ISO you wish to boot is correct, try with a recent Ubuntu release for instance.

Networking

One effortless way is to get an USB WiFi adapter and passing-through the USB device to the VM directly, otherwise, best to look at bridging.

   emerge bridge-utils

Create bridge :

   brctl addbr br0
   brctl addif br0 enp3s0
   dhcpcd br0

Delete bridge :

   ifconfig br0 down
   brctl delif br0 enp3s0
   brctl delbr br0

VM setup & possible bugs

The XML configuration of the VM is located at `/etc/libvirt/qemu`, should you edit it, do it with virsh edit <VM_NAME>, do NOT edit with your editor directly since XML validation is done by virsh (and you want it).

  • Nvidia Code 43 :

Recent nvidia drivers block the driver if it runs on top of an hypervisor, the workaround is to hide KVM :

via cli :

   -cpu [type],kvm=off

via libvirt :

  • Always edit your VM XML definition with virsh edit VM_NAME, not directly via your editor !
   <domain type='kvm'>

...

     <features>
       <kvm>
         <hidden state='on'/>
       </kvm>

...

     </features>
   </domain>

Boot the VM with standard virtualized VGA adapter (using the VM through the QEMU console) and install the nvidia drivers (tested with Nvidia Geforce driver 368.81 WHQL)

You may remove GeForce experience after the installation, specially because GeForce experience 3 is now sending telemetry data.

VirtIO drivers

VirtIO drivers for virtual HDD are available here : https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

Audio

If your IOMMU PCI groups are good enough, passing-through your whole PCI sound chipset is a safe way to get decent audio in your machine.

virt-manager allows you to do that effortlessly

Pci audio passthrough.png

Keyboard & Mouse

You can pass any USB device to the VM so your keyboard and mouse are no exception.

If your IOMMU PCI groups are good enough you can pass a whole USB controller to the VM, allowing you to hotplug USB devices from the VM to your host and vice-versa

Bind & unbind GPU

Is is possible to unbind the vfio_pci module from the passed-through GPU after using so that your Linux machine can use it by spawning another X(org) server on another VT using the nvidia driver

This (unfinished) script describe a possible way to do so, it contains Gentoo-specific workarounds.

#!/bin/bash
# vfio.sh : handle vfio-related operations

MYUSER="user"

function enableNvidia()
{
# unbind a vfio_pci binded NVIDIA gpu and start an X server. 
# execute me as root from a TTY
# https://www.reddit.com/r/vfio

DISPLAYMANAGER="xdm"

# Shutdown the DM running on the intel IGP
service $DISPLAYMANAGER stop

# Remove id's
echo "10de 1380" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "10de 0fbc" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo 1 > /sys/bus/pci/devices/0000:04:00.0/remove
echo 1 > /sys/bus/pci/devices/0000:04:00.1/remove
echo 1 > /sys/bus/pci/rescan

# unload vfio modules
modprobe -r vfio-pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

# load nvidia and sound
modprobe -vv nvidia
#modprobe snd_hda_intel

# gentoo-specific to point ot the proper openGL libs
eselect opengl set 1

# info
lspci -nnk -d 10de:1380
lspci -nnk -d 10de:0fbc
lsmod

# start an X server from the current VT/TTY
# the config file is relative to the /etc/X11/ path
su - $MYUSER -c 'CONSOLE=$(fgconsole) ; exec env LD_LIBRARY_PATH=/usr/lib32/opengl/nvidia/lib/ xinit /usr/bin/startxfce4 -- :1 vt$CONSOLE -config xorg.nvidia.unified.conf -nolisten tcp -verbose 3'
}

function disableAudioOnHost()
{
printf "Reloading vfio modules\n"
su - $MYUSER -c 'pulseaudio -k'
sleep 1
modprobe -r snd_hda_intel
modprobe vfio_pci
modprobe vfio_iommu_type1
modprobe vfio
}

function reloadVfio()
{
echo "10de 1380" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:04:00.0" > /sys/bus/pci/devices/0000:04:00.0/driver/unbind
echo "0000:04:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "10de 1380" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "10de 0fbc" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:04:00.1" > /sys/bus/pci/devices/0000:04:00.1/driver/unbind
echo "0000:04:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
echo "10de 0fbc" > /sys/bus/pci/drivers/vfio-pci/remove_id

sleep 1

# repeat for extra devices
rmmod nvidia_drm
rmmod nvidia_modeset
rmmod nvidia

# function call
disableAudioOnHost

modprobe -v vfio_pci
modprobe -v vfio_iommu_type1
modprobe -v vfio

eselect opengl set 2
service xdm restart
}

function enableAudioOnHost()
{
printf "Enabeling audio driver snd_hda_intel\n"
modprobe -r vfio-pci
modprobe -r vfio_iommu_type1
modprobe -r vfio
#modprobe snd_hda_intel
}

function pci_info()
{
lspci -nnk -d 10de:1380
echo "----------------" 
lspci -nnk -d 10de:0fbc
echo "----------------" 
lspci -nnk -d 8086:1e20
}

function pci_info_curated()
{
printf "GTX 750 Ti GPU :"
lspci -nnk -d 10de:1380 | grep "in use" | cut -d ":" -f 2
if [ $? != 0 ]
then
	printf "No kernel modules in use\n"
fi
printf "Intel HD Audio :"
lspci -nnk -d 8086:1e20 | grep "in use" | cut -d ":" -f 2
if [ $? != 0 ]
then
	printf "No kernel modules in use\n"
fi
}

# handle input
option=$1
if [[ $option == *"--rawinfo"* ]]
then
	pci_info
elif [[ $option == *"--info"* ]]
then
	pci_info_curated
elif [[ $option == *"--nvidia"* ]]
then
	enableNvidia
elif [[ $option == *"--hostaudio"* ]]
then
	enableAudioOnHost
        pci_info_curated
elif [[ $option == *"--vmaudio"* ]]
then
	disableAudioOnHost
	pci_info_curated
elif [[ $option == *"--reload"* ]]
then
	reloadVfio
elif [[ $option == *"--help"* ]]
then
	printf "vfio wrapper script v 0.2\n"
	printf "available commands :\n"
	echo "--info : get curated info about PCI devices and drivers"
	echo "--rawinfo : get raw info about PCI devices and drivers"
	echo "--help : help menu"
	echo "--nvidia : give GPU to Linux with nvidia module"
	echo "--reload : give GPU to VM with vfio_pci module"
	echo "--hostaudio : Enable sound card on Linux"
	echo "--vmaudio : Prepare sound card for VM"
	echo "--help : help menu"
else
	printf "Unknow command"
fi

Common patches explained

  • vgaarb aka i915 patch x-vga=on :

If OVMF (this tutorial) : vgaarb is not needed

If SeaBIOS : Requires vgaarb

  • ACS Override

If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson ACS override patch.

Todo

This area serve as a reminder for future improvements to add to this wiki

- blacklist nvidia module at boot - audio MSI-(X) hack - Clarify OVMF / UEFI install process - libvirtd at startup

Online resources