GPU PCI-Passthrough on Silverblue

Overview

When you have hardware burning a hole in your desk, and you want to run a VM that can utilize physical hardware such as a Graphics Processing Unit, or GPU, this is how you set it up.

References

Docs and Blogs on how-to

• ArchLinux Wiki: PCI Passthrough via OVMF
• GitHub: BigOakley/gpu-passthrough
• mrzk.io Blog: Fedora 32 and GPU Passthrough VFIO
• N00b security Blog: Easy GPU Passthrough using KVM on Fedora

Fedora Discussions

• Fedora Discussions: GPU Pass-through on Silverblue

Hardware

You will need to have a second monitor with this setup. You could try to setup Looking Glass or Sunshine+Moonlight if you wanted to.

Primary GPU for Host

• MSI Radeon RX 6700 XT MECH 2X 12G OC

Secondary GPU for VM

• Zotac GTX 970 Dual Fan - 145w

Preliminary Checks

[!Info] Make sure you have all updates installed before doing any of this. Makes it go faster!

Run this grep command to find if your CPU has virtualization enabled.

sudo grep --color --regexp vmx --regexp svm /proc/cpuinfo

If there is no result, make sure to enable VT-d for Intel or AMD-V for AMD based motherboards. Consult your hardware's instructions on how to do that.

You may need to remove some Nvidia specific options if you had once used an Nvidia GPU.

sudo rpm-ostree kargs \
  --delete-if-present=rd.driver.blacklist=nouveau \
  --delete-if-present=modprobe.blacklist=nouveau \
  --delete-if-present=nvidia-drm.modeset=1 \
  --reboot

Steps for Intel CPU

Ensure CPU has IOMMU enabled

Run the following command.

dmesg | grep -i -e DMAR -e IOMMU

If this doesn't show, make sure to set:

sudo rpm-ostree kargs \
  --append-if-missing="intel_iommu=on" \
  --append-if-missing="iommu=pt" \
  --append-if-missing="rd.driver.pre=vfio_pci" \
  --reboot

as a Kernel parameter

Steps for AMD CPU

Ensure CPU has IOMMU enabled

Run the following command.

dmesg | grep -i -e DMAR -e IOMMU

This should already be enabled for AMD CPUs. Either way, you'll want to do this.

sudo rpm-ostree kargs \
  --append-if-missing="amd_iommu=on" \
  --append-if-missing="iommu=pt" \
  --append-if-missing="rd.driver.pre=vfio_pci" \
  --reboot

Check PCI Bus Groups

#!/usr/bin/env bash

shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

It should spit out stuff that looks like this...

➜  ~ 
> bash ./02-check-pci-bus-groups.sh
IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
[...lots of output...]

IOMMU Group 32:
	0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1)
IOMMU Group 33:
	0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

IOMMU Group 34:
	0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
[...more output...]

You can also get the device IDs using lspci. In this case, I'm looking for my NVIDIA card to pass-through.

➜  ~ 
> lspci -vnn | grep -i --regexp NVIDIA
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])

0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

➜  ~ 
> lspci -vnn | grep -i --regexp Radeon
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1) (prog-if 00 [VGA controller])

Now we'll setup the Kernel Args for disabling the PCI Buses for the GPU.

sudo rpm-ostree kargs \
  --append-if-missing="vfio-pci.ids=10de:13c2,10de:0fbb" \
  --reboot

Then perform a dracut to make sure the initramfs has the Kernel module loaded.

sudo rpm-ostree initramfs \
  --enable \
  --arg="--add-drivers" \
  --arg="vfio-pci" \
  --reboot

You should now see when you perform

sudo lspci -nnv

Should show something similar to...

➜  ~ 
> sudo lspci -vnn
[sudo] password for filbot:
[...lots of output...]

0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1366]
	Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 34
	Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at f6000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] Secondary PCI Express
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1366]
	Flags: bus master, fast devsel, latency 0, IRQ 131, IOMMU group 34
	Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Endpoint, MSI 00
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

[...lots of output...]

[!Warning] If it doesn't say vfio-pci Then you're doing something wrong.

You can also check if the vfio Kernel modules made it in the initramfs by using:

➜  ~ 
> sudo lsinitrd /boot/ostree/fedora-af7516f20c0bee3df3aa77532024d9fb7c207d1b7246d1d4949b047d7ab916d9/initramfs-6.0.15-300.fc37.x86_64.img| grep -i --regexp vfio
Arguments:  --reproducible -v --add 'ostree' --tmpdir '/tmp/dracut' -f --no-hostonly --kver '6.0.15-300.fc37.x86_64' --reproducible -v --add 'ostree' --tmpdir '/tmp/dracut' -f --no-hostonly --add-drivers ' vfio-pci' --kver '6.0.15-300.fc37.x86_64'
drwxr-xr-x   3 root     root            0 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio
drwxr-xr-x   2 root     root            0 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/pci
-rw-r--r--   1 root     root        32140 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/pci/vfio-pci-core.ko.xz
-rw-r--r--   1 root     root         8596 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
-rw-r--r--   1 root     root        19884 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
-rw-r--r--   1 root     root        14880 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/vfio.ko.xz
-rw-r--r--   1 root     root         3552 Dec 31  1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz

The /boot/ostree/fedora-???/initramfs-x.y.z-aaa.fc??.x86_64.img depends on what image is live. You may have two or three depending on how many backup images you keep of your ostree setup.

Install bridge-utils

[!Info] Optional Stuff This is entirely optional. You don't need it.

• getlabsdone.com - How to connect kvm mv to host network
• getlabsdone.com - How to create a Bridge on Fedora

On Fedora, I installed bridge-utils.

sudo rpm-ostree install bridge-utils --reboot

Then follow the blogs listed. However, this will make your host present a different IP address and route traffic differently. I think my issue for my Windows VM was the disk size. See setting up the Virtual Machine further on.

Setup the Virtual Machine

In this we're using qemu-kvm with virt-manager as a GUI to help us.

As of this writing, I'm using qemu-kvm version 7.0.0. This should be sufficient.

➜  ~ 
> qemu-kvm --version
QEMU emulator version 7.0.0 (qemu-7.0.0-12.fc37)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

Download Windows 10 ISO

At the time of this writing I'm using Windows 10 and I am downloading it from this link. Verify it before clicking.

• Microsoft Windows 10 ISO Download

[!Warning] Verify your ISO Download Remember to verify the ISO download you're performing!

Use Virtual Machine Manager

This is going to use virt-manager or Virtual Machine Manager. Either way it's the GUI of the command-line application virt-install.

• O