GNU Linux – how to – kvm qemu pci pass through – GPU Graphic Card or other PCI device – Windows Games on GNU Linux

23.Dec.2017

3D Rendering, Administration / Server, Hardware / Reviews, Virtualization / KVM / VirtualBox / xenserver

pci pass through nic and other hardware directly to vm?

theoretically possible, but it’s pretty hard to setup 🙁

btw: stub was only a workaround until vfio is up and ready

should have TWO! Graphic-Cards:

one onboard for GNU Linux host (does not need to be very powerfull)
one to be passed through to the (Windows gaming?) VM (it will not be available for the host = screen goes dark)
excellent but maybe outdated guide: https://davidyat.es/2016/09/08/gpu-passthrough/

update: gaming inside kvm vm on GNU Linux host? YES!

but TWO (2x) GPUs / dedicated grafix cards are needed! (or one one onboard gpu and one PCIE dedicated GPU card)

“QEMU/KVM VGA PCI Pass-Through of a dedicated AMD Radeon R9 280”

“using a CPU and Motherboard which supports hardware virtualization (Intel VT-x/VT-d or AMD-V) along with two graphics cards, one for the Linux Base and another dedicated PCIe GPU for the Windows Virtual Machine (VM). Gaming experience within the Windows VM is exceptional, yielding a 96% performance return in comparison to native play.”

tested on Intel:

Processor: Intel Core i7-4790 + Mobo: ASUS Z87-Pro + RAM: Crucial Ballistix 32GB (8GBx4)
GPU 1: Intel HD 4600 Integrated Graphics
GPU 2: AMD Radeon R9 280
Monitor1: Connected to Intel HD 4600 (HDMI1) and AMD Radeon R9 280 (HDMI2)
Monitor2: Connected to Intel HD 4600 (HDMI1)

but also tested on AMD:

Debian 12 + lscpu: AMD EPYC 7402P 24xCore CPU
- make sure this is enabled in bios: (screenshots below)
  - AMD Virtualization (AMD-V™) technology is a set of on-chip features that helps improve performance of virtual environments
  - In BIOS, it is named SVM(Secure Virtual Machine) mode, can select enabled or disabled as below shown (default is Enabled for ASUS BIOS) (src: asus.com they should know)
  - make sure IOMMU is enabled in bios

tested on+with:

hostnamectl
  Operating System: Debian GNU/Linux 9 (stretch)
            Kernel: Linux 4.9.0-4-amd64
      Architecture: x86-64

lspci 
00:00.0 Host bridge: Intel Corporation Skylake Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 520 (rev 07)

# search for the device in question

lspci|grep LSI

# or interactive /search with /forward /slash /searchforthis
lspci|less
lspci -vnns 01:00.0                                                                        ---> here is the pciID\
01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: Dell 6Gbps SAS HBA Adapter [1028:1f1c]
Flags: bus master, fast devsel, latency 0, IRQ 68
I/O ports at 4000 [size=256]
Memory at fdb40000 (64-bit, non-prefetchable) [size=64K]
Memory at fdb00000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at fda00000 [disabled] [size=1M]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas

tested those settings:

# IF INTEL CPU: make sure VT-d (Intel Virtualization Technology) is enabled in BIOS
vim /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=oni pci_stub.ids=00:02.0"
GRUB_CMDLINE_LINUX=""

# IF AMD CPU: make sure AMD-V = SVM + IOMMU is enabled in BIOS
vim /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="ipv6.disable=1 amd_iommu=on iommu=pt pci_stub.ids=1000:0072"

# generate new grub config /boot/grub/grub.cfg
# Debian/Ubuntu
update-grub2
# update-grub2 can not be found in Debian10?
# it is probably this problem
# simulate proper root login
su - root
# try again
update-grub2

# RedHat/Fedora/CentOS
grub2-mkconfig -o /boot/grub2/grub.cfg
vim /etc/initramfs-tools/modules

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
# # Syntax: module_name [args ...]
# # must run update-initramfs(8) to effect this change
# # Examples
# # raid1
# sd_mod pci_stub

definitely need a cpu with virtualization support (VT).

egrep '^flags.*(vmx|svm)' /proc/cpuinfo;
# should display some output like

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

egrep '^flags.*(vmx|svm)' /proc/cpuinfo|grep --color -e vmx -e svm;

if no vmx or svm shows up, reboot now to make changes active and check the bios for possible enabling VT support.

sync; sync; sync; reboot;

after reboot, if it’s an intel CPU, the output of should display something like:

 dmesg | grep AMD-Vi
[ 0.020655] AMD-Vi: Unknown option - 'on'
[ 0.111856] AMD-Vi: Using global IVHD EFR:0x58f77ef22294ade, EFR2:0x0
[ 0.722921] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.726519] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.731325] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.734463] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.740948] pci 0000:c0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.741071] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
[ 0.741246] pci 0000:80:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.741367] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
[ 0.741536] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.741658] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
[ 0.741827] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.741949] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
[ 0.742118] AMD-Vi: Interrupt remapping enabled
[ 0.742235] AMD-Vi: X2APIC enabled
[ 0.742383] AMD-Vi: Virtual APIC enabled
[ 1.145770] AMD-Vi: AMD IOMMUv2 loaded and initialized


dmesg | grep pci_stub
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-4-amd64 root=UUID=4287dcc2-278b-4413-9d59-10c1710f35bd ro quiet intel_iommu=oni pci_stub.ids=00:02.0
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-4-amd64 root=UUID=4287dcc2-278b-4413-9d59-10c1710f35bd ro quiet intel_iommu=oni pci_stub.ids=00:02.0

after reboot, if it’s an AMD CPU, the output of should display something like:

dmesg | grep iommu
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.1.0-21-amd64 root=/dev/mapper/ubuntu--gigabyte--vg-root ro ipv6.disable=1 amd_iommu=on iommu=pt vfio-pci.ids=1000:0072
[ 0.020604] Kernel command line: BOOT_IMAGE=/vmlinuz-6.1.0-21-amd64 root=/dev/mapper/ubuntu--gigabyte--vg-root ro ipv6.disable=1 amd_iommu=on iommu=pt vfio-pci.ids=1000:0072
[ 0.674908] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.756470] pci 0000:c0:01.0: Adding to iommu group 0
[ 0.756623] pci 0000:c0:01.1: Adding to iommu group 0
[ 0.756780] pci 0000:c0:02.0: Adding to iommu group 1
[ 0.756959] pci 0000:c0:03.0: Adding to iommu group 2
...
[ 0.775063] pci 0000:07:00.2: Adding to iommu group 57
[ 0.775211] pci 0000:07:00.3: Adding to iommu group 58
[ 0.779987] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 0.780131] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[ 0.780272] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[ 0.780412] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).

inside virt-manager:

open the vm
go to “connection details”
r-click in the list of devices and + add device
under pci-devices select device to pass through

run the VM… if your screen gets dark and even your keyboard passess out… than you only have one graphic card.

maybe install vncserver inside the windows vm prior to that.

install drivers for your graphic card

now you should see device manager should show the correct naming of your chip model…

not just “Default-VGA-Graphic”…. or something is wrong.

fire up some basic 3d benchmark http://www.geeks3d.com/20140304/gputest-0-7-0-opengl-benchmark-win-linux-osx-new-fp64-opengl-4-test-and-online-gpu-database/

or the more advanced 😀 Heaven 4.0 Benchmark

it is amazing that KVM can do this… VirtualBox virtual graphic card adapter is unfortunately not often recognized correctly by 3d applications…

but will need two graphic cards… like one on board and one dedicated PCI device.

network card pass through example:

notice: the passed NIC dissapears from the ip -c a list of NICs on the server

the answers here all suck: https://serverfault.com/questions/446658/assign-individual-nic-to-kvm-guest

and the login-process is a pain X-D (server-fault is one sites of many that ought to share the same login details… but well X-D)

example: possible pitfalls:

vfio host doesn’t support passthrough of host PCI devices
- IOMMU is not enabled in bios
“AMD-Vi: AMD IOMMUv2 functionality not available on this system – This is not a bug”
- don’t be fooled by this message, the hardware might never the less support IOMMU, but settings in /etc/defult/grub might not be correct

pass through SAS adapter:

good luck all involved 🙂

it might not work on the first try