Tutorial How to setup a gaming virtual machine with GPU passthrough (QEMU, KVM, libvirt, and VFIO)

cheuk3

Newbie
Messages
1
Reaction score
0
Hi, thanks for your great guide. I have set up the VM successfully with the vga passthrough but when i play audio in VM. it will cause my whole PC freeze and reboot...I have alreayd tried to set the qemu_pa_sample to 8192 and tried without the vga passthrough, it still freeze the system and reboot...do you have any idea? thanks!

edit: no problem if i use hdmi audio
edit2: ok when using hdmi audio will cause the video and audio lag...but no reboot

Update: switched to gentoo and the issue gone :biggrin:
 
Last edited:

3313

Newbie
Messages
4
Reaction score
0
I have update:

I managed to get this working using Lubuntu 15.04 with Qemu 2.2 and kernel 3.19.

Using script from second post everything works besides sound.

Using script from this link with adding "-soundhw hda" sound is working.

I saw some users get only 3-5 % loss in perfomance. According to 3DMark I have 11% loss with both scipts.

Now I have new problem with both script, when I use more demanding application in guest after some time host and guest freezes.
 

Zelk

Hiding Both Face And Mind
Retired
Messages
1,965
Reaction score
5,328
This is an awesome tut. I'll have to give this a try when I come around to changing my OS to Arch.
 

Amadren

Newbie
Messages
1
Reaction score
0
Awesome tut, I only had BSODs with Windows 7 by following other tuts but everything worked fine after the installation on Windows 10 Pro with yours. I just have to search how to make the audio work and everything will be fine...

P.S: Using Debian Jessie, I had to recompile my kernel with ACS Override and i915 patch, then I had a problem a weird bug about memory and I had to set it to "unlimited" in my config files and it started just fine after that.
 

ssj4gokusama

Newbie
Messages
4
Reaction score
0
Hello, since I've updated my archlinux host with the latest packages, I can't launch my vm anymore. it shows "Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken." Any ideas?

EDIT : it looks like a problem with the libvirt package. For the moment I've done a downgrade of the package with Arch Linux Archive which can found how to do this here . And it's working now

But if someone know how to solve the problem with the latest package (libvirt-1.3.2.1), thanks to help me.
 
Last edited:

lowpro

Professional Abecedarian
Messages
4,593
Reaction score
2,029
Doesn't VMware and (Virtualbox?) support GPU pass through? I'm on win10 as default OS so I'm not sure about Linux.

I want to switch to Linux, but many things easily done in Windows seems hard in Linux without memorizing commands which I don't really want to do as it seems archaic.
 

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Messages
6,518
Reaction score
4,534
Doesn't VMware and (Virtualbox?) support GPU pass through? I'm on win10 as default OS so I'm not sure about Linux.
Any sort of virtualization program that is ran within Windows is considered a type 2 hypervisor and doesn't support PCI passthrough, let alone GPU passthrough. For such a thing you need to be using a type 1 hypervisor so it has direct access to the hardware (type 1 means the hv is running on top of the hardware, where as a type 2 isn't as low and running within the userspace/OS). Anything that is run on Windows simply isn't close enough to the hardware to allow for something like GPU passthrough. Even on Linux, the hypervisor (in this case KVM) isn't running on top of the OS but rather at the kernel level (KVM actually stands for kernel virtual machine).

In order to see support for this on Windows hosts, Microsoft would need to either build support for it the kernel or give users access to the kernel. The later will likely never happen and the former is rather unlikely. If they ever did such a thing, I doubt it would be on consumer versions and only on server distributions of Windows. Even then, it would be very hard for an end user to get it properly configured since they wouldn't have access to the kernel.

I want to switch to Linux, but many things easily done in Windows seems hard in Linux without memorizing commands which I don't really want to do as it seems archaic.
I just realized I forgot to reply to this. The reason it seems hard is because it's something your not used to. Once you get used to it, it is by far and away easier than Windows. The Linux filesystem and the way everything works is extremely intuitive once you get a base understanding. I used Windows for at least 10 years, and have only been using Linux as a main OS for about years. Already, I can do things much more efficiently in Linux than I can on Windows. It's also not like it took me 2 years to learn this, this happened within the span of a few weeks with Linux. This is especially true when it comes to errors and BSODs in Windows. I am still lost half the time I get a BSOD on Windows because with Windows you don't get to see what is going on under the hood. Sure, you have minidumps which can be useful but this is nothing compared to Linux logs (which are almost always in the same directoy) or by a simple command like dmesg. After experiencing the intuitiveness and the amount of customization available on Linux, I don't ever want to go back to Windows.
 
Last edited:
Messages
6
Reaction score
5
Hi everybody,

first of all thank you for the very intersting thread and also thank you to topic creater for this very helpful tutorial.

I am or i try to use to my gpu passthrough with antergos linux and I followed your tutorial. After trying starting VM I got the following error.

Fehler beim Starten der Domain: Interner Fehler: process exited while connecting to monitor: pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
ALSA lib confmisc.c:768:frown:parse_card) cannot find card '0'
ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:frown:snd_func_concat) error evaluating strings
ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:frown:snd_func_refer) error evaluating name
ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:frown:snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:frown:snd_pcm_open_noupdate) U

Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 90, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 126, in tmpcb
callback(*args, **kwargs)
File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn
ret = fn(self, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/domain.py", line 1402, in startup
self._backend.create()
File "/usr/lib/python2.7/site-packages/libvirt.py", line 1035, in create
if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Interner Fehler: process exited while connecting to monitor: pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
ALSA lib confmisc.c:768:frown:parse_card) cannot find card '0'
ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:frown:snd_func_concat) error evaluating strings
ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:frown:snd_func_refer) error evaluating name
ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:frown:snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:frown:snd_pcm_open_noupdate) U


I am not really sure why this error occurs. I followed your steps in point 8 so far but also if I reverte this changes the same error messages appear.

My main xml is following... Do not surprise that I comment the disk part out. There I got some error message that IDE is not supported for qemu. But this I try to fix afterwards.


<?xml version="1.0" encoding="UTF-8"?>
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
<name>gamingvm</name>
<uuid>01bd2ed1-b465-4eba-b6e4-47c6ac8171c6</uuid>
<memory unit="GB">8</memory>
<currentMemory unit="GB">8</currentMemory>
<vcpu placement="static">6</vcpu>
<cpu mode="host-passthrough">
<topology sockets="1" cores="3" threads="2" />
</cpu>
<os>
<type arch="x86_64" machine="q35">hvm</type>
<loader>/usr/share/qemu/bios.bin</loader>
<bootmenu enable="yes" />
</os>
<features>
<hyperv>
<relaxed state="on" />
<vapic state="on" />
<spinlocks state="on" retries="8191" />
</hyperv>
<acpi />
</features>
<clock offset="localtime">
<timer name="hypervclock" present="yes" />
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<interface type='bridge'>
<mac address='52:54:00:a0:41:92'/>
<source bridge='br0'/>
<model type='rtl8139'/>
<rom bar='off'/>
</interface>
<sound model='ich6'/>
<controller type="usb" index="0" />
<controller type="usb" index="1" />
<controller type="usb" index="2" />
<controller type="usb" index="3" />
<controller type="sata" index="0" />
<controller type="pci" index="0" model="pcie-root" />
<controller type="pci" index="1" model="dmi-to-pci-bridge" />
<controller type="pci" index="2" model="pci-bridge" />
<memballoon model="none" />
<sound model="ich6" />
<!--<disk type="file" device="cdrom">
<driver name="qemu" type="raw" />
<source file="/home/gerrit/vm/win7x64-20160308.iso" />
<target dev="hdc" bus="ide" />
<readonly />
<address type="drive" controller="0" bus="1" unit="0" />
</disk>
<disk type="file" device="disk">
<source file="/home/gerrit/vm/VM/win7.img" />
<target dev="vda" bus="ide" />
<address type="drive" bus="0" />
</disk>-->
</devices>
<qemu:commandline>
<qemu:env name="QEMU_PA_SAMPLES" value="4096" />
<qemu:env name="QEMU_AUDIO_DRV" value="pa" />
<qemu:arg value="-device" />
<qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" />
<qemu:arg value="-device" />
<qemu:arg value="vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on" />
<qemu:arg value="-device" />
<qemu:arg value="vfio-pci,host=01:00.1,bus=root.1,addr=00.1" />
<qemu:arg value="-cpu" />
<qemu:arg value="host" />
</qemu:commandline>
</domain>

I hope you can give me any hint.

Thanks and regards
GertPfert
 
Messages
6
Reaction score
5
i already found the clou. The thing is, that my pulse not using tcp instead linux sockets. After adding following line to client.pa from root user it is working.

default-server = unix:/tmp/pulse
 

U83R

Newbie
Messages
1
Reaction score
0
Hello
It seems like a very nice guide :smile:
I've been loosely following the whole Qemu-KVM GPU passthrough idea for a few years now. It sounds really attractive to game without rebooting and use Linux host at the same time almost like alt tab away lol. I hate rebooting!
Anyway I have a few questions that keep me up at night :tongue:

1. Is it possible to switch GPU without rebooting? Say a script to make Linux host switch from integrated Intel GPU to dedicated PCI-e once the VM is shut down? I guess the least what would happen is X11 on host would crash while switching to software rendering while GPUs get restarted and log you out of session? But maybe thats not even possible at all in the first place.
The whole deal is slightly offputing for me cause I have to buy 2 dedicated GPUs if I want to game natively and on Windows VM cause integrated GPU sucks for gaming :frown:
But I guess I could buy them if thats the only way. 1 AMD + 1 Nvidia hehehe

2. How is the stability of both host and VM? Does it require a lot of maintenance? Does VM break after kernel updates? That's another thing I'm worried about. Currently Im running Arch and as it's rolling release rarely but sometimes minor breakages occur after certain updates. Do you just set VM up once and it runs for a very long time or does it break all the time?

Anyway I think I'll be looking into building my next PC around this idea.

EDIT: I'm also thinking of 21:9 Ultra Wide monitor. It would be sick to use 2 inputs on a single Ultra Wide monitor for this particular setup. Just imagine on the left Linux host on the right Windows guest! That would be sick
 
Last edited:

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Messages
6,518
Reaction score
4,534
1. Is it possible to switch GPU without rebooting? Say a script to make Linux host switch from integrated Intel GPU to dedicated PCI-e once the VM is shut down? I guess the least what would happen is X11 on host would crash while switching to software rendering while GPUs get restarted and log you out of session? But maybe thats not even possible at all in the first place.
The whole deal is slightly offputing for me cause I have to buy 2 dedicated GPUs if I want to game natively and on Windows VM cause integrated GPU sucks for gaming :frown:
But I guess I could buy them if thats the only way. 1 AMD + 1 Nvidia hehehe
Yes you can do this but you will need to restart X in order for it to work. It also can occasionally cause kernel panics. I haven't tried it myself, but I suggest checking out this post in my subreddit: https://www.reddit.com/r/VFIO/comments/41pm1q/my_bash_script_for_rebinding_a_secondary_nvidia/

2. How is the stability of both host and VM? Does it require a lot of maintenance? Does VM break after kernel updates? That's another thing I'm worried about. Currently Im running Arch and as it's rolling release rarely but sometimes minor breakages occur after certain updates. Do you just set VM up once and it runs for a very long time or does it break all the time?
There are some kernel updates that cause issues with performance. I personally recommend using a LTS kernel (as shown in the tutorial) so you don't have to worry about this. I've barely had to do any maintenance to the virtual machine. The only time I do is when I want to improve it and optimize it, but I don't believe I've ever had to fix something that suddenly stopped working.
 

aarobc

Newbie
Messages
11
Reaction score
2
Hello again. I've really appreciated the guide. I'm attempting to pass through a USB card and have run into some issues. here are the steps I've taken so far:
detach the card from the host:
Code:
❯ lspci -nn | grep ASMedia
02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
Added the entries to the end of /etc/modprobe.d/pci-stub.conf:
Code:
options pci-stub ids=1002:aab0,1002:6810,1b21:1242
ran "sudo update-initramfs -u" and rebooted.

in the launch script I added the pci devices to DEVLIST
Added the entries for the groups 25 and 29 to /etc/libvirt/qemu.conf
Code:
❯ find /sys/kernel/iommu_groups/ -type l | grep -E '02:00.0|08:00.0'
/sys/kernel/iommu_groups/25/devices/0000:08:00.0
/sys/kernel/iommu_groups/29/devices/0000:02:00.0
and here is where I suspend I'm running into issues: in the xml file to define the vm, after the gpu, I tried adding the lines:
Code:
<qemu:arg value="-device" />                                  
<qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0'/>
<qemu:arg value="-device" />                                  
<qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0'/>
Would anyone be able to point me in a direction to get this working?
 

ssj4gokusama

Newbie
Messages
4
Reaction score
0
Hello again. I've really appreciated the guide. I'm attempting to pass through a USB card and have run into some issues. here are the steps I've taken so far:
detach the card from the host:
Code:
❯ lspci -nn | grep ASMedia
02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
Added the entries to the end of /etc/modprobe.d/pci-stub.conf:
Code:
options pci-stub ids=1002:aab0,1002:6810,1b21:1242
ran "sudo update-initramfs -u" and rebooted.

in the launch script I added the pci devices to DEVLIST
Added the entries for the groups 25 and 29 to /etc/libvirt/qemu.conf
Code:
❯ find /sys/kernel/iommu_groups/ -type l | grep -E '02:00.0|08:00.0'
/sys/kernel/iommu_groups/25/devices/0000:08:00.0
/sys/kernel/iommu_groups/29/devices/0000:02:00.0
and here is where I suspend I'm running into issues: in the xml file to define the vm, after the gpu, I tried adding the lines:
Code:
<qemu:arg value="-device" />                                 
<qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0'/>
<qemu:arg value="-device" />                                 
<qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0'/>
Would anyone be able to point me in a direction to get this working?
Hello, if you're haven't found yet, I think I have the reason why it's not working.
You need to change the adress value if you want to use several pci devices in your vm. I quote for you my qemu:commandline column :
Code:
  <qemu:commandline>
  <qemu:arg value="-device" />
  <qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" />
  <qemu:arg value="-device" />
  <qemu:arg value="vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on" />
  <qemu:arg value="-device" />
  <qemu:arg value="vfio-pci,host=01:00.1,bus=root.1,addr=00.1" />
  <qemu:arg value="-device" />
  <qemu:arg value="vfio-pci,host=04:00.0,bus=root.1,addr=00.2" />
  <qemu:arg value="-device" />
  <qemu:arg value="vfio-pci,host=06:00.0,bus=root.1,addr=00.3" />
  <qemu:arg value="-cpu" />
  <qemu:arg value="host,kvm=off" />
  </qemu:commandline>
You see? addr=00.0 then 00.1 for the next pci, etc...
Normally it would be worked
 

zipeldiablo

Newbie
Messages
2
Reaction score
0
No point using qemu command line here, did the same thing for my vm, just change the bus adress on source and it should work :
Code:
<!-- controler usb card -->
     <hostdev mode='subsystem' type='pci' managed='yes'>
       <source>
         <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</source>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

</hostdev>
On another note, i decided to give a try to the script explained on first page in order to gain some performances.
Cset is working fine but when i launch the vm i got this :
Code:
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dwin10.scope/vcpu0/cpuset.cpus': Permission denied
The error occured when libvirt is trying to vcpupin my vcpu to cpuset.
I run the script as root so i really don't know what the problem is, googling the issue didn't gave me proper solution for this, any idea?

Also nothing is said on how to launch all scripts simultaneously so only one commande has to be run, i tried several options but it seems i have to write a different script for this
 
Last edited:

aarobc

Newbie
Messages
11
Reaction score
2
I'm still having no luck, I suspect the issue is because the usb card shows up in multiple vfio groups with different addresses:
Code:
❯ lspci -nn | grep ASMedia     
02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
So I'm attempting to redo everything under arch. I've used ubuntu for years, but arch has proved a challenge. The problem I'm having is in the tutorial when I get to the part to run the command:
sudo mkinitcpio -p linux-vfio-lts
it gives an error about a file or directory not existing. Any ideas?
 

zipeldiablo

Newbie
Messages
2
Reaction score
0
You cannot expect people to help you understand an error if you don't a copy of the error message mate.
 

aarobc

Newbie
Messages
11
Reaction score
2
Indeed.
I run the command:
pacaur -S qemu libvirt bridge-utils linux-vfio-lts
and it installs just fine, no error messages, I then try to run the subsequent command and get an error:
Code:
❯ sudo mkinitcpio -p linux-vfio-lts
[sudo] password for ac:
/usr/sbin/mkinitcpio: line 264: /etc/mkinitcpio.d/linux-vfio-lts.preset: No such file or directory
==> ERROR: Failed to load preset: `/etc/mkinitcpio.d/linux-vfio-lts.preset'
I've been googling the error for the last few days and haven't found a solution yet, any guidance or suggestions would be greatly appreciated.
 

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Messages
6,518
Reaction score
4,534
Indeed.
I run the command:
pacaur -S qemu libvirt bridge-utils linux-vfio-lts
and it installs just fine, no error messages, I then try to run the subsequent command and get an error:
Code:
❯ sudo mkinitcpio -p linux-vfio-lts
[sudo] password for ac:
/usr/sbin/mkinitcpio: line 264: /etc/mkinitcpio.d/linux-vfio-lts.preset: No such file or directory
==> ERROR: Failed to load preset: `/etc/mkinitcpio.d/linux-vfio-lts.preset'
I've been googling the error for the last few days and haven't found a solution yet, any guidance or suggestions would be greatly appreciated.
Strange that you're installing the linux-vfio-lts kernel but it doesn't come with the preset. Regardless, you can cd to /etc/mkinitcpio.d/ and create a file named linux-vfio-lts.preset with the following:
Code:
# mkinitcpio preset file for the 'linux-vfio-lts' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux-vfio-lts"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/initramfs-linux-vfio-lts.img"
#default_options=""

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/initramfs-linux-vfio-lts-fallback.img"
fallback_options="-S autodetect"
 

EvilWiffles

Newbie
Messages
4
Reaction score
0
Hello,

Thread caught my attention and I'm definitely interested in trying this.
I never really got to use any OS besides Windows intensively and I was always looking for a replacement and this looks like it'll solve a lot of issues. I am running an i7 4790k (Asus Z97-A) and a GTX 970 with 16GB's of RAM, and multiple drives. My main concern is mostly to do with compatibility and drive utilization, as well as being easy to setup.

I'm rather conflicted, I really want to try this though.
 

GrandGamer1

Newbie
Messages
8
Reaction score
0
Hello,

Thread caught my attention and I'm definitely interested in trying this.
I never really got to use any OS besides Windows intensively and I was always looking for a replacement and this looks like it'll solve a lot of issues. I am running an i7 4790k (Asus Z97-A) and a GTX 970 with 16GB's of RAM, and multiple drives. My main concern is mostly to do with compatibility and drive utilization, as well as being easy to setup.

I'm rather conflicted, I really want to try this though.
Nothing to say here really. You will need to check if your CPU supports VT-d/x. Also make sure you have integrated GPU to run Linux on or a second GPU. As long as you don't have some weird hardware peripheral, everything will probably work fine. Other than that, if you have time and the will to do it, backup your Windows partition and try to set it up.

You can also try Virt-manager to easily manage your VM.
 
Top Bottom