How to setup a gaming virtual machine with GPU passthrough (QEMU, KVM, libvirt, and VFIO)

Discussion in 'Linux' started by Monopolyman, Aug 18, 2015 with 161 replies and 148,000 views.

  1. cheuk3

    cheuk3 Newbie

    Messages:
    1
    Ratings:
    0
    Hi, thanks for your great guide. I have set up the VM successfully with the vga passthrough but when i play audio in VM. it will cause my whole PC freeze and reboot...I have alreayd tried to set the qemu_pa_sample to 8192 and tried without the vga passthrough, it still freeze the system and reboot...do you have any idea? thanks!

    edit: no problem if i use hdmi audio
    edit2: ok when using hdmi audio will cause the video and audio lag...but no reboot

    Update: switched to gentoo and the issue gone :biggrin:
     
    Last edited: Mar 1, 2016
  2. 3313

    3313 Newbie

    Messages:
    4
    Ratings:
    0
    I have update:

    I managed to get this working using Lubuntu 15.04 with Qemu 2.2 and kernel 3.19.

    Using script from second post everything works besides sound.

    Using script from this link with adding "-soundhw hda" sound is working.

    I saw some users get only 3-5 % loss in perfomance. According to 3DMark I have 11% loss with both scipts.

    Now I have new problem with both script, when I use more demanding application in guest after some time host and guest freezes.
     
  3. Zelk

    Zelk Hiding Both Face And Mind Retired

    Messages:
    1,961
    Ratings:
    4,210
    This is an awesome tut. I'll have to give this a try when I come around to changing my OS to Arch.
     
    • Like Like x 1
  4. Amadren

    Amadren Newbie

    Messages:
    1
    Ratings:
    0
    Awesome tut, I only had BSODs with Windows 7 by following other tuts but everything worked fine after the installation on Windows 10 Pro with yours. I just have to search how to make the audio work and everything will be fine...

    P.S: Using Debian Jessie, I had to recompile my kernel with ACS Override and i915 patch, then I had a problem a weird bug about memory and I had to set it to "unlimited" in my config files and it started just fine after that.
     
  5. ssj4gokusama

    ssj4gokusama Newbie

    Messages:
    4
    Ratings:
    0
    Hello, since I've updated my archlinux host with the latest packages, I can't launch my vm anymore. it shows "Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken." Any ideas?

    EDIT : it looks like a problem with the libvirt package. For the moment I've done a downgrade of the package with Arch Linux Archive which can found how to do this here . And it's working now

    But if someone know how to solve the problem with the latest package (libvirt-1.3.2.1), thanks to help me.
     
    Last edited: Mar 3, 2016
  6. lowpro

    lowpro Professional Abecedarian

    Messages:
    4,589
    Ratings:
    1,890
    Doesn't VMware and (Virtualbox?) support GPU pass through? I'm on win10 as default OS so I'm not sure about Linux.

    I want to switch to Linux, but many things easily done in Windows seems hard in Linux without memorizing commands which I don't really want to do as it seems archaic.
     
  7. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,339
    Any sort of virtualization program that is ran within Windows is considered a type 2 hypervisor and doesn't support PCI passthrough, let alone GPU passthrough. For such a thing you need to be using a type 1 hypervisor so it has direct access to the hardware (type 1 means the hv is running on top of the hardware, where as a type 2 isn't as low and running within the userspace/OS). Anything that is run on Windows simply isn't close enough to the hardware to allow for something like GPU passthrough. Even on Linux, the hypervisor (in this case KVM) isn't running on top of the OS but rather at the kernel level (KVM actually stands for kernel virtual machine).

    In order to see support for this on Windows hosts, Microsoft would need to either build support for it the kernel or give users access to the kernel. The later will likely never happen and the former is rather unlikely. If they ever did such a thing, I doubt it would be on consumer versions and only on server distributions of Windows. Even then, it would be very hard for an end user to get it properly configured since they wouldn't have access to the kernel.

    I just realized I forgot to reply to this. The reason it seems hard is because it's something your not used to. Once you get used to it, it is by far and away easier than Windows. The Linux filesystem and the way everything works is extremely intuitive once you get a base understanding. I used Windows for at least 10 years, and have only been using Linux as a main OS for about years. Already, I can do things much more efficiently in Linux than I can on Windows. It's also not like it took me 2 years to learn this, this happened within the span of a few weeks with Linux. This is especially true when it comes to errors and BSODs in Windows. I am still lost half the time I get a BSOD on Windows because with Windows you don't get to see what is going on under the hood. Sure, you have minidumps which can be useful but this is nothing compared to Linux logs (which are almost always in the same directoy) or by a simple command like dmesg. After experiencing the intuitiveness and the amount of customization available on Linux, I don't ever want to go back to Windows.
     
    • Like Like x 1
    Last edited: Mar 7, 2016
  8. Gerrit Hartwig

    Gerrit Hartwig Newbie

    Messages:
    6
    Ratings:
    5
    Hi everybody,

    first of all thank you for the very intersting thread and also thank you to topic creater for this very helpful tutorial.

    I am or i try to use to my gpu passthrough with antergos linux and I followed your tutorial. After trying starting VM I got the following error.

    Fehler beim Starten der Domain: Interner Fehler: process exited while connecting to monitor: pulseaudio: pa_context_connect() failed
    pulseaudio: Reason: Connection refused
    pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
    pulseaudio: pa_context_connect() failed
    pulseaudio: Reason: Connection refused
    pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
    ALSA lib confmisc.c:768:frown:parse_card) cannot find card '0'
    ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
    ALSA lib confmisc.c:392:frown:snd_func_concat) error evaluating strings
    ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
    ALSA lib confmisc.c:1251:frown:snd_func_refer) error evaluating name
    ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
    ALSA lib conf.c:4771:frown:snd_config_expand) Evaluate error: No such file or directory
    ALSA lib pcm.c:2266:frown:snd_pcm_open_noupdate) U

    Traceback (most recent call last):
    File "/usr/share/virt-manager/virtManager/asyncjob.py", line 90, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
    File "/usr/share/virt-manager/virtManager/asyncjob.py", line 126, in tmpcb
    callback(*args, **kwargs)
    File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn
    ret = fn(self, *args, **kwargs)
    File "/usr/share/virt-manager/virtManager/domain.py", line 1402, in startup
    self._backend.create()
    File "/usr/lib/python2.7/site-packages/libvirt.py", line 1035, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
    libvirtError: Interner Fehler: process exited while connecting to monitor: pulseaudio: pa_context_connect() failed
    pulseaudio: Reason: Connection refused
    pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
    pulseaudio: pa_context_connect() failed
    pulseaudio: Reason: Connection refused
    pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
    ALSA lib confmisc.c:768:frown:parse_card) cannot find card '0'
    ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
    ALSA lib confmisc.c:392:frown:snd_func_concat) error evaluating strings
    ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
    ALSA lib confmisc.c:1251:frown:snd_func_refer) error evaluating name
    ALSA lib conf.c:4292:frown:_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
    ALSA lib conf.c:4771:frown:snd_config_expand) Evaluate error: No such file or directory
    ALSA lib pcm.c:2266:frown:snd_pcm_open_noupdate) U


    I am not really sure why this error occurs. I followed your steps in point 8 so far but also if I reverte this changes the same error messages appear.

    My main xml is following... Do not surprise that I comment the disk part out. There I got some error message that IDE is not supported for qemu. But this I try to fix afterwards.


    <?xml version="1.0" encoding="UTF-8"?>
    <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
    <name>gamingvm</name>
    <uuid>01bd2ed1-b465-4eba-b6e4-47c6ac8171c6</uuid>
    <memory unit="GB">8</memory>
    <currentMemory unit="GB">8</currentMemory>
    <vcpu placement="static">6</vcpu>
    <cpu mode="host-passthrough">
    <topology sockets="1" cores="3" threads="2" />
    </cpu>
    <os>
    <type arch="x86_64" machine="q35">hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
    <bootmenu enable="yes" />
    </os>
    <features>
    <hyperv>
    <relaxed state="on" />
    <vapic state="on" />
    <spinlocks state="on" retries="8191" />
    </hyperv>
    <acpi />
    </features>
    <clock offset="localtime">
    <timer name="hypervclock" present="yes" />
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <interface type='bridge'>
    <mac address='52:54:00:a0:41:92'/>
    <source bridge='br0'/>
    <model type='rtl8139'/>
    <rom bar='off'/>
    </interface>
    <sound model='ich6'/>
    <controller type="usb" index="0" />
    <controller type="usb" index="1" />
    <controller type="usb" index="2" />
    <controller type="usb" index="3" />
    <controller type="sata" index="0" />
    <controller type="pci" index="0" model="pcie-root" />
    <controller type="pci" index="1" model="dmi-to-pci-bridge" />
    <controller type="pci" index="2" model="pci-bridge" />
    <memballoon model="none" />
    <sound model="ich6" />
    <!--<disk type="file" device="cdrom">
    <driver name="qemu" type="raw" />
    <source file="/home/gerrit/vm/win7x64-20160308.iso" />
    <target dev="hdc" bus="ide" />
    <readonly />
    <address type="drive" controller="0" bus="1" unit="0" />
    </disk>
    <disk type="file" device="disk">
    <source file="/home/gerrit/vm/VM/win7.img" />
    <target dev="vda" bus="ide" />
    <address type="drive" bus="0" />
    </disk>-->
    </devices>
    <qemu:commandline>
    <qemu:env name="QEMU_PA_SAMPLES" value="4096" />
    <qemu:env name="QEMU_AUDIO_DRV" value="pa" />
    <qemu:arg value="-device" />
    <qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" />
    <qemu:arg value="-device" />
    <qemu:arg value="vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on" />
    <qemu:arg value="-device" />
    <qemu:arg value="vfio-pci,host=01:00.1,bus=root.1,addr=00.1" />
    <qemu:arg value="-cpu" />
    <qemu:arg value="host" />
    </qemu:commandline>
    </domain>

    I hope you can give me any hint.

    Thanks and regards
    GertPfert
     
  9. Gerrit Hartwig

    Gerrit Hartwig Newbie

    Messages:
    6
    Ratings:
    5
    i already found the clou. The thing is, that my pulse not using tcp instead linux sockets. After adding following line to client.pa from root user it is working.

    default-server = unix:/tmp/pulse
     
  10. U83R

    U83R Newbie

    Messages:
    1
    Ratings:
    0
    Hello
    It seems like a very nice guide :smile:
    I've been loosely following the whole Qemu-KVM GPU passthrough idea for a few years now. It sounds really attractive to game without rebooting and use Linux host at the same time almost like alt tab away lol. I hate rebooting!
    Anyway I have a few questions that keep me up at night :tongue:

    1. Is it possible to switch GPU without rebooting? Say a script to make Linux host switch from integrated Intel GPU to dedicated PCI-e once the VM is shut down? I guess the least what would happen is X11 on host would crash while switching to software rendering while GPUs get restarted and log you out of session? But maybe thats not even possible at all in the first place.
    The whole deal is slightly offputing for me cause I have to buy 2 dedicated GPUs if I want to game natively and on Windows VM cause integrated GPU sucks for gaming :frown:
    But I guess I could buy them if thats the only way. 1 AMD + 1 Nvidia hehehe

    2. How is the stability of both host and VM? Does it require a lot of maintenance? Does VM break after kernel updates? That's another thing I'm worried about. Currently Im running Arch and as it's rolling release rarely but sometimes minor breakages occur after certain updates. Do you just set VM up once and it runs for a very long time or does it break all the time?

    Anyway I think I'll be looking into building my next PC around this idea.

    EDIT: I'm also thinking of 21:9 Ultra Wide monitor. It would be sick to use 2 inputs on a single Ultra Wide monitor for this particular setup. Just imagine on the left Linux host on the right Windows guest! That would be sick
     
    Last edited: Mar 19, 2016
  11. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,339
    Yes you can do this but you will need to restart X in order for it to work. It also can occasionally cause kernel panics. I haven't tried it myself, but I suggest checking out this post in my subreddit: https://www.reddit.com/r/VFIO/comments/41pm1q/my_bash_script_for_rebinding_a_secondary_nvidia/

    There are some kernel updates that cause issues with performance. I personally recommend using a LTS kernel (as shown in the tutorial) so you don't have to worry about this. I've barely had to do any maintenance to the virtual machine. The only time I do is when I want to improve it and optimize it, but I don't believe I've ever had to fix something that suddenly stopped working.
     
  12. aarobc

    aarobc Newbie

    Messages:
    11
    Ratings:
    2
    Hello again. I've really appreciated the guide. I'm attempting to pass through a USB card and have run into some issues. here are the steps I've taken so far:
    detach the card from the host:
    Code:
    ❯ lspci -nn | grep ASMedia
    02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
    08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
    
    Added the entries to the end of /etc/modprobe.d/pci-stub.conf:
    Code:
    options pci-stub ids=1002:aab0,1002:6810,1b21:1242
    
    ran "sudo update-initramfs -u" and rebooted.

    in the launch script I added the pci devices to DEVLIST
    Added the entries for the groups 25 and 29 to /etc/libvirt/qemu.conf
    Code:
    ❯ find /sys/kernel/iommu_groups/ -type l | grep -E '02:00.0|08:00.0'
    /sys/kernel/iommu_groups/25/devices/0000:08:00.0
    /sys/kernel/iommu_groups/29/devices/0000:02:00.0
    
    and here is where I suspend I'm running into issues: in the xml file to define the vm, after the gpu, I tried adding the lines:
    Code:
    <qemu:arg value="-device" />                                  
    <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0'/>
    <qemu:arg value="-device" />                                  
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0'/>
    
    Would anyone be able to point me in a direction to get this working?
     
  13. ssj4gokusama

    ssj4gokusama Newbie

    Messages:
    4
    Ratings:
    0
    Hello, if you're haven't found yet, I think I have the reason why it's not working.
    You need to change the adress value if you want to use several pci devices in your vm. I quote for you my qemu:commandline column :
    Code:
      <qemu:commandline>
      <qemu:arg value="-device" />
      <qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=01:00.1,bus=root.1,addr=00.1" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=04:00.0,bus=root.1,addr=00.2" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=06:00.0,bus=root.1,addr=00.3" />
      <qemu:arg value="-cpu" />
      <qemu:arg value="host,kvm=off" />
      </qemu:commandline>
    You see? addr=00.0 then 00.1 for the next pci, etc...
    Normally it would be worked
     
  14. zipeldiablo

    zipeldiablo Newbie

    Messages:
    2
    Ratings:
    0
    No point using qemu command line here, did the same thing for my vm, just change the bus adress on source and it should work :
    Code:
    <!-- controler usb card -->
         <hostdev mode='subsystem' type='pci' managed='yes'>
           <source>
             <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </source>
           <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    
    </hostdev>
    
    On another note, i decided to give a try to the script explained on first page in order to gain some performances.
    Cset is working fine but when i launch the vm i got this :
    Code:
    error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dwin10.scope/vcpu0/cpuset.cpus': Permission denied
    The error occured when libvirt is trying to vcpupin my vcpu to cpuset.
    I run the script as root so i really don't know what the problem is, googling the issue didn't gave me proper solution for this, any idea?

    Also nothing is said on how to launch all scripts simultaneously so only one commande has to be run, i tried several options but it seems i have to write a different script for this
     
    Last edited: Apr 10, 2016
  15. aarobc

    aarobc Newbie

    Messages:
    11
    Ratings:
    2
    I'm still having no luck, I suspect the issue is because the usb card shows up in multiple vfio groups with different addresses:
    Code:
    ❯ lspci -nn | grep ASMedia     
    02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
    08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
    
    So I'm attempting to redo everything under arch. I've used ubuntu for years, but arch has proved a challenge. The problem I'm having is in the tutorial when I get to the part to run the command:
    sudo mkinitcpio -p linux-vfio-lts
    it gives an error about a file or directory not existing. Any ideas?
     
  16. zipeldiablo

    zipeldiablo Newbie

    Messages:
    2
    Ratings:
    0
    You cannot expect people to help you understand an error if you don't a copy of the error message mate.
     
  17. aarobc

    aarobc Newbie

    Messages:
    11
    Ratings:
    2
    Indeed.
    I run the command:
    pacaur -S qemu libvirt bridge-utils linux-vfio-lts
    and it installs just fine, no error messages, I then try to run the subsequent command and get an error:
    Code:
    ❯ sudo mkinitcpio -p linux-vfio-lts
    [sudo] password for ac:
    /usr/sbin/mkinitcpio: line 264: /etc/mkinitcpio.d/linux-vfio-lts.preset: No such file or directory
    ==> ERROR: Failed to load preset: `/etc/mkinitcpio.d/linux-vfio-lts.preset'
    
    I've been googling the error for the last few days and haven't found a solution yet, any guidance or suggestions would be greatly appreciated.
     
  18. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,339
    Strange that you're installing the linux-vfio-lts kernel but it doesn't come with the preset. Regardless, you can cd to /etc/mkinitcpio.d/ and create a file named linux-vfio-lts.preset with the following:
    Code:
    # mkinitcpio preset file for the 'linux-vfio-lts' package
    
    ALL_config="/etc/mkinitcpio.conf"
    ALL_kver="/boot/vmlinuz-linux-vfio-lts"
    
    PRESETS=('default' 'fallback')
    
    #default_config="/etc/mkinitcpio.conf"
    default_image="/boot/initramfs-linux-vfio-lts.img"
    #default_options=""
    
    #fallback_config="/etc/mkinitcpio.conf"
    fallback_image="/boot/initramfs-linux-vfio-lts-fallback.img"
    fallback_options="-S autodetect"
    
    
     
  19. EvilWiffles

    EvilWiffles Newbie

    Messages:
    4
    Ratings:
    0
    Hello,

    Thread caught my attention and I'm definitely interested in trying this.
    I never really got to use any OS besides Windows intensively and I was always looking for a replacement and this looks like it'll solve a lot of issues. I am running an i7 4790k (Asus Z97-A) and a GTX 970 with 16GB's of RAM, and multiple drives. My main concern is mostly to do with compatibility and drive utilization, as well as being easy to setup.

    I'm rather conflicted, I really want to try this though.
     
  20. GrandGamer1

    GrandGamer1 Newbie

    Messages:
    8
    Ratings:
    0
    Nothing to say here really. You will need to check if your CPU supports VT-d/x. Also make sure you have integrated GPU to run Linux on or a second GPU. As long as you don't have some weird hardware peripheral, everything will probably work fine. Other than that, if you have time and the will to do it, backup your Windows partition and try to set it up.

    You can also try Virt-manager to easily manage your VM.
     

Share This Page