What's new

Tutorial How to setup a gaming virtual machine with GPU passthrough (QEMU, KVM, libvirt, and VFIO)

  • Thread starter Monopolyman
  • Start date
  • Views 184,278
P

Pictou

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
I am going to cry. It finally worked!

i am going through the setup now (installing windows) and then getting networking up and running and lastly try a few of those tweaks. If all goes according to plan, I am going to write down what I learned from this so hopefully it help others in the future (and improve your tutorial).

EDIT: I have synergy up and the cpu opitimizations going. My question right now is: Do I need to install nvidia drivers in the VM? It seems to error out on me.

Secondly, The VM scaling is off aka it does not fill my full second monitor. Any idea how to fix?

Thank you once again (because this is amazing!)
 
Last edited:
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
I am going to cry. It finally worked!

i am going through the setup now (installing windows) and then getting networking up and running and lastly try a few of those tweaks. If all goes according to plan, I am going to write down what I learned from this so hopefully it help others in the future (and improve your tutorial).

EDIT: I have synergy up and the cpu opitimizations going. My question right now is: Do I need to install nvidia drivers in the VM? It seems to error out on me.

Secondly, The VM scaling is off aka it does not fill my full second monitor. Any idea how to fix?

Thank you once again (because this is amazing!)
Just curious for future reference, did purging and removing dependencies fix it? Or was it something else?

Also yes, you install the Nvidia drivers just like you would on regular windows. That should also fix your resolution issue (hopefully) .
 
P

Pictou

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
Just curious for future reference, did purging and removing dependencies fix it? Or was it something else?

Also yes, you install the Nvidia drivers just like you would on regular windows. That should also fix your resolution issue (hopefully) .

Yes, the purge and reinstall fixed my libvirt issues. It had a lot of config afterwards but it was booted up afterwards.

As far as the drivers, it installed my nvidia drivers but they are obviously not being used because i cannot access the nvidia control panel nor does windows recognizes that I have the most up to date driver. I think this might be a factor:

"kvm=off will hide the kvm hypervisor signature, this is required for NVIDIA cards, since its driver will refuse to work on an hypervisor and result in Code 43 on windows (unless you're using a QUADRO)"

Is this something I should do? (I am not receiving the error code that I know of. Just the driver is not wanting to properly install)

I am considering trying out arch linux again because the debian base is starting to rub off on me. Is the configuration the same or what is different?
 
Last edited:
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
As far as the drivers, it installed my nvidia drivers but they are obviously not being used because i cannot access the nvidia control panel nor does windows recognizes that I have the most up to date driver. I think this might be a factor:

"kvm=off will hide the kvm hypervisor signature, this is required for NVIDIA cards, since its driver will refuse to work on an hypervisor and result in Code 43 on windows (unless you're using a QUADRO)"

Is this something I should do? (I am not receiving the error code that I know of. Just the driver is not wanting to properly install)
Ah yes, I forgot that NVIDIA gives VM users a hard time. Try adding ",kvm=off" to the host CPU argument at the bottom of the XML file. If you are still getting problems, turn off Hyper-v enlightenments. Do so by searching for these elements and changing their values as shown below:
Code:
  <hyperv>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8191'/>
  </hyperv>
and

Code:
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
  </clock>

I am considering trying out arch linux again because the debian base is starting to rub off on me. Is the configuration the same or what is different?
If you could follow this tutorial with a general understanding of what you were doing, doing it on Arch is more or less the same and realitvely easy. In fact, I would even say it's easier because of the AUR. For instance, to patch the kernel with ACS override and the i915 arbiter patch you just use a package manager with AUR support (or manually downlaod from AUR) and install the linux-vfio package from the AUR.

The one thing I don't have yet is sound. This isn't because I tried and failed, but rather because I want to see if I can get JACK properly working and/or find a more elegant solution.
 
P

Pictou

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
If you could follow this tutorial with a general understanding of what you were doing, doing it on Arch is more or less the same and realitvely easy. In fact, I would even say it's easier because of the AUR. For instance, to patch the kernel with ACS override and the i915 arbiter patch you just use a package manager with AUR support (or manually downlaod from AUR) and install the linux-vfio package from the AUR.

The one thing I don't have yet is sound. This isn't because I tried and failed, but rather because I want to see if I can get JACK properly working and/or find a more elegant solution.

I miss aur when using debian based systems. It just is so much more comprehensive. I didn't know much to begin with but all this fiddling has caused me to learn a lot.

Also, for anyone that may be reading this in the future. Do not do what I did and accidentally disable your video driver in windows because your GPU isn’t working. YOu will have to reinstall (like me).
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
I miss aur when using debian based systems. It just is so much more comprehensive. I didn't know much to begin with but all this fiddling has caused me to learn a lot.

Also, for anyone that may be reading this in the future. Do not do what I did and accidentally disable your video driver in windows because your GPU isn’t working. YOu will have to reinstall (like me).
For future reference, I believe you could just emulate use emulated graphics (like QXL) to reinstall the driver.
 
P

Pictou

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
For future reference, I believe you could just emulate use emulated graphics (like QXL) to reinstall the driver.
It was done. I jumped the gun. Thank you for letting me know though.

Also, Virt-manager is pretty nifty if you haven’t already checked it out.
 
Last edited:
R

rayneayami

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
I am having an issue with starting my VM. I am using an radeon hd 5450 for my host card, and a radeon r9 380 for my machine. When I try to start my VM i get the errors:
error: Failed to start domain gaming
error: internal error: process exited while connecting to monitor: Home directory not accessible: Permission denied
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Permission denied
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
 
Last edited:
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
I am having an issue with starting my VM. I am using an radeon hd 5450 for my host card, and a radeon r9 380 for my machine. When I try to start my VM i get the errors:
error: Failed to start domain gaming
error: internal error: process exited while connecting to monitor: Home directory not accessible: Permission denied
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Permission denied
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
Are you running the script as root and have you done step 7c with the proper group number (13)?
 
R

rayneayami

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
Yes, I just double checked my qemu.conf file, and I am both trying root and sudo

Edit:
I got it to work I had to add
"vfio_iommu_type1.allow_unsafe_interrupts=1"

to my grub
 
Last edited:
R

rayneayami

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
I'm getting a no boot device found now, after fixing the above issue

Also, Is there a way to have both host and VM sharing the keyboard/mouse without synergy project
 
Last edited:
B-rad

B-rad

Press On
Legendary Veteran 10th Anniversary Forum Addict
Messages
3,285
Reaction score
936
Points
425
Sin$
0
I was reading this, amazingly great tutorial by the way. And I was thinking, do you believe this would allow someone to create a game-server? Granted you could find a host that allowed it. Noticed it works with "KVM" which is a type of VPS (virtual private server).

Also maybe you could answer my question on exactly how this works. It caught my attention as soon as I read the title. Anyways props on this. It must have taken you a long time to write it.
 
Z61

Z61

Some times our saints are sinners
Retired
Programmer Forum Addict Odysseus' Summit
Messages
5,468
Reaction score
3,429
Points
1,042
Sin$
0
I was reading this, amazingly great tutorial by the way. And I was thinking, do you believe this would allow someone to create a game-server? Granted you could find a host that allowed it. Noticed it works with "KVM" which is a type of VPS (virtual private server).

Also maybe you could answer my question on exactly how this works. It caught my attention as soon as I read the title. Anyways props on this. It must have taken you a long time to write it.
most game servers don't even need to make use of a GPU.
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
I'm getting a no boot device found now, after fixing the above issue

Also, Is there a way to have both host and VM sharing the keyboard/mouse without synergy project
Sorry for the late reply, but are you sure you download the proper .iso and have the correct path? You can also try adding "<boot dev='cdrom'/>" in the <os> element to ensure the VM is booting from the CD.
I was reading this, amazingly great tutorial by the way. And I was thinking, do you believe this would allow someone to create a game-server? Granted you could find a host that allowed it. Noticed it works with "KVM" which is a type of VPS (virtual private server).

Also maybe you could answer my question on exactly how this works. It caught my attention as soon as I read the title. Anyways props on this. It must have taken you a long time to write it.
The answer is yes and no. Yes, you could technically use a driver like QXL or Cirrus and connect to it via a remote client. However, this can introduce a whole lot of problems (ie. latency) while gaming. In order for it to work really well you would need to make your own optimizations to the drivers, and this is why there aren't that many cloud gaming services around. For LAN though, you could use something like Steam in-home streaming and just use the Windows VM to stream games to your host OS or other PCs.

most game servers don't even need to make use of a GPU.
I think he meant cloud gaming servers, not multilayer servers.
 
B-rad

B-rad

Press On
Legendary Veteran 10th Anniversary Forum Addict
Messages
3,285
Reaction score
936
Points
425
Sin$
0
Z61 Z61 Monopolyman Monopolyman

I'm not really a pc gaming guy so I wouldn't know which was why I wanted to ask. I appreciate you letting me know the details. A friend of mine might make good use out of this tutorial. I'll send it his way.
 
PsykoticLeprechaun

PsykoticLeprechaun

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
0
Followed a few different tutorials for doing GPU passthrough and having tried a couple of approaches this one and one other both get me the furthest. I have hit a bit of an issue though: I have created a custom kernel with the VGA arbiter patch (and the pcie_acs_passthrough patch just in-case I need it). Since and before doing so, when I pass the PCI IDs to PCI-Stub for my NVidia card and associated HD audio, on reboot I get an error saying "Your graphics card, display and input devices couldn't be detected." at which point I have to go in via the command line to revert the changes to drm.conf and pci-stub.conf. I think it has something to do with having NVidia drivers already installed on my machine prior to trying this out, but unsure if that's the case and what other possible actions I could take than just outright removing the NVidia drivers.

As a side note: I currently am booting via the iGPU (from UEFI BIOS to login) but then when I get to login, while the video output is coming from the motherboard HDMI (so I assume iGPU is still powering the display) I can attach a second monitor to my GPU which stretches my display (rather than the usual multi-display behaviour of extending) across both monitors. I have also noted activity on the NVidia card through the NVidia control panel.

Edit: thought I would add that I have added the i915.preliminary_hw_support=1 i915.enable_hd_vgaarb=1 flags to the boot line (need both because I built on the 4.2.2 kernel version which doesn't have explicit support for Skylake yet), intel_iommu=on too.
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
Followed a few different tutorials for doing GPU passthrough and having tried a couple of approaches this one and one other both get me the furthest. I have hit a bit of an issue though: I have created a custom kernel with the VGA arbiter patch (and the pcie_acs_passthrough patch just in-case I need it). Since and before doing so, when I pass the PCI IDs to PCI-Stub for my NVidia card and associated HD audio, on reboot I get an error saying "Your graphics card, display and input devices couldn't be detected." at which point I have to go in via the command line to revert the changes to drm.conf and pci-stub.conf. I think it has something to do with having NVidia drivers already installed on my machine prior to trying this out, but unsure if that's the case and what other possible actions I could take than just outright removing the NVidia drivers.

As a side note: I currently am booting via the iGPU (from UEFI BIOS to login) but then when I get to login, while the video output is coming from the motherboard HDMI (so I assume iGPU is still powering the display) I can attach a second monitor to my GPU which stretches my display (rather than the usual multi-display behaviour of extending) across both monitors. I have also noted activity on the NVidia card through the NVidia control panel.

Edit: thought I would add that I have added the i915.preliminary_hw_support=1 i915.enable_hd_vgaarb=1 flags to the boot line (need both because I built on the 4.2.2 kernel version which doesn't have explicit support for Skylake yet), intel_iommu=on too.
Strange that you are seeing that behavior even when you have pci-stub as a dependency for drm. Theoretically, that should ensure that pci-stub is loaded first and NVIDIA drivers won't try to be loaded.

What do you get when you run "dmesg | grep stub" after booting?

You can try to blacklist the driver that the gpu is using. Run "lspci -k | grep VGA -A3" to see what driver the gpu is using. Then add "blacklist [module name]" to /etc/modprobe.d/blacklist.


Edit: Just realized you said your on kernel 4.2. Instead of using pci-stub, you can use vfio-pci. To do so:

1. Run "sudo gedit /etc/modprobe.d/vfio.conf"
2. Add the line "options vfio-pci ids=1002:6810,1002:aab0", except with the ID of your GPU/Audio.
3. Run "sudo gedit /etc/modprobe.d/drm.conf"
4. Add "softdep drm pre: vfio-pci"
5. Run "sudo update-initramfs -u"
6. Reboot
 
Last edited:
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
I have found a much better way to do audio (compared to PA system-wide) and still use PA. Before I explain the way I am using, I do want to mention that using JACK is always an option. However, I found that was very resource heavy since it's really designed to be used in professional audio setups.

What I did:
In my user's default.pa file, I loaded the tcp module and made it so any client connecting from the localhost is authorized. I then opened a terminal as root and changed the default server to the localhost in root's client.conf PA file. Now, whenever a program uses pulseaudio and is run as root, it connects to the PA server run by my unprivileged user. I found this solution here, there you can find a more clear-cut tutorial on what to do.
 
E

emnaea

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
Hi Monopolyman, I'm having trouble attaching my input devices to the VM. When I try to execute the script in step six, I get:

initIO.sh: 2: initIO.sh: [[: not found
error: command 'attach-device' requires <file> option
error: command 'attach-device' requires <file> option

I am using an intel cpu and am trying the OVMF(which I think is running on the GPU-linked monitor).
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
Hi Monopolyman, I'm having trouble attaching my input devices to the VM. When I try to execute the script in step six, I get:

initIO.sh: 2: initIO.sh: [[: not found
error: command 'attach-device' requires <file> option
error: command 'attach-device' requires <file> option

I am using an intel cpu and am trying the OVMF(which I think is running on the GPU-linked monitor).
In step 6d, make sure [Mouse] and [Keyboard] point to the XML files you created in step 6b/c.

If it still doesn't work, post your scripts from step 6 and the output you get.
 
Last edited:
Top Bottom
Login
Register