What's new

Tutorial How to setup a gaming virtual machine with GPU passthrough (QEMU, KVM, libvirt, and VFIO)

  • Thread starter Monopolyman
  • Start date
  • Views 184,274
E

emnaea

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
Hmm I get these with lsusb:

Code:
Bus 002 Device 002: ID 8087:8000 Intel Corp.
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 1532:0037 Razer USA, Ltd
Bus 001 Device 002: ID 8087:8008 Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 0480:a006 Toshiba America Info. Systems, Inc.
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 046d:c326 Logitech, Inc.
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

my keyboard xml file is:
Code:
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x046d'/>
<product id='0xc326'/>
</source>
</hostdev>

mouse:
Code:
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x1532'/>
<product id='0x0037'/>
</source>
</hostdev>

and script:
Code:
#!/bin/bash
if [[ $(virsh list | grep gamingvm) != "" ]]
then
vm=gamingvm
fi

virsh attach-device $vm /home/emnaea/Documents/win10mouse.xml
virsh attach-device $vm /home/emnaea/Documents/win10keyboard.xml

Hmm typing
Code:
 virsh attach-device gamingvm /home/emnaea/Documents/win10keyboard.xml
works manually in the terminal.

---
Actually now that I was able to use a keyboard, I encountered a new problem: when I press enter on continue, the ovmf loops. Selecting Boot Manager or Boot Maintenance Manager leads to a black screen.
One part of the xml file I tried to follow (for nvidia GPUs) is:
Code:
  <qemu:arg value="-cpu" />
  <qemu:arg value="host,kvm=off" />
as well as:
Code:
  <os>
  <type arch="x86_64" machine="q35">hvm</type>
  <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
  <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/>
  <bootmenu enable="yes" />
  </os>

  <features>
  <hyperv>
  <relaxed state="off" />
  <vapic state="off" />
  <spinlocks state="off" />
  </hyperv>
  <kvm>
  <hidden state='on'/>
  </kvm>
  <acpi />
  </features>
  <clock offset="localtime">
  <timer name="hypervclock" present="no" />
  </clock>
 
Last edited:
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
E emnaea The issue with the input script is very strange. Perhaps it's the if statement that is no working for some reason. Try replacing "$vm" with "gamingvm" and see if that works.

I unfortunately don't have any experience with OVMF since I don't have a UEFI card. What graphics card are you passing through? Are you sure it supports UEFI? If so, I'm honestly not quite sure what the problem could be.
 
E

emnaea

Newbie
Messages
0
Reaction score
1
Points
45
Sin$
0
Hmm yeah it works when I replace $vm with gamingvm.

Ah I read incorrectly, my GPU doesn't support UEFI. One more question, how would I apply the ACS/VGA patch? Thanks
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
Hmm yeah it works when I replace $vm with gamingvm.

Ah I read incorrectly, my GPU doesn't support UEFI. One more question, how would I apply the ACS/VGA patch? Thanks
Are you using a debian based distro? If so, you need to manually patch your kernel via the use of make menuconfig. I recommend looking up tutorial on YouTube so it's easy to fix.
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
I've everything set and when I start the shell from step 7 no errors are being generated and the terminal responds: "Domain gamingvm has started. The problem is that I don't get output onto my monitor. I'm using intel integrated gpu for the host and gtx 970 for the guest. I've not installed the ACS override patch or the i915 patch. And I do get a black screen. Are those missing patches the reason i'm getting a black screen or could it be something else? Should i install those patches? If i have to install them, can you please tell me. I'm pretty noob so it would be great if you could help me out!
Thanks
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
So i tested it again and still no output, The terminal does however give a respond: Domain starts succesfully, but im getting the error /home/rwuijster/Desktop/start: line 1: #!/bin/bash: No such file or directory
How do I fix this???
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
So i tested it again and still no output, The terminal does however give a respond: Domain starts succesfully, but im getting the error /home/rwuijster/Desktop/start: line 1: #!/bin/bash: No such file or directory
How do I fix this???
Very strange. It could be because something got messed up with the actual file itself. Try copying what you have in that file into a new file and see if it works. If it still doesn't work, try using #!/usr/bin/env bash instead. However, using env shouldn't make a difference since /bin/bash should work just fine on most Linux distributions.

I've everything set and when I start the shell from step 7 no errors are being generated and the terminal responds: "Domain gamingvm has started. The problem is that I don't get output onto my monitor. I'm using intel integrated gpu for the host and gtx 970 for the guest. I've not installed the ACS override patch or the i915 patch. And I do get a black screen. Are those missing patches the reason i'm getting a black screen or could it be something else? Should i install those patches? If i have to install them, can you please tell me. I'm pretty noob so it would be great if you could help me out!
Thanks
I assume you are using an Intel core (i3/i5/i7) CPU? If so, then you need both of the ACS override patch (because of the CPU) and the i915 patch (because you're using your the iGPU). Unfortunately, I don't have much experience patching/compiling the kernel on Debian since I mainly use Arch and have the AUR at my disposal.

Below are two resources that seem pretty good to me and should be fairly easy to follow:

http://kvm-exp.blogspot.com/2015/06/howto-debian-kernel-i915-vga-arbiter.html

Just do both the ACS override and i915 arbiter patch.
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
Very strange. It could be because something got messed up with the actual file itself. Try copying what you have in that file into a new file and see if it works. If it still doesn't work, try using #!/usr/bin/env bash instead. However, using env shouldn't make a difference since /bin/bash should work just fine on most Linux distributions.


I assume you are using an Intel core (i3/i5/i7) CPU? If so, then you need both of the ACS override patch (because of the CPU) and the i915 patch (because you're using your the iGPU). Unfortunately, I don't have much experience patching/compiling the kernel on Debian since I mainly use Arch and have the AUR at my disposal.

Below are two resources that seem pretty good to me and should be fairly easy to follow:

http://kvm-exp.blogspot.com/2015/06/howto-debian-kernel-i915-vga-arbiter.html

Just do both the ACS override and i915 arbiter patch.

Alright, I'll watch the video and try to patch it. What happens if I don't patch it, will it give no output?
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
So I´ve made an new start file. Made it executable with chmod +x and when I start without sudo the terminal returns this:
./startvm: line 1: #!/bin/bash: No such file or directory
++ modprobe vfio-pci
++ modprobe kvm ignore_msrs=1
++ DEVLIST='0000:01:00.0 0000:01:00.1'
++ for dev in '$DEVLIST'
+++ cat /sys/bus/pci/devices/0000:01:00.0/vendor
++ vendor=0x10de
+++ cat /sys/bus/pci/devices/0000:01:00.0/device
++ device=0x13c2
++ '[' -e /sys/bus/pci/devices/0000:01:00.0/driver ']'
++ echo 0000:01:00.0
./startvm: line 12: /sys/bus/pci/devices/0000:01:00.0/driver/unbind: Permission denied
++ echo 0x10de 0x13c2
./startvm: line 15: /sys/bus/pci/drivers/vfio-pci/new_id: Permission denied
++ for dev in '$DEVLIST'
+++ cat /sys/bus/pci/devices/0000:01:00.1/vendor
++ vendor=0x10de
+++ cat /sys/bus/pci/devices/0000:01:00.1/device
++ device=0x0fbb
++ '[' -e /sys/bus/pci/devices/0000:01:00.1/driver ']'
++ echo 0000:01:00.1
./startvm: line 12: /sys/bus/pci/devices/0000:01:00.1/driver/unbind: Permission denied
++ echo 0x10de 0x0fbb
./startvm: line 15: /sys/bus/pci/drivers/vfio-pci/new_id: Permission denied
++ virsh start gamingvm
error: Failed to start domain gamingvm
error: internal error: early end of file from monitor: possible problem:
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
ALSA lib confmisc.c:768:frown:parse_card) cannot find card '0'
ALSA lib conf.c:4248:frown:_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:frown:snd_func_concat) error evaluating strings
ALSA lib conf.c:4248:frown:_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:frown:snd_func_refer) error evaluating name
ALSA lib conf.c:4248:frown:_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4727:frown:snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2239:frown:snd_pcm_open_noupd

++ set +x


When i do use sudo however I still get the bin/bash problem:
./startvm: 1: ./startvm: #!/bin/bash: not found
+ modprobe vfio-pci
+ modprobe kvm ignore_msrs=1
+ DEVLIST=0000:01:00.0 0000:01:00.1
+ cat /sys/bus/pci/devices/0000:01:00.0/vendor
+ vendor=0x10de
+ cat /sys/bus/pci/devices/0000:01:00.0/device
+ device=0x13c2
+ [ -e /sys/bus/pci/devices/0000:01:00.0/driver ]
+ echo 0000:01:00.0
+ echo 0x10de 0x13c2
+ cat /sys/bus/pci/devices/0000:01:00.1/vendor
+ vendor=0x10de
+ cat /sys/bus/pci/devices/0000:01:00.1/device
+ device=0x0fbb
+ [ -e /sys/bus/pci/devices/0000:01:00.1/driver ]
+ echo 0000:01:00.1
+ echo 0x10de 0x0fbb
+ virsh start gamingvm
Domain gamingvm started

+ set +x
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
I tried to go to the bin/bash folder. I can open the bin folder. The bash folder only appears, if i try to access it, it states that the folder does not exist. Do you have an suggestion or should i just try this on another distro? Like Arch?
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
Alright, I'll watch the video and try to patch it. What happens if I don't patch it, will it give no output?
Yeah. Basically, both of these patches are needed because of the way the iGPU and GPU are situated on the PCI complex. The kernel groups your hardware into IOMMU groups (IOMMU is the generic term for Intel's Vt-d). In order to passthrough a device successfully, you need to attach all the devices in the group to VFIO. We don't want to do this, so we need the ACS overide patch to seperate these devices. This is actually a functionality found on some Xeons, but it doesn't work on Intel i3/5/7 processors. Since the domain seems to be starting fine, you may not actually need this. However, if you're isntalling the i915 patch you might as well get this one as well.

The i915 patch is needed because VGA is a really old really old. Since one GPU is downstream of the other, the driver can confuse the VGA "messages" that are intended to be sent to a specific device. An alternative is using OVMF. This, however, is only an option if your GPU supports UEFI.

I tried to go to the bin/bash folder. I can open the bin folder. The bash folder only appears, if i try to access it, it states that the folder does not exist. Do you have an suggestion or should i just try this on another distro? Like Arch?
Honestly I'm not sure why bash is giving you such an issue. Have you tried using the env line? Also what is the output when you run "echo $BASH" in the terminal?

You can try Arch. Just know that it is fairly different than Debian and the install process is entirely through terminal commands. You can use a a Arch-based distro like ArchBang, Antergos, or Manjero if you have a hard time with the terminal. Once you do get it installed, the process of setting up a gaming VM will be easier, especially applying kernel patches.
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
I fixed the /bin/bash problem. I don't know what was wrong and i didn't have a typo, but if i copied /bin/bash from the terminal and pasted into the startvm file, it didn't give me the error, wut haha. This evening i'll try to patch the kernel and see if that fixes the issue. I was wondering if the stuff the terminal returns is a good sign? or is that not normal?
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
I fixed the /bin/bash problem. I don't know what was wrong and i didn't have a typo, but if i copied /bin/bash from the terminal and pasted into the startvm file, it didn't give me the error, wut haha. This evening i'll try to patch the kernel and see if that fixes the issue. I was wondering if the stuff the terminal returns is a good sign? or is that not normal?
Yes, seeing that the domain has started is a very god sign. This means all the permissions are properly sorted which is one of the most difficult things to get right when it comes to this sort of thing. After you patch the kernel, be sure to also enable the patches in your grub configuration file (I would try just using the i915 first, then doing ACS if you have issues).

If you have an NVIDIA card, be sure to read the troubleshooting steps in the OP.
 
G

Greatrickw

Newbie
Messages
0
Reaction score
0
Points
35
Sin$
7
All right, i'll have a look at the troubleshooting section. By the way, where can i find the kernel boot parameter or grub configuration file?
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
All right, i'll have a look at the troubleshooting section. By the way, where can i find the kernel boot parameter or grub configuration file?
"i915.enable_hd_vgaarb=1" for the i915 patch and "pcie_acs_override=downstream" for the ACS override.
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
okay, but WHERE do i have to enable them? :smile:
Oh sorry. The same place you enabled IOMMU in step 2b.

So that line would be
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream"

Then run "sudo update-grub" and reboot.
 
A

aarobc

Newbie
Messages
0
Reaction score
2
Points
45
Sin$
7
Greatly appreciate the guide, I'm running into an issue though:
First the QEMU bios.bin was missing, so I used the one from seebios.
When I would run the script I'd get the following error:


+ modprobe vfio-pci
+ modprobe kvm ignore_msrs=1
+ DEVLIST='0000:07:00.0 0000:07:00.1'
+ for dev in '$DEVLIST'
++ cat /sys/bus/pci/devices/0000:07:00.0/vendor
+ vendor=0x1002
++ cat /sys/bus/pci/devices/0000:07:00.0/device
+ device=0x67b0
+ '[' -e /sys/bus/pci/devices/0000:07:00.0/driver ']'
+ echo 0000:07:00.0
+ echo 0x1002 0x67b0
+ for dev in '$DEVLIST'
++ cat /sys/bus/pci/devices/0000:07:00.1/vendor
+ vendor=0x1002
++ cat /sys/bus/pci/devices/0000:07:00.1/device
+ device=0xaac8
+ '[' -e /sys/bus/pci/devices/0000:07:00.1/driver ']'
+ echo 0000:07:00.1
+ echo 0x1002 0xaac8
+ virsh start gamingvm
error: Failed to start domain gamingvm
error: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

So out of curiosity, I commented out all the storage devices, and the vm did indeed start up and show output on the second monitor, so I know everything else works.

However, without being able to assign drives the setup is pretty useless, I'm hoping someone can provide me some direction.
Relevent info:
Ubuntu 15.10 64 bit. AMD cpu/gpu. Kernel: 4.2.0-16-generic.
 
Monopolyman

Monopolyman

PC Gamer and Tech Enthusiast
Retired
Mr. Nice Guy Tutorial Creator Jokester
Messages
6,409
Reaction score
4,549
Points
1,105
Sin$
7
Greatly appreciate the guide, I'm running into an issue though:
First the QEMU bios.bin was missing, so I used the one from seebios.
When I would run the script I'd get the following error:


+ modprobe vfio-pci
+ modprobe kvm ignore_msrs=1
+ DEVLIST='0000:07:00.0 0000:07:00.1'
+ for dev in '$DEVLIST'
++ cat /sys/bus/pci/devices/0000:07:00.0/vendor
+ vendor=0x1002
++ cat /sys/bus/pci/devices/0000:07:00.0/device
+ device=0x67b0
+ '[' -e /sys/bus/pci/devices/0000:07:00.0/driver ']'
+ echo 0000:07:00.0
+ echo 0x1002 0x67b0
+ for dev in '$DEVLIST'
++ cat /sys/bus/pci/devices/0000:07:00.1/vendor
+ vendor=0x1002
++ cat /sys/bus/pci/devices/0000:07:00.1/device
+ device=0xaac8
+ '[' -e /sys/bus/pci/devices/0000:07:00.1/driver ']'
+ echo 0000:07:00.1
+ echo 0x1002 0xaac8
+ virsh start gamingvm
error: Failed to start domain gamingvm
error: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

So out of curiosity, I commented out all the storage devices, and the vm did indeed start up and show output on the second monitor, so I know everything else works.

However, without being able to assign drives the setup is pretty useless, I'm hoping someone can provide me some direction.
Relevent info:
Ubuntu 15.10 64 bit. AMD cpu/gpu. Kernel: 4.2.0-16-generic.
In the xml, change it all the ide drives to sata. Then once your booted up change a non OS drive to virtIO and install driver, then change your OS drive to virtIO. If you only use one drive, you can install the virtIO drivers from a windows installation disk
 
Top Bottom
Login
Register