How to setup a gaming virtual machine with GPU passthrough (QEMU, KVM, libvirt, and VFIO)

Discussion in 'Linux' started by Monopolyman, Aug 18, 2015 with 161 replies and 149,539 views.

  1. Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    This tutorial was originally written for debian-based systems. However, that is no longer my daily OS and I consider Arch-based distros to be a much better host system for vfio, especially for beginners. If you want to view the debian-based tutorial view the second post in this thread.

    Prerequisites
    IOMMU compatible CPU and motherboard

    You need a CPU that is capable of IOMMU. On Intel this is called vt-d. To check if your CPU has vt-d, search for it here. On AMD it's called AMD-Vi. Most AMD CPUs and APUs have it. It's also important you have a motherboard that is capable of supporting it and it's enabled. You may need to flash to a different version for it to work.
    Arch-based distro
    This tutorial was written based on what I've done on Arch Linux. The steps should be very similar in most other Arch-based distros (Ie. Manjaro). This can certainly be done in other distros, but the steps will be different aside from the XML.
    Two GPUs (including integrated)
    In order for this to work, you need two GPUs. One is used by the host and one is used by the guest. iGPUs can be used for the host.
    Tutorial

    1. Install necessary packages
    1a. Terminal: "pacman -S gedit"

    1b. Download and install pacaur. If you used an Arch-based distro like Manjaro, it likely comes pre-installed with an AUR helper, such as yaourt. If you installed Arch from scratch, you need to build it yourself. A tutorial on how to build AUR packages in general can be found here. Here is a script to automate the process, however I do not maintain this script and it could be out of date.

    Note: You don't have to use gedit and pacaur and they can be substituted by any alternative. However, this tutorial only uses pacaur and gedit.

    1c. Terminal: "pacaur -S qemu libvirt bridge-utils linux-vfio-lts"

    Note: This command can take awhile (normally between 30 minutes and two hours depending on your system) since you are compiling the kernel.

    Note:
    linux-vfio-lts can be substituted with linux-vfio for the bleeding edge kernel. However, I strongly recommend that you stick with linux-vfio-lts because bugs can be introduced in newer kernels.

    Note:
    linux-vfio-lts or linux-vfio is not needed if you are not using an intel iGPU for the host and you are sure your processor has ACS capabilities. If you don't have a HEDT or Xeon, or you are unsure if your CPU supports ACS, I recommend installing the vfio kernel just in case.
    2. Load necessary modules
    2a. Terminal: "sudo gedit /etc/default/grub" and add "rd.modules-load=vfio-pci intel_iommu=on" inside the quotes on the GRUB_CMDLINE_LINUX_DEFAULT line. Example:
    Code:
    GRUB_CMDLINE_LINUX_DEFAULT="rd.modules-load=vfio-pci intel_iommu=on"
    Note: Add "iommu=pt iommu=1" instead of "intel_iommu=on" for AMD CPUs.

    2b. Terminal "sudo grub-mkconfig -o /boot/grub/grub.cfg"

    2c. Terminal: "sudo gedit /etc/mkinitcpio.conf"

    2d. Add the necessary modules in the MODULES line as shown below
    Code:
    MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
    2e. Terminal: "sudo mkinitcpio -p linux-vfio-lts"
    3. Detach GPU from host
    3a. Terminal: "lspci -nn | grep AMD" (grep NVIDIA if you are passing through an NVIDIA GPU). The output should be something like this
    Code:
    01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] [1002:6810]
    01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]

    3b. Terminal: "sudo gedit /etc/modprobe.d/vfio.conf". Add "options vfio-pci ids=" followed by numbers in the brackets at the end of the lspci output separated with each device separated by commas. Be sure to only include the IDs (one for GPU one for HDMI audio) for the card you want to use in the VM. Example:
    Code:
    options vfio-pci ids=1002:aab0,1002:6810
    3c. Terminal: "sudo mkinitcpio -p linux-vfio-lts"

    3d. Reboot and ensure the GPU is being claimed by vfio. Do this by running "dmesg | grep vfio" and if you see something like below, you are good to go.
    Code:
    [  0.484086] VFIO - User Level meta-driver version: 0.3
    [  0.497805] vfio_pci: add [1002:67b1[ffff:ffff]] class 0x000000/00000000
    [  0.511123] vfio_pci: add [1002:aac8[ffff:ffff]] class 0x000000/00000000
    [ 3985.682858] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
    [ 3985.683003] vfio_ecap_init: 0000:01:00.0 hiding ecap [email protected]
    [ 3985.683010] vfio_ecap_init: 0000:01:00.0 hiding ecap [email protected]
    

    Note: At this point you should have cables plugged into both the host video card and the card that you want to use for gaming.

    4. Create virtual disk(s) for VM
    Terminal: "qemu-img create [Desired Location] [Desired Size]G ". So a 50GiB image in my documents would look like "qemu-img create ~/Documents/disk.img 50G"
    5. Define the virtual machine
    5a. Make an xml file and paste the following code into it.
    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
      <name>gamingvm</name>
      <uuid>01bd2ed1-b465-4eba-b6e4-47c6ac8171c6</uuid>
      <memory unit="GB">[Memory]</memory>
      <currentMemory unit="GB">[Memory]</currentMemory>
      <vcpu placement="static">[Cores*Threads]</vcpu>
      <cpu mode="host-passthrough">
      <topology sockets="1" cores="[Core]" threads="[Threads]" />
      </cpu>
      <os>
      <type arch="x86_64" machine="q35">hvm</type>
      <loader>/usr/share/qemu/bios.bin</loader>
      <bootmenu enable="yes" />
      </os>
      <features>
      <hyperv>
      <relaxed state="on" />
      <vapic state="on" />
      <spinlocks state="on" retries="8191" />
      </hyperv>
      <acpi />
      </features>
      <clock offset="localtime">
      <timer name="hypervclock" present="yes" />
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>destroy</on_crash>
      <devices>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <interface type='bridge'>
      <mac address='52:54:00:a0:41:92'/>
      <source bridge='br0'/>
      <model type='rtl8139'/>
      <rom bar='off'/>
      </interface>
      <sound model='ich6'/>
      <controller type="usb" index="0" />
      <controller type="usb" index="1" />
      <controller type="usb" index="2" />
      <controller type="usb" index="3" />
      <controller type="sata" index="0" />
      <controller type="pci" index="0" model="pcie-root" />
      <controller type="pci" index="1" model="dmi-to-pci-bridge" />
      <controller type="pci" index="2" model="pci-bridge" />
      <memballoon model="none" />
      <sound model="ich6" />
      <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" />
      <source file="[OS disc image]" />
      <target dev="hdc" bus="sata" />
      <readonly />
      <address type="drive" controller="0" bus="1" unit="0" />
      </disk>
      <disk type="file" device="disk">
      <source file="[Primary HDD image]" />
      <target dev="vda" bus="sata" />
      <address type="drive" bus="0" />
      </disk>
      <disk type="file" device="disk">
      <source file="[Secondary HDD image]" />
      <target dev="vdb" bus="sata" />
      <address type="drive" bus="1" />
      </disk>
      </devices>
      <qemu:commandline>
      <qemu:env name="QEMU_PA_SAMPLES" value="4096" />
      <qemu:env name="QEMU_AUDIO_DRV" value="pa" />
      <qemu:arg value="-device" />
      <qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=[GPU],bus=root.1,addr=00.0,multifunction=on,x-vga=on" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=[Audio],bus=root.1,addr=00.1" />
      <qemu:arg value="-cpu" />
      <qemu:arg value="host" />
      </qemu:commandline>
    </domain>

    5b. Replace [OS disc image] with the path to an .iso for an install disc. I highly recommend using Windows 10 considering it's future support for dx12 and it has given me far less problems than Windows 8. This tutorial is also written for Win10, but Windows 8 is practically the same.

    5c. Replace [Memory] with the amount of memory you want the VM to have. Note the value is in Gigabytes. Be sure not to give more than your system ram amount and leave some space for the host.

    5d. Replace [Primary HDD Image] with the path to the image you created in step four. You can also add the path for a second hdd where [Secondary HDD Image]. You can also add more HDDs by copying the <disk></disk> elements and increasing the number of bus and increasing the letter on vdx (so third disk would be bus=2 vdc, fourth would be bus=3, vdd, etc.).

    5e. Choose the amount of cores and threads you want your VM to have. If you're on a processor with 8 threads, I recommend 3 cores and 2 threads (2 threads per core). In this instance, I would change [Cores*Threads] to 6, [Cores] to 3, and [Threads] to 2.

    5f. Change [GPU] and [Audio] to the slot/function numbers from lspci. For example, I would use "01:00.0" and "01:00.1"

    5f. Terminal: "virsh define [Path to XML file]". The terminal should return this if everything is correct.
    Code:
    Domain gamingvm defined from [Path to XML file]

    6. Setting up input devices
    6a. Create a new XML file like the following
    Code:
    <hostdev mode='subsystem' type='usb' managed='no'>
    <source>
    <vendor id='0x[Before Colon]'/>
    <product id='0x[After Colon]'/>
    </source>
    </hostdev>
    6b. Terminal: "lsusb". Identify which device is your keyboard. Then look at the numbers after "ID" from the output. In the XML file, replace [Before Colon] with the set of numbers/letters before the colon, then replace [After Colon] with the set of numbers/letters after the colon. Be sure to keep the 0x. It sounds a bit confusing, but here is an example:

    My keyboard from lsusb
    Code:
    Bus 002 Device 005: ID 413c:2011 Dell Computer Corp. Multimedia Pro Keyboard
    My XML file would to look like this
    Code:
    <hostdev mode='subsystem' type='usb' managed='no'>
    <source>
    <vendor id='0x413c'/>
    <product id='0x2011'/>
    </source>
    </hostdev>
    Note: You could just add the hostdev element in the devices element in the XML file you created in step four and define the vm again. However, putting the USB devices in separate files allows you to attach/detach much easier. This is especially useful if the VM fails to boot and you have no way of interacting with your host OS.

    6c. Repeat that for your mouse. If your mouse/keyboard has multiple entries, be sure to do it all of them (for instance, a wireless mouse would show up twice. Once for the USB cord, and once for the wireless adapter).

    6d. Create a file with the following content
    Code:
    #!/bin/bash
    if [[ $(virsh list | grep gamingvm) != "" ]]
    then
    vm=gamingvm
    fi
    
    virsh attach-device $vm [Mouse]
    virsh attach-device $vm [Keyboard]
    6e. Replace [Mouse] and [Keyboard] with paths to the files you created in the previous steps. If you had to make more than two files, just add lines with the path to those at the end

    6f. Make this file executable by either right clicking it and going into the properties or using "chmod +x".

    6g. Make a copy of the file from 6d, except replace the word “attach” with “detach” so it says “virsh detach-device $vm ...”. This script can be used for detaching your input devices. Make sure it is executable.

    7. Preparing virtual machine
    7a. Create the following file
    Code:
    #!/bin/bash
    if [[ $EUID -ne 0 ]]; then
      echo "This script must be run as root" 1>&2
      su
      exit 0
    fi
    
    set -x
    
    echo 1 > /sys/module/kvm/parameters/ignore_msrs
    virsh start gamingvm
    
    if [[ $(virsh list | grep gamingvm) == "" ]]
    then
      exit 0
    fi
    
    set +x
    
    exit 0
    7b. Terminal: "sudo gedit /etc/libvirt/qemu.conf"

    7c. Add the following lines (or find them and uncomment them)
    Code:
    user = "root"
    group = "root"
    cgroup_device_acl = [
      "/dev/null", "/dev/full", "/dev/zero",
      "/dev/random", "/dev/urandom",
      "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
      "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
      "/dev/vfio/[GROUP]"
    ]
    clear_emulator_capabilities = 0
    
    7d. Terminal: "find /sys/kernel/iommu_groups/ -type l". Now look for what directory the GPU you are passing through is in. Replace "[GROUP]" with whatever number group your device is in.
    8. Preparing PulseAudio
    Note: If you are going to use HDMI audio, you do not need to follow these steps.

    8a.
    Terminal: “cp /etc/pulse/default.pa ~/.pulse/”

    8b. Terminal: "gedit ~/.pulse/"

    8c. Add the following to the end of the file "load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1"

    8d. Terminal: “su"

    8d. Terminal: “mkdir ~/.pulse/"

    8e. Terminal: "echo "default-server = 127.0.0.1" > ~/.pulse/client.conf"
    9. Networking
    Note: This uses netctl which should already be installed. If for some reason it's not, install it via "pacman -S netctl" and configure it by referring to the Arch Wiki page.

    9a. Terminal: “cd /etc/netctl”


    9b. Terminal: "sudo gedit [profile name]" [profile name] is the profile for the device that you use to get your interenet connection. It is likely called wired. If you are unsure run the command "ls" to view the available profiles.

    9c. Change the "IP=... " line to "IP=no". If the Address, DNS, or Gateway lines are present, delete them. Take note of the "Interface=..." line for future reference.

    9d. Terminal: "sudo gedit bridge". This will create a new bridge profile.

    9e. Add the following to file

    Code:
    Description="Bridge connection"
    Interface=br0
    Connection=bridge
    BindsToInterfaces=([Interface])
    IP=static
    Address='192.168.1.202/24'
    Gateway='192.168.1.1'
    ## Ignore (R)STP and immediately activate the bridge
    SkipForwardingDelay=yes
    
    9e. Change [Interface] to the name of the interface you took note of in step 10c. Also feel free to change the IP address to what you want it to be.

    9f. Terminal: "netctl enable bridge"

    9g. Reboot to ensure networking on the host is still working.​

    10. Install The OS
    10a. Look at the "Kernel Patches" section of this tutorial below. Check to see if you need to enable those patches. If you do, enable the patches before you procede.​

    10b. Make the file from step seven executable and run it. If everything goes smoothly, switching to the input of your GPU should display the installation screen. If not, scroll down to the troubleshooting portion of the thread.

    10c. Run the file you made in step six to enable keyboard and mouse.

    10d. Install the OS like usual.

    10e. You now (hopefully) have a VM capable of playing Windows games!
    Kernel Patches
    Note:
    Both of these patches require the linux-vfio-lts or linux-vfio kernel to be installed. This should have been installed on step 1.

    ACS override patch
    Note: This patch is necessary if your CPU doesn't support ACS capabilities. None of regular consumer processors (i3, i5, or i7) support ACS. The only processors that support it are Xeons and some HEDT CPUs.

    1. Terminal: "gedit /etc/default/grub"

    2. On the GRUB_CMDLINE_LINUX_DEFAULT line, add "pcie_acs_override=downstream". Example:
    Code:
    [code]GRUB_CMDLINE_LINUX_DEFAULT="pcie_acs_override=downstream rd.modules-load=vfio-pci intel_iommu=on"
    3. Terminal: "sudo grub-mkconfig -o /boot/grub/grub.cfg"
    i915 VGA arbiter patch
    Note: This step is needed if you are using any intel iGPU for your host system.

    1. Terminal: "gedit /etc/default/grub"

    2. On the GRUB_CMDLINE_LINUX_DEFAULT line, add "i915.enable_hd_vgaarb=1". Example:
    Code:
    [code]GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_hd_vgaarb=1 rd.modules-load=vfio-pci intel_iommu=on"

    Troubleshooting

    This (http://vfio.blogspot.com/2014/08/vfiovga-faq.html) blog post has some of the common errors that you can run into and the solutions. I suggest that is the first place you turn if you run into an issue.

    Black screen on guest and/or host has corrupted graphics
    If you are using an Intel iGPU for the host, you make sure you have the i915 VGA arbiter patch enabled. If you don't want to use the VGA patch, you can set up OVMF.

    Sound isn’t working properly
    The first thing I recommend doing is playing around with the QEMU_PA_SAMPLES. Some may find that a lower value, like 128 or 512, will yield better results (less choppiness).

    NVIDIA drivers not working in guest (or code 43)
    NVIDIA doesn't like their cards being used in virtual machines. To get around this, you need to add ,kvm=off to your -cpu argument. (ie. host,kvm=off). If you are still getting issues, you need to disable Hyper-V enlightenments by deleting or commenting the following elements.
    Code:
    <hyperv>
    <relaxed state='off'/>
    <vapic state='off'/>
    <spinlocks state='off'/>
    </hyperv>
    ...
    
    <clock>
    <timer name='hypervclock' present='no'/>
    </clock>
    
    Tweaks and Optimizations
    GPU performance can’t really get much better (it's already almost bare-metal), but there is a lot of room for improvement in other areas, especially the CPU. There are also some ways that can increase the usability and functionality of your VM.

    Setting up Synergy
    Syngery is a way to share your mouse and keyboard between two computers over a network. This allows you to use both the guest OS and host OS at the same time and have your input work between the two as if it’s a dual monitor setup.

    1. Set a static IP on your guest OS. (http://portforward.com/networking/static-ip-windows-8.htm)

    2. Download Synergy on the guest OS from https://synergy-project.org/. Alternatively, you can compile the source yourself for free.

    3. Configure it so it will act as a server (share mouse+keyboard) and have your linux hostname in the appropriate place when configuring screens.

    4. Download Synergy on the host (either from the website or by using “pacaur -S synergy”)

    5. In the file you created in step 6d, add the line “sudo synergyc [Guest IP]:24800”.

    6. In the detach file created in step 6g, add the lines “killall synergy” and “killall synergyc”.

    Backing memory with hugepages
    This can only be done if you have plenty of free RAM on your system. Using huge pages will ensure that the VMs memory does not get swapped which will increase pages. Certain processors also have larger IOMMU page tables which will be taken advantage of when using hugeapges.

    1.
    Terminal: "sudo gedit /etc/fstab"

    2.
    Add the line "hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0". Make sure this is the only hugetlbfs line.

    3. Terminal: "su"

    4. Terminal: "umount /dev/hugepages"

    5. Terminal: "mount /dev/hugepages" and then run "exit" to exit superuser.

    6. Terminal: "grep Hugepagesize /proc/meminfo". Take note of this size.

    7. Use a calculator to determine how many hugepages you need to back your VM. Be sure to include video RAM. For example, my hugepage size is 2048 kB (2mB) which is what your will likely be. I want the VM to have 10gB and my GPU has 4gB of RAM. That's 14gB total. 14gB = 14336 gB. 14336 mB / 2mB = 7168.

    8. Terminal: "echo [Number from the step above] > /proc/sys/vm/nr_hugepages"

    9. Terminal: "grep HugePages_Total /proc/meminfo" this will display the amount of hugepages. If it is less than you set, you don't have enough free RAM on your computer.

    10. In the script you made in step 7a add "echo [Number from the step above] > /proc/sys/vm/nr_hugepages"

    Note: You can optinonally reclaim the memory for the host after you shut off the VM by running "echo 0 > /proc/sys/vm/nr_hugepages".

    Improve CPU performance with Cpuset and CPU pinning
    CPU pinning allows you to assign real cores to virtual cores, thus increasing the performance of the virtual core. However, we also need to use cpusets to ensure that other host processes don’t interfere with that core.

    1. Terminal: "sudo gedit [XMl file from step five]"

    2. Below the CPU element, add the following:
    Code:
     <cputune>
        <vcpupin vcpu="0" cpuset="2"/>
        <vcpupin vcpu="1" cpuset="3"/>
        <vcpupin vcpu="2" cpuset="4"/>
        <vcpupin vcpu="3" cpuset="5"/>
        <vcpupin vcpu="4" cpuset="6"/>
        <vcpupin vcpu="5" cpuset="7"/>
      </cputune>
    This setup is for a vm with six cores, and it pins each virtual core to a real core. You can change it however you want to fit your needs, just know vcpu is the virtual core and cpuset is the host core(s).

    3. Define the VM

    4. Terminal: “pacaur -S cpuset”

    5. In the executable file you use to attach input devices, add the lines
    Code:
    cset set -c 0-1 -s system
    cset proc -m -f root -t system
    cset proc -k -f root -t system
    
    6. Change the "0-1" to the cores that you want to reserve for your host OS (the ones you didn't use in step two).

    7. In the script you use to detach input devices, add the line "cset set -d system"
    Improve CPU performance via cpupower governor (frequency)
    Note: This is for when your CPU doesn't reach it's max frequency (boost) when gaming. This can force the CPU to boost and give you better performance. It may not be beneficial on all systems depending on the default governor.

    1. Terminal: “pacaur -S cpupower”

    2. Terminal: "cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor". Keep the output handy.

    3. In the attach-device file from step six add the following line
    Code:
    sudo cpupower -c 2-7 frequency-set -g performance
    If you used CPU pinning, replace "2-7" with the numbers of the real cores you pinned the virtual cores to. If not, you can

    4. In the detach-device file from step six, add the following line
    Code:
    sudo cpupower -c 2-7 frequency-set -g [The output you got from step two]
    Using xrandr to control monitors
    xrandr allows us to control which monitors are on and off on Linux. This allows us to automatically turn off the monitor(s) that the GPU uses to play games so we don't have to manually switch inputs.

    1. Terminal: "xrandr". This will tell you the names of your displays.

    2. Identify the name of the monitor(s) you wish to turn off, then in the script you use to start the VM add the line "xrandr --output [NAME OF DISPLAY] -off" with [NAME OF DISPLAY] being the monitor you want to turn off (and switch to the Windows GPU).

    3 (Optional). Depending on how many monitors you have, you may also need to specify which monitor you want to become your primary (assuming the monitor your using for gaming was your primary). To do this, add the line "xrandr --output [NAME OF DISPLAY] --primary".

    4. Save the script and you should be ready to go.
    Using UEFI (OVMF) instead of legacy VGA
    This mainly is useful if you are running into VGA arbitration issues and don't want to use the VGA arbiter patch.

    1. Verify your GPU has a UEFI capable firmware by checking on https://www.techpowerup.com/vgabios/

    2. Download edk2.git-ovmf-x64 from here.

    3. Extract the files to /usr/share. You should now have the following files:
    Code:
    /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd
    /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd
    /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd
    
    4. In your VM's XML file, change the OS element to the following
    Code:
    <os>
    <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/>
    </os>
    
    Note: You may need to boot with emulated graphics (ie. qxl) in order to install the GPU drivers

     
    • Like Like x 16
    • Informative Informative x 4
    Last edited: Jun 19, 2016
  2. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    I will no longer be properly updating this debian-based tutorial. I may make some minor changes here and there, but it is very unlikely that it will be completely up to date. This may change if someone is able to keep in the loop about changes to mainstream debian distros like Ubuntu.

    Prerequisites
    IOMMU compatible CPU and motherboard

    You need a CPU that is capable of IOMMU. On Intel this is called vt-d. To check if your CPU has vt-d, search for it here. On AMD it's called AMD-Vi. Most AMD CPUs and APUs have it. It's also important you have a motherboard that is capable of supporting it and it's enabled. You may need to flash to a different version for it to work.

    Note: If you have a Intel core cpu (i3,i5,i7) you will likely need the ACS override patch. This requires extra steps and recompiling the kernel.
    Debian-based distro
    This tutorial was written based on what I've done in Linux Mint 17.2. The steps should be relatively similar in most other debian-based distros (Ie. Ubuntu). This can certainly be done in other distros, but the steps will be rather different aside from the XML.​

    Two GPUs (including integrated)
    In order for this to work, you need two GPUs. One is used by the host and one is used by the guest. iGPUs can be used for the host.

    Note: If you are using an Intel iGPU for the host, you need the i915 VGA arbiter patch. This requires extra steps and recompiling the kernel.
    Tutorial
    1. Install necessary packages

    Terminal: "sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils qemu-utils"
    2. Load necessary modules
    2a. Terminal: "sudo gedit /etc/modules" and add the following to the file
    Code:
    pci_stub
    vfio
    vfio_iommu_type1
    vfio_pci
    kvm
    kvm_intel
    2b. Terminal: "sudo gedit /etc/default/grub" and add "intel_iommu=on" inside the quotes on the GRUB_CMDLINE_LINUX_DEFAULT line. Example:
    Code:
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"
    Note: Add "iommu=pt iommu=1" for AMD.

    2c. Terminal "sudo update-grub"
    3. Detach GPU from host
    3a. Terminal: "lspci -nn | grep AMD" (grep NVIDIA if you are passing through an NVIDIA GPU). The output should be something like this
    Code:
    01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] [1002:6810]
    01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]

    3b. Terminal: "sudo gedit /etc/modprobe.d/pci-stub.conf". Add "options pci-stub ids= " followed by numbers in the brackets at the end of the lspci output separated with each device separated by commas. Be sure to only include the IDs (one for GPU one for HDMI audio) for the card you want to use in the VM. Example:
    Code:
    options pci-stub ids=1002:aab0,1002:6810
    3c.Terminal: "sudo gedit /etc/modprobe.d/drm.conf". Add the line "softdep drm pre: pci-stub"

    3d. Terminal: "sudo update-initramfs -u"

    3e. Reboot and ensure the GPU is being claimed by pci-stub. Do this by running "dmesg | grep stub" and if you see something like below, you are good.
    Code:
    [ 2.009069] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
    [ 2.009080] pci-stub 0000:01:00.1: claimed by stub
    [ 2.009086] pci-stub: add 1002:6810 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
    [ 2.009092] pci-stub 0000:01:00.0: claimed by stub
    Note: At this point you should have cables plugged into both the host video card and the card that you want to use for gaming.
    4. Create virtual disk(s) for VM
    Terminal: "qemu-img create -f raw -o size=[Desired Size]G [Desired Location]". So a 50GiB image in my documents would look like "qemu-img create -f raw -o size=50G ~/Documents/disk.img"
    5. Define the virtual machine
    5a. Make an xml file and paste the following code into it.
    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
      <name>gamingvm</name>
      <uuid>01bd2ed1-b465-4eba-b6e4-47c6ac8171c6</uuid>
      <memory unit="GB">[Memory]</memory>
      <currentMemory unit="GB">[Memory]</currentMemory>
      <memoryBacking>
      <locked />
      </memoryBacking>
      <vcpu placement="static">[Cores*Threads]</vcpu>
      <cpu mode="host-passthrough">
      <topology sockets="1" cores="[Core]" threads="[Threads]" />
      </cpu>
      <os>
      <type arch="x86_64" machine="q35">hvm</type>
      <loader>/usr/share/qemu/bios.bin</loader>
      <bootmenu enable="yes" />
      </os>
      <features>
      <hyperv>
      <relaxed state="on" />
      <vapic state="on" />
      <spinlocks state="on" retries="8191" />
      </hyperv>
      <acpi />
      </features>
      <clock offset="localtime">
      <timer name="hypervclock" present="yes" />
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>destroy</on_crash>
      <devices>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <sound model='ich6'/>
      <controller type="usb" index="0" />
      <controller type="usb" index="1" />
      <controller type="usb" index="2" />
      <controller type="usb" index="3" />
      <controller type="sata" index="0" />
      <controller type="pci" index="0" model="pcie-root" />
      <controller type="pci" index="1" model="dmi-to-pci-bridge" />
      <controller type="pci" index="2" model="pci-bridge" />
      <memballoon model="none" />
      <sound model="ich6" />
      <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" />
      <source file="[OS disc image]" />
      <target dev="hdc" bus="ide" />
      <readonly />
      <address type="drive" controller="0" bus="1" unit="0" />
      </disk>
      <disk type="file" device="disk">
      <source file="[Primary HDD image]" />
      <target dev="vda" bus="ide" />
      <address type="drive" bus="0" />
      </disk>
      <disk type="file" device="disk">
      <source file="[Secondary HDD image]" />
      <target dev="vdb" bus="ide" />
      <address type="drive" bus="1" />
      </disk>
      </devices>
      <qemu:commandline>
      <qemu:env name="QEMU_PA_SAMPLES" value="4096" />
      <qemu:env name="QEMU_AUDIO_DRV" value="pa" />
      <qemu:arg value="-device" />
      <qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=[GPU],bus=root.1,addr=00.0,multifunction=on,x-vga=on" />
      <qemu:arg value="-device" />
      <qemu:arg value="vfio-pci,host=[Audio],bus=root.1,addr=00.1" />
      <qemu:arg value="-cpu" />
      <qemu:arg value="host" />
      </qemu:commandline>
    </domain>

    5b. Replace [OS disc image] with the path to an .iso for an install disc. I highly recommend using Windows 10 considering it's future support for dx12 and it has given me far less problems than Windows 8. This tutorial is also written for Win10, but Windows 8 is very similar.

    5c. Replace [Memory] with the amount of memory you want the VM to have. Note the value is in Gigabytes. Be sure not to give more than your system ram amount and leave some space for the host.

    5d. Replace [Primary HDD Image] with the path to the image you created in step four. You can also add the path for a second hdd where [Secondary HDD Image]. You can also add more HDDs by copying the <disk></disk> elements and increasing the number of bus and increasing the letter on vdx (so third disk would be bus=2 vdc, fourth would be bus=3, vdd, etc.).

    5e. Choose the amount of cores and threads you want your VM to have. If you're on a processor with 8 threads, I recommend 3 cores and 2 threads (2 threads per core). In this instance, I would change [Cores*Threads] to 6, [Cores] to 3, and [Threads] to 2.

    5f. Change [GPU] and [Audio] to the slot/function numbers from lspci. For example, I would use "01:00.0" and "01:00.1"

    5f. Terminal: "virsh define [Path to XML file]". The terminal should return this if everything is correct.
    Code:
    Domain gamingvm defined from [Path to XML file]

    6. Setting up input devices
    6a. Create a new XML file like the following
    Code:
    <hostdev mode='subsystem' type='usb' managed='no'>
    <source>
    <vendor id='0x[Before Colon]'/>
    <product id='0x[After Colon]'/>
    </source>
    </hostdev>
    6b. Terminal: "lsusb". Identify which device is your keyboard. Then look at the numbers after "ID" from the output. In the XML file, replace [Before Colon] with the set of numbers/letters before the colon, then replace [After Colon] with the set of numbers/letters after the colon. Be sure to keep the 0x. It sounds a bit confusing, but here is an example:

    My keyboard from lsusb
    Code:
    Bus 002 Device 005: ID 413c:2011 Dell Computer Corp. Multimedia Pro Keyboard
    My XML file would to look like this
    Code:
    <hostdev mode='subsystem' type='usb' managed='no'>
    <source>
    <vendor id='0x413c'/>
    <product id='0x2011'/>
    </source>
    </hostdev>
    Note: You could just add the hostdev element in the devices element in the XML file you created in step four and define the vm again. However, putting the USB devices in separate files allows you to attach/detach much easier. This is especially useful if the VM fails to boot and you have no way of interacting with your host OS.

    6c. Repeat that for your mouse. If your mouse/keyboard has multiple entries, be sure to do it all of them (for instance, a wireless mouse would show up twice. Once for the USB cord, and once for the wireless adapter).

    6d. Create a file with the following content
    Code:
    #!/bin/bash
    if [[ $(virsh list | grep gamingvm) != "" ]]
    then
    vm=gamingvm
    fi
    
    virsh attach-device $vm [Mouse]
    virsh attach-device $vm [Keyboard]
    6e. Replace [Mouse] and [Keyboard] with paths to the files you created in the previous steps. If you had to make more than two files, just add lines with the path to those at the end

    6f. Make this file executable by either right clicking it and going into the properties or using chmod.

    6g. Make a copy of the file from 6d, except replace the word “attach” with “detach” so it says “virsh detach-device $vm ...”. This script can be used for detaching your input devices.

    7. Preparing virtual machine
    7a. Create the following file
    Code:
    #!/bin/bash
    set -x
    modprobe vfio-pci
    modprobe kvm ignore_msrs=1
    DEVLIST="0000:01:00.0 0000:01:00.1"
    
    for dev in $DEVLIST;do
    vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
    device=$(cat /sys/bus/pci/devices/$dev/device)
    if [ -e /sys/bus/pci/devices/$dev/driver ]
    then
    echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    
    echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
    
    done
    
    virsh start gamingvm
    set +x
    7b. Terminal: "sudo gedit /etc/libvirt/qemu.conf"

    7c. Add the following lines (or find them and uncomment them)
    Code:
    user = "root"
    group = "root"
    cgroup_device_acl = [
      "/dev/null", "/dev/full", "/dev/zero",
      "/dev/random", "/dev/urandom",
      "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
      "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
      "/dev/vfio/[GROUP]"
    ]
    clear_emulator_capabilities = 0
    
    7d. Terminal: "find /sys/kernel/iommu_groups/ -type l". Now look for what directory the GPU you are passing through is in. Replace "[GROUP]" with whatever number group your device is in.
    8. Preparing PulseAudio
    Note: If you are going to use HDMI audio, you do not need to follow these steps.

    8a.
    Terminal: “sudo gedit /etc/init/pulseaudio.conf”

    8b. Uncomment (remove the # in front of) the "start on run level... " line

    8c. Terminal: “sudo adduser [your user] pulse-access && sudo adduser libvirt-qemu pulse-access && sudo adduser root pulse-access” and reboot

    8d. Terminal: “sudo gedit /etc/apparmor.d/abstractions/libvirt-qemu”

    8e. Find the “/{dev,run}/shm r,”. Add a w after the r and write two new lines, like so:
    Code:
    /{dev,run}/shm rw,
    /{dev,run}/shm/pulse* rw,
    /var/lib/libvirt/.pulse-cookie rwk,

    9. Install The OS
    9a. Make the file from step seven executable and run it. If everything goes smoothly, switching to the input of your GPU should display the installation screen. If not, scroll down to the troubleshooting portion of the thread.

    9b. Run the file you made in step six to enable keyboard and mouse.

    9c. Install the OS like usual. Be aware that the networking has not yet been setup​

    10. Networking
    Note: This requires the use of /etc/network/interfaces. Using this file for network configuration will make your network interfaces unmanageable via the network-manager GUI.

    10a. Terminal: “sudo gedit /etc/network/interfaces”
    Note: This next step assumes you are using ethernet for your internet connection. If you use WiFi, visit this link for information on how to set it up in the interfaces file.

    10b. Edit the file to
    Code:
    auto lo
    iface lo inet loopback
    auto eth0
    iface eth0 inet manual
    auto br0
    iface br0 inet dhcp
    bridge_ports eth0

    10c.
    Terminal: “sudo gedit [XML from step five]”

    10d. Add the following inside the <devices> tag
    Code:
    <interface type='bridge'>
    <mac address='52:54:00:a0:41:92'/>
    <source bridge='br0'/>
    <model type='rtl8139'/>
    </interface>
    10e. Terminal: “virsh define [XML from step five]”

    Note: In general, every time you make edits to the XML file you have to redefine the VM. You could also use virsh edit.

    10f. Reboot your PC and the VM should now start and connect to your local network with its own IP address.​

    Troubleshooting
    This (http://vfio.blogspot.com/2014/08/vfiovga-faq.html) blog post has some of the common errors that you can run into and the solutions. I suggest that is the first place you turn if you run into an issue.

    Black screen on guest and/or host has corrupted graphics
    If you are using an Intel iGPU for the host, you need the i915 VGA arbiter patch. To use it, you must recompile the kernel with the patch and then put "i915.enable_hd_vgaarb=1" in your kernel boot parameters.

    If you are having this issue on a non-Intel GPU. You can try this patch.

    If you don't want to use the VGA patch, you can set up OVMF.

    Sound isn’t working properly
    The first thing I recommend doing is playing around with the QEMU_PA_SAMPLES. Some may find that a lower value, like 128 or 512, will yield better results (less choppiness).

    Most games run fine, but some games run extremely poorly
    Aside from the generic response of “check your GPU drivers”, also be sure you are on a kernel version higher than 3.15. Some games (Dirty Bomb and Borderlands are the ones I know of) use debug registers and tend to really hurt the performance. A fix to this was added in kernel 3.15.\

    NVIDIA drivers not working in guest (or code 43)
    NVIDIA doesn't like their cards being used in virtual machines. To get around this, you need to add ,kvm=off to your -cpu argument. (ie. host,kvm=off). If you are still getting issues, you need to disable Hyper-V enlightenments.
    Code:
    <hyperv>
    <relaxed state='off'/>
    <vapic state='off'/>
    <spinlocks state='off'/>
    </hyperv>
    
    ...
    
    <features>
    <kvm>
    <hidden state='on'/>
    </kvm>
    </features>
    
    ...
    
    <clock>
    <timer name='hypervclock' present='no'/>
    </clock>
    
    
    
    Tweaks and Optimizations
    GPU performance can’t really get much better (it's already almost bare-metal), but there is a lot of room for improvement in other areas, especially the CPU. There are also some ways that can increase the usability and functionality of your VM.

    Setting up Synergy
    Syngery is a way to share your mouse and keyboard between two computers over a network. This allows you to use both the guest OS and host OS at the same time and have your input work between the two as if it’s a dual monitor setup.

    1. Set a static IP on your guest OS. (http://portforward.com/networking/static-ip-windows-8.htm)

    2. Buy and download Synergy on the guest OS from https://synergy-project.org/. Alternatively, you can compile the source yourself for free.

    3. Configure it so it will act as a server (share mouse+keyboard) and have your linux hostname in the appropriate place when configuring screens.

    4. Download Synergy on the host (either from the website or by using “sudo apt-get install synergy”)

    5. In the file you created in step 6d, add the line “sudo synergyc [Guest IP]:24800”.

    6. In the detach file created in step 6g, add the lines “killall synergy” and “killall synergyc”.

    Improve CPU performance with Cpuset and CPU pinning
    CPU pinning allows you to assign real cores to virtual cores, thus increasing the performance of the virtual core. However, we also need to use cpusets to ensure that other host processes don’t interfere with that core.

    1. Terminal: "sudo gedit [XMl file from step five]"

    2. Below the CPU element, add the following:
    Code:
     <cputune>
        <vcpupin vcpu="0" cpuset="2"/>
        <vcpupin vcpu="1" cpuset="3"/>
        <vcpupin vcpu="2" cpuset="4"/>
        <vcpupin vcpu="3" cpuset="5"/>
        <vcpupin vcpu="4" cpuset="6"/>
        <vcpupin vcpu="5" cpuset="7"/>
      </cputune>
    This setup is for a vm with six cores, and it pins each virtual core to a real core. You can change it however you want to fit your needs, just know vcpu is the virtual core and cpuset is the host core(s).

    3. Define the VM

    4. Terminal: “sudo apt-get install linux-tools-common linux-tools-generic”

    5. In the executable file you use to attach input devices, add the lines
    Code:
    cset set -c 0-1 -s system
    cset proc -m -f root -t system
    cset proc -k -f root -t system
    
    6. Change the "0-1" to the cores that you want to reserve for your host OS (the ones you didn't use in step two).

    7. In the script you use to detach input devices, add the line "cset set -d system"
    Improve CPU performance via cpupower governor (frequency)
    Note: This is for when your CPU doesn't reach it's max frequency (boost) when gaming. This can force the CPU to boost and give you better performance. It may not be beneficial on all systems depending on the default governor.

    1. Terminal: “sudo apt-get install linux-tools-common linux-tools-generic”

    2. Terminal: "cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor". Keep the output handy.

    3. In the attach-device file from step six add the following line
    Code:
    sudo cpupower -c 2-7 frequency-set -g performance
    If you used CPU pinning, replace "2-7" with the numbers of the real cores you pinned the virtual cores to. If not, you can

    4. In the detach-device file from step six, add the following line
    Code:
    sudo cpupower -c 2-7 frequency-set -g [The output you got from step two]
    Using xrandr to control monitors
    xrandr allows us to control which monitors are on and off on Linux. This allows us to automatically turn off the monitor(s) that the GPU uses to play games so we don't have to manually switch inputs.

    1. Terminal: "xrandr". This will tell you the names of your displays.

    2. Identify the name of the monitor(s) you wish to turn off, then in the script you use to start the VM add the line "xrandr --output [NAME OF DISPLAY] -off" with [NAME OF DISPLAY] being the monitor you want to turn off (and switch to the Windows GPU).

    3 (Optional). Depending on how many monitors you have, you may also need to specify which monitor you want to become your primary (assuming the monitor your using for gaming was your primary). To do this, add the line "xrandr --output [NAME OF DISPLAY] --primary".

    4. Save the script and you should be ready to go.
    Using UEFI instead of legacy VGA (OVMF)
    This mainly is useful if you are running into VGA arbitration issues and don't want to use the VGA arbiter patch.

    1. Verify your GPU has a UEFI capable firmware by checking on https://www.techpowerup.com/vgabios/

    2. Download edk2.git-ovmf-x64 from here.

    3. Extract the files to /usr/share. You should now have the following files:
    Code:
    /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd
    /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd
    /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd
    
    4. In your VM's XML file, change the OS element to the following
    Code:
    <os>
    <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/>
    </os>
    
    Note: You may need to boot with emulated graphics (ie. qxl) in order to install the GPU drivers

     
    • Like Like x 1
    Last edited: Jan 3, 2016
  3. Joshy

    Joshy Legend Premium

    Messages:
    6,877
    Ratings:
    4,335
    This is a long one, I might need a whole new PC for this dude.
     
  4. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    Thank you so much for this tutorial. I just found out about this method a few days ago and all the tutorials I found were out-of-date. This also works great because we are both using the same base system.

    Much Appreciated!
     
    • Like Like x 1
  5. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    Glad you find it helpful. Have you successfully installed the VM yet?

    Just let me know if you run into any problems and I'll do my best to find a solution.
     
    • Like Like x 1
  6. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    I have not attempted it yet. I am waiting for the weekend to give it a try because I just ordered a second graphics card and, of course, some sort of time constraints that happen during the week. I was definitely going to post here if it was successful or if I had any issues.
     
  7. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    Hello!

    Unfortunately, my new gpu was DOA so I went ahead with the process with my existing card. I did have some issues and I did not want to start the installation on the VMdisk until I made sure I have it addressed.

    In step 7d, the command "find /sys/kernel/iommu_groups/ -type l" is supposed to find the group for my graphics card. I ran it and I received no results. Link to image from my Dropbox.

    I also had a question about Step 8 [configuring pulse audio] after the reboot, I noticed that it turned all my audio devices in one pulseaudio server. Not that I am complaining at this point because it doesn't seem to have changed anything important. Just curious if this has any implications for use down the road.

    Thank you again for this and I cannot wait to be able to use this. (sorry for the double post, but I wanted to make sure you got an alert saying I posted a new reply)
     
    Last edited: Sep 7, 2015
  8. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    That's strange, what CPU do you have? If you're on an Intel CPU, ensure it supprots vt-d and is enabled in BIOS and in the kernel boot parameters (step 2b). If it's AMD, in step 2b try adding "iommu=pt iommu=1" instead. I don't own an AMD CPU, so I was unaware that such as astep was necessary (as per http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM).

    If you have done the above, check to make sure IOMMU is being loaded upon booting up. Open up a terminal and run the command "dmesg | grep IOMMU" and see what the output is.

    I'm not going to lie, I really have a tough time wraping my head around PulseAudio. However, as far as I know, there aren't any major downsides in this use case.\

    Edit: http://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/WhatIsWrongWithSystemWide/

    The link above outlines why PA advises you to not run it system wide. Most of the reasons are security-related. However, that shouldn't really be much of an issue in most home enviorments.
     
    • Like Like x 1
    Last edited: Sep 7, 2015
  9. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    That is good to know about PulseAudio.

    I thought I should share how I got past it for whoever may have the same issue.
    It turns out that even if your hardware supports vt-d you still have to enable it (as stated in the prerequisites). I scoured my Bios/UEFI for a way to enable it. It stated it was supported but there was no place I found to enable it. After googling for hours and an unneeded bios update I found the option inside Northbridge Configuration. Check there first if you have a ASROCK motherboard.

    So I finally got past that section and many revisions later (all done in accordance to the instructions) I finally get to this:

    Code:
    [email protected]:~ > ./Step7a-PreparingVM
    ++ modprobe vfio-pci
    ++ modprobe kvm ignore_msrs=1
    ++ virsh start gamingvm
    error: Failed to start domain gamingvm
    error: internal error: early end of file from monitor: possible problem:
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/1: No such file or directory
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
    
    
    ++ set +x

    I was trying to track this down myself since it mentioned the groups (that I ran into that previously).
    Code:
    [email protected]:~ > lspci -nn | grep NVIDIA
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 660] [10de:11c0] (rev a1)
    01:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)
    [email protected]:~ > find /sys/kernel/iommu_groups/ -type l
    /sys/kernel/iommu_groups/0/devices/0000:00:00.0
    /sys/kernel/iommu_groups/1/devices/0000:00:01.0
    /sys/kernel/iommu_groups/1/devices/0000:01:00.0
    /sys/kernel/iommu_groups/1/devices/0000:01:00.1
    /sys/kernel/iommu_groups/2/devices/0000:00:02.0
    /sys/kernel/iommu_groups/3/devices/0000:00:14.0
    /sys/kernel/iommu_groups/4/devices/0000:00:16.0
    /sys/kernel/iommu_groups/5/devices/0000:00:1a.0
    /sys/kernel/iommu_groups/6/devices/0000:00:1b.0
    /sys/kernel/iommu_groups/7/devices/0000:00:1c.0
    /sys/kernel/iommu_groups/8/devices/0000:00:1c.3
    /sys/kernel/iommu_groups/9/devices/0000:00:1c.4
    /sys/kernel/iommu_groups/10/devices/0000:00:1c.5
    /sys/kernel/iommu_groups/11/devices/0000:00:1c.7
    /sys/kernel/iommu_groups/12/devices/0000:00:1d.0
    /sys/kernel/iommu_groups/13/devices/0000:00:1f.0
    /sys/kernel/iommu_groups/13/devices/0000:00:1f.2
    /sys/kernel/iommu_groups/13/devices/0000:00:1f.3
    /sys/kernel/iommu_groups/14/devices/0000:03:00.0
    /sys/kernel/iommu_groups/15/devices/0000:04:00.0
    /sys/kernel/iommu_groups/16/devices/0000:05:00.0
    /sys/kernel/iommu_groups/17/devices/0000:07:00.0
    
    I also ensured I had it correctly marked in my /etc/libvirt/qemu.conf.
    Code:
    user = "root"
    group = "root"
    cgroup_device_acl = [
      "/dev/null", "/dev/full", "/dev/zero",
      "/dev/random", "/dev/urandom",
      "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
      "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
      "/dev/vfio/1"
    ]
    
    #All these were added at the bottom. I did not
    #uncomment them out individually.

    Hopefully this is my error because that means it can be fix. (lol.)

    Thank you for your assistance again!
     
  10. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    Ah, I should probably mention that in the tutorial. I also have an ASRock motherbaord and needed to udpate the firmware as well.


    It looks like the script to attach the device to VFIO isn't running. I've ran into this before, and I'm honestly not quite sure why it happens (and only happened sometimes for me).

    In the script, replace
    Code:
    for dev in "[email protected]"; do
            vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
            device=$(cat /sys/bus/pci/devices/$dev/device)
            if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                    echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
            fi
            echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
    done
    
    with

    Code:
    DEVLIST="0000:01:00.0 0000:01:00.1"
    
    for dev in $DEVLIST;do
    vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
    device=$(cat /sys/bus/pci/devices/$dev/device)
    if [ -e /sys/bus/pci/devices/$dev/driver ]
    then
    echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    
    echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
    
    done
    
    Just replace the numbers in DEVLIST with the numbers for your GPU/AUDIO
     
  11. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    I added the code as stated and I am still getting a similar response. I am not using my grpahics card at all at this time. Main monitor plugged into Integrated graphics and secondary monitor is plugged into graphics card. I have done nothing to disable the card (except is any of this steps did that for me).

    Here is copies of all my files that have been created so far. Maybe I did something in them? (and yes, my graphics card is reading as 0000:01:00.0 and 0000:01:00.1. Just wanted to state it so you don't think I am just using your example numbers.)

    Created Files

    (also, we have the same motherboard. So I had to do the same thing.)
     
    Last edited: Sep 7, 2015
  12. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    First off, in the script used to start the machine, you have duplicate lines at the start and begenning.
    Code:
    virsh start gamingvm
    set +x
    virsh start gamingvm
    set +x
    
    and
    Code:
    #!/bin/bash
    set -x
    modprobe vfio-pci
    modprobe kvm ignore_msrs=1
    #!/bin/bash
    set -x
    modprobe vfio-pci
    modprobe kvm ignore_msrs=1
    
    Secondly, can you post the output you get when post the script? Make sure that IOMMU is loaded and the card is being claimed by pci-stub on boot (dmesg | grep IOMMU and dmesg | grep pci-stub)

    Edit: Also, if you grep IOMMU and see the following message "No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform" you need to add "vfio_iommu_type1.allow_unsafe_interrupts=1" to your kerenl boot params (where you have intel_iommu=on)
     
    Last edited: Sep 7, 2015
  13. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    First off, I am sorry about the double listings in the file. Those files have been edited so much today (as there was other issues that I was able to fix). I hate to have such a simple item be overlooked.

    Here is the results from each command.
    Code:
    [email protected]:~ > dmesg | grep IOMMU
    [    0.000000] Intel-IOMMU: enabled
    [    0.095958] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
    [    0.095962] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
    [    0.096033] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
    [    0.681288] IOMMU 0 0xfed90000: using Queued invalidation
    [    0.681289] IOMMU 1 0xfed91000: using Queued invalidation
    [    0.681290] IOMMU: Setting RMRR:
    [    0.681297] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
    [    0.681574] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9522000 - 0xc9554fff]
    [    0.681589] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9522000 - 0xc9554fff]
    [    0.681601] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9522000 - 0xc9554fff]
    [    0.681609] IOMMU: Prepare 0-16MiB unity mapping for LPC
    [    0.681614] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
    The command has returned this answer since it was ran in following the instructions.
    Code:
    [email protected]:~ > dmesg | grep pci-stub
    [    0.841575] pci-stub: add 10DE:11C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
    [    0.841584] pci-stub 0000:01:00.0: claimed by stub
    [    0.841589] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
    [    0.841594] pci-stub 0000:01:00.1: claimed by stub
    
    Then the output so we can see what errors are arising.
    Code:
    [email protected]:/media/sol/Virtual-10/SystemFiles > sudo ./Step7a-PreparingVM
    + modprobe vfio-pci
    + modprobe kvm ignore_msrs=1
    + DEVLIST='0000:01:00.0 0000:01:00.1'
    + for dev in '$DEVLIST'
    ++ cat /sys/bus/pci/devices/0000:01:00.0/vendor
    + vendor=0x10de
    ++ cat /sys/bus/pci/devices/0000:01:00.0/device
    + device=0x11c0
    + '[' -e /sys/bus/pci/devices/0000:01:00.0/driver ']'
    + echo 0000:01:00.0
    + echo 0x10de 0x11c0
    + for dev in '$DEVLIST'
    ++ cat /sys/bus/pci/devices/0000:01:00.1/vendor
    + vendor=0x10de
    ++ cat /sys/bus/pci/devices/0000:01:00.1/device
    + device=0x0e0b
    + '[' -e /sys/bus/pci/devices/0000:01:00.1/driver ']'
    + echo 0000:01:00.1
    + echo 0x10de 0x0e0b
    + virsh start gamingvm
    error: Failed to start domain gamingvm
    error: internal error: early end of file from monitor: possible problem:
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f6a215d5b40, 0x0, 0x80000000, 0x7f6813200000) = -12 (Cannot allocate memory)
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f6a215d5b40, 0x100000000, 0x15ce00000, 0x7f6893200000) = -12 (Cannot allocate memory)
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 1
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
    qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
    qemu-system-x86_64: -device v
    
    + set +x
     
  14. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    Wow, I feel like an idiot. In the tutorial, I forgot to disable clear_emulator_capabilities. With this still enabled, it won't let you run QEMU as root and thus you hit the memory limits of your unprivelerged user.

    To fix this:
    Go back into the qemu.conf file from 7c and add the line "clear_emulator_capabilities = 0" (or unccoment it).

    Hopefully you'll be able to now actually launch the VM without any errors coming from the script.

    The bad news is that looking at yoursetup, you will likely need to patch the kernel with some fixes in order to get the GPU working properly within the host. This is something I've only done on Arch, but I believe I should be able to help you out if you need to.
     
  15. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    Well, good and bad news.

    The good news is that when I added that line it went and ran aka started the domain. Though the monitor that is hooked directly to the graphics card detects absolutely no signal. While trying to see if I could get this fixed my computer frozen up on me.

    I had to hard reboot to get back and now it is giving me an error message that I thought I fixed (I don't know how I did it, but it went away and now it is back.)
    Code:
    [email protected]:/media/sol/Virtual-10/SystemFiles > sudo ./Step7a-PreparingVM1
    [sudo] password for sol:
    + modprobe vfio-pci
    + modprobe kvm ignore_msrs=1
    + DEVLIST='0000:01:00.0 0000:01:00.1'
    + for dev in '$DEVLIST'
    ++ cat /sys/bus/pci/devices/0000:01:00.0/vendor
    + vendor=0x10de
    ++ cat /sys/bus/pci/devices/0000:01:00.0/device
    + device=0x11c0
    + '[' -e /sys/bus/pci/devices/0000:01:00.0/driver ']'
    + echo 0000:01:00.0
    + echo 0x10de 0x11c0
    + for dev in '$DEVLIST'
    ++ cat /sys/bus/pci/devices/0000:01:00.1/vendor
    + vendor=0x10de
    ++ cat /sys/bus/pci/devices/0000:01:00.1/device
    + device=0x0e0b
    + '[' -e /sys/bus/pci/devices/0000:01:00.1/driver ']'
    + echo 0000:01:00.1
    + echo 0x10de 0x0e0b
    + virsh start gamingvm
    error: failed to connect to the hypervisor
    error: no valid connection
    error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
    
    + set +x

    I am going to do a bit more checking here on my side and will edit my post if anything changes.

    EDIT: So I went looking for a possible cause for this problem. According to the message, I looked into libvirt-bin as the possible problem.

    In my googling, it seems like I needed to verify that the daemon was running.

    The daemon is not running. I went and tried to start it manually. Here is my sttempts and their results:
    Code:
    [email protected]:/media/sol/Virtual-10/SystemFiles > ps -ef | grep libvirt
    sol  3345  1917  0 22:46 ?  00:00:00 /usr/sbin/libvirtd -d
    sol  4668  3408  0 23:12 pts/2  00:00:00 grep --colour=auto libvirt
    [email protected]:/media/sol/Virtual-10/SystemFiles > libvirtd
    2015-09-09 05:48:57.556+0000: 3531: info : libvirt version: 1.2.2
    2015-09-09 05:48:57.556+0000: 3531: error : virPidFileAcquirePath:411 : Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable
    [email protected]:/media/sol/Virtual-10/SystemFiles > /etc/init.d/libvirt-bin start
    * Starting libvirt management daemon libvirtd                                           Giving up waiting for /var/run/libvirt/libvirt-sock.
    [email protected]:/media/sol/Virtual-10/SystemFiles > ls /var/run/libvirt -lh
    total 0
    drwx------ 2 root root 60 Sep  8 22:43 network
    [email protected]:/media/sol/Virtual-10/SystemFiles > libvirtd -d
    libvirtd: error: Unable to obtain pidfile. Check /var/log/messages or run without --daemon for more info.
    /var/run/libvirt/libvirt-sock does not exist. I went into the file system and checked.
    That last message prompted me to check for error logs. I found them but am not sure what to take out from them. I also located an error log for the gamingvm domain and a log that makes me kinda worried about (it states it destroyed gamingvm. See the end of shutdown.log)

    Here is the logs. I posted them in my Dropbox. All of them were located in "/var/log/libvirt/"

    EDIT2:
    From various places I have looked, it looks like I will need to start over again and/or patch my kernel as well. (While I mention it, what about my setup causes the need for such patching? Just curious.)

    I also saw in some similar tutorials that they blacklisted their GPU's to enable vga pasthrough. It looks like I need to enable i915 VGA Passthrough. I am trying to find this patch and how to modify the kernel.

    I also found a nifty spreadsheet of GPU data for use in KVM. Then what to you think think of virt-manager for libvirt-bin?
     
    Last edited: Sep 9, 2015
  16. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    Just restarting the libvirt service and daemon should fix the issue, the same thing happened to me when I was setting this up on arch
    Code:
    service libvirtd restart
    /etc/init.d/libvirt restart
    
    Let me know if doing that solves the issue.

    A few things
    1. I assume you are using your iGPU for you host graphics. Because of this, you will likely need the i915 VGA Arbiter patch. For an indepth explanation to this, you can read Alex Williamson's (main contributer of VFIO) blog post about VGA arbiration. A super simplfied explanation is that the iGPU is inbetween the CPU and other PCIe devices. This means other devices are on the same bus which can cause confusion with legacy VGA arbiration (choosing which VGA device to use).

    2. In order to avoid using this patch with one dedicated GPU, you would need to have a graphics card that supports UEFI so you could use OVMF. Unfortunately, the 660 does not support UEFI.

    3. Then, it looks like you might be needing the ACS override patch. In order to properly use VFIO, you need bind all the devices in the same IOMMU group as your GPU. The "0000:00:01.0" device is the CPU's PCIe root port. Ufortunately, Intel core chips do not natively support the sereration of the root port. The ACS override patch essentially spoofs the kernel into thinking you can do this isolation on core CPUs. More in-depth: http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html
     
  17. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    It did not. I did try it but it has never done anything for me.
    Code:
    [email protected]:~ > service libvirtd restart
    libvirtd: unrecognized service
    [email protected]:~ > /etc/init.d/libvirt restart
    bash: /etc/init.d/libvirt: No such file or directory
    [email protected]:~ > /etc/init.d/libvirt-bin restart
    * Restarting libvirt management daemon /usr/sbin/libvirtd  /usr/sbin/libvirtd: error: Unable to obtain pidfile. Check /var/log/messages or run without --daemon for more info.
      [fail]
    
    

    Sorry about my long post about libvirt-bin. I was trying anything to get it working. I am not an expert but I do try and figure things out.

    EDIT: Found some great resources on patching the kernel. I am building it as I watch it in the konsole (terminal). Hardest part was finding the patches. I will share how I did it once I get some sleep if you would like to add it into the tutorial (if you don’t already know how. Very similar to arch from what I gather.)

    EDIT2: Finished patching and updating the kernel. The libvirt commands are still popping up and causing trouble.
     
    Last edited: Sep 10, 2015
  18. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    The issues you're having with libvirt are rather strange. I'm honestly not sure what could be causing them, are you sure you successfully installed libvirt-bin? Also make sure you run those commands as root.

    In order to make use of the patches, add "pcie_acs_override=downstream i915.enable_hd_vgaarb=1" to the same place as 2b.
     
  19. Pictou

    Pictou Newbie

    Messages:
    13
    Ratings:
    1
    Lines added to file for the patches.

    Secondly, it looks like some how my libvirt daemon is not running and I cannot get it running (not matter how many command I try or pull off the internet. I even uninstalled and reinstalled libvirt-bin. I might have to use the purge command just to try and get this working.

    Code:
    [email protected]:/media/sol/Virtual-10/SystemFiles > sudo /etc/init.d/libvirt restart
    sudo: /etc/init.d/libvirt: command not found
    [email protected]:/media/sol/Virtual-10/SystemFiles > sudo service libvirtd restart
    libvirtd: unrecognized service
    [email protected]:/media/sol/Virtual-10/SystemFiles > sudo service libvirt-bin restart
    stop: Unknown instance:
    libvirt-bin stop/post-start, (post-start) process 3442
    [email protected]:/media/sol/Virtual-10/SystemFiles > sudo /etc/init.d/libvirt-bin restart
    * Restarting libvirt management daemon /usr/sbin/libvirtd                                                    [ok]
    [email protected]:/media/sol/Virtual-10/SystemFiles > 

    Considering just reinstalling Linux Mint and starting all over with a clean machine.
     
    Last edited: Sep 10, 2015
  20. OP
    Monopolyman

    Monopolyman PC Gamer and Tech Enthusiast Retired

    Messages:
    6,525
    Ratings:
    4,335
    That is just extremely odd... First I would try puring libvirt-bin and all its dependancies
    Code:
    sudo apt-get purge --auto-remove libvirt-bin
    
    and then reinstall libvirt-bin.

    When installing, make sure there isn't any conflicting dependencies.
     

Share This Page