Virtualization: VGA-passthrough

A little experiment of mapping VGA device into guest Windows OS via qemu-kvm.

This is article is appeared because I like play in video games but not so many developers of AAA games are support the Linux-like OS.

This is not 'best practice' -- just my notes, so please do not judge me strongly. Please feel free to leave any comments!

Overview

All there steps I had make on the same hardware. I test this how-to on LinuxMint 17, Ubuntu 15.10 and Ubuntu 16.04 with kernels since 4.1.0-040100rc2-generic up to current 4.4.0-36-generic. The qemu version is from 2.0.0 to 2.5.0.

Also I've tried to use Windows 10 as guest system but it proved to be less performance so I have still keep using Windows 7.

Hardware
  • CPU AMD Phenom(tm) II X4 955 Processor
  • VGA0 AMD/ATI Radeon HD 6770
  • VGA1 AMD/ATI Radeon R7 240
  • Motherboard Asus M5A97 R2.0

Concept

I created the guest machine and install the Windows 7 on it by using virt-manager (just because its much easy) but later I will use the qemu-kvm hypervisor for this guest.

At first I will use AMD/ATI Radeon HD 6770 video card for the guest machine and AMD/ATI Radeon R7 240 for the host.
Then I'll change video card -- R7 240 for guest, HD 6770 for host.

Configuration

No matter what video card we are using for guest and host, we need to check out a virtualization support of our hardware.

zombig@z0 ~ $ dmesg | grep -iE "AMD-Vi|IOMMU"  
[    1.287603] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.287637] AMD-Vi: Interrupt remapping enabled
[    1.287831] AMD-Vi: Initialized for Passthrough Mode
[    1.928462] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    1.928465] AMD IOMMUv2 functionality not available on this system
Note: you may need to include IOMMU support in BIOS!  

OK! That looks good!But if you don't see anything it is bad -- you need to upgrade your hardware.

Then you need to load vfio and kvm kernel modules. Since kernel 4.1 you can use vfio-pci directly instead of pci-stub.

zombig@z0 ~ $ cat /etc/modules  
lp  
rtc  
vfio  
vfio_iommu_type1  
vfio_pci  
kvm  
kvm_amd  

Also you need to enable IOMMU support in kernel, increase loop devices and setup IOMMU version 1. This is my grub config.

zombig@z0 ~ $ cat /etc/default/grub | grep -iE "GRUB_CMDLINE_LINUX"  
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"  
GRUB_CMDLINE_LINUX="amd_iommu=on max_loop=64 iommu=pt iommu=1 amd_iommu=fullflush"  
Define your video card

For a start we will pass-through Radeon HD 6770.

zombig@z0 ~ $ lspci -nn | grep -iE "Radeon"  
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 6770] [1002:68ba]  
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]  
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240] [1002:6613]  
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]  

As I write before I will use HD 6770 card so I need device 01:00.0 (vendor id 1002, device id 68ba) and 01:00.1 (audio on VGA, vendor id 1002, device id aa58). Lets add this ids to the vfio-pci module.

zombig@z0 ~ $  cat /etc/modprobe.d/vfio-pci.conf  
options vfio-pci ids=1002:68ba,1002:aa58  

Now just update initramfs, grub and reboot!

zombig@z0 ~ $  sudo update-initramfs -u  
zombig@z0 ~ $  sudo update-grub  
zombig@z0 ~ $  sudo init 6  

After reboot we can see:

zombig@z0 ~ $ dmesg | grep -iE "vfio"  
[    1.860455] VFIO - User Level meta-driver version: 0.3
[    1.879835] vfio_pci: add [1002:68ba[ffff:ffff]] class 0x000000/00000000
[    1.895787] vfio_pci: add [1002:aa58[ffff:ffff]] class 0x000000/00000000

Also you can verify that your devices are really use vfio with lspci command:

zombig@z0 ~ $ lspci -nnk  
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 6770] [1002:68ba]  
  Subsystem: XFX Pine Group Inc. Device [1682:3152]
  Kernel driver in use: vfio-pci
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]  
  Subsystem: XFX Pine Group Inc. Device [1682:aa58]
  Kernel driver in use: vfio-pci

Now you can start virtual machine!

Change the Card!

Now lets change the card for guest from HD 6770 to R7 240.

All configurations is same as I write at previous post but with new device and vendor ids.

For now it's:

zombig@z0 ~ $ lspci -nnvk | grep 06:00  
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240] [1002:6613] (prog-if 00 [VGA controller])  
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]  

Change ids for vfio-pci:

zombig@z0 ~ $ cat /etc/modprobe.d/vfio-pci.conf  
options vfio-pci ids=1002:6613,1002:aab0  
And reboot!  

Now let's check whether all okay...

dmesg | grep -iE "vfio"  
[    1.861046] VFIO - User Level meta-driver version: 0.3
[    1.884809] vfio_pci: add [1002:6613[ffff:ffff]] class 0x000000/00000000
[    1.900821] vfio_pci: add [1002:aab0[ffff:ffff]] class 0x000000/00000000

Now you can start virtual machine!

Start Virtual Machine!

I start my guest Windows with qemu, e.g.:

zombig@z0 ~ $ qemu-system-x86_64 \  
    -boot menu=on \
    -enable-kvm \
    -M pc-1.3 \
    -m $mem \
    -cpu host \
    -rtc base=localtime \
    -smp 4,sockets=1,cores=4,threads=1 \
    -bios /usr/share/qemu/bios.bin \
    -drive file=$vm_path,id=disk1,format=raw \
    -netdev tap,id=net0 -device e1000,netdev=net0,mac=52:54:00:26:7F:96 \
    -device vfio-pci,host=01:00.0 \
    -device vfio-pci,host=01:00.1 \
    -soundhw hda \
    -vnc :1

Note: do not add vfio-pci device while the Windows installation. You must have are already installed a guest OS!

When a OS has booted you will see your device in "Device Manager". Just download and install drivers!
When installation has completed you can connect monitor directly to output interface of your card.

Troubleshooting

Windows logo is freezed on boot

Don't worry! Windows will boot normally, if you want to connect to guest just use vnc and start VM without graphics.

vfio-pci can't handle with video device driver

If you look R7 240 information now you will see that device still using radeon driver but not vfio-pci (damn!).

zombig@z0 ~ $ lspci -nnvk  
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240] [1002:6613] (prog-if 00 [VGA controller])  
  Subsystem: ASUSTeK Computer Inc. Device [1043:04c5]
  Flags: fast devsel, IRQ 11
  Memory at b0000000 (64-bit, prefetchable) [disabled] [size=256M]
  Memory at fe700000 (64-bit, non-prefetchable) [disabled] [size=256K]
  I/O ports at c000 [disabled] [size=256]
  Expansion ROM at fe740000 [disabled] [size=128K]
  Capabilities: <access denied>
  Kernel driver in use: radeon

06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]  
  Subsystem: ASUSTeK Computer Inc. Device [1043:aab0]
  Flags: bus master, fast devsel, latency 0, IRQ 14
  Memory at fe760000 (64-bit, non-prefetchable) [size=16K]
  Capabilities: <access denied>
  Kernel driver in use: vfio-pci

Why?! Because on boot the kernel is loading DRM which loading the Radeon driver. Then the Radeon driver is initializing device so vfio-pci can't stub it.

zombig@z0 ~ $ dmesg | grep -iE "radeon|vfio"  
[ 1.890178] [drm] radeon kernel modesetting enabled.
[ 1.890659] radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used)
[ 1.890661] radeon 0000:01:00.0: GTT: 1024M 0x0000000040000000 - 0x000000007FFFFFFF
[ 1.924710] [drm] radeon: irq initialized.
...
[ 2.393721] [drm] Initialized radeon 2.39.0 20080528 for 0000:01:00.0 on minor 0
[ 2.393766] radeon 0000:06:00.0: enabling device (0000 -> 0003)
[ 2.643959] radeon 0000:06:00.0: VRAM: 4096M 0x0000000000000000 - 0x00000000FFFFFFFF (4096M used)
[ 2.643962] radeon 0000:06:00.0: GTT: 1024M 0x0000000100000000 - 0x000000013FFFFFFF
...
[ 2.662693] [drm] radeon: irq initialized.
...
[ 3.934510] VFIO - User Level meta-driver version: 0.3
[ 3.934517] vfio_pci: add [1002:6613[ffff:ffff]] class 0x000000/00000000
[ 3.934529] vfio_pci: add [1002:aab0[ffff:ffff]] class 0x000000/00000000
Solution

You can manage of booting process of kernel modules (see man modprobe.d). Working option for me:

zombig@z0 ~ $ cat /etc/modprobe.d/drm.conf  
softdep drm pre: vfio vfio_pci post: radeon  
softdep drm_kms_helper pre: vfio vfio_pci post: radeon

install drm /sbin/modprobe vfio; /sbin/modprobe vfio_pci  

And same for radeon:

zombig@z0 ~ $ cat /etc/modprobe.d/radeon.conf  
softdep radeon pre: vfio vfio_pci

install radeon /sbin/modprobe vfio; /sbin/modprobe vfio_pci  

With this options we says to radeon and drm that they must load vfio and vfio_pci modules before itself. Do not forget to update the initramfs and reboot.

verify
zombig@z0 ~ $ dmesg | grep -iE "drm|radeon|vfio"  
[    1.862472] VFIO - User Level meta-driver version: 0.3
[    1.880848] vfio_pci: add [1002:6613[ffff:ffff]] class 0x000000/00000000
[    1.896882] vfio_pci: add [1002:aab0[ffff:ffff]] class 0x000000/00000000
[    1.902817] [drm] Initialized drm 1.1.0 20060810
[    1.926959] [drm] radeon kernel modesetting enabled.
...

Now all should work!

Keyboard and mouse.

You can make USB pass-through too. For example use: -usb -device usb-host,hostbus=X,hostaddr=Y
To see usb-device use lsusb.