How to get FreeBSD + Bhyve + SR-IOV to work

In my case with the X552 10GBASE-T NIC. Updating the driver may be optional. I used FreeBSD 12.1-RELEASE-p10 and ix(4) driver 3.3.14.

Preparations

Download and install updated Intel network from here. Do not use package intel-ix-kmod, as it is not built with SR-IOV support.
Edit the Makefile to build the SR-IOV version.
Compile and copy it to /boot/modules/if_ix_updated.ko.
Edit /usr/share/examples/bhyve/vmrun.sh and remove four lines
```
if [ $tap_total -eq 0 ] ; then
    tap_total=1
    tap_dev0="${DEFAULT_TAPDEV}"
fi
```
This removes the minimum of one vtnet interface.

Checks

Check the device you want to use, in thise case ix0, shows up in /dev/iov. If it does not, you have the wrong device driver, BIOS-setting, CPU, PCI-root bridge. The device needs SR-IOV capability and the PCIe-bridges need Access Control Service capability. SR-IOV needs to be enabled in the BIOS.

Check that module vmm.ko loads. If it does not, you have incorrect BIOS-setting or a CPU that is too old or unsupported. VT-d or AMD-Vi need to be enabled in the BIOS to enable the IOMMU.

Configuration

/boot/loader.conf excerpt

vmm_load="YES"
if_ix_updated_load="YES"

/etc/rc.conf excerpt

iovctl_files="/etc/iovctl-ix0.conf"

/etc/iovctl-ix0.conf

PF {
	device: "ix0";
	num_vfs: 4;
}
DEFAULT {
	passthrough: true;
	allow-set-mac: true;
	allow-promisc: true;
}

This gives you four ppt devices that you passthrough to the VMs. Then do pciconf list and find the passthrough device IDs, for example

# pciconf -l | grep ^ppt
ppt0@pci0:3:0:128:	class=0x020000 card=0x15ad15d9 chip=0x15a88086 rev=0x00 hdr=0x00
ppt1@pci0:3:0:130:	class=0x020000 card=0x15ad15d9 chip=0x15a88086 rev=0x00 hdr=0x00
ppt2@pci0:3:0:132:	class=0x020000 card=0x15ad15d9 chip=0x15a88086 rev=0x00 hdr=0x00
ppt3@pci0:3:0:134:	class=0x020000 card=0x15ad15d9 chip=0x15a88086 rev=0x00 hdr=0x00

Here you see PCI device 3:0:128 for the first device. That number you put in the configuration of the first VM.

/vm/friet1-ppt.sh

sh /usr/share/examples/bhyve/vmrun.sh -m 2048 -d friet1.dsk -p 3/0/128 -c 2 friet1

Further work

Aggregated Ethernet
VLANs
Speed and load tests, compared to vtnet

References

As with a lot of things, once you know how to do them, they are easy.

And now for something completely different: do not put untagged (ix0.7) interfaces in a bridge. Performance will be horrible. Ping works, but there is packet loss, resulting in lots of TCP resends and lots of duplicate ACKs. 8kB/s is optimistic. Just put the tagged interface (ix0) in the bridge and configure the VLAN in the guest. This has security implications if the guest is untrusted.

webmaster@itsx / f+b+s / created 2020-9-26 / last update 2020-9-26