I am experimenting with local AI models on FreeBSD, and I must say so far it has been a joy!
There is but one key issue. My test server has four Radeon MI50 GPUs with 32GB, all identical models and all reporting the same BIOS version and DeviceID.
I'm running 15.0-RELEASE with AMDGPU from drm-kmod which attaches just fine and I have four /dev/dri/cardX and /dev/dri/renderX devices. I can see the cards in pciconf. However, when I run any vulkan application only two devices have drivers attached.
I tried updating to drm-latest-kmod, but then none of the GPUs get a vulkan driver attached, and I am stuck with llvm-pipe.
I have tried running Fedora Server on the hardware and that does correctly provide four vulkan devices, so the hardware seems fine. I see that a similar problem have plagued NVidia in the past, but I have not found anything to correct the issue.
Any thoughts on how I might coerce vulkan into loading drivers for all the cards will be greatly appreciated.
There is but one key issue. My test server has four Radeon MI50 GPUs with 32GB, all identical models and all reporting the same BIOS version and DeviceID.
I'm running 15.0-RELEASE with AMDGPU from drm-kmod which attaches just fine and I have four /dev/dri/cardX and /dev/dri/renderX devices. I can see the cards in pciconf. However, when I run any vulkan application only two devices have drivers attached.
I tried updating to drm-latest-kmod, but then none of the GPUs get a vulkan driver attached, and I am stuck with llvm-pipe.
I have tried running Fedora Server on the hardware and that does correctly provide four vulkan devices, so the hardware seems fine. I see that a similar problem have plagued NVidia in the past, but I have not found anything to correct the issue.
Any thoughts on how I might coerce vulkan into loading drivers for all the cards will be greatly appreciated.
Bash:
$ MESA_VK_DEVICE_SELECT=list vulkaninfo
WARNING: [Loader Message] Code 0 : Path to given binary /usr/local/lib/libvulkan_intel.so was found to differ from OS loaded path /usr/local/lib/l
ibvulkan_intel-devel.so
WARNING: [Loader Message] Code 0 : Path to given binary /usr/local/lib/libvulkan_radeon.so was found to differ from OS loaded path /usr/local/lib/
libvulkan_radeon-devel.so
WARNING: [Loader Message] Code 0 : Path to given binary /usr/local/lib/libvulkan_intel_hasvk.so was found to differ from OS loaded path /usr/local
/lib/libvulkan_intel_hasvk-devel.so
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
selectable devices:
GPU 0: 1002:66a0 "AMD Radeon Graphics (RADV VEGA20)" discrete GPU 0000:83:00.0
GPU 1: 1002:66a0 "AMD Radeon Graphics (RADV VEGA20)" discrete GPU 0000:c6:00.0
GPU 2: 10005:0 "llvmpipe (LLVM 19.1.7, 256 bits)" CPU 0000:00:00.0
Bash:
$ pciconf -l -vvv |grep -B 4 VGA
vgapci0@pci0:198:0:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0x1002 device=0x66a0 subvendor=0x1002 subdevice=0x081e
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Vega 20 [Radeon Pro/Radeon Instinct]'
class = display
subclass = VGA
--
vgapci1@pci0:200:0:0: class=0x030000 rev=0x41 hdr=0x00 vendor=0x1a03 device=0x2000 subvendor=0x1a03 subdevice=0x2000
vendor = 'ASPEED Technology, Inc.'
device = 'ASPEED Graphics Family'
class = display
subclass = VGA
--
vgapci2@pci0:131:0:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0x1002 device=0x66a0 subvendor=0x1002 subdevice=0x081e
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Vega 20 [Radeon Pro/Radeon Instinct]'
class = display
subclass = VGA
--
vgapci3@pci0:67:0:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0x1002 device=0x66a0 subvendor=0x1002 subdevice=0x081e
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Vega 20 [Radeon Pro/Radeon Instinct]'
class = display
subclass = VGA
--
vgapci4@pci0:3:0:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0x1002 device=0x66a0 subvendor=0x1002 subdevice=0x081e
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Vega 20 [Radeon Pro/Radeon Instinct]'
class = display
subclass = VGA