Laptop with AMD 'Cezanne' graphics + Nvidia RTX 3060

I use hybrid setup with the latest nVidia driver 515.57, works pretty well.
With what? There are zero plausible use cases for it. Well, maybe Blender with Eevee, but that's really niche.

As I said I can have two screens configured in xorg.conf, one screen per each GPU. In this case two GPU drivers load well, but this setup does not allow render offloading. No matter what I try, I cannot fit two GPUs into one screen setup.
What is this "one screen setup"?

I can have either "nvidia" driver or "amdgpu" driver loaded by Xorg at a moment, but not both. If I set "modesetting" driver for iGPU, then "nvidia" driver for dGPU never gets probed.
Did you notice nvidia-hybrid-graphics puts nvidia_drv.so outside of default search path?

Can you please share your Xorg.0.log with working setup of two GPUs ?
I only have one GPU at the moment.
 
I get some vt backtrace when loading amdgpu on 14.0:

Code:
amdsmn0: <AMD Family 17h System Management Network> on hostb0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb0
[drm] amdgpu kernel modesetting enabled.
drmn0: <drmn> on vgapci0
vgapci0: child drmn0 requested pci_enable_io
vgapci0: child drmn0 requested pci_enable_io
[drm] initializing kernel modesetting (RENOIR 0x1002:0x1636 0x17AA:0x507F 0xC3).
drmn0: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
[drm] register mmio base: 0xFD300000
[drm] register mmio size: 524288
[drm] add ip block number 0 <soc15_common>
[drm] add ip block number 1 <gmc_v9_0>
[drm] add ip block number 2 <vega10_ih>
[drm] add ip block number 3 <psp>
[drm] add ip block number 4 <smu>
[drm] add ip block number 5 <gfx_v9_0>
[drm] add ip block number 6 <sdma_v4_0>
[drm] add ip block number 7 <dm>
[drm] add ip block number 8 <vcn_v2_0>
[drm] add ip block number 9 <jpeg_v2_0>
drmn0: Fetched VBIOS from VFCT
amdgpu: ATOM BIOS: 113-RENOIR-025
drmn0: successfully loaded firmware image 'amdgpu/renoir_sdma.bin'
[drm] VCN decode is enabled in VM mode
[drm] VCN encode is enabled in VM mode
[drm] JPEG decode is enabled in VM mode
[drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
drmn0: VRAM: 512M 0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
drmn0: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
drmn0: AGP: 267419648M 0x000000F800000000 - 0x0000FFFFFFFFFFFF
[drm] Detected VRAM RAM=512M, BAR=512M
[drm] RAM width 128bits DDR4
[TTM] Zone  kernel: Available graphics memory: 7955644 KiB
[TTM] Zone   dma32: Available graphics memory: 2097152 KiB
[TTM] Initializing pool allocator
[drm] amdgpu: 512M of VRAM memory ready
[drm] amdgpu: 3072M of GTT memory ready.
[drm] GART: num cpu pages 262144, num gpu pages 262144
[drm] PCIE GART of 1024M enabled (table at 0x000000F400900000).
drmn0: successfully loaded firmware image 'amdgpu/renoir_asd.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_pfp.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_me.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_ce.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_rlc.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_mec.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_mec2.bin'
drmn0: successfully loaded firmware image 'amdgpu/renoir_dmcub.bin'
[drm] Loading DMUB firmware via PSP: version=0x01000000
drmn0: successfully loaded firmware image 'amdgpu/renoir_vcn.bin'
[drm] Found VCN firmware Version ENC: 1.7 DEC: 4 VEP: 0 Revision: 17
drmn0: Will use PSP to load VCN firmware
[drm] reserve 0x400000 from 0xf41f800000 for PSP TMR
drmn0: SMU is initialized successfully!
[drm] kiq ring mec 2 pipe 1 q 0
[drm] Display Core initialized with v3.2.104!
[drm] DMUB hardware initialized: version=0x01000000
lkpi_iic0: <LinuxKPI I2C> on drmn0
iicbus0: <Philips I2C bus> on lkpi_iic0
iic0: <I2C generic I/O> on iicbus0
ugen1.3: <vendor 0x8087 product 0x0029> at usbus1
lkpi_iic1: <LinuxKPI I2C> on drmn0
iicbus1: <Philips I2C bus> on lkpi_iic1
iic1: <I2C generic I/O> on iicbus1
lkpi_iic2: <LinuxKPI I2C> on drmn0
iicbus2: <Philips I2C bus> on lkpi_iic2
iic2: <I2C generic I/O> on iicbus2
[drm] VCN decode and encode initialized successfully(under DPG Mode).
[drm] JPEG decode initialized successfully.
drmn0: SE 1, SH per SE 1, CU per SH 8, active_cu_number 6
[drm] fb mappable at 0x410CE0000
[drm] vram apper at 0x410000000
[drm] size 8294400
[drm] fb depth is 24
[drm]    pitch is 7680
VT: Replacing driver "efifb" with new "fb".
taskqueue_drain with the following non-sleepable locks held:
exclusive sleep mutex vtdev (vtdev) r = 0 (0xffffffff81aea698) locked @ /usr/src/sys/dev/vt/vt_core.c:3051
stack backtrace:
#0 0xffffffff80c5c815 at witness_debugger+0x65
#1 0xffffffff80c5d97a at witness_warn+0x3fa
#2 0xffffffff80c4f113 at taskqueue_drain+0x33
#3 0xffffffff84451933 at vt_kms_postswitch+0x73
#4 0xffffffff80a314fd at vt_fb_init+0xfd
#5 0xffffffff80a36b48 at vt_replace_backend+0x118
#6 0xffffffff80a31603 at vt_fb_attach+0x13
#7 0xffffffff844635bb at linux_register_framebuffer+0xeb
#8 0xffffffff84464a62 at __drm_fb_helper_initial_config_and_unlock+0x492
#9 0xffffffff8401d9a6 at amdgpu_fbdev_init+0xe6
#10 0xffffffff840125e0 at amdgpu_device_init+0x1d70
#11 0xffffffff84029536 at amdgpu_driver_load_kms+0x16
#12 0xffffffff8401ca5f at amdgpu_pci_probe+0x1ef
#13 0xffffffff80e635f1 at linux_pci_attach_device+0x431
#14 0xffffffff80c27021 at device_attach+0x3c1
#15 0xffffffff80c26bd0 at device_probe_and_attach+0x70
#16 0xffffffff80c28be7 at bus_generic_driver_added+0x67
#17 0xffffffff80c246a9 at devclass_driver_added+0x39
Sleeping on "tq_drain" with the following non-sleepable locks held:
exclusive sleep mutex vtdev (vtdev) r = 0 (0xffffffff81aea698) locked @ /usr/src/sys/dev/vt/vt_core.c:3051
stack backtrace:
#0 0xffffffff80c5c815 at witness_debugger+0x65
#1 0xffffffff80c5d97a at witness_warn+0x3fa
#2 0xffffffff80bf6e05 at _sleep+0x55
#3 0xffffffff80c4f1db at taskqueue_drain+0xfb
#4 0xffffffff84451933 at vt_kms_postswitch+0x73
#5 0xffffffff80a314fd at vt_fb_init+0xfd
#6 0xffffffff80a36b48 at vt_replace_backend+0x118
#7 0xffffffff80a31603 at vt_fb_attach+0x13
#8 0xffffffff844635bb at linux_register_framebuffer+0xeb
#9 0xffffffff84464a62 at __drm_fb_helper_initial_config_and_unlock+0x492
#10 0xffffffff8401d9a6 at amdgpu_fbdev_init+0xe6
#11 0xffffffff840125e0 at amdgpu_device_init+0x1d70
#12 0xffffffff84029536 at amdgpu_driver_load_kms+0x16
#13 0xffffffff8401ca5f at amdgpu_pci_probe+0x1ef
#14 0xffffffff80e635f1 at linux_pci_attach_device+0x431
#15 0xffffffff80c27021 at device_attach+0x3c1
#16 0xffffffff80c26bd0 at device_probe_and_attach+0x70
#17 0xffffffff80c28be7 at bus_generic_driver_added+0x67
lock order reversal: (Giant after non-sleepable)
 1st 0xffffffff81aea698 vtdev (vtdev, sleep mutex) @ /usr/src/sys/dev/vt/vt_core.c:3051
 2nd 0xffffffff81a02b40 Giant (Giant, sleep mutex) @ /usr/src/sys/kern/kern_synch.c:232
lock order Giant -> vtdev established at:
#0 0xffffffff80c5bb0d at witness_checkorder+0x32d
#1 0xffffffff80bc51b4 at __mtx_lock_flags+0x94
#2 0xffffffff80a36115 at vt_upgrade+0x365
#3 0xffffffff80b71a08 at mi_startup+0x1e8
#4 0xffffffff8038b023 at btext+0x23
lock order vtdev -> Giant attempted at:
#0 0xffffffff80c5c3dd at witness_checkorder+0xbfd
#1 0xffffffff80bc51b4 at __mtx_lock_flags+0x94
#2 0xffffffff80bf712f at _sleep+0x37f
#3 0xffffffff80c4f1db at taskqueue_drain+0xfb
#4 0xffffffff84451933 at vt_kms_postswitch+0x73
#5 0xffffffff80a314fd at vt_fb_init+0xfd
#6 0xffffffff80a36b48 at vt_replace_backend+0x118
#7 0xffffffff80a31603 at vt_fb_attach+0x13
#8 0xffffffff844635bb at linux_register_framebuffer+0xeb
#9 0xffffffff84464a62 at __drm_fb_helper_initial_config_and_unlock+0x492
#10 0xffffffff8401d9a6 at amdgpu_fbdev_init+0xe6
#11 0xffffffff840125e0 at amdgpu_device_init+0x1d70
#12 0xffffffff84029536 at amdgpu_driver_load_kms+0x16
#13 0xffffffff8401ca5f at amdgpu_pci_probe+0x1ef
#14 0xffffffff80e635f1 at linux_pci_attach_device+0x431
#15 0xffffffff80c27021 at device_attach+0x3c1
#16 0xffffffff80c26bd0 at device_probe_and_attach+0x70
#17 0xffffffff80c28be7 at bus_generic_driver_added+0x67
start FB_INFO:
type=11 height=1080 width=1920 depth=32
pbase=0x410ce0000 vbase=0xfffff80410ce0000
name=drmn0 flags=0x0 stride=7680 bpp=32
end FB_INFO
drmn0: ring gfx uses VM inv eng 0 on hub 0
drmn0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
drmn0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
drmn0: ring comp_1.2.0 uses VM inv eng 5 on hub 0
drmn0: ring comp_1.3.0 uses VM inv eng 6 on hub 0
drmn0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
drmn0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
drmn0: ring comp_1.2.1 uses VM inv eng 9 on hub 0
drmn0: ring comp_1.3.1 uses VM inv eng 10 on hub 0
drmn0: ring kiq_2.1.0 uses VM inv eng 11 on hub 0
drmn0: ring sdma0 uses VM inv eng 0 on hub 1
drmn0: ring vcn_dec uses VM inv eng 1 on hub 1
drmn0: ring vcn_enc0 uses VM inv eng 4 on hub 1
drmn0: ring vcn_enc1 uses VM inv eng 5 on hub 1
drmn0: ring jpeg_dec uses VM inv eng 6 on hub 1
vgapci0: child drmn0 requested pci_get_powerstate
sysctl_warn_reuse: can't re-use a leaf (hw.dri.debug)!
lkpi_iic3: <LinuxKPI I2C> on drm1
iicbus3: <Philips I2C bus> on lkpi_iic3
iic3: <I2C generic I/O> on iicbus3
lkpi_iic4: <LinuxKPI I2C> on drm3
iicbus4: <Philips I2C bus> on lkpi_iic4
iic4: <I2C generic I/O> on iicbus4
[drm] Initialized amdgpu 3.40.0 20150101 for drmn0 on minor 0
 
Do I need CURRENT? Will it work there? -- I built and installed the 14.0 kernel. No difference.
I'm using 13.1-RELEASE now and it works fine with 5.10-lts amdgpu driver. I noticed that my GPU has a different ChipID which is 0x1638 where yours is 0x1636.

Have your tried some Linux on your machine ? Can it handle GPU properly ? If so, with what DRM driver and Xorg version ?
 
I noticed that my GPU has a different ChipID which is 0x1638 where yours is 0x1636.
Strange, the 1636 is listed in drivers/gpu/drm/amd/display/include/dal_asic_id.h, while the 1638 isn't. (I found it)
Searching for 0x1636 I found this who apparently succeeded with drm-kmod 5.5-stable, but that version doesn't compile with today's 13.1-STABLE.

... I built the commits of kernel (13.0) and drm-kmod (5.5) he advised and get exactly the same result as before ...
 
Strange, the 1636 is listed in drivers/gpu/drm/amd/display/include/dal_asic_id.h, while the 1638 isn't. (I found it)
Searching for 0x1636 I found this who apparently succeeded with drm-kmod 5.5-stable, but that version doesn't compile with today's 13.1-STABLE.

... I built the commits of kernel (13.0) and drm-kmod (5.5) he advised and get exactly the same result as before ...
You should use drm-kmod 5.10-lts from the git repo. The one that is in FreeBSD repo does not work with our GPUs.

Here's what I have installed:

Code:
rz@butterfly:~/drm-kmod % git status
On branch 5.10-lts
Your branch is up to date with 'origin/5.10-lts'.

nothing to commit, working tree clean
rz@butterfly:~/drm-kmod % git remote -v
origin    https://github.com/freebsd/drm-kmod.git (fetch)
origin    https://github.com/freebsd/drm-kmod.git (push)

rz@butterfly:~/drm-kmod % git log
commit ecbad2e93ea96fab832b1d144b7dcf839b32f36e (HEAD -> 5.10-lts, origin/5.10-lts)
Author: Emmanuel Vadot <manu@FreeBSD.org>
Date:   Wed Jul 27 11:40:47 2022 +0200

    linuxkpi: Use power_supply.h from base
    
    It's present since 13.1
    
    Sponsored by:   Beckhoff Automation GmbH & Co. KG
    
    (cherry picked from commit 833208195d4880b95ce4fb01a09e48af50813ab1)
 
Code:
rz@butterfly:~/drm-kmod % git status
commit ecbad2e93ea96fab832b1d144b7dcf839b32f36e (HEAD -> 5.10-lts, origin/5.10-lts)
Author: Emmanuel Vadot <manu@FreeBSD.org>
Date:   Wed Jul 27 11:40:47 2022 +0200
All the same here, but I think you are one commit behind ;)
And whether I use the master branch or 5.10-lts doesn't change the results for me ...

Is there a comprehensive list of Options I can play with? Are there more than:
Code:
Section "Device"
#Section "OutputClass"
        Identifier "Card0"
       Driver "amdgpu"
#Driver "scfb"
#       Driver "modesetting"
#    BusID "PCI:4:0:0"

#      Option "AccelMethod" "none"
#    Option "MigrationHeuristic" "greedy"
#       Option "SWcursor" "Off"
#       Option "Accel" "Off"
#       Option "ZaphodHeads" "string"
       Option "DRI" "3"
#       Option "EnablePageFlip" "Off"
#       Option "TearFree" "On"
#       Option "VariableRefresh" "On"
#       Option "AsyncFlipSecondaries" "Off"
#      Option "ShadowPrimary" "On"
EndSection
 
I made a diff of the linux and freebsd Xorg.0.log.
The only difference I see is the smaller(!) gart and vram size on linux - does it mean anything?
The linux kernel module has an option to restrict gartsize ... can I do something like this on FreeBSD?

... There is hw.amdgpu.gartsize, but setting it doesn't change anything.
And setting the graphics memory in BIOS changes the vram amounts, also in linux, but doesn't help.

Code:
--- xorg.fbsd    2022-08-09 14:48:33.322869000 +0200
+++ xorg.linux    2022-08-09 14:48:46.885883000 +0200
@@ -1,7 +1,8 @@
 
 r 1.21.1.4
 rsion 11, Revision 0
-Current Operating System: FreeBSD thinkpad 13.1-STABLE FreeBSD 13.1-STABLE #13 stable/13-n252050-9a0d922f57f: Sun Aug  7 13:02:24 CEST 2022     root@thinkpad:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
+Current Operating System: Linux manjaro 5.15.55-1-MANJARO #1 SMP PREEMPT Fri Jul 15 09:45:39 UTC 2022 x86_64
+Kernel command line: BOOT_IMAGE=/boot/vmlinuz-x86_64 lang=en_US keytable=de tz=UTC misobasedir=manjaro misolabel=MANJARO_XFCE_2135 quiet systemd.show_status=1 apparmor=1 security=apparmor driver=free nouveau.modeset=1 i915.modeset=1 radeon.modeset=1
 
 (II) Module ABI versions:
@@ -43,31 +43,59 @@
     X.Org Video Driver: 25.2
     X.Org XInput driver : 24.4
     X.Org Server Extension : 10.0
-(--) PCI:*(4@0:0:0) 1002:1636:17aa:507f rev 195, Mem @ 0x460000000/268435456, 0x470000000/2097152, 0xfd300000/524288, I/O @ 0x00001000/256, BIOS @ 0x????????/65536

+(--) PCI:*(4@0:0:0) 1002:1636:17aa:507f rev 195, Mem @ 0x460000000/268435456, 0x470000000/2097152, 0xfd300000/524288, I/O @ 0x00001000/256
[ICODE]1e2e5000[/ICODE]

 (II) Module amdgpu: vendor="X.Org Foundation"
-    compiled for 1.21.1.4, module version = 22.0.0
+    compiled for 1.21.1.3, module version = 22.0.0
     Module class: X.Org Video Driver
     ABI class: X.Org Video Driver, version 25.2

@@ -79,11 +107,11 @@


-(II) AMDGPU(0): glamor X acceleration enabled on AMD RENOIR (DRM 3.40.0, 13.1-STABLE, LLVM 13.0.1)
+(II) AMDGPU(0): glamor X acceleration enabled on AMD RENOIR (LLVM 14.0.6, DRM 3.42, 5.15.55-1-MANJARO)
 (II) AMDGPU(0): glamor detected, initialising EGL layer.
 (==) AMDGPU(0): TearFree property default: auto
 (==) AMDGPU(0): VariableRefresh: disabled
@@ -142,19 +170,21 @@

 (II) AMDGPU(0): Output eDP using initial mode 1920x1080 +0+0
-(II) AMDGPU(0): mem size init: gart size :bfa7e000 vram size: s:1e2e5000 visible:1e2e5000
+(II) AMDGPU(0): mem size init: gart size :bf6ea000 vram size: s:1df2d000 visible:1df2d000
 
I made a diff of the linux and freebsd Xorg.0.log.
The only difference I see is the smaller(!) gart and vram size on linux - does it mean anything?
The linux kernel module has an option to restrict gartsize ... can I do something like this on FreeBSD?
I'm not an expert in these matters, but what version of DRM driver do you run on Linux ( uname -r) ?
 
FreeBSD:
[drm] Initialized amdgpu 3.40.0 20150101 for drmn0 on minor 0

Linux:
[drm] Initialized amdgpu 3.42.0 20150101 for 0000:04:00.0 on minor 0

an older gparted iso works with 3.40:

AMDGPU(0): glamor X acceleration enabled on AMD RENOIR (DRM 3.40.0, 5.10.0-8-amd64, LLVM 11.0.1)

it even works with 3.39 (gparted 1.1):
AMDGPU(0): glamor X acceleration enabled on AMD RENOIR (DRM 3.39.0, 5.9.0-3-amd64, LLVM 11.0.0)
 
Top