For as long as I remember when nvidia-modeset was used I have always had to configure xorg.conf to specify nvidia to get X working. Otherwise X would continually try the wrong driver, and fail. Of course nvidia-drm module also fixes this, but I am happy to report that xlibre-nvidia-driver-580.119.02_1 fixes the issue for nvidia-modeset as well. It's nice to see something like this finally, after all these years even though it is a minor inconvenience get fixed for FreeBSD to where things just work more out of box now.