AFAIK, you should use kld_list="nvidia-modeset" on rc.conf instead.
Also, you can use /boot/loader.conf.local instead of /boot/loader.conf, so you can make the .local yours and let the loader.conf alone.
Two thoughts. The nvidia driver fom the nvidia site might be unknown to the freebsd package manager pkg, as such uninstall might be problematic.
Also to remove entries from loader.conf you sometimes have to boot into single user mode which is an annoyance.