Underneath most Xorg (and Wayland compositor) drivers is libdrm.
You can draw directly with it. I ported a
gameboy emulator to do just that including raw keyboard access. Before that we were using
syscons(4) and
vgl(3). But the later was broken when the newer console system was introduced and subsequently removed.
The reason why we tend not to is because IPC is poor (programs need to share the framebuffer). X11 does way more than is needed for a consumer playing games or browsing the web. But for enterprise / hetrogeneous environments it is useful.
I do have an early implementation of vgl(3) ontop of libdrm but I don't really have time to finish it off and tend to not use FreeBSD for client stuff currently.