Figured it out. It appears to be incorrectly documented in syscons(4). Setting the high 16 bits of hint.sc.0.flags will get you no where. You need to set hint.sc.0.vesa_mode in /boot/loader.conf. For example, the kernel sets 1024x768x16 for me with this:
0x0117 is taken from
(MODE_279)
A little bit better than allscreens_flags in rc.conf.
Code:
hint.sc.0.flags=0x0180
hint.sc.0.vesa_mode=0x0117
0x0117 is taken from
# vidcontrol -i mode
(MODE_279)
A little bit better than allscreens_flags in rc.conf.