That depends on the distro. Slackware and Archlinux, for instance, do not install anything you don't want. The bigger distros are trying to cater to "Joe Sixpack" for some strange reason.
Anyway, I have run into one problem: no guest additions for FreeBSD as a guest in Virtualbox. It makes me...