setting memory usage limit

Is possible? This happened to me:

I have a FreeBSD box, with 4GB of RAM, where I run VirtualBox headless for 2 virtual machines
(Windows 2003 server and Windows 7). I assign 1.5GB for each one, for now no problem,
but when the memory saturate, I see it with freecolor -om minutes before, the box does not respond to anything. Not ssh and even with a keyboard plugged in. The only way is shutting down by the power button.

So my question is, is there any way to say to FreeBSD:
"I have 4GB of Ram, and I want to use 3GB for applications (not part of the userland), the rest is for the system"