To run X, you need the whole xorg: /usr/ports/x11/xorg. Some window managers don't depend on xorg explicitly, possibly because they might also be used with XFree86. xfce4 is like that.
Anyway, install xorg. There's also x11/xorg-minimal, which I have not tried.