Xfce Xfce (soft of) hangs after short inactivity


I'm experiencing strange behavior with Xfce: after short inactivity (5-10 mins), the WM sort of hangs on me: I can move the mouse and the currently windows do refresh (e.g. with terminal open with htop). However I can't click on anything and the keyboard doesn't work (I can Ctrl-Alt-F1 though to switch to the terminal).

I turned off everything in the Screensaver and Power manager Xfce settings sections.

Any ideas?

EDIT: BTW, I'm using the "latest" FreeBSD repos with Xfce at version 4.16.1, could that be the reason?