Even restarting X doesn't help. I have to reboot to get the device working normally for a few days hours.