No suggestions, but i have exactly the same problem as you do.
All my hardware is different, but the setup is mostly the same(small file server, only used by me, etc).
One thing i have found is that "top" will say "Mem: 962M Active" even when i have closed all programs, and nothing should be...