Exactly. Moving time forward is safe: Of you for example jump from 2 in the afternoon to 3 in the afternoon, then absolutely nothing happened between 2 and 3, but computers know how to handle delays.
Moving time backwards is unsafe. If at 3pm you go back to 2pm, and there are two events, one at 2:15 and one at 2:45, nobody can tell which happened first. A program like Make might get confused by that, and for example think a source code that was last changed at 2:15 must have been compiled already, because the object file was created at 2:45. But since the 2:15 might be from the second time around and the 2:45 from the first time, that could be wrong.
If you have to move time backwards, make sure that nothing time sensitive is running at the time. For example do it in single-user mode during a reboot. Or run NTP all the time, then the whole problem won't even arise.