That's fixable in BIOS, you should be able to specify that the USB sticks boot first. But yeah, if you remove the stick that the BIOS saw last time, then the BIOS boot order will revert back to Windows on the HDD.
A BIOS will boot whatever it can see.
