I don't understand. If the SD card is not retained, then there is de-facto no SD card in the Pi. Why would it make a difference whether the card that is not in there is a Raspbian, FreeBSD or purple elefant card? By "Rainbow Screen" I presume you mean the screen that has all the colors in a big rectangle or square? If I remember right, that comes up if there is no readable boot image, so boot does not go on. Which matches your description that there is no SD card in the slot.
Have you tried some tape (masking tape, scotch tape) to hold the card in?