So, how do we make this somehow safe?
There are a lot of (silly) black and white answers above. But the real world is not black and white, it's complex and colored.
If you have a server that's "hosted" (in whatever fashion, ranging from a short-lived task on a VM somewhere in an AWS data center, to a wire cage with customer-controlled lock in a data center), then the people who have physical control of the infrastructure can, in extreme cases, get control of that server. By the same logic, if your server is connected to any network, than the people who control that network can in theory also do whatever they want to you. If you look only at extreme cases, nothing is safe.
But by the same logic, nothing is safe at other locations either. If you have your server at your house or your business, a burglar can come and steal it. Sure, you could encrypt the disks, but a smart burglar could come and install a listening device on whatever way you enter the decryption password (keyboard or network cable), come back a month later having sniffed the password, and then steal your computer.
In the real world, you have to do tradeoffs. I would say that having your server in a well-managed data center run by is on average very safe; probably much safer than having it in a badly managed hosted environment, and probably considerably safer than in typical residential environments.