Solved Restore A Corrupted Jail

Hi,

imagine you run apache in a jail. Someone breaks in.

Is it possible to detect the intrusion somehow (checksums on binaries, user counting), inform the admin via mail, shut down the jail, restore the last snapshot of the jails mountpoint and start it again** automatically?

** maybe with a new firewall rule that blocks the attackers IP for 12 hours.

Or is this science fiction?
 
Why would you restore a known bad configuration? They got in, so it has a bug, restoring the bugged version is just bad mojo. You need to investigate how they got in and plug the hole before putting it back online. Or else it'll be hacked just as fast again.
 
Why would you restore a known bad configuration? They got in, so it has a bug, restoring the bugged version is just bad mojo. You need to investigate how they got in and plug the hole before putting it back online. Or else it'll be hacked just as fast again.
But you reduce downtime. "Show must go on." ;) Of course, you have to take care about fixing the bug, but this can be done parallel, right?
 
For about 5 minutes until it's hacked again. Then taken offline, restored, put back online and hacked again in 5 minutes. Repeat ad infinitum.
🤣 No, not that way. This might happen when you run windows in your jail ...
 
For about 5 minutes until it's hacked again. Then taken offline, restored, put back online and hacked again in 5 minutes. Repeat ad infinitum.
Just kick out and ban attackers until the fix is done. they may try another way ... they will always try. you cannot stop 'em. it is about how you deal with their attacks keeping up your app.
 
Just kick out and ban attackers until the fix is done.
Attackers often use proxies, so they'll just change their IP and attack again. It's like playing whack-a-mole. Any IP address you block will be pointless, they'll just popup on a different IP.
it is about how you deal with their attacks keeping up your app.
Yes, and just taking it down, and restoring a known broken configuration isn't the way to deal with it. You take it down, fix the hole then put it back online. The best course of action is of course not letting it get hacked in the first place. So that means diligently audit it before putting it online. Test, test and test again.
 
Hi,

imagine you run apache in a jail. Someone breaks in.

Is it possible to detect the intrusion somehow (checksums on binaries, user counting), inform the admin via mail, shut down the jail, restore the last snapshot of the jails mountpoint and start it again** automatically?

** maybe with a new firewall rule that blocks the attackers IP for 12 hours.

Or is this science fiction?
Is this possible anyway? If "yes", how to do it? Thank you.
 
Given enough time and resources anything is possible.

Using an IDS/IPS and/or something like Wireguard could detect the attack. Then have it kick off a script that takes down the jail, restores a snapshot (ZFS is useful) and starts it again.
Let's start with the easy one: How to take down a jail?
 
Can you shut down the OS inside the jail?
Sure, just a regular shutdown(8) will do the trick. But that typically only shuts down the processes inside the jail. Anything mounted for the jail (devfs(5) for example) will still be mounted, IP aliases will still be bound to interfaces, etc. So the jail isn't 'completely' shutdown as would be the case with a 'physical' machine.
 
In a jail, is it possible to detect that you are inside a jail? In vms you can take a look at the number of interupts and some cpu-registers to find out.
 
In a jail, is it possible to detect that you are inside a jail?
Code:
root@molly:~ # bastille console ports
[ports]:
root@ports:~ # sysctl -d security.jail.jailed
security.jail.jailed: Process in jail?
root@ports:~ # sysctl security.jail.jailed
security.jail.jailed: 1
root@ports:~ # logout

root@molly:~ # sysctl security.jail.jailed
security.jail.jailed: 0
 
Given enough time and resources anything is possible.

Using an IDS/IPS and/or something like Wireguard could detect the attack. Then have it kick off a script that takes down the jail, restores a snapshot (ZFS is useful) and starts it again.
Wireguard ?
 
Code:
root@molly:~ # bastille console ports
[ports]:
root@ports:~ # sysctl -d security.jail.jailed
security.jail.jailed: Process in jail?
root@ports:~ # sysctl security.jail.jailed
security.jail.jailed: 1
root@ports:~ # logout

root@molly:~ # sysctl security.jail.jailed
security.jail.jailed: 0
Shouldn't the jail be a restricted place with limited content/tools? Wouldn't it be better, if you could not see that you are inside a jail (for debug it is okay, I guess)?
 
Wireguard ?
Oops, brain fart. I actually meant Tripwire (security/tripwire)

Shouldn't the jail be a restricted place with limited content/tools?
It's a form of virtualization. It can be as complete or bare as you want or need it to be.

Wouldn't it be better, if you could not see that you are inside a jail (for debug it is okay, I guess)?
VMs know they're a VM too, I don't see a problem with that. Puppet for example has an $::is_virtual fact you can check. Quite useful as you often don't need to run certain services inside a VM or want to explicitly run certain services because it's a VM (think AWS/VMWare tools for example).
 
Attackers often use proxies, so they'll just change their IP and attack again. It's like playing whack-a-mole. Any IP address you block will be pointless, they'll just popup on a different IP.
Yeah sure, but the attackers can also run out of proxies with unbanned ips if you "kick and ban".
 
Oops, brain fart. I actually meant Tripwire (security/tripwire)
Description:Tripwire is a tool that aids system administrators and users in monitoring a designated set of files for any changes. Used with system files on a regular (e.g., daily) basis, Tripwire can notify system administrators of corrupted or tampered files, so damage control measures can be taken in a timely manner.
😁 Brain fart ... 🤣
Tripwire looks good.
 
Yeah sure, but the attacker can also run out of proxies with unbanned ips if you "kick and ban".
With botnets growing to thousands of computers that's quite unlikely. They're not using "proper" proxies, they're leveraging their army of botnet servers.
 
Back
Top