Removed Everything(?) - Where to start?

[markosolo]

So I'm busy cleaning up our main box which hosts my works 3 main websites when I decide I want to get rid of postfix and start again, redo the whole mail side of things from scratch. This box runs FreeBSD 8.2 + Apache 2.2 Worker + PHP + Postfix + MySQL + IPFW and I think that's it....

I'm about to cleanup when I get a phone call about an emergency meeting, I let them know I'll be there in a sec, i really wanted to get this server sorted. Rushing now, I run the following in a moment of madness: # pkg_delete postfix/*

instead of: # pkg_delete postfix\*

I didn't even notice until I saw a warning about MySQL and then I realised and poo'd my pants... I killed the process straight away - everything looked fine, all the sites were running, etc so I ran off to the meeting I was now a few minutes late for. When I got back I brought up a separate SSH to the box and went to ssh in - the public key auth no longer worked... I put in the password and still couldn't login... I looked across at my other session that was sudosh to root and thought what the heck(completely forgetting about earlier) and rebooted the box. Anyways the box comes back up, still no SSH, so I console to it in VMware and try login as root, I get in but it cant find bash so it logs me out. I rebooted into single user mode and sweet I can see all the users still, unfortunately they all use bash which is now gone dammit. So now I cant get onto this box and I don't know what to do, so here are my questions:

1. I noticed /var and /home (is in var) were both empty when booted into single user mode - have I somehow deleted all the files? Or does it just not mount /var[/fi;le] in single user? The files are the most important at this stage...
   [*]What's the recommended course of action from here? With linux i would boot to a live cd change /etc/passwd so that root uses /bin/sh and then reboot, install what I need and copy off all the files before re-evaluating... Can I do this with BSD? What live-cd should I use?
   [*]Am I better off just getting the files off the box and then moving to a new box with a clean install? Or rebuilding this one?
   [*]Do you think there is any way of recovering the mysql databases?
   [*]Any tips? ARRGGHH I'm so worried!

Cheers!
Mark

 

[SirDice]

1. I noticed /var and /home(is in var) were both empty when booted into single user mode - have I somehow deleted all the files? Or does it just not mount /var in single user? The files are the most important at this stage...

In single user mode only the root (/) filesystem is mounted, read-only.

2. Whats the recommended course of action from here? With linux i would boot to a live cd change /etc/passwd so that root uses /bin/sh and then reboot, install what I need and copy off all the files before re-evaluating... Can I do this with BSD? What live-cd should I use?

Boot to single user mode then:

fsck -y
mount -u /
mount -a -t ufs
swapon

And change root's shell.

3. Am I better off just getting the files off the box and then moving to a new box with a clean install? Or rebuilding this one?

No need to do a clean install, just # pkg_delete -a and reinstall your ports.

4. Do you think there is any way of recovering the mysql databases?

The database is still there. It won't be removed if you remove MySQL.

 

[markosolo]

Thanks SirDice. I tried what you said and it looks like the changes I made to /etc/passwd didn't take affect... I made the changes in vi, saved and rebooted and then when the box came back up it still wouldn't let me login, was looking for bash... Any tips?

 

[SirDice]

I tried what you said and it looks like the changes I made to /etc/passwd didn't take affect... I made the changes in vi, saved and rebooted and then when the box came back up it still wouldn't let me login, was looking for bash... Any tips?

NEVER edit /etc/passwd directly! Not only won't this update /etc/pwd.db you'll also run the risk of completely nuking the file.

Use chsh(1), pw(8) or vipw(8).

 

[wblock@]

pkg_delete postfix/*

That wouldn't remove anything. Likely this went one of two ways. The first is that there was a space before \* and you deleted all packages. That's easy enough to fix, just back up /usr/local/etc and then install all the needed ports again. Or restore the whole system from backups. You have backups, right?

The other possibility is that you used rm instead of pkg_delete. In that case, restore from backups, and stop using rm for package management.

I looked across at my other session that was sudosh to root and thought what the heck(completely forgetting about earlier) and rebooted the box. Anyways the box comes back up, still no SSH, so I console to it in VMware and try login as root, I get in but it cant find bash so it logs me out.

Rebooting to "fix things" is a Windowsism, setting root's shell to bash is a Linuxism, and you've discovered they're both mistakes.

1. I noticed /var and /home(is in var) were both empty when booted into single user mode - have I somehow deleted all the files? Or does it just not mount /var in single user? The files are the most important at this stage...

Single-user doesn't mount everything, that's all. Mount the other filesystems from /etc/fstab manually.

 

[phoenix]

I'm about to cleanup when I get a phone call about an emergency meeting, I let them know I'll be there in a sec, i really wanted to get this server sorted. Rushing now, I run the following in a moment of madness: # pkg_delete postfix/*

instead of: # pkg_delete postfix\*

And this is why you should never use shell wildcards with pkg tools. They all support globbing internally via -x. The correct command to use would have been:
# pkg_delete -xi postfix

-x to remove any packages with "postfix" anywhere in the name
-i to ask for confirmation before actually removing it

When in doubt, read the pkg_delete(1) man page first.

Anyways the box comes back up, still no SSH, so I console to it in VMware and try login as root, I get in but it cant find bash so it logs me out.

And now you know why you should never change root's shell. Leave it as tcsh. If you really need a more interactive shell than that, then configure the toor account. That's why it's there.

• I noticed /var and /home (is in var) were both empty when booted into single user mode - have I somehow deleted all the files? Or does it just not mount /var[/fi;le] in single user? The files are the most important at this stage...

Only the / filesystem is mounted (as read-only) when you enter single-user mode. You have to manually remount the / filesystem read/write, and then manually mount the rest of the filesystems.

• What's the recommended course of action from here? With linux i would boot to a live cd change /etc/passwd so that root uses /bin/sh and then reboot, install what I need and copy off all the files before re-evaluating... Can I do this with BSD? What live-cd should I use?

Mount the filesystems, enable networking, then check the contents of either /var/db/pkg or the output of # pkg_info to see what's missing, and then install the missing apps. Be sure to backup /usr/local/etc first (just in case).

As for a LiveCD, definitely download and make a few copies of the Frenzy LiveCD. It's a FreeBSD 8.1-based system, perfect for emergency system administration.

 

[wblock@]

Whoops, forgot to mention mfsBSD, a memory-based FreeBSD system that's nice for repairs.

 

[markosolo]

Thanks for all your help, got it all sorted.