Stuck on "Mounting local file systems..."

D

dazealex

Hi everyone,

I've been enjoying FreeBSD 9.0 for the most part over the past several months using ZFS. I come from the Ubuntu/Debian world, so it has been a bit of a learning curve.

Anyhow, I was trying to upgrade my ports when it seems to gotten stuck. I was using SSH, so I am not sure what happened exactly. Now when the system restarts, it gets stuck on the
Code: 
Mounting local file systems:.
part and I hear hard drives thrashing, but can't SSH into the system or remotely access it -- which is a pain consider its location in the basement. To show you my patience, I even let the system proceed along for a few days but it was still stuck at the mounting local file systems part.

Is there anyway to see what's going on? I did go into single user mode, did a [cmd=]mount -o rw /[/cmd] and commented out the NFS share mounts, and left everything else such as the / and swap mounts. However, no luck.

How can I fix this? Is there a way to connect to the system via serial cable attached to another box (a Ubuntu machine)?

Thanks,

Daze
 
I hear hard drives thrashing
Click to expand...

Yikes! It sounds like you may have a disk failure. You can verify this with smartmontools (smartctl --all /dev/disk). You will probably need to boot from a live media. Parted Magic works pretty well (also has a graphical smartmontools shell).

By the way, you can press ^C during startup to skip a rc.d script.
 
Is there a way to disable the auto-mounting of the ZFS mount? I'll run smartctl and get back to you on that. However, the root file system is on a 30GB OCZ SSD.
 
You can boot in single-user mode. At the boot loader prompt, type boot -s.
You will be booted in a minimal FreeBSD environment, rc.d scripts will not be run, and only the root filesystem will be mounted.
 
OK, so I reinstalled FreeBSD. Booted off of it just fine.

I think the problem is pertaining to my ZFS raidz1 mount. It's as follows:

Code: 
# zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
raid           1.43T   133G  1.26T  /raid
raid/Pictures  41.3G   133G  41.3G  /raid/Pictures
raid/iTunes     133G   133G   133G  /raid/iTunes
raid/ports     2.48M   133G  2.48M  /raid/ports

I can mount every mount just fine, except for the base, i.e. /raid. So when I issue this command:

# zfs mount raid

It just sits there. I do have de-dupe and compression enabled. And the machine is a dual core AMD (from a years back) with 2GB RAM.

What to do next?

Thanks for your help so far guys!

Daze
 
Update: Now the machine is frozen. I can ping it, but I can't SSH to it anymore. My screen session is also frozen, obviously... :(

Should I add more memory? Is that the problem considering I have dedup and compression enabled? The whole RAID isn't very big, it consists of 4x500GB Seagate drives. Two have the same firmware revisions, and the other two don't... Of these, two are running at SATA 2, and the other at SATA 1 speeds. I'm sure this has performance implications... But at this point, I just want my data to be available. I think going this route was way overkill with the hardware I have. Thoughts?

Daze
 
Your pool is almost full (>90% used). There is only 2G RAM with dedup turned on. It is a disaster waiting to happen. Try to run

# zpool import -o readonly=on raid

in single user mode. Then backup all of your data before proceeding further. Afterward, plan ahead before going on, e.g., you may start all over again, or you may replace current disks with larger capacity ones and turn off dedup.
 
Hi t1066,

My board has two free DIMM slots (4 in total). I could potentially add 2 sticks of 1GBx2, making it a total of 4GB RAM. Would this amount of memory help with a dedup setup or is the problem more on the disk usage size (>90% utilization) side?

I do think I learned a lesson here. I should have stuck to simple compression on ZFS, and chosen only specific pools for dedup.

My next question, can I replace the existing 4 drives (4x500GB) with let's say, 4x2GB without having to manually copy stuff around? Does ZFS support this? From my reading, it does not seem so. I really need the compression feature... (

Thanks,

Daze
 
Hi dazealex,

According to the ZFS wikipedia, 4GB RAM still may not be enough in your case. If you can find an SSD of more than 64GB capacity, adding it to the pool as L2ARC may be a better choice. In whatever route you went, you should test the stability of your system by backup some files and trimming the original pool. In addition, it would improve the performance of your pool so that the following steps would be faster. If you can finish the above steps without any problems, you should start to replace the 500GB disks with new 2TB disks using zpool replace. You want to do this one by one and it may take a few hours to a whole day or more for each disks. After replacing all the disks, then a simple zpool import, zpool export will expand the capacity of your pool to 6TB.

Good luck.
 
Hi t1066,

I do have a 128GB SSD lying around doing nothing. I can hook that up in a jiffy and see what happens.

Thanks so much for the excellent advice so far.

I'll report back shortly.

Daze
 
I added a 128GB SSD cache drive, and tried remounting. No go, system probably ran out of memory again and became unresponsive.

I haven't tried the import command in single user mode yet. But my next step would be to get 4GB RAM, and hope that the existing 2GB work with it, making it a total of 6GB RAM.
 
As promised, I wanted to report back that I was able to zfs mount after adding 4GB more RAM to machine (now has 6GB in total).

Thank you for all the suggestions and friendliness! I think you've made my decision to use FreeBSD look like a good one.

Daze
 
Wow. You have 1.4 TB of data stored in a pool with dedupe enabled, in a machine with only 2 GB of RAM, and you are wondering why it won't finish importing the pool after a crash?

You'll need to add at least 16 GB of RAM into that machine in order to get the thing to mount. The issue is that it needs to load the complete DDT (dedupe table) into ARC in order to completely load the malfunctioning filesystem. It needs to double-check every block in the DDT before it will load the filesystem. Which means you need a lot of RAM (260 bytes per entry in the DDT) to hold it.

We just went through this with one box with 24 GB of RAM that couldn't load a 5 MB filesystem. Turns out, the DDT for that pool is 27 GB in size. Had to grab RAM from another box to boost it to 32 MB temporarily.

Find another box with at least 16 GB (preferably 32 GB) of RAM. Stick the drives in that box, and try to import the pool. Once it's imported successfully, export/import it a couple of times to make sure.

Then you can move the drives back to the old machine and things will work.

HOWEVER, you really, really, really need to either get more RAM into that box, or disable dedupe and destroy/recreate the pool so there's no deduplication involved.

EDIT: I see you got lucky and were able to import it with only 6 GB. But don't take that as a sign you can continue with only 6 GB of RAM with dedupe enabled. The DDT will only grow over time.
 
Hi Freddie,

You're right. Deduping in "basement" level machines is a disaster waiting to happen. I am better off with just enabling compression. ZFS, rather, de-duplication isn't the panacea that I thought it was.

Thanks for the support everyone!

Daze
 
You must log in or register to reply here.
Back
Top