ZFS Files/dirs in pool missing after reboot HELP!

Avery Freeman · Apr 19, 2018

11.1-Current freshly created VM in vSphere with passed-through LSI 2008 and 4x 2TB HGST 7k2000s, worked great in a CentOS VM previously, decided to migrate it to FreeBSD because smbshare is irreperrably broken in CentOS (could not be imported, unsupported feature set from ZoL).

~~But now I'm starting to think maybe it was a bad idea ...~~

My pool will seem fine, until I reboot, then upon boot I go to `/path/of/pool` and it's empty.

I haven't had time to properly nail down what's going on, it seems beyond me at this point. I've deleted all file-system created directories out of the pool so all there should be are datasets created with `zfs create pool/dataset`, and the files in them.

Exporting the pool and then immediately importing seems to work but is a frustratingly bad solution. Anyone ever seen this before?

Also, seek times seem awfully slow once pool is imported. Does that have to do with the recent import? It's killing me.

Edit: I overlooked a significant flag which in my experience was not required to mount ZFS in Linux, and it did require me having help because apparently it was staring me right in the face and I just didn't see it (I guess for me it's RTFMx2 or 3).

in /etc/rc.conf add

Code:

zfs_enable="YES"

Re: Slow pool, I think it was still loading in the background from being mounted while I was trying to use it, because now it's stunningly fast.

Thanks for your help and sorry for the dumb question, I'm sure there will be more.

ShelLuser · Apr 20, 2018

Pretty much impossible to comment on this because you're hardly giving us any useful information.

For example, what does: zpool status tell you? Or zfs list? How did you even determine /path/of/pool? I mean, if you installed FreeBSD using the default settings then there wouldn't be one: the main pool would be mounted on root (/). So... are your filesystems even mounted?

SirDice · Apr 20, 2018

Avery Freeman said:
11.1-Current

That version doesn't exist. It's 11.1-RELEASE, 11.1-STABLE or 12.0-CURRENT. And yes, it's an important distinction.

gkontos · Apr 20, 2018

You need to add zfs_enable="YES" in your /etc/rc.conf file.

Avery Freeman · Apr 21, 2018

I know, I feel really dumb about this. My FreeBSD-fu is so rusty, I'm sure I've forgotten more than I remember. Thanks for your help.

Avery Freeman · Apr 21, 2018

ShelLuser said:
Pretty much impossible to comment on this because you're hardly giving us any useful information.

For example, what does: zpool status tell you? Or zfs list? How did you even determine /path/of/pool? I mean, if you installed FreeBSD using the default settings then there wouldn't be one: the main pool would be mounted on root (/). So... are your filesystems even mounted?

You're right, sorry about that. For future reference:

# zpool status

Code:

  pool: qpool
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Apr 19 12:00:28 2018
config:

        NAME        STATE     READ WRITE CKSUM
        qpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0

errors: No known data errors

# zfs get mounted

Code:

NAME                    PROPERTY  VALUE    SOURCE
qpool                   mounted   yes      -
qpool/hdhomerun         mounted   yes      -
qpool/videos            mounted   yes      -
qpool/videos/Movies     mounted   yes      -
qpool/videos/TV         mounted   yes      -
qpool/videos/Tutorials  mounted   yes      -
qpool/videos/old_TV     mounted   yes      -

also

Code:

# zfs list -t filesystem
NAME                     USED  AVAIL  REFER  MOUNTPOINT
qpool                    382G  3.14T    23K  /qpool
qpool/hdhomerun          222G  3.14T   222G  /qpool/hdhomerun
qpool/videos             161G  3.14T    24K  /qpool/videos
qpool/videos/Movies     44.9G  3.14T  44.9G  /qpool/videos/Movies
qpool/videos/TV           23K  3.14T    23K  /qpool/videos/TV
qpool/videos/Tutorials  11.3G  3.14T  11.3G  /qpool/videos/Tutorials
qpool/videos/old_TV      104G  3.14T   104G  /qpool/videos/old_TV

because

Code:

# cat /etc/rc.conf | grep zfs

zfs_enable="YES"

ralphbsz · Apr 21, 2018

OK, everything looks good: Your pool exists, all the devices in the pool are online, everything is mounted. Makes sense, since ZFS is enabled in /etc/rc.conf. Please explain what the problem is exactly? You said "files/dirs missing", what does that exactly mean? What does "/path/of/pool" refer to? What do you mean when you say "I've deleted all file-system created directories out of the pool"? ZFS file systems don't always have file-system created directories.

ShelLuser · Apr 21, 2018

Avery Freeman said:
You're right, sorry about that. For future reference:

Ok, that helps. So do I assume right that when you mentioned /path/of/pool you're referring to /qpool and /qpool/videos?

That leaves one important question: where does your base system reside on? Is that another ZFS pool or did you use UFS? And if it's ZFS: did you set this up yourself or using the installer?

I have a hunch as to what could be going on but I can't back this up... If you have your base system installed in another ZFS pool (check /boot/loader.conf for more hints on that) then it is possible that the time of mounting could become an issue.

For example... I have zroot/var mounted on /var and I also have zroot/var/db and zroot/var/db/pkg. So here's the thing: the very moment I try to mount zroot/imapd onto /var/db/imapd by merely changing its mountpoint then it'll fail. I end up with an empty /var/db/imapd. The only way to solve this was to either create zroot/var/db/imapd (which I didn't want) or... add an entry to /etc/fstab and set up a 'delayed mount' (use the late option, see also fstab(5)).

The cause of the problem was that /var/db/imapd was getting mounted before /var/db was mounted. Both were different filesystems and one basically depended on the other, but got mounted too early.

So now I can't help but wonder if you're experiencing the same thing. In your example qpool possibly depending on your base system to be present while it might not be mounted at that given time during boot. It seems a bit odd, but... that's why this is simply a theory

Avery Freeman · Apr 22, 2018

ShelLuser said:
Ok, that helps. So do I assume right that when you mentioned /path/of/pool you're referring to /qpool and /qpool/videos?

That leaves one important question: where does your base system reside on? Is that another ZFS pool or did you use UFS? And if it's ZFS: did you set this up yourself or using the installer?

I have a hunch as to what could be going on but I can't back this up... If you have your base system installed in another ZFS pool (check /boot/loader.conf for more hints on that) then it is possible that the time of mounting could become an issue.

For example... I have zroot/var mounted on /var and I also have zroot/var/db and zroot/var/db/pkg. So here's the thing: the very moment I try to mount zroot/imapd onto /var/db/imapd by merely changing its mountpoint then it'll fail. I end up with an empty /var/db/imapd. The only way to solve this was to either create zroot/var/db/imapd (which I didn't want) or... add an entry to /etc/fstab and set up a 'delayed mount' (use the late option, see also fstab(5)).

The cause of the problem was that /var/db/imapd was getting mounted before /var/db was mounted. Both were different filesystems and one basically depended on the other, but got mounted too early.

So now I can't help but wonder if you're experiencing the same thing. In your example qpool possibly depending on your base system to be present while it might not be mounted at that given time during boot. It seems a bit odd, but... that's why this is simply a theory

Hi,

I sincerely appreciate your thoughtfulness on the matter. It's not as complicated as you think, it's just me not putting zfs_enable="YES" in rc.conf.

The boot drive is UFS, I opted to go with it since it's a VM in ESXi and I was just going to do snapshotting, etc. through vCenter. Therefore zfs_enable="YES" wasn't included in the default installer setup.

The zpool is a group of 4 drives on a passthrough LSI 2008. Which is quite lovely, but is the lone pool on the system.

But maybe the theory you have will help someone else who has a setup like that. Perhaps they will discover this thread at some point looking for help.

Avery Freeman · Apr 22, 2018

ralphbsz said:
OK, everything looks good: Your pool exists, all the devices in the pool are online, everything is mounted. Makes sense, since ZFS is enabled in /etc/rc.conf. Please explain what the problem is exactly? You said "files/dirs missing", what does that exactly mean? What does "/path/of/pool" refer to? What do you mean when you say "I've deleted all file-system created directories out of the pool"? ZFS file systems don't always have file-system created directories.

There's no problem anymore, I just didn't have zfs_enable="YES" in rc.conf . In Linux ZFS pools are generally mounted automatically without any intervention, so I was confused and didn't realize I had to configure anything in FreeBSD.

Thanks everyone for your help

ZFS Files/dirs in pool missing after reboot HELP!

Administrator