ZFS mounting issue and GPT structure

bsduni · Sep 1, 2013

Hi!

I m new to FreeBSD (though I've been using Ubuntu for a number of years).

Unfortunately I have to start exploring FreeBSD with a tough assignment. I have got a i386 based server with 2 x 250 GB HDDs and disks have (had) boot, swap, and ZFS (for /) file systems. Those disks were mirrored. All of a sudden, both the disks failed to get mounted, and as a result server stopped working. Whenever the system is started, it gets stuck at a point where it asks for manual mount points. No manual mount entries are seem to be helpful. The HDDs can be seen in the /dev list. And, gpart show provides sectors correctly. diskinfo -c /dev/adaXX provides the right values.

Then I tried the following:

I booted the server via Live USB, and then tried to mount the disks. It is not successful. (Shows invalid entry for the /dev/adaXX.)
(With the limited amount of info available) I tried to look for any corruptions in the GPT entry by comparing primary and secondary GPT headers. None of the GPT headers (primary and secondary) did not match with another - among the same disk or between disks.

Since the HDDs have some valuable data, I have been asked to revive it somehow. Therefore I would be grateful if you could give any useful information on the following:

Any tools or commands that can be used to test HDDs (I have tried combinations of dd, md, ...).
Any information on the GPT header architecture, and how an anomaly can be detected.
Any spcial commands or steps to be used to mount the HDD with ZFS.

Cheers!

Vije

J65nko · Sep 1, 2013

Because we have no idea how the ZFS system has been configured, you really should provide some info about the system

Please post the output of the following commands. If a command specifies a disk like ada0 repeat the command for the other disk(s). Use [code] and [/code] tags wrapped around the output.

# uname -a
# fdisk ada0
# bsdlabel ada0
# diskinfo -v ada0
# ls -l /dev/gpt
# ls -l /dev/ufs*
# gpart show
# gpart list
# gmirror status
# gmirror list
# cat /etc/fstab
# cat /boot/loader.conf
# cat /etc/rc.conf
# ls -l /etc/zfs*
# ls -l /boot/zfs*
# ls -l /boot/gpt*
# zpool get all
# zpool status
# zpool list
# zfs list

bsduni · Sep 2, 2013

Thank you very much for the prompt response.

Here are the outputs taken by booting on FreeBSD live USB:

Code:

uname -a
  FreeBSD  9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:15:25 UTC 2012     [email]root@obrian.cse.buffalo.edu[/email]:/usr/obj/usr/src/sys/GENERIC  i386

Code:

fdisk ada0
  ******* Working on device /dev/ada0 *******
  parameters extracted from in-core disklabel are:
  cylinders=484406 heads=16 sectors/track=63 (1008 blks/cyl)
   
  Figures below won't work with BIOS for partitions not in cyl 1
  parameters to be used for BIOS calculations are:
  cylinders=484406 heads=16 sectors/track=63 (1008 blks/cyl)
   
  Media sector size is 512
  Warning: BIOS sector numbering starts with sector 1
  Information from DOS bootblock is:
  The data for partition 1 is:
  sysid 238 (0xee),(EFI GPT)
      start 1, size 488281249 (238418 Meg), flag 80 (active)
        beg: cyl 0/ head 1/ sector 1;
        end: cyl 1023/ head 255/ sector 63
  The data for partition 2 is:
  <UNUSED>
  The data for partition 3 is:
  <UNUSED>
  The data for partition 4 is:
  <UNUSED>

Code:

fdisk ada1
  ******* Working on device /dev/ada1 *******
  parameters extracted from in-core disklabel are:
  cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl)
   
  Figures below won't work with BIOS for partitions not in cyl 1
  parameters to be used for BIOS calculations are:
  cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl)
   
  Media sector size is 512
  Warning: BIOS sector numbering starts with sector 1
  Information from DOS bootblock is:
  The data for partition 1 is:
  sysid 238 (0xee),(EFI GPT)
      start 1, size 488397167 (238475 Meg), flag 80 (active)
        beg: cyl 0/ head 1/ sector 1;
        end: cyl 1023/ head 255/ sector 63
  The data for partition 2 is:
  <UNUSED>
  The data for partition 3 is:
  <UNUSED>
  The data for partition 4 is:
  <UNUSED>

Code:

# bsdlabel ada0
  bsdlabel: /dev/ada0: no valid label found
# bsdlabel ada1[/B]
  bsdlabel: /dev/ada1: no valid label found

Code:

diskinfo -v ada0
  ada0
        512               # sectorsize
        250000000000      # mediasize in bytes (232G)
        488281250         # mediasize in sectors
        0                 # stripesize
        0                 # stripeoffset
        484406            # Cylinders according to firmware.
        16                # Heads according to firmware.
        63                # Sectors according to firmware.
        WD-WCANK3390327   # Disk ident.
   
   
# diskinfo -v ada1
  ada1
        512               # sectorsize
        250059350016      # mediasize in bytes (232G)
        488397168         # mediasize in sectors
        0                 # stripesize
        0                 # stripeoffset
        484521            # Cylinders according to firmware.
        16                # Heads according to firmware.
        63                # Sectors according to firmware.
        WD-WCAT15483764   # Disk ident.

Code:

ls -l /dev/gpt
  total 0
  crw-r-----  1 root  operator    0, 106 Sep  2 13:54 disk0
  crw-r-----  1 root  operator    0,  95 Sep  2 13:54 disk1
  crw-r-----  1 root  operator    0, 104 Sep  2 13:54 swap0
  crw-r-----  1 root  operator    0,  93 Sep  2 13:54 swap1

Code:

ls -l /dev/ufs
  total 0
  crw-r-----  1 root  operator    0, 116 Sep  2 13:54 FreeBSD_Install

Code:

gpart show
  =>       34  488281183  ada0  GPT  (232G)
           34        128     1  freebsd-boot  (64k)
          162    8388608     2  freebsd-swap  (4.0G)
      8388770  479892447     3  freebsd-zfs  (228G)
   
  =>       34  488397101  ada1  GPT  (232G)
           34        128     1  freebsd-boot  (64k)
          162    8388608     2  freebsd-swap  (4.0G)
      8388770  479892447     3  freebsd-zfs  (228G)
    488281217     115918        - free -  (56M)
   
  =>       0  15633408  da0  BSD  (7.5G)
           0   1095504    1  freebsd-ufs  (534M)
     1095504  14537904       - free -  (7G)

Code:

gpart list
  Geom name: ada0
  modified: false
  state: OK
  fwheads: 16
  fwsectors: 63
  last: 488281216
  first: 34
  entries: 128
  scheme: GPT
  Providers:
  1. Name: ada0p1
     Mediasize: 65536 (64k)
     Sectorsize: 512
     Stripesize: 0
     Stripeoffset: 17408
     Mode: r0w0e0
     rawuuid: 975812e7-7950-11df-a5bc-001372fd7595
     rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
     label: (null)
     length: 65536
     offset: 17408
     type: freebsd-boot
     index: 1
     end: 161
     start: 34
  2. Name: ada0p2
     Mediasize: 4294967296 (4.0G)
     Sectorsize: 512
     Stripesize: 0
     Stripeoffset: 82944
     Mode: r0w0e0
     rawuuid: c3c9311c-7950-11df-a5bc-001372fd7595
     rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
     label: swap1
     length: 4294967296
     offset: 82944
     type: freebsd-swap
     index: 2
     end: 8388769
     start: 162
  3. Name: ada0p3
     Mediasize: 245704932864 (228G)
     Sectorsize: 512
     Stripesize: 0
     Stripeoffset: 82944
     Mode: r0w0e0
     rawuuid: e32a063c-7950-11df-a5bc-001372fd7595
     rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
     label: disk1
     length: 245704932864
     offset: 4295050240
     type: freebsd-zfs
     index: 3
     end: 488281216
     start: 8388770
  Consumers:
  1. Name: ada0
     Mediasize: 250000000000 (232G)
     Sectorsize: 512
     Mode: r0w0e0
   
  Geom name: ada1
  modified: false
  state: OK
  fwheads: 16
  fwsectors: 63
  last: 488397134
  first: 34
  entries: 128
  scheme: GPT
  Providers:
  1. Name: ada1p1
     Mediasize: 65536 (64k)
     Sectorsize: 512
     Stripesize: 0
     Stripeoffset: 17408
     Mode: r0w0e0
     rawuuid: 95ac6611-7950-11df-a5bc-001372fd7595
     rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
     label: (null)
     length: 65536
     offset: 17408
     type: freebsd-boot
     index: 1
     end: 161
     start: 34
  2. Name: ada1p2
     Mediasize: 4294967296 (4.0G)
     Sectorsize: 512
     Stripesize: 0
     Stripeoffset: 82944
     Mode: r0w0e0
     rawuuid: bcad63e6-7950-11df-a5bc-001372fd7595
     rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
     label: swap0
     length: 4294967296
     offset: 82944
     type: freebsd-swap
     index: 2
     end: 8388769
     start: 162
  3. Name: ada1p3
     Mediasize: 245704932864 (228G)
     Sectorsize: 512
     Stripesize: 0
     Stripeoffset: 82944
     Mode: r0w0e0
     rawuuid: 0d944bff-7951-11df-a5bc-001372fd7595
     rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
     label: disk0
     length: 245704932864
     offset: 4295050240
     type: freebsd-zfs
     index: 3
     end: 488281216
     start: 8388770
  Consumers:
  1. Name: ada1
     Mediasize: 250059350016 (232G)
     Sectorsize: 512
     Mode: r0w0e0
   
  Geom name: da0
  modified: false
  state: OK
  fwheads: 255
  fwsectors: 63
  last: 15633407
  first: 0
  entries: 8
  scheme: BSD
  Providers:
  1. Name: da0a
     Mediasize: 560898048 (534M)
     Sectorsize: 512
     Mode: r1w0e1
     rawtype: 7
     length: 560898048
     offset: 0
     type: freebsd-ufs
     index: 1
     end: 1095503
     start: 0
  Consumers:
  1. Name: da0
     Mediasize: 8004304896 (7.5G)
     Sectorsize: 512
     Mode: r1w0e1

Code:

gmirror status
  gmirror: Command 'status' not available.

Code:

gmirror list[/B]
  gmirror: Command 'list' not available.

Code:

cat /etc/fstab
  /dev/ufs/FreeBSD_Install / ufs ro,noatime 1 1

Code:

cat /boot/loader.conf
  cat: /boot/loader.conf: No such file or directory

Code:

cat /etc/rc.conf
  sendmail_enable="NONE"
  hostid_enable="NO"

Code:

ls -l /etc/zfs
  total 0

Code:

ls -l /boot/zfs /boot/zfsboot /boot/zfsloader
  -r--r--r--  1 root  wheel   66048 Jan  3  2012 /boot/zfsboot
  -r-xr-xr-x  1 root  wheel  299008 Jan  3  2012 /boot/zfsloader
   
  /boot/zfs:
  total 0

Code:

ls -l /boot/gptboot /boot/gptzfsboot
  -r--r--r--  1 root  wheel  17011 Jan  3  2012 /boot/gptboot
  -r--r--r--  1 root  wheel  40563 Jan  3  2012 /boot/gptzfsboot

Code:

zpool status
  no pools available

Code:

zpool list
  no pools available

Code:

zfs list
  no datasets available

Note that I was told the system was working without any issue. I was told that, once they connected an external harddisk to take some data back up and every thing stopped working. Since then, if the server is booted by itself (without USB live), it goes passing "Welcome to FreeBSD!", and all the boot options. Then the server stops at the command prompt "mountroot>". No options (as suggested) make the server move forward.

Please let me know any anomaly could be detected from these outputs.

Thank you in advance!

jem · Sep 2, 2013

Also show us the output of zpool import without any arguments. This should load the ZFS modules and probe the disks, then list any pools found.

bsduni · Sep 4, 2013

Thank you @jem!

zpool import helped me access the hard disks while on FreeBSD Live (USB). After importing, zpool status showed an error and fixed it with zpool clear. Yet, still the server is neither booting from the harddisks (without Live USB), nor the harddisks get mounted automatically while booting with Live USB. The following are the results that have changed after zpool import. Also note that some commands stopped working as given below:

Code:

[cmd]df[/cmd]

  Filesystem                1K-blocks    Used     Avail Capacity  Mounted on
  /dev/ufs/FreeBSD_Install     545975  509590     -7293   101%    /
  devfs                             1       1         0   100%    /dev
  /dev/md0                      31772     272     28960     1%    /var
  /dev/md1                      19740      28     18136     0%    /tmp
  zroot/tmp                 105628096      73 105628023     0%    /tmp
  zroot/usr                 106766167 1138144 105628023     1%    /usr
  zroot/usr/ports           105961617  333594 105628023     0%    /usr/ports
  zroot/usr/ports/distfiles 106038375  410352 105628023     0%    /usr/ports/distfiles
  zroot/usr/ports/packages  105744540  116517 105628023     0%    /usr/ports/packages
  zroot/usr/src             105949029  321006 105628023     0%    /usr/src
  zroot/var                 107168775 1540752 105628023     1%    /var
  zroot/var/empty           105628042      19 105628023     0%    /var/empty
  zroot/var/tmp             105628248     225 105628023     0%    /var/tmp

Before import it was like:

Code:

[cmd]df[/cmd]
  Filesystem               1K-blocks   Used  Avail Capacity  Mounted on[/color]
  /dev/ufs/FreeBSD_Install    545975 509590  -7293   101%    /
  devfs                            1      1      0   100%    /dev
  /dev/md0                     31772    268  28964     1%    /var
  /dev/md1                     19740     28  18136     0%    /tmp
  /dev/da1s1                 1020976 874784 146192    86%    /tmp/usb1

zpool get all

Not working, and suggests command options which indicate that the above is correct.

diskinfo -v ada0

Code:

  /usr/sbin/diskinfo: Exec format error. Binary file not executable.

Note: many commands become non-executable after importing the zpool. I guess that the newly mounted directory /usr (from zpool) is causing the issue.

diskinfo -v ada1

Code:

  /usr/sbin/diskinfo: Exec format error. Binary file not executable.

Note: The same issue as above.

ls -l /dev/gpt

Code:

  total 0
  crw-r-----  1 root  operator    0,  93 Sep  4 10:01 swap0
  crw-r-----  1 root  operator    0, 104 Sep  4 10:01 swap1

Before zpool import it was like

ls -l /dev/gpt

Code:

  total 0
  crw-r-----  1 root  operator    0, 106 Sep  2 13:54 disk0
  crw-r-----  1 root  operator    0,  95 Sep  2 13:54 disk1
  crw-r-----  1 root  operator    0, 104 Sep  2 13:54 swap0
  crw-r-----  1 root  operator    0,  93 Sep  2 13:54 swap1

gmirror status

Code:

  gmirror: Command 'status' not available.

Note: Again, no idea why 'status' is not working, while it is shown as an acceptable option.

gmirror list

Code:

  gmirror: Command 'list' not available.

Note: The same issue as above.

ls -l /etc/zfs

Code:

  total 0

ls -l /boot/zfs /boot/zfsboot /boot/zfsloader

Code:

  -r--r--r--  1 root  wheel   66048 Jan  3  2012 /boot/zfsboot
  -r-xr-xr-x  1 root  wheel  299008 Jan  3  2012 /boot/zfsloader
   
  /boot/zfs:
  total 0

ls -l /boot/gptboot /boot/gptzfsboot

Code:

  -r--r--r--  1 root  wheel  17011 Jan  3  2012 /boot/gptboot
  -r--r--r--  1 root  wheel  40563 Jan  3  2012 /boot/gptzfsboot

Before zpool import,

ls -l /etc/zfs

Code:

  total 0

zpool status

Code:

    pool: zroot
   state: ONLINE
  status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
  action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
     see: [url]http://www.sun.com/msg/ZFS-8000-9P[/url]
   scan: none requested
  config:
   
        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     3
   
  errors: No known data errors

Note: the error was fixed with zpool clear. And, no more errors seen.

zpool list

Code:

  NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
  zroot   228G   124G   104G    54%  1.00x  ONLINE  -

zfs list

Code:

  NAME                        USED  AVAIL  REFER  MOUNTPOINT
  zroot                       124G   101G  5.52G  legacy
  zroot/home                 44.9G   101G  76.5K  /home
  zroot/home/ahmad           2.11G   911M  2.11G  /home/ahmad
  zroot/home/aorgerie         455M   569M   455M  /home/aorgerie
  zroot/home/athipatl        2.60M   497M  2.60M  /home/athipatl
  zroot/home/bmor            38.5K   500M  38.5K  /home/bmor
  zroot/home/bsk             6.46G  1.54G  6.46G  /home/bsk
  zroot/home/caleckie         311M   713M   311M  /home/caleckie
  zroot/home/darryl          2.50G  5.50G  2.50G  /home/darryl
  zroot/home/davis           4.25G  3.75G  4.25G  /home/davis
  zroot/home/dbatt           3.57M   496M  3.57M  /home/dbatt
  zroot/home/fajalali         714M   310M   714M  /home/fajalali
  zroot/home/git             95.9M   101G  95.9M  /home/git
  zroot/home/hanly           3.10G   921M  3.10G  /home/hanly
  zroot/home/jse              418K   500M   418K  /home/jse
  zroot/home/jthomps         38.5K   500M  38.5K  /home/jthomps
  zroot/home/lburchat        37.5K   500M  37.5K  /home/lburchat
  zroot/home/lha              505M   519M   505M  /home/lha
  zroot/home/lsptune          471M   553M   471M  /home/lsptune
  zroot/home/mzfeng          38.5K   500M  38.5K  /home/mzfeng
  zroot/home/nhohn            831M   193M   831M  /home/nhohn
  zroot/home/pabry           81.8M   942M  81.8M  /home/pabry
  zroot/home/pfarrell        37.5K  1024M  37.5K  /home/pfarrell
  zroot/home/pin             1.95M  1022M  1.95M  /home/pin
  zroot/home/rbowden         39.5K  1024M  39.5K  /home/rbowden
  zroot/home/rmuharar        4.45M   496M  4.45M  /home/rmuharar
  zroot/home/rst             14.1M  1010M  14.1M  /home/rst
  zroot/home/sdey            61.5K   500M  61.5K  /home/sdey
  zroot/home/seminars        19.0G   101G  19.0G  /home/seminars
  zroot/home/sgeadmin        70.5K   101G  70.5K  /home/sgeadmin
  zroot/home/slow            54.4M   970M  54.4M  /home/slow
  zroot/home/taoye           1.87G   128M  1.87G  /home/taoye
  zroot/home/tjbr             451M   573M   451M  /home/tjbr
  zroot/home/tscbuilder        30K   101G    30K  /home/tscbuilder
  zroot/home/www             1.69G   101G  1.69G  /home/www
  zroot/jails                6.70G   101G  6.70G  /jails
  zroot/media                  31K   101G    31K  legacy
  zroot/scratch              49.2G  30.8G  49.2G  /scratch
  zroot/tftpboot             13.7G   101G  13.7G  /tftpboot
  zroot/tmp                    78K   101G    78K  /tmp
  zroot/usr                  2.21G   101G  1.09G  /usr
  zroot/usr/ports             840M   101G   326M  /usr/ports
  zroot/usr/ports/distfiles   401M   101G   401M  /usr/ports/distfiles
  zroot/usr/ports/packages    114M   101G   114M  /usr/ports/packages
  zroot/usr/src               313M   101G   313M  /usr/src
  zroot/var                  1.47G   101G  1.47G  /var
  zroot/var/empty              19K   101G    19K  /var/empty
  zroot/var/tmp               225K   101G   225K  /var/tmp

I would be grateful for any tips/clues of how to fix the server and bring it up again.

As I said earlier, when the server is booted without live USB, it shows Boot options. And then, stops with

Code:

mountroot>

No boot options let the server pass that stage.

Note that loader variables remain blank as shown below.

Loader variables:

Code:

vfs.root.mountfrom=
vfs.root.mountfrom.options=

Is that blank loader variables causing any issue? I tried various options by setting them with different possible values.

bsduni · Sep 5, 2013

In addition, when I forcibly mount the zpool using # zpool import -f zroot some directories fail to get mounted, and the error messages are as shown below:

Code:

cannot mount '/home': failed to create mountpoint
cannot mount '/home/adams': failed to create mountpoint
cannot mount '/home/john': failed to create mountpoint
cannot mount '/home/kat': failed to create mountpoint
...
cannot mount '/scratch': failed to create mountpoint
cannot mount '/tftpboot': failed to create mountpoint

Any reasons that could be guessed for this?

jem · Sep 5, 2013

When you import your zpool under the live USB environment, you need to specify the altroot option to mount it under /mnt, otherwise it gets mounted over the top of the USB root filesystem and you lose access to it.

The USB root filesystem is probably mounted read-only, hence why new mointpoint directories can not be created for your /home, /scratch and /tftpboot datasets. altroot should fix this too, as you'll be mounting everything under /mnt.

SirDice · Sep 5, 2013

At the "mountroot>" prompt, try booting the system with zfs:zroot. Looking at the zfs list you provided that seems to be the filesystem it used to boot from.

bsduni · Sep 6, 2013

Thank you again for the useful information and guidance.

I tried the altroot option with zpool import -f -R /tmp/tmpzroot zroot, and got the following error:

Code:

cannot share 'zroot/home': share(1M) failed
cannot share 'zroot/home/adams': share(1M) failed
cannot share 'zroot/home/john': share(1M) failed
cannot share 'zroot/home/kat': share(1M) failed
...
cannot share 'zroot/scratch': share(1M) failed
cannot share 'zroot/tftpboot': share(1M) failed

I also tried zfs:zroot at the mountroot> prompt and I am still getting the same error message as follows:

Code:

Trying to mount root from zfs:zroot
ROOT MOUNT ERROR:
If you have invalid mount options, reboot, and first try the following from the loader prompt:

    set vfs.vfs.root.mountfrom.options=rw

and then remove invalid mount options from /ect/fstab.

Loader variables:
vfs.root.mountfrom=zfs:zroot
vfs.root.mountfrom.options=rw

Manual root filesystem specification:
   <fstype>:<device> ....

mountroot>

In between, the following information on the system might be useful:

FreeBSD Version 8.3 (on the HDDs) and 9.0 on the USB live
ZFS storage pool version 28
ZFS filesystem version 5

I would be grateful if any help can be provided to make the server up and running again.

jem · Sep 6, 2013

Boot into your USB environment again and import the pool (with an altroot set). Ignore the "cannot share" errors.

Then do zpool get bootfs zroot and check that this is set to the dataset containing your root filesystem.

Then check for the existence of /{altrootdir}/boot/zfs/zpool.cache.

bsduni · Sep 8, 2013

Thank you again.

Being a person who only took over the server after the failure and without the complete picture/information on the original configuration I'm in a bit of a dilemma. All I know is the server was working fine on the FreeBSD 8.3 platform and stopped working after connecting external harddisks (via USB) in an attempt to make a backup.

While I appreciate your great help, still I could not bring it up despite I follow all the instructions given in this thread. In addition, I am also going through a number of threads on the FreeBSD Forum that I think might be related to this failure with no luck.

Here are the command outcomes that might give further information on the system:

After zpool import -f -R /tmp/tmpzroot zroot where tmpzroor is the altrootdir

Code:

[cmd]zpool get bootfs zroot[/cmd]
NAME   PROPERTY  VALUE   SOURCE
zroot  bootfs    zroot   local

I could not find the boot directory under tmpzroot. Instead boot is found under the subdirectory /tmp/tmpzroot/jails/mroot. And, zpool.cache under any of those subdirectories (see the find command output at last). The following are some of the outputs:

Code:

[cmd]ls /tmp/tmpzroot[/cmd]
home   jails   scratch   tftpboot  tmp   usr   var

[cmd]ls /tmp/tmpzroot/jails[/cmd]
jspace   mroot   mroot-8.1   skel   www   

[cmd]ls /tmp/tmpzroot/jails/mroot[/cmd]
.cshrc   .profile   COPYRIGHT   bin   boot   dev   etc   home   lib   libexec   media
mnt   proc   rescue   root   s   sbin   sys   tmp   usr   var   

[cmd]ls /tmp/tmpzroot/jails/mroot/boot[/cmd] 
beastie.4th   boot   boot0   boot0sio   boot1   boot2   cdboot   defaults   device.hints   firmware
frames.4th   gptboot   gptzfsboot   kernel   loader   loader.4th   loader.help   loader.rc   mbr  
modules   pmbr   pxeboot   screen.4th   support.4th   zfs   zfsboot   

[cmd]ls /tmp/tmpzroot/jails/mroot/boot/zfs[/cmd]
      (No files)

[cmd]cd /tmp/tmpzroot[/cmd]
[cmd]find . â€“name zpool*[/cmd] ## An attempt to find the zpool.cache location
find: No match.

Any other hints?

J65nko · Sep 8, 2013

I get the impression that the server ZFS install is done using the instructions of https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot. Existence of a /etc/fstab similar to the one in that wiki entry, with an entry for swap, could confirm that

EDIT: A ZFS mirror install using the same method is Installing FreeBSD Root on ZFS (Mirror) using GPT

If I were in your position, instead of trying to get the server up again, I would first concentrate on backing up the all the data and configuration. You could do a FreeBSD 8.3 install using those instructions from the wiki. That would give your more confidence and knowledge what to look for. You will then also have a second system to do create a complete backup with zfs send and zfs receive.

Re: mountpoints

From your posting:

Code:

  NAME                        USED  AVAIL  REFER  MOUNTPOINT
  zroot                       124G   101G  5.52G  [highlight]legacy[/highlight]
  [snip]
  zroot/media                  31K   101G    31K  [highlight]legacy[/highlight]

The section "Mount Points" of the zfs(8) manual page explains how ZFS automatically manages file systems and handles the mounting automatically without the need of editing /etc/fstab. Then it says the following:

Code:

If needed, ZFS file systems can also be managed with traditional tools
     (mount(8), umount(8), fstab(5)).  If a file system's mount point is set
     to [color=blue]legacy[/color], ZFS makes no attempt to manage the file system, and the admin-
     istrator is responsible for mounting and unmounting the file system.

The administrator can handle the mounting by setting the bootfs property or by creating suitable entries in the /etc/fstab file, or with # mount -t zfs zroot /mnt.

RE: 8.3 versus 9.0

You also should be careful with allowing the 9.0 live USB stick to modify the ZFS pool and/or datasets, when the server is FreeBSD 8.3. When I was messing with the 9.2 BETA on a second disk of my 9.1 box, 9.1 complained about some ZFS incompatibility.

When I started with ZFS of FreeBSD 9.1 about six weeks ago, I got tired of paging through the ZFS manual pages, so I printed them. You also may find it helpful to have a printed copy of the zfs(8), zpool(8) and zdb(8) man pages available.

J65nko · Sep 9, 2013

RE: zpool.cache

If the cache file went missing you can recreate the cache file:

Code:

[cmd=#]zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache -d /dev/gpt zroot[/cmd] 
cannot import 'zroot': pool may be in use from other system
use '-f' to import anyway
[cmd=#]zpool import -f -o altroot=/mnt -o cachefile=/tmp/zpool.cache -d /dev/gpt zroot[/cmd]

Your pool will then be mounted on /mnt and you will have a /tmp/zpool.cache that you copy to /mnt/boot/zfs.

J65nko · Sep 10, 2013

You can find out information about your ZFS pool with # zpool history

The following is from a Installing FreeBSD Root on ZFS (Mirror) using GPT for FreeBSD 8.3.

Code:

History for 'wiki':
2013-09-10.00:11:49 zpool create -f -m none -o cachefile=/tmp/zpool.cache wiki mirror /dev/gpt/disk6.nop /dev/gpt/disk7.nop
2013-09-10.00:12:05 zpool export wiki
2013-09-10.00:12:47 zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache -d /dev/gpt wiki
2013-09-10.00:14:10 zpool set bootfs=wiki wiki
2013-09-10.00:14:19 zfs set checksum=fletcher4 wiki
2013-09-10.00:14:19 zfs set atime=off wiki
2013-09-10.00:14:24 zfs set mountpoint=/ wiki
2013-09-10.00:14:57 zfs create -o compression=on -o exec=on -o setuid=off wiki/tmp
2013-09-10.00:14:58 zfs create wiki/usr
2013-09-10.00:14:58 zfs create wiki/usr/home
2013-09-10.00:14:59 zfs create -o compression=lzjb -o setuid=off wiki/usr/ports
2013-09-10.00:14:59 zfs create -o compression=off -o exec=off -o setuid=off wiki/usr/ports/distfiles
2013-09-10.00:15:00 zfs create -o compression=off -o exec=off -o setuid=off wiki/usr/ports/packages
2013-09-10.00:15:00 zfs create -o compression=lzjb -o exec=off -o setuid=off wiki/usr/src
2013-09-10.00:15:01 zfs create wiki/var
2013-09-10.00:15:01 zfs create -o compression=lzjb -o exec=off -o setuid=off wiki/var/crash
2013-09-10.00:15:02 zfs create -o exec=off -o setuid=off wiki/var/db
2013-09-10.00:15:02 zfs create -o compression=lzjb -o exec=on -o setuid=off wiki/var/db/pkg
2013-09-10.00:15:03 zfs create -o exec=off -o setuid=off wiki/var/empty
2013-09-10.00:15:04 zfs create -o compression=lzjb -o exec=off -o setuid=off wiki/var/log
2013-09-10.00:15:04 zfs create -o compression=gzip -o exec=off -o setuid=off wiki/var/mail
2013-09-10.00:15:05 zfs create -o exec=off -o setuid=off wiki/var/run
2013-09-10.00:15:10 zfs create -o compression=lzjb -o exec=on -o setuid=off wiki/var/tmp
2013-09-10.00:20:39 zfs set mountpoint=legacy wiki
2013-09-10.00:20:39 zfs set mountpoint=/tmp wiki/tmp
2013-09-10.00:20:40 zfs set mountpoint=/usr wiki/usr
2013-09-10.00:20:45 zfs set mountpoint=/var wiki/var

bsduni · Sep 10, 2013

Thank you so much @J65nko! Your advice is really useful.

I got the data backup. And, then today I used two harddisks to install a fresh and similar system (skipped jail). The new installation gave me a good experience to understand some basics of ZFS/mirror.
I could see three files were missing in their expected directories on the old disks, namely:
* loader.conf,
* zfsloader, and
* zpool.cache.

I will try your suggestions and will come back with my findings.

jem · Sep 10, 2013

bsduni said:
After zpool import -f -R /tmp/tmpzroot zroot where tmpzroor is the altrootdir

Code:

[cmd]zpool get bootfs zroot[/cmd] NAME PROPERTY VALUE SOURCE zroot bootfs zroot local

I could not find the boot directory under tmpzroot. Instead boot is found under the subdirectory /tmp/tmpzroot/jails/mroot.

If you don't have a /boot directory in the root dataset of the pool, then it is unlikely that the pool contained a bootable FreeBSD installation to begin with.

J65nko · Sep 11, 2013

jem said:
If you don't have a /boot directory in the root dataset of the pool, then it is unlikely that the pool contained a bootable FreeBSD installation to begin with.

Not if you use ZFS Boot Environments

From one of my installation logs:

Code:

[cmd=#]zfs list[/cmd]
NAME                 USED  AVAIL  REFER  MOUNTPOINT
super               4.13G  3.56T   144K  none
super/ROOT           288K  3.56T   144K  none
super/ROOT/default   144K  3.56T   144K  /mnt
super/swap          4.13G  3.57T    72K  -

root@:/tmp [cmd=#] make install[/cmd]
Fri Aug 16 20:45:42 UTC 2013 : Unpacking /usr/freebsd-dist/base.txz .......
tar --unlink -xpJf /usr/freebsd-dist/base.txz -C /mnt

Here FreeBSD is installed in super/ROOT/default and not in the root dataset of the pool super.

I wonder whether the previous administrator used a script like @vermaden's sysutils/beadm or manageBE. These scripts manipulate loader.conf and zpool.cache.

@bsduni, can you find(1) a script similar to beadm or manageBE in the system somewhere? Did they use a script to backup the system to the USB drive? There could be an error in that script.

Consider the output you posted previously:

Code:

 ls /tmp/tmpzroot/jails/mroot
.cshrc   .profile   COPYRIGHT   bin   boot   dev   etc   home   lib   libexec   media
mnt   proc   rescue   root   s   sbin   sys   tmp   usr   var   

 ls /tmp/tmpzroot/jails/mroot/boot 
beastie.4th   boot   boot0   boot0sio   boot1   boot2   cdboot   defaults   device.hints   firmware
frames.4th   gptboot   gptzfsboot   kernel   loader   loader.4th   loader.help   loader.rc   mbr  
modules   pmbr   pxeboot   screen.4th   support.4th   zfs   zfsboot

So it looks like a complete FreeBSD install in mroot. On the other hand it is quite normal for a jail directory. Are there any other directories like this?

Although there is no directory GENERIC and no zfsloader in the boot directory, I wonder whether the following /boot/loader.conf would boot the system:

Code:

zfs_load=YES
vfs.root.mountfrom="zfs:zroot/jails/mroot

Of course you also would need to copy a regenerated ZFS zpool.cache into the zroot/jails/mroot/boot/zfs directory.
Does zroot/jails/mroot/etc/rc.conf contain

Code:

zfs_enable=YES

? That would be another requirement.

But maybe it would be best to first duplicate the original server completely and try these modifications on the duplicate server first.

bsduni · Sep 11, 2013

Thanks again!

I could NOT find either beadm or manageBE among the files (on the system in trouble). But, I was told that some attempts on system backup were made. Yet no exact information is available.

Here are some additional findings:

When I issued zpool history I could see too many executions of zfs destroy after a particular date (14-04-2013) as shown below (only a part is given here, but there are hundreds of such lines). I m yet to confirm whether this date is related with the day the server failed.

Code:

[cmd]# zpool history[/cmd]
History for 'zroot':
2010-06-16.14:13:43 zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk1
2013-04-14.16:10:35 zfs destroy zroot@zfsbkup.20130414
2013-04-14.16:10:46 zfs destroy zroot/home@zfsbkup.20130414
2013-04-14.16:10:58 zfs destroy zroot/home/aorgerie@zfsbkup.20130414
2013-04-14.16:11:53 zfs destroy zroot/home/adams@zfsbkup.20130414
2013-04-14.16:14:06 zfs destroy zroot/home/john@zfsbkup.20130414
2013-04-14.16:14:47 zfs destroy zroot/home/kat@zfsbkup.20130414
...

The server also has two possible locations with installation files: zroot/jails/mroot/ and zroot/jails/mroot-8.1/. Both directories (and the subdirectories) do not have loader.conf, zfsloader, and zpool.cache.
I created loader.conf and zpool.cache as suggested and put them in the right places. Still the server fails to boot by itself.
I also found that no rc.conf as expected with zroot/jails/mroot/etc/rc.conf or zroot/jails/mroot-8.1/etc/rc.conf directories.

Therefore, now I am focusing on a duplicate system using two new harddisks. I have got two new 500 GB disks instead of 250 GB ones (as they were in the server). I only used 228 GB for ZFS as in the old disks. Nevertheless, after installing FreeBSD 8.3 only I noticed that the old system was on version 8.1 (but, I was told 8.3).

Upon installation and the new system working, I tried to boot with a combination of one new disk (with working installation) and one old (failed system). The system got hanged. I am planning to do some more experiments as the data backup up has been taken.

I will comeback with my findings.

Thanks again for all of your support.

J65nko · Sep 12, 2013

Some years ago when FreeBSD could not boot from ZFS yet, you had to use an UFS partition to boot the system. Some people simply used a Compact Flash or USB device for booting. Are there any other disk devices in the server that could have been used to boot?

In the listing you posted there is a zroot/tftpboot dataset.

Code:

  NAME                        USED  AVAIL  REFER  MOUNTPOINT
  zroot                       124G   101G  5.52G  legacy
  [snip]
  zroot/tftpboot             13.7G   101G  13.7G  /tftpboot

Typically this directory is used for serving boot files to PXE or netboot clients. But it made me think that although unlikely, there is a theoretical possibility that the server was booted via the network.

Does # find . -name 'messages*' after importing the pool produce anything interesting? If that does not, I am afraid that instead of backing up the server to the USB drive, the USB drive contents was transferred to the server, although it would be strange that in that case there is no trace of any /etc/rc.conf or /etc/fstab.

Before you spend too much time in setting up a duplicate server with FreeBSD 8.3, please check the Supported FreeBSD releases page. Unless there is a compelling reason to stay with the FreeBSD 8 series, I think you would be better off by using 9.1 or the upcoming 9.2 release.

bsduni · Sep 13, 2013

I'm grateful for your continuous support @J65nko!

I tried to bring the server up as it was. Since only a very limited amount information was available about the previous configuration/history, it became a real challenge. After trying so many possible fixes (thanks to you all and FreeBSD forum discussion threads), still the server is not booting up for over four weeks. In a way I am happy that I could learn a lot about ZFS, mirrors, and FreeBSD.

While I'm a bit exhausted, my boss is frustrated

. So, I had to move on, decided to go fresh with FreeBSD 9.1, and have done the installation with the new disks (exactly as you suggested). I also have documented configuration and taken backup of important files for any future references. Now, planning to migrate files to new system as the requirement arises.

At the same time, I will try my luck during the week-ends with the old system (disks) and will update here if I find anything interesting / a break-through.

A million thanks!

ZFS mounting issue and GPT structure

Administrator