Solved Creating ZFS RaidZ1 problems - Rebuilding my FreeBSD server

[fitz]

Hello Forum,

As a 'newbie' to FreeBSD, I recently had some 'irrecoverable' problems with my (third/fourth) FreeBSD build.

I have 5 x 1.5TB HDDs for data storage, and, rather than simply installing FreeNAS, I wanted to build my own FreeBSD server ustilising ZFS for my RAID array upon these 5 HDDs.

This re-build, I am trying to install FreeBSD onto a 'Kingston Data Traveler" USB key (there are no more SATA interfaces - and - although there are 'other' options - I like the idea, especially, being a HP Gen7 Microserver, it has a 'purpose' internal USB slot.

On my initial build, I installed ZFS to all 5 HDDs using the installer, however, realised that following any OS error - a re-build would result in the loss of my DATA across all 5 HDDs, and, I wanted to keep them isolated as DATA drives.

The build onto the USB key went fine (if slow).

Manually I prepared my 5 x HDDs using:

gpart show
gpart delete -i 1 ada0 ... 2 ada0 ... 3 ada0 ... 1 ada1 etc.
gpart destroy ada0 ... ada1 etc.
gpart create -s gpt ada0 ... ada1 ... ada2 etc.

Then I created my ZFS conatiner and then dataset which I named Media:

zpool create tank raidz ada0 ada1 ada2 ada3 ada4
zfs create tank/Media
zfs list
zpool status

All appeared OK so far.

When trying to resolve NFS sharing, (another post from me!), I read /var/log/messages kindly pointed out to me by DutchDaemon.

The result is as follows:

Dec 14 10:49:46 backup5 kernel: da0: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SCSI-4 device
Dec 14 10:49:46 backup5 kernel: da0: Serial Number C86000BDBA09EE303A28006D
Dec 14 10:49:46 backup5 kernel: da0: 40.000MB/s transfers
Dec 14 10:49:46 backup5 kernel: da0: 7441MB (15240576 512 byte sectors: 255H 63S/T 948C)
Dec 14 10:49:46 backup5 kernel: da0: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
Dec 14 10:49:46 backup5 kernel: GEOM: ada0: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: ada0: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS801263: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS801263: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: ada1: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: ada1: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: SMP: AP CPU #1 Launched!
Dec 14 10:49:46 backup5 kernel: Timecounter "TSC-low" frequency 1098195211 Hz quality 800
Dec 14 10:49:46 backup5 kernel: GEOM: ada2: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: ada2: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: ada3: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: ada3: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: ada4: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: ada4: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS800648: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS800648: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS801253: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS801253: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS800654: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-S1XWJDWS800654: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-WD-WCAZAL476104: the primary GPT table is corrupt or invalid.
Dec 14 10:49:46 backup5 kernel: GEOM: diskid/DISK-WD-WCAZAL476104: using the secondary instead -- recovery strongly advised.
Dec 14 10:49:46 backup5 kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
Dec 14 10:49:46 backup5 kernel: ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
Dec 14 10:49:46 backup5 kernel: to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
Dec 14 10:49:46 backup5 kernel: ZFS filesystem version: 5
Dec 14 10:49:46 backup5 kernel: ZFS storage pool version: features support (5000)
Dec 14 10:49:46 backup5 kernel: uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.04, addr 2> on usbus0
Dec 14 10:49:47 backup5 mountd[489]: can't get address info for host 192.168.98.0/255.255.255.0(rw,no_root_squash)
Dec 14 10:49:47 backup5 mountd[489]: bad host 192.168.98.0/255.255.255.0(rw,no_root_squash), skipping
Dec 14 10:49:47 backup5 mountd[489]: bad exports list line /tank/Media 192.168.98.0/255.255.255.0(rw,no_root_squash)
Dec 14 10:54:39 backup5 su: fitz to root on /dev/pts/0

Please could someone advise as to where I am going wrong such that I have created the above HDDs/GPT errors (the NFS exports error would be a 'nice' addition, but, my primary concern is the HDD/GPT issue!)

Using the installer to create my ZFS RAIDz across all 5 HDDs is tempting, but, I would rather learn how to do this successfully from the command line if possible!

Thank you everyone.

 

[wblock@]

It is pointless to partition a disk and then overwrite that partition with ZFS. The error messages show that the primary GPT table at the beginning of the disk was overwritten by ZFS, and the secondary remained.

To fix this, make sure any data to be preserved is backed up elsewhere. Then recover the GPT partitioning scheme on each disk:
gpart recover ada0

At this point, it's just a normal GPT disk without ZFS. Decide whether you really want GPT partitioning on disks that will be completely used for ZFS. If you do, create ZFS partitions that fill the whole disk:
gpart destroy -F ada0
gpart create -s gpt ada0
gpart add -t freebsd-zfs ada0

When creating the pool, use the partitions (ada0p1), not the whole disk (ada0).

However, many people find no advantage to having GPT partitioning on a disk that is completely used for ZFS. In that case, make sure any old partitioning is destroyed before using the whole disk for ZFS:
gpart destroy -F ada0
zpool create tank raidz ada0 ...

 

[fitz]

You're a gent - thank you very much wblock@, now I can progress - it makes perfect sense now that you explain it - I simply 'didn't see the wood for the trees' and was blinded by the magical 'ZFS' term...

 

[fitz]

Worked a treat