Boot fails on FreeBSD BIOS/GPT boot after several months with 8TB disk installation replacing 1.5 TB disk

W

waltr

I have used FreeBSD since about Release 2. It was upgraded over time and occasionally refreshed with fresh installs and the hardware replaced from time to time.
Most recently I decided to do a fresh install of Release 13 from 12 and to convert a mixed UFS/ZFS installation to a pure ZFS/Bios installation with a new 8TB disk. Things looked good for a few months and blew up last week. I'm going to rebuild (from fresh) to R15 since the old system is perking along just fine. My thoughts are to return to the hybrid, buy a new small disk, or as a last resort, get a new MB which has its own complications.

A fresh installation of BIOS based ZFS on the 8TB disk and a boot setup worked for about 5 months when it lost the boot loader stage 3. The older mixed disk boots just fine, but will not boot nor mount the newer version due to ZFS upgrades.

The R12 version does not support the R13 gpt ZFS disk (shown below) but a USB live system will mount both disks. The primary use the new disk was to collect large files (2- 200 GB) from other data collection systems and archive them for eventual analysis.

When I bought the 8TB disk I was unaware of a problem that has been reported with BIOS based boot system whereby there Boot Stage 3 apparently gets lost beyond a certain disk address space that has been debated on these forums. So, I'd like to use the new disk without having to upgrade the motherboard for now (future budget though), if that is possible. The thoughts have been that the 512k limit on the MBR space truncates a high order address bit and the boot loader gets lost once a number of ZFS re-writes take place. So, I'm thinking the best way to use this disk is to partition it into 2 TB or smaller on the one hand or on the other hand keep the large ZFS at the end of the disk and set up some UFS space at the beginning and go from there. But I'm a little bit gun shy after the disk boot went away. The data is safe and I can get it off if won't boot using the USB R15 boot, but I'd rather not go through that again.

I'm looking for a best practices thoughts and suggestions before I go zapping the new 8TB disk and ending up with the same problem in a couple of months.

Early in many of those discussions discussion revolved around bad disks v. hardware v. software but ultimately the consensus was that BIOS and bigger than 2 TB MBR systems were not a happy marriage. So, I'm looking for advice on best practices for configuring a system either hybrid UFS/ZFS or pure ZFS.

Here's the original disk (working for years):

Code: 
dev01@usr/uz0 # gpart show -p ada0
=>        34  3907029101    ada0  GPT  (1.8T)
          34           6          - free -  (3.0K)
          40          94  ada0p1  freebsd-boot  (47K)
         134        1914          - free -  (957K)
        2048     4194304  ada0p2  freebsd-ufs  (2.0G)
     4196352    25165824  ada0p3  freebsd-swap  (12G)
    29362176    10485760  ada0p4  freebsd-ufs  (5.0G)
    39847936    10485760  ada0p5  freebsd-ufs  (5.0G)
    50333696  1048576000  ada0p6  freebsd-ufs  (500G)
  1098909696  2808119439  ada0p7  freebsd-zfs  (1.3T)
Code: 
/uz0 # zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
uz0   1.30T   732G   604G        -         -      -    54%  1.00x  ONLINE  -
root@waltr:/uz0 # zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
uz0              732G   562G  5.60G  /uz0
uz0/backups      365G   562G   365G  /uz0/backups
uz0/db           391M   562G   388M  /db
uz0/mbp0         295G   562G   294G  /uz0/mbp0
uz0/var         43.0G   562G  39.5K  /uz0/var
uz0/var/cache     25K   562G    25K  /uz0/var/cache
uz0/var/crash     25K   562G    25K  /uz0/var/crash
uz0/var/db      1.16G   562G  1.14G  /uz0/var/db
uz0/var/db/pkg  18.8M   562G  18.8M  /uz0/var/db/pkg
uz0/var/empty   41.8G   562G  41.8G  /uz0/var/empty
uz0/var/log       25K   562G    25K  /uz0/var/log
uz0/var/mail      25K   562G    25K  /uz0/var/mail
uz0/var/run       25K   562G    25K  /uz0/var/run
uz0/var/tmp     46.3M   562G  46.3M  /uz0/var/tmp
uz0/www         22.8G   562G  22.8G  /www

Code: 
# df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs    1.9G    284M    1.5G    16%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/gpt/tmpfs     4.8G     84M    4.4G     2%    /tmp
/dev/gpt/usrfs     484G    333G    113G    75%    /usr
/dev/gpt/varfs     4.8G    452M    4.0G    10%    /var
procfs             4.0K    4.0K      0B   100%    /proc
uz0/www            585G     23G    562G     4%    /www
uz0                568G    5.6G    562G     1%    /uz0
uz0/db             562G    388M    562G     0%    /db
uz0/var            562G     40K    562G     0%    /uz0/var
uz0/mbp0           856G    294G    562G    34%    /uz0/mbp0
uz0/var/cache      562G     25K    562G     0%    /uz0/var/cache
uz0/var/tmp        562G     46M    562G     0%    /uz0/var/tmp
uz0/var/crash      562G     25K    562G     0%    /uz0/var/crash
uz0/var/mail       562G     25K    562G     0%    /uz0/var/mail
uz0/var/db         563G    1.1G    562G     0%    /uz0/var/db
uz0/var/log        562G     25K    562G     0%    /uz0/var/log
uz0/var/empty      604G     42G    562G     7%    /uz0/var/empty
uz0/var/run        562G     25K    562G     0%    /uz0/var/run
uz0/var/db/pkg     562G     19M    562G     0%    /uz0/var/db/pkg
uz0/backups        927G    365G    562G    39%    /uz0/backups

And the new disk GPT info:
Code: 
/uz0 # gpart show -p ada1
=>         40  15628053088    ada1  GPT  (7.3T)
           40         1024  ada1p1  freebsd-boot  (512K)
         1064          984          - free -  (492K)
         2048      4194304  ada1p2  freebsd-swap  (2.0G)
      4196352  15623856128  ada1p3  freebsd-zfs  (7.3T)
  15628052480          648          - free -  (324K)
The later stage boot seems to get lost on a loader error 5 after a number of large dataset writes/revisions (~ 500 GB data file transfers). This never happened on the smaller disk/Release 12 setup. I'm thinking I need to repartition the disk to include a root mountpoint after the freebsd-boot partition (/dev/ada1p1). Or do I just need to break up the ada1p3 into < 2TB segment/partitions?

If anyone has any insight or can point me to a good reading source, that will help me figure out how to best live within my current HW limits, I would very much appreciate the help. I have no preconceptions on how to best do this, except I'm not a fan of repetitive work, and I am a fan of reliability which is why I use FreeBSD. Thanks, folks.
 
Error 5 is unknown file system. Do you read also something that speaks of unsupported feature?
I would try to update the BIOS bootloader.
 
  • Like
Reactions: mer
I am aware of the backwards compatibility issue with 13 v. 12. I can mount both versions using R15 on the USB LiveFS which is how I recovered the data on the new drive.

Emrion, I have not read of anything per se that speaks of any unsupported feature. From what I gather, the problem is with the MBR/BIOS problem, the direct addressing space provided by the BIOS is limited to about 2.2 TB. (source Dell discussion from 2013, references available if permitted on this forum). This means UEFI but this board does not support UEFI. The processor is a 64 bit AMD chip, and the FreeBSD doesn't care once it is booted up and running. It can read the whole disk, but in the above ada1 gpart config once ZFS rewrites a leaf outside the 2.2 TB addressing space, BIOS gets lost. The exact way it loses the path to the boot loader is unclear, but I suspect it is doing an internal two's complement add and the high order bit is shifted out of its address space registers. The BIOS is the problem and unfortunately, I'm not sure Gigabyte even has an update for this bios on this board (Award bios, unsure of exact version except its old and this system supports an embedded system which we are replacing but not there yet).

So, my thoughts are, that I should do the following:
gpart the new disk into a UFS section with the boot loader stable at address block 40 or so, limited to 512k as an MBR.
add a root directory ufs mount point for the basics after the MBR restricted to any reasonable size for foreseeable growth of the FreeBSD kernel and associated routines.
add a var directory ufs mount point for some growth and for routines that might need it that is protected from the ZFS philosophy of leaf style re-writes of large data transfers (files) to keep the boot system within the address space of the hardware limited bios.
Then add the user/application space in the sequential <= 2.2 TB partitions.

Alternatively, since ZFS and the FBSD kernel do support large disks (proven since it worked until the boot segment was moved in the new drive and then it didn't), create the UFS BIOS boot record mandated space I need at the beginning of the drive and make the rest a zpool/zfs space for the application data collection and processing.

This is where I need insight into how this part of the FBSD system works pretty much down at the hardware level. Once I understand this, I think I can work it out. My other approach will be to experiment with it to see what I can make work, then if anyone is interested, report it back.

(FYI, my background is decades of programming beginning with PDP-8s with 12 bit register/word lengths, real RISC with 8 total instructions and direct addressing of 2^12 - 1 words back in the 1960s), and of course I do the modern stuff too but never learned much about dealing with the boot loaders of pcs. PDP-8s, -10s and 11s used binary switches on the front panel to toggle a rudimentary binary bootstrap loader into memory which was replaced when memory moved from magnetic cores and wires to solid state in the late 1970s. We've come a long way.)
 
waltr said:
From what I gather, the problem is with the MBR/BIOS problem, the direct addressing space provided by the BIOS is limited to about 2.2 TB. (source Dell discussion from 2013, references available if permitted on this forum).
Click to expand...
I think it's not relevant. Your disk is and has been fully addressed by your bios.
 
waltr said:
So, my thoughts are, that I should do the following:
gpart the new disk into a UFS section with the boot loader stable at address block 40 or so, limited to 512k as an MBR.
Click to expand...
Bootloaders are starting from block 40 and sized 512k right now. MBR vs. GPT has nothing to do with this. And the master boot record itself will be at block 0, always. FreeBSD's PMBR will load the bootcode from freebsd-boot (well within reach of that potential 2TB pitfall).

One thing you may have forgotten, is zfs_load="YES" in /boot/loader.conf? And make sure you actually used gptzfsboot(8) for freebsd-boot, not gptboot(8).
 
Good to know. I now have some better hardware info on the system using dmidecode.

# dmidecode 3.4
Scanning /dev/mem for entry point.
SMBIOS 2.4 present.
60 structures occupying 3000 bytes.
Table at 0x000F0100.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Award Software International, Inc.
Version: F2
Release Date: 02/11/2011
Address: 0xE0000
Runtime Size: 128 kB
ROM Size: 2 MB
Characteristics:
ISA is supported
PCI is supported
PNP is supported
APM is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/360 kB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
AGP is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Targeted content distribution is supported

Handle 0x0001, DMI type 1, 27 bytes
Click to expand...
The BIOS is an Award Bios ca. 2011 Rev F2. Reviewing the Award/Gigabyte literature on this specific board, they list F2 as Feb 2011 release, and an F3 version as March 2011. The interesting description on the F3 release is this: 3TB+ HDD support. Support Turbo HDX function.

I'll follow up with Gigabyte and see if anyone there knows anything more about this and consider how to best proceed. dmidecode did not report the hardware version of the MB so I don't know if it's an older version or the last one produced. The most recent version was listed as a beta version. The immediate prior version F4 has a description: "Update Hybrid EFI." I learned something new this week.

Thanks folks for getting me pointed in a good direction with this old but still functional and (mostly) adequate hardware. Appreciate your help.
 
So, your system was working with a 8 TB disk for a few months and suddenly lose its ability to read above 2 or 3 TB of the same disk? Is it what you're saying or something escapes me?
 
Assuming your boot codes are up-to-date.

As SirDice pointed, legacy BIOS boot often have potential 2TB pitfalls that disallows next stage boot codes to read /boot/{zfs}loader if its actual in-disk location (even if partially) beyonds 2TB limit, or disallows loader to load kernel and/or kmods specified in /boot/loader.conf including /boot/loader.conf itself (even if partially) placed beyond 2TB limit.

To summarize, anything read BEFORE kernel starts (handed over from loader) are needed inside 2TB limit on legacy BIOS.

This doesn't matter for UEFI boots.
 
You must log in or register to reply here.
Back
Top