Boot fails on FreeBSD BIOS/GPT boot after several months with 8TB disk installation replacing 1.5 TB disk

W

waltr

I have used FreeBSD since about Release 2. It was upgraded over time and occasionally refreshed with fresh installs and the hardware replaced from time to time.
Most recently I decided to do a fresh install of Release 13 from 12 and to convert a mixed UFS/ZFS installation to a pure ZFS/Bios installation with a new 8TB disk. Things looked good for a few months and blew up last week. I'm going to rebuild (from fresh) to R15 since the old system is perking along just fine. My thoughts are to return to the hybrid, buy a new small disk, or as a last resort, get a new MB which has its own complications.

A fresh installation of BIOS based ZFS on the 8TB disk and a boot setup worked for about 5 months when it lost the boot loader stage 3. The older mixed disk boots just fine, but will not boot nor mount the newer version due to ZFS upgrades.

The R12 version does not support the R13 gpt ZFS disk (shown below) but a USB live system will mount both disks. The primary use the new disk was to collect large files (2- 200 GB) from other data collection systems and archive them for eventual analysis.

When I bought the 8TB disk I was unaware of a problem that has been reported with BIOS based boot system whereby there Boot Stage 3 apparently gets lost beyond a certain disk address space that has been debated on these forums. So, I'd like to use the new disk without having to upgrade the motherboard for now (future budget though), if that is possible. The thoughts have been that the 512k limit on the MBR space truncates a high order address bit and the boot loader gets lost once a number of ZFS re-writes take place. So, I'm thinking the best way to use this disk is to partition it into 2 TB or smaller on the one hand or on the other hand keep the large ZFS at the end of the disk and set up some UFS space at the beginning and go from there. But I'm a little bit gun shy after the disk boot went away. The data is safe and I can get it off if won't boot using the USB R15 boot, but I'd rather not go through that again.

I'm looking for a best practices thoughts and suggestions before I go zapping the new 8TB disk and ending up with the same problem in a couple of months.

Early in many of those discussions discussion revolved around bad disks v. hardware v. software but ultimately the consensus was that BIOS and bigger than 2 TB MBR systems were not a happy marriage. So, I'm looking for advice on best practices for configuring a system either hybrid UFS/ZFS or pure ZFS.

Here's the original disk (working for years):

Code: 
dev01@usr/uz0 # gpart show -p ada0
=>        34  3907029101    ada0  GPT  (1.8T)
          34           6          - free -  (3.0K)
          40          94  ada0p1  freebsd-boot  (47K)
         134        1914          - free -  (957K)
        2048     4194304  ada0p2  freebsd-ufs  (2.0G)
     4196352    25165824  ada0p3  freebsd-swap  (12G)
    29362176    10485760  ada0p4  freebsd-ufs  (5.0G)
    39847936    10485760  ada0p5  freebsd-ufs  (5.0G)
    50333696  1048576000  ada0p6  freebsd-ufs  (500G)
  1098909696  2808119439  ada0p7  freebsd-zfs  (1.3T)
Code: 
/uz0 # zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
uz0   1.30T   732G   604G        -         -      -    54%  1.00x  ONLINE  -
root@waltr:/uz0 # zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
uz0              732G   562G  5.60G  /uz0
uz0/backups      365G   562G   365G  /uz0/backups
uz0/db           391M   562G   388M  /db
uz0/mbp0         295G   562G   294G  /uz0/mbp0
uz0/var         43.0G   562G  39.5K  /uz0/var
uz0/var/cache     25K   562G    25K  /uz0/var/cache
uz0/var/crash     25K   562G    25K  /uz0/var/crash
uz0/var/db      1.16G   562G  1.14G  /uz0/var/db
uz0/var/db/pkg  18.8M   562G  18.8M  /uz0/var/db/pkg
uz0/var/empty   41.8G   562G  41.8G  /uz0/var/empty
uz0/var/log       25K   562G    25K  /uz0/var/log
uz0/var/mail      25K   562G    25K  /uz0/var/mail
uz0/var/run       25K   562G    25K  /uz0/var/run
uz0/var/tmp     46.3M   562G  46.3M  /uz0/var/tmp
uz0/www         22.8G   562G  22.8G  /www

Code: 
# df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs    1.9G    284M    1.5G    16%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/gpt/tmpfs     4.8G     84M    4.4G     2%    /tmp
/dev/gpt/usrfs     484G    333G    113G    75%    /usr
/dev/gpt/varfs     4.8G    452M    4.0G    10%    /var
procfs             4.0K    4.0K      0B   100%    /proc
uz0/www            585G     23G    562G     4%    /www
uz0                568G    5.6G    562G     1%    /uz0
uz0/db             562G    388M    562G     0%    /db
uz0/var            562G     40K    562G     0%    /uz0/var
uz0/mbp0           856G    294G    562G    34%    /uz0/mbp0
uz0/var/cache      562G     25K    562G     0%    /uz0/var/cache
uz0/var/tmp        562G     46M    562G     0%    /uz0/var/tmp
uz0/var/crash      562G     25K    562G     0%    /uz0/var/crash
uz0/var/mail       562G     25K    562G     0%    /uz0/var/mail
uz0/var/db         563G    1.1G    562G     0%    /uz0/var/db
uz0/var/log        562G     25K    562G     0%    /uz0/var/log
uz0/var/empty      604G     42G    562G     7%    /uz0/var/empty
uz0/var/run        562G     25K    562G     0%    /uz0/var/run
uz0/var/db/pkg     562G     19M    562G     0%    /uz0/var/db/pkg
uz0/backups        927G    365G    562G    39%    /uz0/backups

And the new disk GPT info:
Code: 
/uz0 # gpart show -p ada1
=>         40  15628053088    ada1  GPT  (7.3T)
           40         1024  ada1p1  freebsd-boot  (512K)
         1064          984          - free -  (492K)
         2048      4194304  ada1p2  freebsd-swap  (2.0G)
      4196352  15623856128  ada1p3  freebsd-zfs  (7.3T)
  15628052480          648          - free -  (324K)
The later stage boot seems to get lost on a loader error 5 after a number of large dataset writes/revisions (~ 500 GB data file transfers). This never happened on the smaller disk/Release 12 setup. I'm thinking I need to repartition the disk to include a root mountpoint after the freebsd-boot partition (/dev/ada1p1). Or do I just need to break up the ada1p3 into < 2TB segment/partitions?

If anyone has any insight or can point me to a good reading source, that will help me figure out how to best live within my current HW limits, I would very much appreciate the help. I have no preconceptions on how to best do this, except I'm not a fan of repetitive work, and I am a fan of reliability which is why I use FreeBSD. Thanks, folks.
 
Error 5 is unknown file system. Do you read also something that speaks of unsupported feature?
I would try to update the BIOS bootloader.
 
  • Like
Reactions: mer
I am aware of the backwards compatibility issue with 13 v. 12. I can mount both versions using R15 on the USB LiveFS which is how I recovered the data on the new drive.

Emrion, I have not read of anything per se that speaks of any unsupported feature. From what I gather, the problem is with the MBR/BIOS problem, the direct addressing space provided by the BIOS is limited to about 2.2 TB. (source Dell discussion from 2013, references available if permitted on this forum). This means UEFI but this board does not support UEFI. The processor is a 64 bit AMD chip, and the FreeBSD doesn't care once it is booted up and running. It can read the whole disk, but in the above ada1 gpart config once ZFS rewrites a leaf outside the 2.2 TB addressing space, BIOS gets lost. The exact way it loses the path to the boot loader is unclear, but I suspect it is doing an internal two's complement add and the high order bit is shifted out of its address space registers. The BIOS is the problem and unfortunately, I'm not sure Gigabyte even has an update for this bios on this board (Award bios, unsure of exact version except its old and this system supports an embedded system which we are replacing but not there yet).

So, my thoughts are, that I should do the following:
gpart the new disk into a UFS section with the boot loader stable at address block 40 or so, limited to 512k as an MBR.
add a root directory ufs mount point for the basics after the MBR restricted to any reasonable size for foreseeable growth of the FreeBSD kernel and associated routines.
add a var directory ufs mount point for some growth and for routines that might need it that is protected from the ZFS philosophy of leaf style re-writes of large data transfers (files) to keep the boot system within the address space of the hardware limited bios.
Then add the user/application space in the sequential <= 2.2 TB partitions.

Alternatively, since ZFS and the FBSD kernel do support large disks (proven since it worked until the boot segment was moved in the new drive and then it didn't), create the UFS BIOS boot record mandated space I need at the beginning of the drive and make the rest a zpool/zfs space for the application data collection and processing.

This is where I need insight into how this part of the FBSD system works pretty much down at the hardware level. Once I understand this, I think I can work it out. My other approach will be to experiment with it to see what I can make work, then if anyone is interested, report it back.

(FYI, my background is decades of programming beginning with PDP-8s with 12 bit register/word lengths, real RISC with 8 total instructions and direct addressing of 2^12 - 1 words back in the 1960s), and of course I do the modern stuff too but never learned much about dealing with the boot loaders of pcs. PDP-8s, -10s and 11s used binary switches on the front panel to toggle a rudimentary binary bootstrap loader into memory which was replaced when memory moved from magnetic cores and wires to solid state in the late 1970s. We've come a long way.)
 
waltr said:
From what I gather, the problem is with the MBR/BIOS problem, the direct addressing space provided by the BIOS is limited to about 2.2 TB. (source Dell discussion from 2013, references available if permitted on this forum).
Click to expand...
I think it's not relevant. Your disk is and has been fully addressed by your bios.
 
waltr said:
So, my thoughts are, that I should do the following:
gpart the new disk into a UFS section with the boot loader stable at address block 40 or so, limited to 512k as an MBR.
Click to expand...
Bootloaders are starting from block 40 and sized 512k right now. MBR vs. GPT has nothing to do with this. And the master boot record itself will be at block 0, always. FreeBSD's PMBR will load the bootcode from freebsd-boot (well within reach of that potential 2TB pitfall).

One thing you may have forgotten, is zfs_load="YES" in /boot/loader.conf? And make sure you actually used gptzfsboot(8) for freebsd-boot, not gptboot(8).
 
Good to know. I now have some better hardware info on the system using dmidecode.

# dmidecode 3.4
Scanning /dev/mem for entry point.
SMBIOS 2.4 present.
60 structures occupying 3000 bytes.
Table at 0x000F0100.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Award Software International, Inc.
Version: F2
Release Date: 02/11/2011
Address: 0xE0000
Runtime Size: 128 kB
ROM Size: 2 MB
Characteristics:
ISA is supported
PCI is supported
PNP is supported
APM is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/360 kB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
AGP is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Targeted content distribution is supported

Handle 0x0001, DMI type 1, 27 bytes
Click to expand...
The BIOS is an Award Bios ca. 2011 Rev F2. Reviewing the Award/Gigabyte literature on this specific board, they list F2 as Feb 2011 release, and an F3 version as March 2011. The interesting description on the F3 release is this: 3TB+ HDD support. Support Turbo HDX function.

I'll follow up with Gigabyte and see if anyone there knows anything more about this and consider how to best proceed. dmidecode did not report the hardware version of the MB so I don't know if it's an older version or the last one produced. The most recent version was listed as a beta version. The immediate prior version F4 has a description: "Update Hybrid EFI." I learned something new this week.

Thanks folks for getting me pointed in a good direction with this old but still functional and (mostly) adequate hardware. Appreciate your help.
 
So, your system was working with a 8 TB disk for a few months and suddenly lose its ability to read above 2 or 3 TB of the same disk? Is it what you're saying or something escapes me?
 
Assuming your boot codes are up-to-date.

As SirDice pointed, legacy BIOS boot often have potential 2TB pitfalls that disallows next stage boot codes to read /boot/{zfs}loader if its actual in-disk location (even if partially) beyonds 2TB limit, or disallows loader to load kernel and/or kmods specified in /boot/loader.conf including /boot/loader.conf itself (even if partially) placed beyond 2TB limit.

To summarize, anything read BEFORE kernel starts (handed over from loader) are needed inside 2TB limit on legacy BIOS.

This doesn't matter for UEFI boots.
 
In addition to the 2TB issue, is your 8TB hard drive CMR or SMR?

It is a complete waste of time using ZFS on a shingled drive. I did it once on a laptop with a brand new 1TB 2.5" SMR HD before I knew anything about SMR. Write performance got progressively worse as data was copied to the drive. It wasn't long before the shingled drive started to misbehave on reads. I swapped out the SMR HD for a SSD to get rid of the problem and I haven't bought a SMR drive since.
 
Not quite. The system was installed in April, booted and from time to time restarted without problems on the 8 TB drive. There were zero errors or signs of misbehavior. The problem became apparent when a failing data acquisition system was repeatedly dumped until its replacement was received. This was a fairly large block transfer of data onto the system. As long as the system was not restarted, it ran fine. If I rebooted it from the USB livefs I could restart and the non-boot activities were all fully functional and operated as expected. ONLY the boot failed. One board poster called it the BIOS Time Bomb, which would go off at an indeterminate unpredictable time in the future, and I agree based on my understanding of OpenZFS's description of how zfs works.

I did discuss this same issue with Gigabyte after the correspondence of this morning (UTC-6). I just received an answer. Herewith the conversation, confirming my original theory and its proper solution. Namely make a small partition at the beginning of the disk dedicated to UFS similar to what I did with the original system disk which had been working for years, then make the remainder zfs, thus preserving space in fixed UFS that the old BIOS can live with.

I have the above ref. MB with Rel. F2 of the firmware (Feb 2011) running. Recently upgraded/replaced bad disk with 8TB disk not knowing there was a 3 TB limitation on BIOS Boot. System is FreeBSD 15 with ZFS. System initally booted and ran just fine with BIOS boot, but after several months and about 3 TB of data save/re-write ops failed to find boot loader for Stage 3 FBSD boot. I cannot determine the board HW version, your BIOS probably needs an upgrade to F3 or F4. How do I check compat/vers/and procedure to try a Firmware update. FBSD in latest incarnations uses UEFI boot or MBR boot. FBSD/ZFS once booted does recognize and mount the full single partition zfs disk (max part= 7.92TB), and data was recoverable when mounted of alternative (USB) live-fs chip. I did not test beyond the first 3TB boundary on this drive. The boot failed on a load error 5 cannot find bootloader, which I suspect is due to zfs moving files beyond its address register range.

So, if I were to gpart partition the disk into a series of 2 TB partitions and place an MBR at the beginning of one and a UNIX root mountpoint on UFS would that protect the BIOS boot sectors from zfs, and then create a zpool beyond that space that the FBSD/zfs does recognize? It take several months for this system to fail and I'm reluctant to put critical data on it until I'm sure it's stable. Thanks. 9/3/2026 11:43 AM
Click to expand...
From Gigabyte:
Answer:
Dear Customer,

What you're seeing is expected on a legacy‑BIOS‑only board when using a very large disk with ZFS.

The disk and ZFS pool are fine; the limitation is in how the old BIOS and MBR‑style bootloader address sectors. They can only reliably access data in roughly the first 2 TB of the drive. When ZFS moved boot‑related files or metadata beyond that range, the BIOS could no longer find the Stage 3 bootloader, resulting in the “load error 5 / cannot find bootloader” message.

The reliable workaround is to:

Keep a small boot partition (a few GB) at the beginning of the disk (well under 2 TB).

Put /boot (and optionally a small UFS /) on that partition.
Click to expand...

I have seen a number of posts on this around the internet boards but none have addressed the question as clearly and directly as Gigabyte just did. This confirms my understanding of what was going wrong, why it was going wrong and that it was indeed restricted to a hardware design decision in the original BIOS boot which was designed for and handicapped by Intel style x86 processors in the days of old. How I so wish IBM's pc offering had gone with the Motorola 68000 back in the day.

Again, I thank you fine folks for the guidance and giving me a compass to find the answer and I hope this will help others similarly situated in finding a good solution and saving a large pile of still functional and productive motherboards from the scrapyards. The knowledge found in these sites is much appreciated.
Put the ZFS root/data pool on the rest of the disk.

That way, everything the BIOS and bootloader need to read is safely within the addressable range, while ZFS can still use the full 8 TB for data. You do not need to split the disk into multiple 2 TB partitions; one small early boot partition plus one large ZFS partition is sufficient.

Updating the BIOS should not address this issue, but you can try it if you like as it won't hurt.

9/3/2026 6:36 PM
Click here to submit your valuable feedback
Product Info
Product Name: GA-890GPA-UD3H (3.0)
BIOS Ver: F2
Purchase Dealer: unk.
Model: AMD Phenom II X4 955
Brand: Kingston
Size: 16
Power Supply: ATX
Memory Part No.: unk
Click to expand...
 
T-Aoki said:
Assuming your boot codes are up-to-date.

As SirDice pointed, legacy BIOS boot often have potential 2TB pitfalls that disallows next stage boot codes to read /boot/{zfs}loader if its actual in-disk location (even if partially) beyonds 2TB limit, or disallows loader to load kernel and/or kmods specified in /boot/loader.conf including /boot/loader.conf itself (even if partially) placed beyond 2TB limit.

To summarize, anything read BEFORE kernel starts (handed over from loader) are needed inside 2TB limit on legacy BIOS.

This doesn't matter for UEFI boots.
Click to expand...
Agreed. Gigabyte just confirmed precisely that and its solution. See post #10.
 
vmb said:
In addition to the 2TB issue, is your 8TB hard drive CMR or SMR?

It is a complete waste of time using ZFS on a shingled drive. I did it once on a laptop with a brand new 1TB 2.5" SMR HD before I knew anything about SMR. Write performance got progressively worse as data was copied to the drive. It wasn't long before the shingled drive started to misbehave on reads. I swapped out the SMR HD for a SSD to get rid of the problem and I haven't bought a SMR drive since.
Click to expand...
It is not. We do buy for performance and spend the money to get it. In this case, we would have bought a similar 2TB disk suitable for NAS r/w many ops because the nature of the application is a large amount of data (raster type) in a very large number of files. There were no 2TB disks of appropriate quality available when we needed them, at any price so we elected to get the bigger disk.
 
What could be important for this kind of situations is that ZFS doesn't promise putting everything needed to be read by bootcodes / loader to be placed at the top of the pool.

This means, for larger than 2TB pools, even if theoretically sufficent size are inside 2TB limit, /boot/loader.conf, kernel or kmods could be placed outside of 2TB limit, thus, bootcodes / loader fail to read them on legacy BIOS boots.

When I've been using legacy BIOS environment with over 8GiB (was actually 8GiB limit of CHS compatible addressing) of disk, I habitally created partition for root file system (at the era, I've used UFS for FreeBSD) inside BIOS limit, and create anything others (excluding freebsd-boot partition on GPT scheme) after it. For example, freebsd-boot at the top, several gigabytes smaller than that can be fit inside 8GiB, swap and others (/home, /usr/ports/distfiles etc.).
No version other than Japanese, but if you can read by translating, this page may be helpful to understand.
 
Is your SATA set in AHCI in the bios?
MBR in BIOS is limited to 32bit LBA address which is around 2^32 sectors ~2.2TB (512/512e) some old HBA/RAID controllers were using 32+16 (48bits) for LBA so they can increase the size of the supported disks but still they are limited on the boot disk to be under 2TB. Check if there's a new BIOS update for your motherboard and check if it's set to AHCI / Native and not in compatible/IDE mode.
 
VladiBG said:
Is your SATA set in AHCI in the bios?
MBR in BIOS is limited to 32bit LBA address which is around 2^32 sectors ~2.2TB (512/512e) some old HBA/RAID controllers were using 32+16 (48bits) for LBA so they can increase the size of the supported disks but still they are limited on the boot disk to be under 2TB. Check if there's a new BIOS update for your motherboard and check if it's set to AHCI / Native and not in compatible/IDE mode.
Click to expand...
see post #11 above. Gigabyte has a couple of updates but again the problem is now confirmed: a hardware limitation in the Bios processor design, not in the firmware, and the workaround is given. For booting purposes a small UFS partition at the beginning of the disk to keep the boot stage 3 code within the address space reliably available and to keep ZFS away from this space will solve (and so far does) solve the problem.
 
Ah, I forgot about AHCI, as it's default (unless defaulted to RAID, but the firmware / BIOS level RAID may be exposing volumes mimic'ing AHCI) for years. But old enough motherboards could default differently.
 
waltr said:
see post #11 above. Gigabyte has a couple of updates but again the problem is now confirmed: a hardware limitation in the Bios processor design, not in the firmware, and the workaround is given. For booting purposes a small UFS partition at the beginning of the disk to keep the boot stage 3 code within the address space reliably available and to keep ZFS away from this space will solve (and so far does) solve the problem.
Click to expand...
Another way would to have 2 ZFS volumes, small one that have bootfs and fit in the limitation, and another to be imported from boot pool.

In this case, be careful NOT to include a copy of /boot in the 2nd pool not to let bootcode to find the 2nd pool as a candidate to boot.
 
waltr said:
It is not. We do buy for performance and spend the money to get it. In this case, we would have bought a similar 2TB disk suitable for NAS r/w many ops because the nature of the application is a large amount of data (raster type) in a very large number of files. There were no 2TB disks of appropriate quality available when we needed them, at any price so we elected to get the bigger disk.
Click to expand...
There are hard drives sold as NAS drives (for NTFS) and there are a subset of NAS drives that are suitable for ZFS. There is no problem using larger than 2TB drives with ZFS if they do not use SMR technology. ZFS can be used with SMR drives but performance degrades with each write that uses a sector that has already been used. Additionally, resilvering is unlikely to complete or be successful if it does. ZFS scrub may also trigger a data integrity issue requiring a resilver.

The ZFS/SMR problems are more likely to occur on file systems with many smaller files and on workloads that write often to the file system. A workload that writes just two 3TB files to an 8TB single stripe ZFS pool once only will unlikely have a problem with an SMR drive. I don't know what would happen if one of those files has a few bytes change and needs an update. I guess as long as there are virgin sectors on the hard drive all will be OK. I also don't know what would happen, and unlikely to ever try using SMR drives in ZFS RAIDZ1 or RAIDZ2 with duplicated writes (2 copies of each file).

My advice to anyone wishing to purchase a modern magnetic hard drive for use with ZFS where the drive was manufactured after September 2013, is to have written into the condition of sale that the drives must be 100% compatible with OpenZFS 2.x. If such a guarantee is not obtained, ZFS compatibility can only be assumed if the magnetic drive is CMR.

If you think I am wrong, humor me by writing to your magnetic hard drive manufacturer quoting your drive model number and serial numbers and asking for them to state if those drives are 100% compatible with OpenZFS 2.x . Hopefully, the answer you get back is YES, they are CMR drives.
 
waltr said:
Herewith the conversation, confirming my original theory and its proper solution. Namely make a small partition at the beginning of the disk dedicated to UFS similar to what I did with the original system disk which had been working for years, then make the remainder zfs, thus preserving space in fixed UFS that the old BIOS can live with.
Click to expand...
It would be better to use a 32GB to 2TB SSD as a ZFS-on-root boot drive, or a pair of SSD mirrored as a ZFS-on-root just for the operating system. Put your data on a second zpool with the geometry that you desire on CMR hard drives. This keeps the operating system functional if/when a magnetic drive fails. It also makes your data zpool portable to another server if an operating system SSD fails.
 
SMR is good for large files that consists of multiple blocks (updated with read-modify-write atomically), with quite limited number of fragments (of block, not of sector) exists in single HDD. Or in case quite rarely updated.
This means it is NOT good at copy on write operations of ZFS.
Best for storing huge audio / video files that are not at all updated and incrementally accepts new files, as SMR eases to increase capacity.

CMR is good for all other cases, including ZFS.
 
T-Aoki already talked about the boot limitation and how to overcome it. I want to write it more clearly.

The boot blocks and the loader rely on the firmware (the BIOS) for reading from disk and the BIOS is limited to 32-bit disk sector addresses which gives access to the first 2 TB of the boot disk only. It is part of the design of ZFS that changes to files and directories lead to new copies of file system metadata being written to disk instead of overwriting older copies. At a certain point in time the latest version of the metadata that is necessary for finding and reading the loader from ZFS got written beyond the 2 TB boundary.

One solution is to use a small UFS filesystem for booting and a big ZFS pool for data.

Another solution, which I recommend, is to have a small ZFS pool for booting (contained entirely within the first 2 TB of the disk) and a big ZFS pool for data. Using ZFS for booting allows you to make use of the boot environments feature.

Note that a 'small' ZFS pool for booting can be 1.5 TB, for example, and can contain not only the /boot directory but also everything that comprises the FreeBSD base system plus all the 3rd party software you need, installed from ports or packages.
 
You must log in or register to reply here.
Back
Top