ZFS gptzfsboot slow boot on FreeBSD13, gptzfsboot: error 128 on FreeBSD12.2

A

abaumgaertner

On a FreeBSD13-p5 system with two zfs pools, each 6x18TB (ada0-ada11) with 4k sectors (sector size logical 4096, physical 4096, offset 0), gptzfsboot takes about 3 minutes to reach the bootstrap loader.
There are no error messages.

Code: 
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
BIOS drive H: is disk5
-
Waiting for 3 minutes ... booting continues normally afterwards

FreeBSD/x86 bootstrap loader, Revision 1.1

On the same system currently running FreeBSD 12.2-RELEASE-p11, I get these errors from gptzfsboot and it takes about 10 minutes to reach bootstrap loader.
This set of errors repeats for each of the 4 internal 18TB disks and takes about 15 seconds per line ( error 128 = timeout).
Code: 
gptzfsboot: error 128 lba 1
gptzfsboot: error 128 lba 1
gptzfsboot: error 128 lba 32
gptzfsboot: error 128 lba 544
gptzfsboot: error 128 lba 16513056
gptzfsboot: error 128 lba 16513568
gptzfsboot: error 128 lba 1
gptzfsboot: error 128 lba 16514056
gptzfsboot: error 128 lba 1

Takes about 10 minutes to get here ... booting continues normally afterwards

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
BIOS drive H: is disk5

System boots from SDD (ada12). For the test with FreeBSD13, a 2nd SSDs was used. zpools were not upgraded.
To have the latest gptzfsboot, updated to FreeBSD13-p5 and updated boot code on 2nd SSD:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada12

Sector size changed from 512 to 4096 bytes on all pool devices ( Seagate ST18000NM000J-2TV103 ) with :
SeaChest_Lite_x86_64-portbld-freebsd12.0 --setSectorSize 4096 --confirm this-will-erase-data -d /dev/ada[n]

System has Bios with MBR.

The pool disks do not contain any partitions and used directly in the zpool:
Code: 
zpool status Pool_A Pool_B
  pool: Pool_A
 state: ONLINE
  scan: resilvered 5.13T in 1 days 17:49:10 with 0 errors on Sun Nov  7 04:56:49 2021
config:

        NAME        STATE     READ WRITE CKSUM
        Pool_A      ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada6    ONLINE       0     0     0
            ada8    ONLINE       0     0     0
            ada10   ONLINE       0     0     0

errors: No known data errors

  pool: Pool_B
 state: ONLINE
  scan: resilvered 5.16T in 1 days 18:01:02 with 0 errors on Sun Nov  7 05:09:45 2021
config:

        NAME        STATE     READ WRITE CKSUM
        Pool_B      ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada7    ONLINE       0     0     0
            ada9    ONLINE       0     0     0
            ada11   ONLINE       0     0     0

errors: No known data errors

Resilvering due to recent capacity upgrade of the zpool.

No issues with 4k sectors in normal operation or during Bios POST.

Any ideas how to speed up gptzfsboot?
 
abaumgaertner said:
This set of errors repeats for each of the 4 internal 18TB disks and takes about 15 seconds per line ( error 128 = timeout).
Click to expand...
abaumgaertner said:
System boots from SDD (ada12). For the test with FreeBSD13, a 2nd SSDs was used.
Click to expand...
Is that SSD you used to boot FreeBSD 12 on the same controller as those 4 internal disks? Are you sure that SSD is still good? Or the cable to that SSD?
 
SirDice said:
Is that SSD you used to boot FreeBSD 12 on the same controller as those 4 internal disks? Are you sure that SSD is still good? Or the cable to that SSD?
Click to expand...
Both SSDs are fine. Disconnected the 4 internal 18TB HDDs: System boots without issue on both releases. Reconnected one internal drive: lba errors only appear once. Reconnected all four drives: 4 sets of lba errors.

The issue started with the capacity upgrade.
 
The other day I was saying that it's not possible to legacy boot off the native 4Kn disk as pmbr assumes 512 sector size. Please can you share the output of smartctl and diskinfo for one of your boot disk ? smartctl -i /dev/daX and diskinfo -v /dev/daX.

How does the connection to all disks look like ? How is it connected ?
 
In the mean time error you see is coming from stand/i386/common/drv.c from function drvread(). Error 128 means "timeout, not ready". Possible conflicts till the drives settle ? That's why I asked about the physical connections.
 
_martin said:
Possible conflicts till the drives settle ? That's why I asked about the physical connections.
Click to expand...
That's why I wondered if that SSD is on the same controller as those drives. I've had dodgy drives completely hang up and hose the bus. Which resulted in all sorts of weird and wonderful errors until I physically removed that bad drive.

A dodgy extender could possible also cause various issues, it might work fine with 3 disks but adding the 4th and the whole thing just starts acting up.

abaumgaertner said:
System boots from SDD (ada12). For the test with FreeBSD13, a 2nd SSDs was used. zpools were not upgraded.
Click to expand...
Is that ada12 (with FreeBSD 12) still connected to the system when you booted FreeBSD 13 from the other disk? Or did you swap out those disks?
 
I agree, that's why I was also curious about the physical setup of those disks, what was expand (added).
I also really hope abaumgaertner shares the diskinfo of the boot drive, I'm wondering if disk is faking the 512 sectors size or not (i.e. 512e).
 
+1 for dying disk or wonky controller/cable/extender

Have seen this on several occasions especially with SATA drives that refuse to acknowledge they are dead and keep delaying/blocking boot/io and interfering with other disks on the same controller.


Wild guess: what disk controller/HBA is used?
abaumgaertner said:
System has Bios with MBR.
Click to expand...
This sounds very much like some ancient hardware. Several old controllers/HBAs are known to have bugs/limitations regarding big drives (e.g. almost everything supported by mpt(4) comes to mind...)
 
SMART overall-health self-assessment test result: PASSED for both boot SSDs and all zfs pool devices

After a full backup, I resilvered the zfs pool for the capacity upgrade 3 times, swapping out 2 drives each time.
No issues during the resilvering and normal operation - 2-3TB were already written to the enlarged pool.

There is only one boot SSD installed. System runs on FreeBSD12.2. For yesterdays testing with FreeBSD13, I swapped the SSDs out of the system for each test.

To avoid interference (cables, bus, controller, etc.) between zfs pool HDDs and Boot-SSD during gptzfsboot, I've removed the boot SSD from the system and installed FreeBSD12.2 and FreeBSD13 on two new USB Sticks (to boot) for further testing.
Exactly the same results.


12.2-RELEASE-p11
FreeBSD bsd12test 12.2-RELEASE-p7 FreeBSD 12.2-RELEASE-p7 GENERIC amd64

USB Boot Stick A:
smartctl -i /dev/da0​
Code: 
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/da0: Unknown USB bridge [0x8564:0x1000 (0x1100)]
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

diskinfo -v /dev/da0​
Code: 
/dev/da0
        512             # sectorsize
        30579621888     # mediasize in bytes (28G)
        59725824        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        3717            # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        JetFlash Transcend 32GB # Disk descr.
        XXXXXXXXXXNSXW6R        # Disk ident.
        No              # TRIM/UNMAP support
        Unknown         # Rotation rate in RPM
        Not_Zoned       # Zone Mode


13.0-RELEASE-p5
FreeBSD bsd13test 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

USB Boot Stick B:
smartctl -i /dev/da0​
Code: 
smartctl 7.2 2020-12-30 r5155 [FreeBSD 13.0-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/da0: Unknown USB bridge [0x8564:0x1000 (0x1100)]
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

diskinfo -v /dev/da0​
Code: 
/dev/da0
        512             # sectorsize
        30579621888     # mediasize in bytes (28G)
        59725824        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        3717            # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        JetFlash Transcend 32GB # Disk descr.
        XXXXXXXXXX1DMRMT        # Disk ident.
        umass-sim0      # Attachment
        No              # TRIM/UNMAP support
        Unknown         # Rotation rate in RPM
        Not_Zoned       # Zone Mode

HDD18TB, 4k Sectors, Pool Member:
smartctl -i /dev/ada0​
Code: 
smartctl 7.2 2020-12-30 r5155 [FreeBSD 13.0-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST18000NM000J-2TV103
Serial Number:    XXXXXKFS
LU WWN Device Id: 5 000c50 0dbe5a144
Firmware Version: SN02
User Capacity:    18,000,207,937,536 bytes [18.0 TB]
Sector Size:      4096 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Nov 16 20:29:04 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

diskinfo -v /dev/ada0​
Code: 
/dev/ada0
        4096            # sectorsize
        18000207937536  # mediasize in bytes (16T)
        4394582016      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        4359704         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        ST18000NM000J-2TV103    # Disk descr.
        XXXXXKFS        # Disk ident.
        ahcich0         # Attachment
        No              # TRIM/UNMAP support
        7200            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

gpart backup ada0​
gpart: No such geom: ada0.​

gpart list ada0​
Code: 
gpart: Class 'PART' does not have an instance named 'ada0'.


boot SSD (same device model for FreeBSD12.2 and FreeBSD13 Test):
smartctl -i /dev/ada12
Code: 
smartctl 7.2 2020-12-30 r5155 [FreeBSD 13.0-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     SandForce Driven SSDs
Device Model:     Corsair Force GS
Serial Number:    13247925000017XXXXXX
LU WWN Device Id: 0 000000 000000000
Firmware Version: 5.05
User Capacity:    240,057,409,536 bytes [240 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
TRIM Command:     Available, deterministic
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS, ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Nov 16 21:45:07 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

diskinfo -v /dev/ada12​
Code: 
/dev/ada12
        512             # sectorsize
        240057409536    # mediasize in bytes (224G)
        468862128       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        465141          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Corsair Force GS        # Disk descr.
        13247925000017XXXXXX    # Disk ident.
        ahcich1         # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode



I've retested the scenario but replaced the 18TB HDD with a 16TB Seagate HDD with logical 512 Sector Size with zfs pool residing in GPT partition from a smiliar pool on another system: zfs-raid2, 6x16TB
These errors are correct as the 5 other devices from the zfs pool are not present in the 2nd system.
These errors are not displayed with the 18TB 4k Sector HDD. So it seems gptzfsboot is unable to read data from 4k sector disk.

13.0-RELEASE-p5
FreeBSD bsd13test 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

16TB HDD, gptzfsboot:
Code: 
BIOS drive C: is disk0
BIOS drive D: is disk1
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
zio_read error: 5
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
ZFS: can not read MOS config

HDD 16TB, 512 Sectors, Pool Member:
smartctl -i /dev/ada0​
Code: 
smartctl 7.2 2020-12-30 r5155 [FreeBSD 13.0-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST16000VN001-2RV103
Serial Number:    XXXXXRBQ
LU WWN Device Id: 5 000c50 0b6bcae2d
Firmware Version: SC61
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Nov 16 21:01:47 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
diskinfo -v /dev/ada0​
Code: 
/dev/ada0
        512             # sectorsize
        16000900661248  # mediasize in bytes (15T)
        31251759104     # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        31003729        # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        ST16000VN001-2RV103     # Disk descr.
        XXXXXRBQ        # Disk ident.
        ahcich0         # Attachment
        No              # TRIM/UNMAP support
        7200            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

gpart backup ada0​
Code: 
GPT 128
1   freebsd-zfs         128 31251758936
gpart list ada0​
Code: 
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 31251759063
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 16000900575232 (15T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   efimedia: HD(1,GPT,245dc297-d851-11e9-a541-001517320c9c,0x80,0x746bfff58)
   rawuuid: 245dc297-d851-11e9-a541-001517320c9c
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 16000900575232
   offset: 65536
   type: freebsd-zfs
   index: 1
   end: 31251759063
   start: 128
Consumers:
1. Name: ada0
   Mediasize: 16000900661248 (15T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

6x 16TB ST16000VN001-2RV103 for comparison
zpool status Pool_A​
Code: 
  pool: Pool_A
 state: ONLINE
  scan: scrub canceled on Mon Jan  4 01:58:39 2021
config:

        NAME         STATE     READ WRITE CKSUM
        Pool_A       ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            ada0p1   ONLINE       0     0     0
            ada2p1   ONLINE       0     0     0
            ada4p1   ONLINE       0     0     0
            ada6p1   ONLINE       0     0     0
            ada9p1   ONLINE       0     0     0
            ada11p1  ONLINE       0     0     0

errors: No known data errors

I see 2 differences between the configs:
working vs slow gptzfsboot
  1. logical sector size 512 (Seagate 16TB) vs 4k (Seagate 18TB)

  2. zfs pool members with gpt partitions vs raw device without gpt
    => ada[x]p1 vs ada[x]
 
abaumgaertner said:
So it seems gptzfsboot is unable to read data from 4k sector disk
Click to expand...
That's correct -- all parts of i386 loader assume 512 as a sector size and have it hardcoded into the asm code. If disk can't fake it (i.e. stanard 512e) you can't legacy boot it. That's why I was interested to see the output from those commands. To boot of 4k disk you need to use UEFI.

If I understood your last reply correctly you introduced new device (usb) to the boot during tests. More interesting test would be to disconnect data pool and boot off the ssd pool only. For that I'm lost a bit in your reply -- what disks are used in the boot pool ? What does gpart show say about them ? Can you test the boot with the boot pool only?

From output above you mentioned ada12. Is that the single disk in the boot pool ?
 
All boot pools are zfs stripe, single device - same setup with SDD and USB Stick.
All boot devices have 512 byte native sector size - no emulation.
Boot device selection is done in BIOS. It's a supported combination to have 512byte sector size device for booting next to 4k sector size devices for zfs pools.

gpart show points out the 2nd difference: the 512byte zfs pool devices have a gpt partition structure and the zfs pool resides in ada[x]p1 while the 4k devices have not and the zfs pool resides on the whole device ada[x].

The issue is limited to gptzfsboot. Kernel and drivers work correctly with 512byte and 4k sectors mix during normal operation in FreeBSD12.2

Boot devices summary:

USB Boot Stick A - testing with 12.2-RELEASE-p11
diskinfo -v /dev/da0​
Code: 
/dev/da0
        512             # sectorsize
        30579621888     # mediasize in bytes (28G)
        59725824        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        3717            # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        JetFlash Transcend 32GB # Disk descr.
        XXXXXXXXXXNSXW6R        # Disk ident.
        No              # TRIM/UNMAP support
        Unknown         # Rotation rate in RPM
        Not_Zoned       # Zone Mode

USB Boot Stick B - testing with 13.0-RELEASE-p5:
diskinfo -v /dev/da0​
Code: 
/dev/da0
        512             # sectorsize
        30579621888     # mediasize in bytes (28G)
        59725824        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        3717            # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        JetFlash Transcend 32GB # Disk descr.
        XXXXXXXXXX1DMRMT        # Disk ident.
        umass-sim0      # Attachment
        No              # TRIM/UNMAP support
        Unknown         # Rotation rate in RPM
        Not_Zoned       # Zone Mode


boot SSD (same device model for FreeBSD12.2 and FreeBSD13 Test):
diskinfo -v /dev/ada12​
Code: 
/dev/ada12
        512             # sectorsize
        240057409536    # mediasize in bytes (224G)
        468862128       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        465141          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Corsair Force GS        # Disk descr.
        13247925000017XXXXXX    # Disk ident.
        ahcich1         # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode
 
I never said you can't have native 4k pools, I just pointed out you can't legacy boot out of pool with 4k sector due to hardcoded sector size in the bootloader code. This is not the issue here but it was not clear how you're booting, at least not from your initial question. Thanks for sharing that, we don't need to focus on that part.

However if BIOS is UEFI capable it's worth a try to boot like that.

Did you try to boot the system without data pools ?
 
I tried to emulate your situation in my LAB:
Code: 
root@lab20:~ # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  19.5G  1.36G  18.1G        -         -     0%     6%  1.00x    ONLINE  -
spool   144T  2.58M   144T        -         -     0%     0%  1.00x    ONLINE  -
root@lab20:~ #

root@lab20:~ # zpool status
  pool: rpool
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      da0p3     ONLINE       0     0     0

errors: No known data errors

  pool: spool
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    spool       ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        da1     ONLINE       0     0     0
        da2     ONLINE       0     0     0
        da3     ONLINE       0     0     0
        da4     ONLINE       0     0     0
        da5     ONLINE       0     0     0
        da6     ONLINE       0     0     0
        da7     ONLINE       0     0     0
        da8     ONLINE       0     0     0
With disks set a as:
Code: 
# for i in `seq 0 8`; do diskinfo da$i; done
da0    512    21474836480    41943040    4096    0    2610    255    63
da1    4096    19799799234560    4833935360    0    0    300898    255    63
da2    4096    19799799234560    4833935360    0    0    300898    255    63
da3    4096    19799799234560    4833935360    0    0    300898    255    63
da4    4096    19799799234560    4833935360    0    0    300898    255    63
da5    4096    19799799234560    4833935360    0    0    300898    255    63
da6    4096    19799799234560    4833935360    0    0    300898    255    63
da7    4096    19799799234560    4833935360    0    0    300898    255    63
da8    4096    19799799234560    4833935360    0    0    300898    255    63
#

And boot disk da0
Code: 
# gpart show da0
=>      40  41942960  da0  GPT  (20G)
        40    614400    1  efi  (300M)
    614440      1024    2  freebsd-boot  (512K)
    615464  41327536    3  freebsd-zfs  (20G)
#

# uname -a
FreeBSD lab20 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
#
I have no problem booting fast. Of course this is emulated environment so it is with its limitation. But bootcode had no problems going forward.
 
You must log in or register to reply here.
Back
Top