UFS problem booting from 2nd drive

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#1
I used dump/restore commands to copy a custom freebsd 11.1 (upgraded from 10.3) to a new server. After I restore the files I create gmirror (RAID-1) and copy the bootcode to both disks:

Code:
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1
When I select from BIOS to boot using 1st disk (ada0) then it boots without issue. If I select to boot from 2nd disk (ada1) then it hangs and restarts again and again.

Code:
gpart show

=>        40  7814037088  ada0  GPT  (3.6T)
          40         472     1  freebsd-boot  (236K)
         512     8388608     2  freebsd-ufs  (4.0G)
     8389120   134217728     3  freebsd-swap  (64G)
   142606848    33554432     4  freebsd-ufs  (16G)
   176161280   134217728     5  freebsd-ufs  (64G)
   310379008    33554432     6  freebsd-ufs  (16G)
   343933440  1073741824     7  freebsd-ufs  (512G)
  1417675264  6396361856     8  freebsd-ufs  (3.0T)
  7814037120           8        - free -  (4.0K)

=>        40  7814037088  ada1  GPT  (3.6T)
          40         472     1  freebsd-boot  (236K)
         512     8388608     2  freebsd-ufs  (4.0G)
     8389120   134217728     3  freebsd-swap  (64G)
   142606848    33554432     4  freebsd-ufs  (16G)
   176161280   134217728     5  freebsd-ufs  (64G)
   310379008    33554432     6  freebsd-ufs  (16G)
   343933440  1073741824     7  freebsd-ufs  (512G)
  1417675264  6396361856     8  freebsd-ufs  (3.0T)
  7814037120           8        - free -  (4.0K)

The message I get during boot is:

gptboot: invalid primary GPT header
gptboot: invalid backup GPT header
gptboot: unable to load GPT


Any idea what the issue could be?
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#2
Assuming ada1p2 is your / partition you might need to set bootme flag.

gpart set -a bootme -i 2 ada1
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#3
I will try it but don't think this is the problem. From gptboot manpage:

Code:
For normal usage, the user does not have to set or manage any of the
partition attributes.  gptboot will boot from the first UFS partition
found.
Code:
If neither bootonce nor bootme attributes are found on any partitions,
booting is attempted from the first freebsd-ufs partition on the disk.
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#4
Yes, it was just a guess. For me I've never needed to set it manually but I've read others needed to for whatever reason.

Didn't see that one at first
gptboot: invalid primary GPT header
gptboot: invalid backup GPT header
gptboot: unable to load GPT
Is that when you try to boot from ada1?
I've never seen the primary and the backup invalid so the following might not work, but try i.
gpart recover ada1

When you created the GPT layout on ada1, did you start from scratch with
gpart destroy -F ada1 ?
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#5
I use dump to take the backup from another server, then boot with mfsbsd and run:

Code:
gpart destroy -F ada0
gpart destroy -F ada1

gpart create -s gpt ada0
gpart add -b 40 -s 472 -t freebsd-boot ada0
gpart add -s 4G -t freebsd-ufs ada0
gpart add -s 64G -t freebsd-swap ada0
gpart add -s 16G -t freebsd-ufs ada0
gpart add -s 64G -t freebsd-ufs ada0
gpart add -s 16G -t freebsd-ufs ada0
gpart add -s 512G -t freebsd-ufs ada0
gpart add -t freebsd-ufs ada0

newfs -S 4096 -f 4096 -b 32768 ada0p2
newfs -S 4096 -f 4096 -b 32768 -U ada0p4
newfs -S 4096 -f 4096 -b 32768 -U ada0p5
newfs -S 4096 -f 4096 -b 32768 -U ada0p6
newfs -S 4096 -f 4096 -b 32768 -U ada0p7
newfs -S 4096 -f 4096 -b 32768 -U ada0p8

mount /dev/ada0p2 /mnt
cd /mnt
mkdir var tmp usr home home2
mount /dev/ada0p4 /mnt/tmp
mount /dev/ada0p5 /mnt/var
mount /dev/ada0p6 /mnt/usr
mount /dev/ada0p7 /mnt/home
mount /dev/ada0p8 /mnt/home2

cd /mnt/home2
fetch http://xxx.xxx.xxxx.xxx/FreeBSD-11.1-p1-20170905.tar.gz
tar xf FreeBSD-11.1-p1-20170905.tar.gz
rm FreeBSD-11.1-p1-20170905.tar.gz
mv FreeBSD-11.1-p1-20170905 dump

cd /mnt/tmp && restore -r -f /mnt/home2/dump/tmp.dump && rm restoresymtable
cd /mnt/usr && restore -r -f /mnt/home2/dump/usr.dump && rm restoresymtable
cd /mnt/var && restore -r -f /mnt/home2/dump/var.dump && rm restoresymtable
cd /mnt/home && restore -r -f /mnt/home2/dump/home.dump && rm restoresymtable
cd /mnt && restore -r -f /mnt/home2/dump/root.dump && rm restoresymtable
cd

gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada0

umount /mnt/tmp
umount /mnt/var
umount /mnt/usr
umount /mnt/home
umount /mnt/home2
umount /mnt

gmirror label -vb load gm0p2 /dev/ada0p2
gmirror label -vb load gm0p3 /dev/ada0p3
gmirror label -vb load gm0p4 /dev/ada0p4
gmirror label -vb load gm0p5 /dev/ada0p5
gmirror label -vb load gm0p6 /dev/ada0p6
gmirror label -vb load gm0p7 /dev/ada0p7
gmirror label -vb load gm0p8 /dev/ada0p8

shutdown -r now
Then the server come back online and run:

Code:
sysctl kern.geom.debugflags=17
gpart backup ada0 | gpart restore -F ada1

gmirror insert gm0p2 /dev/ada1p2
gmirror insert gm0p3 /dev/ada1p3
gmirror insert gm0p4 /dev/ada1p4
gmirror insert gm0p5 /dev/ada1p5
gmirror insert gm0p6 /dev/ada1p6
gmirror insert gm0p7 /dev/ada1p7
gmirror insert gm0p8 /dev/ada1p8

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

shutdown -r now
Now if I configure bios to boot from 1st disk it works, if I configure it to boot from 2nd disk it doesn't boot.
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#7
I can't see anything wrong with what you did, but obviously somethings wrong with GPT on your ada1.
Do you have softupdates journaling enabled on your disks?
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#9
ada0 and ada1 are the same models and have the same bootcode.

BIOS has default settings, booting from UEFI, then Network boot, then USB disk, then hard disks.

If I don't run "gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1" and the disk boot priority is ada1 first and then ada0 it boots. If I write the bootcode to ada1 then the problem exist.

diff from hex dump show no differences:

Code:
hexdump -C -n 512 /dev/ada0p1 > hex0
hexdump -C -n 512 /dev/ada1p1 > hex1
diff hex0 hex1
diff from dd shows no differences:

Code:
dd if=/dev/ada0 of=./bootcode_ada0 count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.013218 secs (38736 bytes/sec)

dd if=/dev/ada1 of=./bootcode_ada1 count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.012451 secs (41120 bytes/sec)

diff bootcode_ada0 bootcode_ada1
I install the bootcode in another server in both disks again and the boot was successful from 2nd drive. These 2 servers have only different motherboard (the server with the problem has Fujitsu D3401-H2 and the server without the problem has Fujitsu D3401-H1) and different disks (the server with the problem has Seagate ST4000NM0245-1Z2107 and the the server without the problem has Seagate ST4000NM0024-1HT178). The BIOS for server with problem is V5.0.0.12 R1.8.0 for D3401-H2x (Release Date: 05/15/2017) and BIOS for server without problem is V5.0.0.11 R1.21.0 for D3401-H1x (Release Date: 05/15/2017).
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#10
I have no explanation for this, you've done right and testet everything.
Last and only thing is maybe the disk is not ready in time. Staggered spinup enabled? Sata controller configuration in the BIOS...? I see the motherboard has M.2 support, those usually share some lanes with sata ports and that's usually set to 'auto'. I that's possible I would set it fixed to sata. But really I'm out of ideas.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#11
The datacenter swap the first with the second disk. Now it doesn't boot if the new first disk (previously was the second) is selected in BIOS. If I configure BIOS to boot from second disk (previously was the first) it boots. So bootcode is not installed correctly in previously second disk? It doesn't make sense.
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#12
I can't imagine why the bootcode should not work on that one disk.
The way you installed the bootcode was right and it works with the other disk.
I would try to start over setting up that one disk from scratch, if you then again get those GPT errors the first sector
of the disk may be bad.
If that would be the case, you could consider setting up to boot in UEFI mode, as it does not rely on reading from sector 0/MBR.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#13
Finally the problem is not related to this server. I did a fresh install in another server "that didn't have the same issue" and it finally has the same issue.

As you can see in my previous messages I install bootcode in first disk while in mfsbsd 11.0:

Code:
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada0
and when I finish the installation and reboot in FreeBSD 11.1 I install the bootcode for 2nd disk:

Code:
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1
This way the problem exist.

But if I install bootcode for both disks while in mfsbsd 11.0 then I can boot from both disks:

Code:
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada0
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada1
So my conclusion is that I can install the bootcode during the installation from mfsbsd 11.0 but not from freebsd 11.1

The mfsbsd version is:

Code:
FreeBSD rescue 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
But if I don't install the bootcode on 2nd disk during the installation from mfsbsd and after the installation boot on mfsbsd again and run:

Code:
mount /dev/ada0p2 /mnt
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada1
then it doesn't boot from second disk.

So the question is why it works if I install bootcode during the installation but not after I reboot back in mfsbsd?
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#14
gpart bootcode -b /boot/pmbr -p /mnt/boot/gptboot -i 1 ada1
I don't understand why you install MBR code from /boot/ and partition code from /mnt/.
Didn't notice that before now. Use the code from /boot/ for both. Kind of confusing...
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#15
You are right, the correct commands is:

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1

I type it correctly on the server but I mistype it here.

I edit my previous message.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#16
Another strange thing:

If I do the installation using mfsbsd 10.3 and write the bootcode to 1st disk during the installation and then reboot to freebsd 11.3 and write the bootcode for 2nd disk I can boot from both disks.

I suspect now the mfsbsd 11.0 provided by the datacenter. I will try to create a mfsbsd 11.1 on my own and boot from it. Then I will try to do the installation again.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#17
I boot using mfsbsd 11.1 but the same issue happens. Any idea why it only works with mfsbsd 10.3?

Even with mfsbsd 10.3 used for installation and writing the bootcode for first disk, I write the bootcode for second disk after installation using FreeBSD 11.3. It doesn't make sense why it works this way.
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#18
BIOS has default settings, booting from UEFI, then Network boot, then USB disk, then hard disks.
I thought twice about that and what you really do all the time is writing bootcode to boot in BIOS mode.
Though most motherboards will still probe to boot in UEFI and BIOS mode, no matter what you set in to in the BIOS
this can be quite confusing when troubleshooting.

I suggest two things: set to boot in BIOS mode first in your BIOS (disable CSM completely) or leave it as it is and install UEFI boot code to the disk that refuses to boot.

That's easy to do:
Code:
gpart modify -t efi -i1 ada1
gpart bootcode -p /boot/boot1.efi -i1 ada1
dd if=/dev/zero of=/dev/ada1 bs=512 count=1  (clear MBR)
I just testet it on my computer and it works, switched flawlessly to UEFI boot and cleared out MBR.
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#20
Here is is:
Code:
gpart show ada0
=>       40  937703008  ada0  GPT  (447G)
         40     409600     1  efi  (200M)
     409640  928628736     2  freebsd-ufs  (443G)
  929038376    8388608     3  freebsd-swap  (4.0G)
  937426984     276064        - free -  (135M)
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#22
There is no need to disable Boot Option #4 (this would only take effect when an UEFI bootloader could not be found)
Just leave Boot Option #1 as it is (this will search for an efi bootloader in the order specified at the bottom...)

Now go to "Hard Disk Drive BBS Priorities" at the bottom and set to boot from the harddisk with the 'efi' partition.
This will most likely be SATA1 (SATA0 should be the one which still has BIOS bootcode).
Move SATA1 in the first position here.
 

k.jacker

Well-Known Member

Thanks: 263
Messages: 494

#24
Yes, as it only says [UEFI Device].
That means you have do specify the drive from "Hard Disk Drive BBS Priorities".

If it's possible to choose something like [UEFI, SATA1, Seagate .....] directly in "Boot Option #1", that would also do.
Then you can leave the "Hard Disk Drive BBS Priorities" menu alone.

That really depends on BIOS.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Thanks: 9
Messages: 137

#25
I change the commands I type to mfsbsd but I can't boot:

Code:
gpart destroy -F ada0
gpart destroy -F ada1

dd if=/dev/zero of=/dev/ada0 bs=512 count=1
dd if=/dev/zero of=/dev/ada1 bs=512 count=1

gpart create -s gpt ada0
gpart add -b 40 -s 472 -t efi ada0
gpart add -s 4G -t freebsd-ufs ada0
gpart add -s 64G -t freebsd-swap ada0
gpart add -s 16G -t freebsd-ufs ada0
gpart add -s 64G -t freebsd-ufs ada0
gpart add -s 16G -t freebsd-ufs ada0
gpart add -s 512G -t freebsd-ufs ada0
gpart add -t freebsd-ufs ada0

newfs -S 4096 -f 4096 -b 32768 ada0p2
newfs -S 4096 -f 4096 -b 32768 -U ada0p4
newfs -S 4096 -f 4096 -b 32768 -U ada0p5
newfs -S 4096 -f 4096 -b 32768 -U ada0p6
newfs -S 4096 -f 4096 -b 32768 -U ada0p7
newfs -S 4096 -f 4096 -b 32768 -U ada0p8

mount /dev/ada0p2 /mnt
cd /mnt
mkdir var tmp usr home home2
mount /dev/ada0p4 /mnt/tmp
mount /dev/ada0p5 /mnt/var
mount /dev/ada0p6 /mnt/usr
mount /dev/ada0p7 /mnt/home
mount /dev/ada0p8 /mnt/home2

cd /mnt/home2
fetch http://xxx.xxx.xxx.xxx/images/FreeBSD-11.1-p1-20170904.tar.gz
tar xf FreeBSD-11.1-p1-20170904.tar.gz
rm FreeBSD-11.1-p1-20170904.tar.gz
mv FreeBSD-11.1-p1-20170904 dump

cd /mnt/tmp && restore -r -f /mnt/home2/dump/tmp.dump && rm restoresymtable
cd /mnt/usr && restore -r -f /mnt/home2/dump/usr.dump && rm restoresymtable
cd /mnt/var && restore -r -f /mnt/home2/dump/var.dump && rm restoresymtable
cd /mnt/home && restore -r -f /mnt/home2/dump/home.dump && rm restoresymtable
cd /mnt && restore -r -f /mnt/home2/dump/root.dump && rm restoresymtable
cd

gpart bootcode -p /mnt/boot/boot1.efi -i1 ada0

umount /mnt/tmp
umount /mnt/var
umount /mnt/usr
umount /mnt/home
umount /mnt/home2
umount /mnt

gmirror label -vb load gm0p2 /dev/ada0p2
gmirror label -vb load gm0p3 /dev/ada0p3
gmirror label -vb load gm0p4 /dev/ada0p4
gmirror label -vb load gm0p5 /dev/ada0p5
gmirror label -vb load gm0p6 /dev/ada0p6
gmirror label -vb load gm0p7 /dev/ada0p7
gmirror label -vb load gm0p8 /dev/ada0p8

shutdown -r now
 

Attachments

Top