Boot from SAN and Multipath

S

serveroneblu

Hello,

Our organization has acquired HP DL380 with one Dual Port HBA Qlogic Qlogic2562 8GB (HPAJ764A)

The server has been connected to the Storage Area Network (3Par) and I have installed FreeBSD 10.0 STABLE (amd64) on a LUN with the ZFS filesystem. I have used ZFS partitions with Pool Name: BootSAN and GPT schema on da1 so I get:
Code: 
Filesystem              Size    Used   Avail Capacity  Mounted on
BootSAN/ROOT/default     45G    1.2G     44G     3%    /
devfs                   1.0K    1.0K      0B   100%    /dev
BootSAN/tmp              44G    176K     44G     0%    /tmp
BootSAN/usr/home         44G    144K     44G     0%    /usr/home
BootSAN/usr/ports        45G    828M     44G     2%    /usr/ports
BootSAN/usr/src          45G    545M     44G     1%    /usr/src
BootSAN/var              44G    220M     44G     0%    /var
BootSAN/var/crash        44G    148K     44G     0%    /var/crash
BootSAN/var/log          44G    328K     44G     0%    /var/log
BootSAN/var/mail         44G    148K     44G     0%    /var/mail
BootSAN/var/tmp          44G    200K     44G     0%    /var/tmp

The boot from SAN is working perfectly. I am able to see all four paths for the LUN 0:
Code: 
camcontrol devlist |grep VV
<3PARdata VV 3210>                 at scbus3 target 0 lun 0 (pass1,da1)
<3PARdata VV 3210>                 at scbus3 target 1 lun 0 (pass3,da2)
<3PARdata VV 3210>                 at scbus4 target 0 lun 0 (pass5,da3)
<3PARdata VV 3210>                 at scbus4 target 1 lun 0 (pass7,da4)

gpart output is:
Code: 
# gpart show
=>       34  104857533  da1  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

=>       34  104857533  da2  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

=>       34  104857533  da3  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

=>       34  104857533  da4  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

I have activated geom_multipath and Qlogic driver directly at boot:
Code: 
cat /boot/loader.conf
geom_multipath_load="YES"
isp_load="YES"

But when I try to enable the multipath:
Code: 
sysctl kern.geom.debugflags=0x10
gmultipath label -v BOOT /dev/da1 /dev/da2 /dev/da3 /dev/da4

In dmesg I see:
Code: 
GEOM: multipath/BOOT: corrupt or invalid GPT detected.
GEOM: multipath/BOOT: GPT rejected -- may not be recoverable.
GEOM: diskid/MULTIPATH-%201653047: corrupt or invalid GPT detected.
GEOM: diskid/MULTIPATH-%201653047: GPT rejected -- may not be recoverable.

Code: 
gmultipath status
          Name   Status  Components
multipath/BOOT  OPTIMAL  da2 (ACTIVE)
                         da3 (PASSIVE)
                         da1 (PASSIVE)
                         da2 (PASSIVE)

gpart status
                        Name  Status  Components
                       da1p1      OK  da1
                       da1p2      OK  da1
                       da1p3      OK  da1
             multipath/BOOTa      OK  multipath/BOOT

First problem, the configuration looks OK but when I detach a fibre, the OS will hang. Second, after a reboot the OS won't boot, I have this message on the console:
Code: 
FreeBSD/x86 boot
Default 0:ad(0,a)/boot/kernel/kernel
Boot:

The system cannot find the boot

I tried some commands, but I'm failing miserably :)

- echo "0:da(1,a)/boot/loader" > /boot.config
- gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

In /boot/loader.conf:
Code: 
vfs.root.mountfrom="zfs:/BOOT"

My question is: is it possible to have write access on a FS like / at boot via the geom_multipath device?

Thank you for your help.
 
I think there are a few problems here (although I don't have any experience with multipath and booting from fibre-channel)
  1. It appears that gmultipath writes metadata to the end of the disk. This is probably colliding with the backup GPT table. I'm not sure of the "correct" way to deal with this. The gmultipath man page doesn't mention anything about using it in conjunction with GPT. Maybe you could pass the GPT partition to the create command instead, so that gmultipath metadata is stored at the end of the partition and not at the end of the disk?
    gmultipath label -v BOOT da1p2 da2p2 etc
  2. The pool will need to be told to use the multipath device. If you created the pool before multipath, it will be using something like /dev/da1p2 (You can see this info in zpool status output). If that one device disappears (which it will if the relevant path disappears), the pool and system will stop working. The pool needs to use the /dev/multipath/BOOT device, which is the one that will use any of the available paths. I'm not sure how you can change this at this point without a bit of messing about. It would of been easiest to create the multipath device, then use that to create the pool during installation.
  3. The entry in /boot/loader.conf should look something like the below (although if it's booting, I suspect you have a new enough version of FreeBSD that it is using the bootfs property rather than relying on this loader variable).
    Code: 
    vfs.root.mountfrom="zfs:BootSAN/ROOT/default"
 
Ciao usdmatt!
Thank you very much. Using this command on the boot partition the GPT was ok without errors: # gmultipath label -v BOOT /dev/da1p1 /dev/da2p1 /dev/da3p1 /dev/da4p1. Now I would like to boot with the new label "BOOT", so I tried this command:
# disklabel -Brw /dev/multipath/BOOT auto
# mount -o rw /dev/multipath/BOOT /

I tried also with this entry in /boot/loader.conf:
Code: 
vfs.root.mountfrom="zfs:/dev/multipath/BOOT"
but it does not work.

How can I use the new label "BOOT" for / at boot via the geom_multipath(8) device?

Is it possible to change the pool from "BootSAN" to "BOOT"?

Code: 
# zpool status output:
  pool: BootSAN
state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        BootSAN                                       ONLINE       0     0     0
          gptid/40ca71b7-6040-11e4-89c5-38eaa715862c  ONLINE       0     0     0
 
You must log in or register to reply here.
Back
Top