Mirror problems on 9.0 to 9.1 upgrade

gjpc · Nov 8, 2013

I updated my 9.0 to 9.1 using freebsd-update yielding me:

Code:

 FreeBSD cserve 9.1-RELEASE-p7 FreeBSD 9.1-RELEASE-p7 #0: Mon Sep  9 21:34:37 UTC 2013     [email]root@amd64-builder.daemonology.net[/email]:/usr/obj/usr/src/sys/GENERIC  amd64

The 9.0 system had three mirrored partitions on two Seagate drives:

Code:

=>        34  1953525101  ada2  GPT  (931G)
          34           6        - free -  (3.0k)
          40         128     1  freebsd-boot  (64k)
         168  1935671296     2  freebsd-ufs  (923G)
  1935671464    17852416     3  freebsd-swap  (8.5G)
  1953523880        1255        - free -  (627k)

The system has mounting problems while booting:

Code:

Nov  8 14:45:45 cserve kernel: ada0 at ata2 bus 0 scbus0 target 0 lun 0
Nov  8 14:45:45 cserve kernel: ada0: <ST1000DM003-9YN162 CC4D> ATA-8 SATA 3.x device
Nov  8 14:45:45 cserve kernel: ada0: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
Nov  8 14:45:45 cserve kernel: ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
Nov  8 14:45:45 cserve kernel: ada0: Previously was known as ad4
Nov  8 14:45:45 cserve kernel: cd0 at ata1 bus 0 scbus3 target 0 lun 0
Nov  8 14:45:45 cserve kernel: cd0: <ATAPI DVD A  DH20A3S 9P53> Removable CD-ROM SCSI-0 device 
Nov  8 14:45:45 cserve kernel: cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 8192bytes)
Nov  8 14:45:45 cserve kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
Nov  8 14:45:45 cserve kernel: ada1 at ata2 bus 0 scbus0 target 1 lun 0
Nov  8 14:45:45 cserve kernel: ada1: <ST33000651NS 0001> ATA-8 SATA 3.x device
Nov  8 14:45:45 cserve kernel: ada1: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
Nov  8 14:45:45 cserve kernel: ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
Nov  8 14:45:45 cserve kernel: ada1: Previously was known as ad5
Nov  8 14:45:45 cserve kernel: ada2 at ata3 bus 0 scbus1 target 0 lun 0
Nov  8 14:45:45 cserve kernel: ada2: <ST1000DM003-9YN162 CC4D> ATA-8 SATA 3.x device
Nov  8 14:45:45 cserve kernel: ada2: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
Nov  8 14:45:45 cserve kernel: ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
Nov  8 14:45:45 cserve kernel: ada2: Previously was known as ad6
Nov  8 14:45:45 cserve kernel: SMP: AP CPU #2 Launched!
Nov  8 14:45:45 cserve kernel: SMP: AP CPU #1 Launched!
Nov  8 14:45:45 cserve kernel: Timecounter "TSC-low" frequency 16405956 Hz quality 800
Nov  8 14:45:45 cserve kernel: GEOM_RAID: Promise: Array Promise created.
Nov  8 14:45:45 cserve kernel: GEOM_MIRROR: Device mirror/p1 launched (1/1).
Nov  8 14:45:45 cserve kernel: GEOM_MIRROR: Device mirror/p2 launched (1/1).
Nov  8 14:45:45 cserve kernel: GEOM_MIRROR: Device mirror/p3 launched (1/1).
Nov  8 14:45:45 cserve kernel: Root mount waiting for: GRAID
Nov  8 14:45:45 cserve last message repeated 29 times
Nov  8 14:45:45 cserve kernel: GEOM_RAID: Promise: Force volume start due to timeout.
Nov  8 14:45:45 cserve kernel: GEOM_RAID: Promise: Disk ada0 state changed from NONE to ACTIVE.
Nov  8 14:45:45 cserve kernel: GEOM_RAID: Promise: Subdisk Logical Drive 1:1-ada0 state changed from NONE to ACTIVE.
Nov  8 14:45:45 cserve kernel: GEOM_RAID: Promise: Volume started.
Nov  8 14:45:45 cserve kernel: Root mount waiting for: GRAID
Nov  8 14:45:45 cserve kernel: GEOM_RAID: Promise: Volume Logical Drive 1 state changed from STARTING to BROKEN.
Nov  8 14:45:45 cserve kernel: Trying to mount root from ufs:/dev/mirror/p2 [rw]...
Nov  8 14:45:45 cserve kernel: .

I fouled things up by performing a gmirror forget for each of the mirror partitions. My gmirror status is now:

Code:

     Name    Status  Components
mirror/p1  COMPLETE  ada2p1 (ACTIVE)
mirror/p2  COMPLETE  ada2p2 (ACTIVE)
mirror/p3  COMPLETE  ada2p3 (ACTIVE)

and my mounted volumes are just plain scary, root is not even shown:

Code:

/dev/mirror/p2 on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
/dev/ad5s1 on /usr/sambashare/craid/Media (ufs, local)

Here is gpart of the all the disks:

Code:

[root@cserve ~]# gpart show  ada0
gpart: No such geom: ada0.
[root@cserve ~]# gpart show  ada1
=>        34  5860533101  ada1  GPT  (2.7T)
          34  5860533101     1  freebsd  (2.7T)

[root@cserve ~]# gpart show  ada2
=>        34  1953525101  ada2  GPT  (931G)
          34           6        - free -  (3.0k)
          40         128     1  freebsd-boot  (64k)
         168  1935671296     2  freebsd-ufs  (923G)
  1935671464    17852416     3  freebsd-swap  (8.5G)
  1953523880        1255        - free -  (627k)

I think I am booted off ada0s1 but not really sure. Any suggestions how I can fix this?

kpa · Nov 8, 2013

The problem is that the disks somehow have Promise soft-RAID metadata on them and the GENERIC kernel on 9.1 has the graid(8) driver enabled by default and it detects the Promise array before gmirror(8) has chance of doing its own tasting.

Try this at the loader prompt:

Code:

set kern.geom.raid.enable=0
boot

If that makes ada0 usable again put the setting in loader.conf(5).

wblock@ · Nov 8, 2013

GEOM_RAID is activated because it found old motherboard RAID metadata on the drives.

Use the sysctl(8) shown by @kpa to temporarily disable the kernel module.
Back up the filesystems. This is not optional. You may have already done it.
Use graid(8) remove or delete to remove the metadata (see http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom-graid.html).
Remove the entry in /boot/loader.conf and reboot.
Test data on the mirrors if they still exist. If not, recreate and restore from the backup in step 2.

gjpc · Nov 9, 2013

Thanks @kpa and @wblock@. @kpa's solution is no joy. I am working on @wblock@'s solution very carefully. (I considered the mirror the backup so I am moving the data off machine.)

I will report on completion.

gjpc · Nov 9, 2013

I have been able to get the system to boot with no errors with no RAID mirror enabled. I have

Code:

kern.geom.raid.enable=0

in loader.conf and changed the fstab to point at the ada0p2.

When I boot the FreeBSD 9.1 CDROM there are RAID error messages that the Promise RAID is broken.

Code:

Root mount waiting for: GRAID  <-- many times
GEOM_RAID: Promise: Force volume stat due to timeout.
GEOM_RAID: Promise: Disk ada0 state changed from NONE to ACTIVE
GEOM_RAID: Promise: Subdisk Logical Drive 1:1-ada0 state changed from NONE to ACTIVE
GEOM_RAID: Promise: Volume Started
Root mount waiting for: GRAID
GEOM_RAID: Promise: Volume Logical Drive 1 state changed from STARTING to BROKEN.
Trying to mount root from cd9660:/dev/iso9660/FREEBSD_INSTALL [ro]

When I enter the shell, graid status returns nothing, there is no RAID meta data that can be removed. All of the gmirror commands return "not available". These are the same results I get on the disk booted system.

Any ideas that may get my mirror back?

gjpc · Nov 11, 2013

Looks like I am in an catch-22 situation. I see no way out of this short of wiping both drives stem to stern and doing a fresh install. I submitted a bug report.

The sad part is that I did the update to have pkg not report:

Code:

/usr/local/sbin/pkg: Undefined symbol "_ThreadRuneLocale"[root@cserve ~]

and it still does.

wblock@ · Nov 11, 2013

Boot with mfsBSD. Turn off graid(8) tasting before it boots. Mount the partitions and copy the data elsewhere. Wipe out the Promise RAID metadata. I don't know how long it is or whether it is at the start or end of the disk, but that information is probably on the net. Repartition the disk and restore the data.

Optional: if you still have the Promise controller, connect the drives to that and use the BIOS option on it to destroy the RAID.

gjpc · Nov 11, 2013

wblock@ said:
Boot with mfsBSD. Turn off graid(8) tasting before it boots. Mount the partitions and copy the data elsewhere. Wipe out the Promise RAID metadata. I don't know how long it is or whether it is at the start or end of the disk, but that information is probably on the net. Repartition disk and restore data.

Optional: if you still have the Promise controller, connect the drives to that and use the BIOS option on it to destroy the RAID.

I think I already said that...

wblock@ · Nov 11, 2013

Said what? That you tried mfsBSD, or that you still had the Promise controller?