Solved zroot on USB drives

Hello,

I'm facing the following issue with FreeBSD 10.1: I want to install FreeBSD on x3 USB flash drives and have them all mirrored in ZFS. It installs all fine and boots, the problem is that when I unplug one of the USB drivers, the BSD detects it's gone:
Code:
kernel: ugen1.4: <ADATA> at usbus1 (disconnected)
kernel: umass2: at uhub0, port 3, addr 4 (disconnected)
kernel: (da2:umass-sim2:2:0:0): WRITE(10). CDB: 2a 00 00 5c 3c 22 00 00 08 00
kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
kernel: (da2:umass-sim2:2:0:0): Retrying command
kernel: (da2:umass-sim2:2:0:0): WRITE(10). CDB: 2a 00 00 5c 3c 22 00 00 08 00
kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
kernel: (da2:umass-sim2:2:0:0): Retrying command
kernel: (da2:umass-sim2:2:0:0): WRITE(10). CDB: 2a 00 00 5c 3c 22 00 00 08 00
kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
kernel: (da2:umass-sim2:2:0:0): Retrying command
kernel: (da2:umass-sim2:2:0:0): WRITE(10). CDB: 2a 00 00 5c 3c 22 00 00 08 00
kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
kernel: (da2:umass-sim2:2:0:0): Retrying command
kernel: (da2:umass-sim2:2:0:0): WRITE(10). CDB: 2a 00 00 5c 3c 22 00 00 08 00
kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
kernel: (da2:umass-sim2:2:0:0): Error 5, Retries exhausted
kernel: da2 at umass-sim2 bus 2 scbus5 target 0 lun 0
kernel: da2: <ADATA USB Flash Drive 1100> s/n 24327082000200CB detached
Notice the missing
Code:
...Periph destroyed
And after I plug the device back in nothing happens, it's not getting picked up by the OS. No device files get created and it's as if the device was not there. There are no messages in dmesg or /var/log/messages.

All of the devices are partitioned with GPT and ZFS detects that the device has been removed.
Code:
root@sm1:~ # zpool status
  pool: zroot
state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: resilvered 112K in 0h0m with 0 errors on Tue Nov 18 11:37:59 2014
config:

        NAME                     STATE     READ WRITE CKSUM
        zroot                    DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            8189580355720541689  REMOVED      0     0     0  was /dev/gpt/zfs2
            gpt/zfs1             ONLINE       0     0     0
            gpt/zfs0             ONLINE       0     0     0
It does not matter which device from the pool I unplug. The same thing happens to all of them. Removing or detaching the device from the pool does not help either. One thing that I also noticed is that if I boot the BSD with one device attached, and after that I plug and mirror the other they work as expected when I unplug them:
Code:
kernel: (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 5c 35 d2 00 00 08 00
kernel: (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
kernel: (da0:umass-sim0:0:0:0): Retrying command
kernel: (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 5c 35 d2 00 00 08 00
kernel: (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
kernel: (da0:umass-sim0:0:0:0): Retrying command
kernel: (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 5c 35 d2 00 00 08 00
kernel: (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
kernel: (da0:umass-sim0:0:0:0): Retrying command
kernel: da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
kernel: da0: <ADATA USB Flash Drive 1100> s/n 2432708200010198 detached
kernel: (da0:umass-sim0:0:0:0): Periph destroyed
And after I plug it in, I get all the device files as expected. Any help would be appreciated.
 
I found out what is the problem. Since the installation of FreeBSD puts swap partitions on all three drives, and puts them all in /etc/fstab they all get activated at boot time, and this "holds" the device files in FreeBSD, removing the entries from fstab and rebooting the system, and everything works perfectly and as expected.
 
Back
Top