Problem with shutdown and external storage

FreeBSD 9 on external HDD (rack) freezes during both shutdown and reboot since upgraded from 9.0 RC3 to 9.0 stable (any version).

On RC3, the reboot command has been working fine and the power button on the laptop even stopped the HDD in the rack before the laptop powered off.

On stable 9.0 both reboot command and shutdown (I press power button on laptop) freeze. I have to press and hold the power button for 4 seconds to turn off the computer after it freezes.

The last lines I see on console:
Code:
usbus4: Controller shutdown complete
usbus5: Controller shutdown
....
usbus6: Controller shutdown complete
usbus7: Controller shutdown
This is where it freezes.

Nothing else is connected to the laptop via USB (maybe the internal webcam).

I have another computer with RC3 and same USB storage racks and reboot works on that. On the laptop I updated and rebuilt the kernel and the world several times since stable is out.

The root filesystem is ZFS with ZFS boot.

Code:
FreeBSD xena 9.0-STABLE FreeBSD 9.0-STABLE #0: Fri Feb 10 15:02:26 CET 2012     root@xena:/usr/obj/usr/src/sys/GENERIC  amd64
Code:
Bus /dev/usb Device /dev/ugen3.3: ID 174c:55aa ASMedia Technology Inc. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x174c ASMedia Technology Inc.
  idProduct          0x55aa 
  bcdDevice            1.00
  iManufacturer           2 DMI
  iProduct                3 External HDD
  iSerial                 1 201007230107
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk (Zip)
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Input/output error
Device Status:     0x0001
  Self Powered
 
UPDATE:

ACPI shutdown and reboot commands work fine after a fresh install of FreeBSD 9 stable on the same machine. However a world upgrade still caused the same issue. I upgraded the system based on the handbook step by step, no cheating except that the second invocation of mergemaster had a -F. All files except passwd and group but including a lot of rc scripts were upgraded in /etc.
 
I can confirm the same issue on another computer after building and installing kernel and world.

The computer has totally different hardware than the first one. The USB storage controllers have different chipsets but are from same brand (Sharkoon). The system isn't installed on the USB drives. There is just another ZFS pool on them for backup.

The installation was done with FreeBSD 9 RC3 disk. I have been able to reboot the system properly until I rebuilt the world.

Not it halts at "usb controller shutdown" and I have to press power button for 4 seconds.
 
Greetings,

How does the halt(8) command work for you? It sounds like there's an endless wait to complete flushing everything to disk on one, or more of your drives. I'm also suspicious that something in your /etc/ has changed after your world upgrade(s). I suppose this would include /usr/local/etc/ as well.

Something else occurs to me. Have you run fsck(8)? Run reboot -s and fsck -f to see if their might be any anomalies — couldn't hurt. :)

Best wishes.

Chris
 
Hello Chris_H,

All my FreeBSD systems run on ZFS root and I often run scrub. I can reproduce this issue with every USB HDD I have: WD Passport, Sharkoon Rapid-Case USB, Sharkoon Rapid-Case USB. It occurs on three different computers of mine. It doesn't even matter if the system is running from the USB HDD or not. Without USB HDD plugged in all my FreeBSD systems shut down properly.
 
I'm seeing what appears to be the same problem on 9.0-STABLE with "normal" root filesystem (UFS), but an external USB drive carrying ZFS on it.

When I try to # shutdown -p now or # reboot now

I get through the Syncing disks, vnodes remaining...stage, then all the USB shuts down "properly" except for usbus4 (the "last" one on my machine).

Code:
Waiting (max 60 seconds) for system process `syncer' to stop...Syncing disks, vnodes remaining...17 17 17 17 17
 8 5 6 6 6 6 6 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 done
All buffers synced.
Uptime: 14h58m35s
usbus0: Controller shutdown
uhub0: at usbus0, port 1, addr 1 (disconnected)
usbus0: Controller shutdown complete
usbus1: Controller shutdown
uhub1: at usbus1, port 1, addr 1 (disconnected)
usbus1: Controller shutdown complete
usbus2: Controller shutdown
uhub2: at usbus2, port 1, addr 1 (disconnected)
usbus2: Controller shutdown complete
usbus3: Controller shutdown
uhub3: at usbus3, port 1, addr 1 (disconnected)
usbus3: Controller shutdown complete
usbus4: Controller shutdown
(Hardware reset switch at this point)

The only thing that looks "fishy" to me in the boot sequence is GEOM: da0: corrupt or invalid GPT detected.
Since the device is fully dedicated to ZFS and was used on other systems prior to its current use, it doesn't surprise me a lot, but it is curious.

That this occurs with conventional (UFS) root suggests that it is a ZFS-related problem, not something about the ZFS root configuration.

Code:
CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1600.03-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x106c2  Family = 6  Model = 1c  Stepping = 2
  Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,
  SSE2,SS,HTT,TM,PBE>
  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
Code:
uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0xb880-0xb89f
 irq 20 at device 29.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0xbc00-0xbc1f irq 17 at device 29.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0xc000-0xc01f irq 18 at device 29.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0xc080-0xc09f irq 19 at device 29.3 on pci0
uhci3: LegSup = 0x2f00
usbus3 on uhci3
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xefdffc00-0xefdfffff irq 20 at device 29.7 on pci0
usbus4: EHCI version 1.0
usbus4 on ehci0
Code:
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
ada0 at ata2 bus 0 scbus1 target 0 lun 0
ada0: <SAMSUNG HD103UJ 1AA01113> ATA-7 SATA 2.x device
ada0: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ata2 bus 0 scbus1 target 1 lun 0
ada1: <OCZ-VERTEX2 1.29> ATA-8 SATA 2.x device
ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada1: 57241MB (117231408 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad5
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
uhub4: 8 ports with 8 removable, self powered
Root mount waiting for: usbus4
Root mount waiting for: usbus4
ugen4.2: <Western Digital> at usbus4
umass0: <Western Digital My Passport 071A, class 0/0, rev 2.00/20.11, addr 2> on usbus4
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:4:0:-1: Attached to scbus4
Trying to mount root from ufs:/dev/ada0s2a [rw]...
da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
da0: <WD My Passport 071A 2011> Fixed Direct Access SCSI-4 device 
da0: 40.000MB/s transfers
da0: 305218MB (625086464 512 byte sectors: 255H 63S/T 38909C)
ses0 at umass-sim0 bus 0 scbus4 target 0 lun 1
ses0: <WD SES Device 2011> Fixed Enclosure Services SCSI-4 device 
ses0: 40.000MB/s transfers
ses0: SCSI-3 SES Device
GEOM: da0: corrupt or invalid GPT detected.
GEOM: da0: GPT rejected -- may not be recoverable.
ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable,
            add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 5
ZFS storage pool version 28
re0: link state changed to UP
 
I can replicate the problem with UFS root/filesystem and with the only USB device attached being a flash card in reader used for ZFS. There is no funky pre-partitioned WD external drive involved and nothing "fishy" in the boot log.

# dd if=/dev/zero of=/dev/da1 bs=64k

# zpool create stick /dev/da1

is enough to get the hang on shutdown / reboot

Code:
[jeff@port16 ~]$ uname -a
FreeBSD port16.pn.wagsky.com 9.0-STABLE FreeBSD 9.0-STABLE #0: Mon Apr 23 08:07:43 PDT 2012     
root@port16.pn.wagsky.com:/usr/obj/usr/src/sys/JMK.2012-04-23  i386
[jeff@port16 ~]$ cat /usr/src/sys/i386/conf/JMK.2012-04-23
include		GENERIC

ident		JMK-2012-04-23

nocpu		I486_CPU
nocpu		I586_CPU

# For ZFS

options		KVA_PAGES=512

# For vimage in jails

nooptions	SCTP
options		VIMAGE

Has someone already opened a gnats ticket?
 
http://www.freebsd.org/cgi/query-pr.cgi?pr=167685

Working on a memstick build of the failing version, as well as one using src/release/generate-release.sh to see if this can be made easier to reproduce.

Edit -- Yes, if you build a memstick, you don't even need to install to replicate this. Just boot, go Live CD, then
Code:
dd if=/dev/zero of=/dev/da1 bs=64k
zpool create stick /dev/da1
reboot now

Bisecting now...
 
Still working on this, but bisection is suggesting that


Code:
r229100 | hselasky | 2011-12-31 06:33:15 -0800 (Sat, 31 Dec 2011) | 6 lines

MFC r228709, r228711 and r228723:
- Add missing unlock of USB controller's lock, when
doing shutdown, suspend and resume.
- Add code to wait for USB shutdown to be executed at system shutdown.
- Add sysctl which can be used to skip this waiting.

may have caused the problem to become visible (the problem, I am guessing, is that ZFS is not letting loose of the USB system).

As a workaround (unconfirmed), you might want to try
# sysctl hw.usb.no_shutdown_wait=1
 
Interesting. I'm seeing this on a new Acer 4752-6431 with UFS root and ZFS on a USB drive while a six year old Acer 3623NWxMI has no issues whatsoever. As I'm copying my FreeBSD to the new computer I'll see if it hangs after install and without the USB drive.
 
I'm relieved to see I am not alone :)

I am now seeing the same problem on my home server:

Code:
sim@area51> uname -a
FreeBSD area51 9.0-STABLE FreeBSD 9.0-STABLE #3: Sat Jun  2 00:17:51 BST 2012     
root@area51:/usr/obj/usr/src/sys/AREA51  amd64

fitted with a 2TB Hitachi external USB 3.0 drive over a USB 3.0 interface card:

Code:
umass1: <vendor 0x4971 product 0x1013, class 0/0, rev 3.00/48.96, addr 2> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0x0000
umass1:7:1:-1: Attached to scbus7
da1 at umass-sim1 bus 1 scbus7 target 0 lun 0
da1: <Hitachi Hitachi HDS72302 A800> Fixed Direct Access SCSI-4 device 
da1: 400.000MB/s transfers
da1: 1907727MB (3907024896 512 byte sectors: 255H 63S/T 243201C)

The external drive contains a simple ZFS pool on an encrypted GELI:

Code:
  pool: zext
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zext        ONLINE       0     0     0
          da1.eli   ONLINE       0     0     0

errors: No known data errors

Similar symptoms: the shutdown process hangs at:

Code:
usbus0: Controller Shutdown


I haven't looked into it in detail yet, although I'll probably end up wrenching my back to hold down the awkwardly positioned power button sooner or later... :\

sim
 
Setting
Code:
hw.usb.no_shutdown_wait=1
fixes this issue for me, so looks like there's a bug in the code introduced by r229100 (8.3-RELEASE).
 
  • Thanks
Reactions: sim
I just ran across this problem -- any idea what version of FreeBSD it's going to be fixed in?

(The workaround works, obviously, but I prefer to leave the core configuration as close to default as possible).
 
Back
Top