ZFS import pool ahcich0: timeout (NanoBSD 8.2 -> 9.1)

allan_sundry · Mar 20, 2013

Hi,

Many times when I had to make changes to the system, I boot from the USB drive to the new build NanoBSD then import the pool to the system. In the transition from 8.2 to 9.1, I do the same - boot from the USB drive new build NanoBSD 9.1. When I import the pool in the new system with:

Code:

# zfs import tank

system for about 10 seconds does not respond, then the message:

Code:

ahcich0: Timeout on slot 0 port 0
ahcich0: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 58 serr 00000000 cmd 0000c017
(ada0:ahcich0:0:0:0): READ_MUL. ACB: C4 00 00 00 00 40 00 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: Command timeout
(ada0:ahcich0:0:0:0): Retrying command

message is repeated.

Please help me with a problem importing a pool in the new system.

System:
- NanoBSD 8.2->9.1 on USB drive
- Jails on ZFS mirror

ZFS v15

Code:

# zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank   464G  11.8G   452G     2%  ONLINE  -

Code:

# zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada1    ONLINE       0     0     0

errors: No known data errors

In NanoBSD 8.2 I used ATA_CAM:

Code:

# ATA controllers
device          ahci            # AHCI-compatible SATA controllers
device          ata             # Legacy ATA/SATA controllers
options         ATA_CAM         # Handle legacy controllers with CAM
options         ATA_STATIC_ID   # Static device numbering
device          mvs             # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA
device          siis            # SiliconImage SiI3124/SiI3132/SiI3531 SATA

In NanoBSD 9.1 ATA_CAM has been used by default.

TUNING:

Code:

# cat /boot/loader.conf
# BOOT
loader_logo=none          # Disable the loader logo
autoboot_delay=2          # Reduce the autoboot delay to 2 seconds
hw.ata.atapi_dma=0        # Disable DMA for ATAPI (CD) drive
hw.ata.ata_dma=0          # Disable DMA for hard-drive and CF
hw.ata.wc=0               # Disable disk write caching
net.fibs=16               # Numbers of FIB
kern.cam.boot_delay=10000 # Fix booting from USB device bug

# GEOM 
kern.geom.raid.enable=0   # Fix GEOM_RAID in GENERIC

HARDWARE:

Code:

    ahci0@pci0:0:31:2:      class=0x010601 card=0x868015d9 chip=0x26818086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'LSI LOGIC, 62089A2, LSISAS1068 B0, T 0620, WE 119200.1 (62089A2)'
    class      = mass storage
    subclass   = SATA
    cap 01[70] = powerspec 2  supports D0 D3  current D0
    cap 12[a8] = SATA Index-Data Pair

Code:

# dmesg | grep ahci 
ahci0: <Intel ESB2 AHCI SATA controller> port 0x18c0-0x18c7,0x1894-0x1897,0x1898-0x189f,0x1890-0x1893,0x18a0-0x18bf mem 0xd8500400-0xd85007ff irq 19 at device 31.2 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0

Code:

# smartctl -a /dev/ada0
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 7K500
Device Model:     HDS725050KLA360
Serial Number:    KRVN0AZBGKP2RJ
LU WWN Device Id: 5 000cca 20ec80978
Firmware Version: K2AOAB0A
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 1
Local Time is:    Wed Mar 20 09:59:39 2013 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   160   160   050    Pre-fail  Offline      -       202
  3 Spin_Up_Time            0x0007   140   140   024    Pre-fail  Always       -       499 (Average 499)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       104
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   020    Pre-fail  Offline      -       54
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       20807
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       102
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       463
193 Load_Cycle_Count        0x0012   100   100   050    Old_age   Always       -       463
194 Temperature_Celsius     0x0002   157   157   000    Old_age   Always       -       35 (Min/Max 16/52)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   253   000    Old_age   Always       -       0

Code:

# smartctl -a /dev/ada1
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 7K500
Device Model:     HDS725050KLA360
Serial Number:    KRVP67ZBGTLUEF
LU WWN Device Id: 5 000cca 20ecb2f8e
Firmware Version: K2AOAB0A
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 1
Local Time is:    Wed Mar 20 09:59:41 2013 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   099   099   016    Pre-fail  Always       -       131072
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       110
  3 Spin_Up_Time            0x0007   150   150   024    Pre-fail  Always       -       393 (Average 540)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       92
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   134   134   020    Pre-fail  Offline      -       32
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       20476
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       90
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       454
193 Load_Cycle_Count        0x0012   100   100   050    Old_age   Always       -       454
194 Temperature_Celsius     0x0002   141   141   000    Old_age   Always       -       39 (Min/Max 19/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   253   000    Old_age   Always       -       0

allan_sundry · Mar 20, 2013

Because with NanoBSD 8.2 + ZFS server already spent about a year I think the issue is not related to hardware.
I suspect that the problem may be in the parameter vfs.zfs.txg.timeout:

NanoBSD 8.2:

Code:

# sysctl -a | grep vfs.zfs.txg.timeout
vfs.zfs.txg.timeout: 30

FreeBSD 9.1:

Code:

# sysctl -a | grep vfs.zfs.txg.timeout
vfs.zfs.txg.timeout: 5

olav · Mar 20, 2013

I've had that problem several times. I has always been a bad sata/ide cable causing it. Try to replace it.

allan_sundry · Mar 21, 2013

Today to test the boot with mfsBSD 9.1 amd64 - pool imported without problems.
Ie problem somewhere in the image NanoBSD 9.1 amd64.

NANOBSD.CONF

Code:

NANO_PMAKE="make -j 4"

CONF_WORLD='
CC=clang
CXX=clang++
CPP=clang-cpp
NO_WERROR=
WERROR=
NO_FSCHG=
'

CONF_WORLD='
WITHOUT_ACCT=
WITHOUT_APM=
WITHOUT_ASSERT_DEBUG=
WITHOUT_ATM=
WITHOUT_AUDIT=
WITHOUT_AUTHPF=
WITHOUT_BIND=
WITHOUT_BIND_DNSSEC=
WITHOUT_BIND_ETC=
WITHOUT_BIND_LIBS_LWRES=
WITHOUT_BIND_MTREE=
WITHOUT_BIND_NAMED=
WITHOUT_BIND_UTILS=
WITHOUT_BLUETOOTH=
WITHOUT_CALENDAR=
WITHOUT_CTM=
WITHOUT_CVS=
WITHOUT_DICT=
WITHOUT_FLOPPY=
WITHOUT_FREEBSD_UPDATE=
WITHOUT_GAMES=
WITHOUT_GCOV=
WITHOUT_GDB=
WITHOUT_GPIB=
WITHOUT_HTML=
WITHOUT_IPFILTER=
WITHOUT_IPX=
WITHOUT_IPX_SUPPORT=
WITHOUT_KERBEROS=
WITHOUT_KERBEROS_SUPPORT=
WITHOUT_LIB32=
WITHOUT_LPR=
WITHOUT_NCP=
WITHOUT_NDIS=
WITHOUT_NIS=
WITHOUT_NLS=
WITHOUT_NLS_CATALOGS=
WITHOUT_NS_CACHING=
WITHOUT_OPENSSH=
WITHOUT_QUOTAS=
WITHOUT_RCMDS=
WITHOUT_RESCUE=
WITHOUT_SENDMAIL=
WITHOUT_SSP=
WITHOUT_WIRELESS=
WITHOUT_WIRELESS_SUPPORT=
WITHOUT_WPA_SUPPLICANT_EAPOL=
'

NANO_MEDIASIZE=15665140
NANO_DRIVE=da0
NANO_KERNEL=NANOBSD
NANO_CONFSIZE=204800
NANO_RAM_ETCSIZE=204800
NANO_RAM_TMPVARSIZE=204800
NANO_SECTS=63
NANO_HEADS=255
NANO_BOOTLOADER="boot/boot0"

# ADD PORTS
add_port () {
        port=`echo $1 | sed -e 's/\//_/'`
                # Check if package allready exist
                # Need to:
                # 1. check ARCH of this package!
                # 2. Add a trap
                cd /usr/ports/$1
                PKG_NAME=`make package-name`
                if [ -f ${NANO_OBJ}/ports/packages/All/${PKG_NAME}.tbz ]; then
                # Pkg file found: Generate add_pkg_NAME function
                        eval "
                                add_pkg_${port} () {
                                        #Need to create ${NANO_OBJ}/ports in this add_pkg_${port} function
                                        if [ ! -d \${NANO_OBJ}/ports ]; then
                                                mkdir -p \${NANO_OBJ}/ports/distfiles
                                                mkdir -p \${NANO_OBJ}/ports/packages
                                        fi
                                        mkdir -p \${NANO_WORLDDIR}/usr/ports/packages
                                        mkdir -p \${NANO_WORLDDIR}/usr/ports/distfiles
                                        mount -t unionfs -o noatime \${NANO_OBJ}/ports/packages \
                                                \${NANO_WORLDDIR}/usr/ports/packages
                                        mount -t unionfs -o noatime \${NANO_OBJ}/ports/distfiles \
                        \${NANO_WORLDDIR}/usr/ports/distfiles
                                        chroot \${NANO_WORLDDIR} /bin/sh -exc \
                                'cd /usr/ports/packages/All;pkg_add -F ${PKG_NAME}.tbz'
                                        umount \${NANO_WORLDDIR}/usr/ports/packages
                                        umount \${NANO_WORLDDIR}/usr/ports/distfiles
                                        rmdir \${NANO_WORLDDIR}/usr/ports/packages
                                        rmdir \${NANO_WORLDDIR}/usr/ports/distfiles
                                        rmdir \${NANO_WORLDDIR}/usr/ports
                                }
                        customize_cmd add_pkg_${port}
                "
                else
                # No pkg file: Generate add_port_NAME function
                eval "
                        add_port_${port} () {
                                        #Need to create ${NANO_OBJ}/ports in this add_port_${port} function
                                        if [ ! -d \${NANO_OBJ}/ports ]; then
                                                mkdir -p \${NANO_OBJ}/ports/distfiles
                                                mkdir -p \${NANO_OBJ}/ports/packages
                                        fi
                        mkdir -p \${NANO_WORLDDIR}/usr/ports/packages
                                        mkdir -p \${NANO_WORLDDIR}/usr/ports/distfiles
                        mount -t unionfs -o noatime \${NANO_SRC} \
                       \${NANO_WORLDDIR}/usr/src
                        mount -t unionfs -o noatime /usr/ports \
                       \${NANO_WORLDDIR}/usr/ports
                                        mount -t unionfs -o noatime \${NANO_OBJ}/ports/packages \
                                                \${NANO_WORLDDIR}/usr/ports/packages
                                        mount -t unionfs -o noatime \${NANO_OBJ}/ports/distfiles \
                        \${NANO_WORLDDIR}/usr/ports/distfiles
                        mkdir -p \${NANO_WORLDDIR}/dev
                        mount -t devfs devfs \${NANO_WORLDDIR}/dev
                        mkdir -p \${NANO_WORLDDIR}/usr/pobj
                        mkdir -p \${NANO_WORLDDIR}/usr/workdir
                        cp /etc/resolv.conf \${NANO_WORLDDIR}/etc/resolv.conf
                                        # Improvement: Don't know why package-recursive don't works here
                                        # If a port have NO_PACKAGE, comment-it
                    if grep -q NO_PACKAGE \${NANO_WORLDDIR}/usr/ports/$1/Makefile; then
                        mv \${NANO_WORLDDIR}/usr/ports/$1/Makefile \${NANO_WORLDDIR}/usr/ports/$1/Makefile.nanobsd
                        sed 's/NO_PACKAGE/\#NO_PACKAGE/g' \${NANO_WORLDDIR}/usr/ports/$1/Makefile.nanobsd > \${NANO_WORLDDIR}/usr/ports/$1/Makefile
                    fi
                        chroot \${NANO_WORLDDIR} /bin/sh -exc \
                        'env TARGET=${NANO_ARCH} TARGET_ARCH=${NANO_ARCH} MACHINE=${NANO_ARCH} UNAME_p=${NANO_ARCH} UNAME_m=${NANO_ARCH} \
                                                make WRKDIRPREFIX=/usr/workdir FETCH_ARGS=-pRr -C /usr/ports/$1 \
                        package BATCH=yes -DWITHOUT_X11 -DNOPORTDOCS -DNO_INSTALL_MANPAGES -DNOPORTEXAMPLES -DWITHOUT_NLS $2 clean'
                        rm \${NANO_WORLDDIR}/etc/resolv.conf
                        rm -rf \${NANO_WORLDDIR}/usr/obj
                        rm -rf \${NANO_WORLDDIR}/usr/pobj
                        rm -rf \${NANO_WORLDDIR}/usr/workdir
                        umount \${NANO_WORLDDIR}/dev
                                        umount \${NANO_WORLDDIR}/usr/ports/packages
                                        umount \${NANO_WORLDDIR}/usr/ports/distfiles
                        umount \${NANO_WORLDDIR}/usr/ports
                        umount \${NANO_WORLDDIR}/usr/src
                                        rmdir \${NANO_WORLDDIR}/usr/ports/packages
                                        rmdir \${NANO_WORLDDIR}/usr/ports/distfiles
                        rmdir \${NANO_WORLDDIR}/usr/ports
                        }
                        customize_cmd add_port_${port}
                "
                fi
}

# PORTS
add_port "net/quagga" 
add_port "security/openssh-portable"
add_port "sysutils/ezjail"
add_port "sysutils/zfs-stats"
add_port "net/openntpd"

# CUSTOMIZE
my_custom () {
        # Relocate /root/.ssh to /etc/dot.root.ssh
        # This permit to save ssh keys (only /etc and /usr/local/etc are saved)
        mkdir -p ${NANO_WORLDDIR}/etc/dot.ssh.root
        ln -s ../etc/dot.ssh.root ${NANO_WORLDDIR}/root/.ssh
        # Relocate /var/cron to /etc/cron
        # This permit to save crontab (only /etc and /usr/local/etc are saved)
        mkdir -p ${NANO_WORLDDIR}/etc/cron
        rm -rf ${NANO_WORLDDIR}/var/cron
        ln -s ../etc/cron ${NANO_WORLDDIR}/var/cron
        # Relocate /home and /usr/home to /etc/home (only /etc and /usr/local/etc are saved)
        chroot ${NANO_WORLDDIR} /bin/sh -exc \
        'mkdir -p /etc/home && ln -s /etc/home /home && ln -s /etc/home /usr/home'
        # Relocate /boot/zfs to /etc/zfs
        mkdir -p ${NANO_WORLDDIR}/etc/zfs
        rmdir ${NANO_WORLDDIR}/boot/zfs
        ln -s ../etc/zfs ${NANO_WORLDDIR}/boot/zfs
        # Set Quagga permission
        chroot ${NANO_WORLDDIR} /bin/sh -exc \
        'chown -R quagga:quagga /usr/local/etc/quagga'

        #------ Copy config to image
        cp ${NANO_TOOLS}/Customize/Files/boot/loader.conf ${NANO_WORLDDIR}/boot/loader.conf
        cp ${NANO_TOOLS}/Customize/Files/etc/sysctl.conf ${NANO_WORLDDIR}/etc/sysctl.conf

customize_cmd my_custom

KERNEL 8.2/9.1:

Code:

 # add Generic kernel
include    GENERIC
ident      NANOBSD

options    IPFIREWALL
options    IPFIREWALL_VERBOSE
options    IPFIREWALL_VERBOSE_LIMIT=5
options    IPFIREWALL_DEFAULT_TO_ACCEPT
options    IPDIVERT
options    IPFIREWALL_FORWARD
options    IPFIREWALL_NAT
options    LIBALIAS
options    DUMMYNET
options    HZ=1000

# QUAGGA with MD5
options TCP_SIGNATURE
options IPSEC
device crypto

# ATA controllers (for 8.2 only)
device          ahci            
device          ata             
options         ATA_CAM         
options         ATA_STATIC_ID   
device          mvs             
device          siis

TUNING:
- /boot/loader.conf 8.2/9.1

Code:

loader_logo=none          
autoboot_delay=2          
hw.ata.atapi_dma=0        
hw.ata.ata_dma=0          
hw.ata.wc=0               
net.fibs=16               
kern.cam.boot_delay=10000 
kern.geom.raid.enable=0

- /etc/sysctl.conf 8.2/9.1

Code:

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.drop_redirect=1
net.inet.icmp.log_redirect=1
net.inet.ip.redirect=0
net.inet.tcp.drop_synfin=1

net.inet.ip.forwarding=1

kern.coredump=0

allan_sundry · Mar 29, 2013

Today is no longer available one from the hard drive - I think that its periodic disappearance of the system led to errors when importing the pool. Error was hardware and coincided with the update system.

ZFS import pool ahcich0: timeout (NanoBSD 8.2 -> 9.1)

allan_sundry

allan_sundry

olav

allan_sundry

allan_sundry