Unable to mount one zfs FS from a zpool

E

exseven

Hello, I have a server running FreeBSD 9-current built on March 23rd (generic kernel) with a 3TB striped zpool (6x500gb) drives (used for mirror space). Up until recently this has been okay, with some minimal issues, however last week the machine became unresponsive. I was able to make TCP connections and get ICMP echo responses, but no CLI would come up.

Once the machine rebooted it would no longer mount my ZFS filesystems. I booted into single user mode, was able to mount one of the filesystems (tank/varlog) but the important one (tank/ftp 1.5TB used) would not come up and the system 'froze' again. A zpool scrub came up clean. Rebooted and left the machine to turn and see what would happen if anything. At first I received and error for "not enough space to start" or similar.

I have now disabled ZFS (zfs_enable=NO) and tried over the weekend to mount the offending filesystem and now have error "aacd0 hard error cmd=read", when mounting top(1) was running and would show the zfs process change between tx->tx and zio-> states.

I assume the hard error means one of the hard drives went bad but maybe somebody here would be able to help in case I'm missing something.


Also, Dedup is on for the pool and compression=gzip for tank/varlog only. Nothing out of the ordinary shows in zpool history.


Code: 
 FreeBSD 9.0-CURRENT #1: Thu Mar 24 08:26:39 UTC 2011
    nick@:/usr/obj/usr/src/sys/GENERIC amd64
WARNING: WITNESS option enabled, expect reduced performance.
CPU: Intel(R) Pentium(R) D CPU 3.00GHz (3010.71-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0xf44  Family = f  Model = 4  Stepping = 4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x649d<SSE3,DTES64,MON,DS_CPL,EST,CNXT-ID,CX16,xTPR>
  AMD Features=0x20000800<SYSCALL,LM>
  TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2046611456 (1951 MB)

Code: 
zpool status
  pool: tank
 state: ONLINE
 scan: scrub repaired 0 in 8h15m with 0 errors on Sat Apr  9 02:08:20 2011
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  aacd0     ONLINE       0     0     0
	  aacd1     ONLINE       0     0     0
	  aacd2     ONLINE       0     0     0
	  aacd3     ONLINE       0     0     0
	  ad8       ONLINE       0     0     0
	  ad10      ONLINE       0     0     0

errors: No known data errors

zfs list
NAME          USED  AVAIL  REFER  MOUNTPOINT
tank         1.38T  1.36T    31K  none
tank/ftp     1.37T  1.36T  1.37T  /ftp
tank/varlog  50.0M  9.95G  50.0M  /var/log
 
Hi,

I'd guess it's a disk related hardware issue, i.e. controller or cable or array backplane etc. It's odd that the scrub finished ok though, but actually I have a problem with an eSATA array that I was also able to do a successful scrub but which when ZFS was under load it fails. I know it's hardware as I've had the same disks running in a different array without any issues for a month, the faulty shelf fails within a few hours of use,

ta Andy.
 
I was able to export the zpool and change the 4 disks off the Adaptec PCI-X controller and onto the onboard Marvell SATA (had to add atamarvell to the loader.conf). Importing the pool caused the same issue as I assume that the filesystems are attempting to mount.

The console is now spitting out Poll Timeout errors for one of the disks, I'll have to match it with the AHCI and I guess remove the disk and rebuild the pool.
 
I rebuilt the array as a raidz, same mount points and variables. The array worked great up until the other day when the machine became unresponsive again. I took this opportunity today to rebuildworld and install a new kernel.


The tank/ftp zfs is still unable to mount once more (rebuilt however), when mounting manually CTRL+T shows below and is stuck at that point.

Code: 
sudo zfs mount tank/ftp
load: 0.19  cmd: zfs 1642 [tx->tx_sync_done_cv)] 105.62r 0.00u 1.70s 0% 2504k
load: 0.18  cmd: zfs 1642 [tx->tx_sync_done_cv)] 106.62r 0.00u 1.70s 0% 2504k
load: 0.18  cmd: zfs 1642 [tx->tx_sync_done_cv)] 107.28r 0.00u 1.70s 0% 2504k
 
zdb -DD doesn't seem to be working either right now. I know my dedup ratio was around 1.03 on that filesystem. I'm going to burn a solaris livecd and see if I can mount in that, at least gets me to know if my FS I broke or something up with FreeBSD-current.

Code: 
zdb
tank:
    version: 28
    name: 'tank'
    state: 0
    txg: 4
    pool_guid: 11262982702397889338
    hostid: 2180312168
    hostname: 'less.cogeco.net'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 11262982702397889338
        create_txg: 4
        children[0]:
            type: 'raidz'
            id: 0
            guid: 2796696943285313607
            nparity: 1
            metaslab_array: 30
            metaslab_shift: 34
            ashift: 9
            asize: 1997098975232
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 6658705604927529512
                path: '/dev/aacd0'
                phys_path: '/dev/aacd0'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 4056742938320947436
                path: '/dev/aacd1'
                phys_path: '/dev/aacd1'
                whole_disk: 1
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 11568426481116823656
                path: '/dev/aacd2'
                phys_path: '/dev/aacd2'
                whole_disk: 1
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 2455739288150171986
                path: '/dev/aacd3'
                phys_path: '/dev/aacd3'
                whole_disk: 1
                create_txg: 4
 
Actually, if I leave it long enough I guess there is some output.


Code: 
zdb -DD tank    
DDT-sha256-zap-duplicate: 208552 entries, size 343 on disk, 165 in core
DDT-sha256-zap-unique: 8384127 entries, size 317 on disk, 148 in core

DDT histogram (aggregated over all DDTs):

bucket              allocated                       referenced          
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    8.00M   1003G   1003G   1002G    8.00M   1003G   1003G   1002G
     2     203K   23.9G   23.9G   23.8G     409K   48.2G   48.2G   48.1G
     4      903    108M    108M    108M    3.78K    459M    459M    459M
     8       22   2.38M   2.38M   2.38M      225   24.7M   24.7M   24.7M
    16        3    129K    129K    129K       61   3.64M   3.64M   3.65M
    32       23   2.50M   2.50M   2.50M      761   81.7M   81.7M   81.7M
    64       25   2.84M   2.84M   2.83M    2.05K    241M    241M    241M
   128        2    128K    128K    129K      352   24.0M   24.0M   24.0M
   256        8    896K    896K    896K    2.82K    323M    323M    323M
   512        1    128K    128K    128K      513   64.1M   64.1M   64.1M
    8K        1    128K    128K    128K    8.36K   1.05G   1.05G   1.04G
 Total    8.19M   1.00T   1.00T   1.00T    8.41M   1.03T   1.03T   1.03T

dedup = 1.03, compress = 1.00, copies = 1.00, dedup * compress / copies = 1.03

zdb -DD tank/ftp
Dataset tank/ftp [ZPL], ID 37, cr_txg 12, 1.03T, 1011613 objects
 
Your dedup looks ok to me. I don't think that the problem is there.
I also noticed that your kernel is running with WITNESS option and with 2G of RAM.

How busy is that ftp server ?
Are you limiting your ARC ?
Do you transfer mainly small files ?

George
 
In the last kernel build I removed witness, however the unfortunate 2GB of RAM remains.

The FTP is not that busy, HTTP can get > 400 active file transfers. Average file is about 13MB. I wasn't limiting ARC but using the arc_summary.pl script I could see about 400MB for arcsize.
 
You must log in or register to reply here.
Back
Top