Strange issues with rtorrent and raid0

wonslung · Oct 30, 2010

I have a server with 2 hard drives, which I set up with mirrors for everything except /home, which is raid0 using geom_stripe.

Here is some console output to show my setuop better.

Code:

=>        34  1465149101  ad4  GPT  (699G)
          34         128    1  freebsd-boot  (64K)
         162     2097152    2  freebsd-ufs  (1.0G)
     2097314     2097152    3  freebsd-swap  (1.0G)
     4194466     2097152    4  freebsd-ufs  (1.0G)
     6291618     4194304    5  freebsd-ufs  (2.0G)
    10485922    16777216    6  freebsd-ufs  (8.0G)
    27263138  1437885997    7  freebsd-ufs  (686G)

=>        34  1465149101  ad6  GPT  (699G)
          34         128    1  freebsd-boot  (64K)
         162     2097152    2  freebsd-ufs  (1.0G)
     2097314     2097152    3  freebsd-swap  (1.0G)
     4194466     2097152    4  freebsd-ufs  (1.0G)
     6291618     4194304    5  freebsd-ufs  (2.0G)
    10485922    16777216    6  freebsd-ufs  (8.0G)
    27263138  1437885997    7  freebsd-ufs  (686G)

Code:

/dev/mirror/root              989M    319M    591M    35%    /
devfs                         1.0K    1.0K      0B   100%    /dev
/dev/mirror/usr               7.7G    1.1G    6.1G    15%    /usr
/dev/mirror/var               1.9G    434M    1.4G    24%    /var
/dev/mirror/tmp               989M     18K    910M     0%    /tmp
procfs                        4.0K    4.0K      0B   100%    /proc
devfs                         1.0K    1.0K      0B   100%    /var/named/dev
/dev/stripe/homea             1.3T    1.2T     37G    97%    /usr/home

I noticed that rtorrent kept crashing, so I decided to check my logs and I see a TON of errors....what's strange is, it seems to write data fine, and when I use ftp to fetch the data, it works. I'm wondering if there is anything OTHER than a bad hard drive which could cause this type of error.

Code:

Oct 30 03:33:19 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=1009669442
Oct 30 03:33:19 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=1005984096256, length=65536)]error = 5
Oct 30 03:33:19 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 03:33:19 ns304819 kernel: vm_fault: pager read error, pid 63230 (rtorrent)
Oct 30 03:33:52 ns304819 kernel: pid 63230 (rtorrent), uid 5001: exited on signal 6 (core dumped)
Oct 30 03:35:30 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=862804642
Oct 30 03:35:30 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=855594549248, length=65536)]error = 5
Oct 30 03:35:30 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 03:35:30 ns304819 kernel: vm_fault: pager read error, pid 64145 (rtorrent)
Oct 30 03:35:56 ns304819 kernel: pid 64145 (rtorrent), uid 5001: exited on signal 6 (core dumped)
Oct 30 03:44:20 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=1004455242
Oct 30 03:44:20 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=1000644751360, length=65536)]error = 5
Oct 30 03:44:20 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 03:44:20 ns304819 kernel: vm_fault: pager read error, pid 64169 (rtorrent)
Oct 30 03:45:20 ns304819 kernel: pid 64169 (rtorrent), uid 5001: exited on signal 6 (core dumped)
Oct 30 03:50:08 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=957033554
Oct 30 03:50:08 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=952084938752, length=65536)]error = 5
Oct 30 03:50:08 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 03:50:08 ns304819 kernel: vm_fault: pager read error, pid 64220 (rtorrent)
Oct 30 03:50:32 ns304819 kernel: pid 64220 (rtorrent), uid 5001: exited on signal 6 (core dumped)
Oct 30 03:58:35 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=951747306
Oct 30 03:58:35 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=946671808512, length=65536)]error = 5
Oct 30 03:58:35 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 03:58:35 ns304819 kernel: vm_fault: pager read error, pid 64256 (rtorrent)
Oct 30 03:59:00 ns304819 kernel: pid 64256 (rtorrent), uid 5001: exited on signal 6 (core dumped)
Oct 30 04:01:11 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=1269316978
Oct 30 04:01:11 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=1271863148544, length=16384)]error = 5
Oct 30 04:01:11 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 04:01:11 ns304819 kernel: vm_fault: pager read error, pid 64296 (rtorrent)
Oct 30 04:01:21 ns304819 kernel: pid 64296 (rtorrent), uid 5001: exited on signal 6 (core dumped)
Oct 30 04:15:04 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=1031200706
Oct 30 04:15:04 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=1028032110592, length=65536)]error = 5
Oct 30 04:15:04 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 04:15:04 ns304819 kernel: vm_fault: pager read error, pid 64322 (rtorrent)
Oct 30 04:15:16 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=891555618
Oct 30 04:15:16 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=885035548672, length=16384)]error = 5
Oct 30 04:15:18 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=891555618
Oct 30 04:15:18 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=885035548672, length=16384)]error = 5
Oct 30 04:15:21 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=891555618
Oct 30 04:15:21 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=885035548672, length=16384)]error = 5
Oct 30 04:15:25 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=891555634
Oct 30 04:15:25 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=885035565056, length=16384)]error = 5
Oct 30 04:15:28 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=1205166242
Oct 30 04:15:28 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=1206172794880, length=65536)]error = 5
Oct 30 04:15:28 ns304819 kernel: vnode_pager_getpages: I/O read error
Oct 30 04:15:28 ns304819 kernel: vm_fault: pager read error, pid 64322 (rtorrent)
Oct 30 04:15:30 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=891555634
Oct 30 04:15:30 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=885035565056, length=16384)]error = 5
Oct 30 04:15:32 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=891555634
Oct 30 04:15:32 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=885035565056, length=16384)]error = 5
Oct 30 04:15:39 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=1310655698
Oct 30 04:15:39 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=1314194014208, length=16384)]error = 5
Oct 30 04:15:50 ns304819 kernel: ad6: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=859565682
Oct 30 04:15:50 ns304819 kernel: g_vfs_done():stripe/homea[READ(offset=852277821440, length=16384)]error = 5

And here is the output of dmesg: http://pastie.org/1259565

I would just assume it's a bad hard drive but: everything worked fine with another OS before I installed FreeBSD, and it seems to have no problem writing the data, and reading it when I use an ftp client.

Thanks for any help

richardpl · Oct 30, 2010

Check reported LBA - if you can read and/or write them.

If you can not both of those that means you disk is slowly dying. Any data present on such LBAs can not be recovered. (you need backup)

wonslung · Oct 30, 2010

richardpl said:
Check reported LBA - if you can read and/or write them.

If you can not both of those that means you disk is slowly dying. Any data present on such LBAs can not be recovered. (you need backup)

I'm sorry to ask such an obviously silly question but HOW do I check the reported LBA?

tingo · Oct 30, 2010

If you haven't done so already, you should run smartctl from sysutils/smartmontools on that drive. If smartmontools says the drive is failing, you should replace it before it fails.

richardpl · Oct 30, 2010

wonslung said:
I'm sorry to ask such an obviously silly question but HOW do I check the reported LBA?

Use dd(1) with right offset parameters. Offset is already available so just apply litle math.

If you have time you can run: # dd if=/dev/XXXX of=/dev/null bs=YYYMB and hope that command doesn't make your system unreliable if error happens again.

Strange issues with rtorrent and raid0

wonslung

richardpl

wonslung

tingo

richardpl