ZFS "errors: Permanent errors have been detected in the following files:"

T

thavinci

More joys on ZFS not only has it turned my server into Jelly it seems to be contributing too massive dataloss slowly.



Im getting more and more of the following:

Code: 
zpool status -v
  pool: TwoTB1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        TwoTB1        ONLINE       0     0   416
          label/2tb1  ONLINE       0     0   832

errors: Permanent errors have been detected in the following files:

        /mnt/2tb1/DVD/SBSP_VIKING_SIZED_ADVENTURE_EU.iso
        TwoTB1:<0x47>
        TwoTB1:<0x4c>
        /mnt/2tb1/DVD/NOW_DVD_5.ISO
        /mnt/2tb1/DVD/DARWIN_AWARDS.ISO
        /mnt/2tb1/DVD/BEYONBORDERS.ISO
        /mnt/2tb1/DVD/BEST_OF_BLUE.ISO
        /mnt/2tb1/DVD/STRANGE_WILDERNESS.ISO
        /mnt/2tb1/DVD/FOUR_BROTHERS_EURO.ISO
        /mnt/2tb1/DVD/HONEYMOONERS_EUR.ISO

What could be causing zfs to corrupt files now?
 
You have a single-drive pool? No redundancy whatsoever? Have you at least set copies property to higher than 1? If not, why do you think this is the fault of ZFS, and not a dying harddrive? If ZFS has no redundancy to work with to repair things, all it can do is alert you to corrupted data that it has found.
 
These are brand new drives, and i did not experience such issues before zfs and cannot imagine drives are both failing after a week +- with smart passing perfectly and no relocated sectors for either drive.

Both 2Tb Drives have same issue. Reason why they are simply volumes are because i already have a raidz setup with 6x 1.5tb and was going to wait for a few more 2tb's before proceeding with next phase.
 
Has the computer crashed recently? Have the drives been disconnected while the computer was running? Things like that?
 
Oh damn. Yes the computer crashes daily because of ZFS, i didn't worry too much because everywhere i read it says it can't get corrupt because of COW feature. Was i wrong? Was one of the reasons why i also opted to put single drives on ZFS.
 
thavinci said:
the computer crashes daily because of ZFS
Click to expand...

Are you quite certain of that? I've got two boxes with all of their filesystems on ZFS and they're rock solid.

Or are you using it on i386 without having tuned it?
 
Yup i used to use 8.0-Release AMD64 tried zfs.
Before this all FS where UFS and no issues.
Then Came zfs:
zpool commands would lock up and i would loose control of pool leading to only way out destroying pool. That was a Bug apparently.
Then crashes related to memory but admittedly that was way i tuned zfs, ran out of memory and boom.
Then came huge issues with write speeds after overcoming that one.
Constant lockups http://forums.freebsd.org/showthread.php?t=15710
I cannot write to volume and watch a movie off it...

Inbetween all this i moved to 8.1-RC1,8.1-RC2 and now 8.1-REL

Tried new drives, on different controllers. Single Drives no raidz all same issues.

Then after update to 8.1-Release to try solve that, now the daily crashes.
http://forums.freebsd.org/showthread.php?t=16325


Now My two single zfs volumes is suffering data-loss. Managed to retrieve most files on one, but now the other....

So far ive lost 80% of all my data due to a combination of zfs/FreeBSD and poor manufacturing standards of drives larger than 1Tb.
 
Im not sure what's to blame here. Practically in tears about the data and the fact that i just keep putting more money into this and it simply doesnt work as it's supposed to.

Just as a side note as part of the ordeal i had one nasty surprise during the conversion from UFS.
Ive even had a rather supprising episode where a hard-drive kept getting more bad sectors while in zfs raidz on bsd but worked fine without climbing bad sectors in windows. That turned out to be the Hard-Drive brackets Fan somehow messing things up!!!! Never heard of that one!
 
thavinci said:
Now My two single zfs volumes is suffering data-loss. Managed to retrieve most files on one, but now the other....
Click to expand...

Well, i would look at the snapshots, if you made some. Maybe you can recover some more files. But iam afraid with the video files you wont have any luck because they are usually written only once. If there is no write (after a snapshot) then there is no copy.

thavinci said:
So far ive lost 80% of all my data due to a combination of zfs/FreeBSD and poor manufacturing standards of drives larger than 1Tb.
Click to expand...

I use the same zfs/FreeBSD combination with drives up to 2TB on 3 servers with up to 24TB of disk space. One of them with cache and zil log devices. After configuring them (not as single drive pools) i stress test them for two weeks continuously with 16-32 single processes reading/writing at the same time. The load was high and the disks permanent busy. I saw no hardware problems and no zfs problems like checksum errors. After that i was sure that the zfs/freebsd with 2TB drives will work stable on my hardware.

Only one time i've got one corrupted file but this error was identified as a cash flush error from the controller. A copy of the file from the last backup solved this.

If I were you I would buy a third drive and make a raidz1 with the 3 drives. After stress test the raidz1 pool some days long without errors, making snapshots every day and a scrub every week, the zfs/FreeBSD combination should work good. During the test especially look for harddrive/controller errors and zfs checksum errors.

Dont get me wrong, i wont say there are no problems. But if you rtfm and really test your combination before you used it maybe you wont be in that situation. So please dont blame zfs/freebsd because it works good for a lot of people. First blame yourself because there seems to be no backup ;)
 
Some weird result here was i unplugged both 2tb drives plugged them into same controller card only one at a time and rebooting server.
Did scrub on each drive again individually and now no errors and copied all files off successfully.

No explanation for this but that's what happened. As for having backups. I did have everything on drives and tested for few days before putting the data on however as you can imagine i had 7.5Tb of data initially it was hard keeping it off during testing.

Now as for the zfs/bsd and hardware issues. No doubt i don't just blame bsd & zfs however the recent events have hurt both pocket and drive to actually get it working. Many of the issues weir indeed zfs related. I still cannot get acceptable writing speed without huge pauses in-between that make watching a simple video off drive impossible as it also pauses read process during that. That is zfs/bsd because it occurs with a simple zfs volume, no raid, and not on ufs. As for the daily crash, it MIGHT and i mean MIGHT just have been coincidental that i am actually having memory issues. I did test with a bootable disk and a memory testing programme but have a suspicion it's when temp changes during night that a stick doesn't make contact nicely anymore. I have yet to confirm this and is just my bad luck if this is what's happening that it just happened to start when i commenced zfs testing.

I ALSO during testing had that harddrive sitution where the fan in bay was causing bad sectors on one of the seagate drives in raidz setup. This was endless issues as BSD eventually disconnected the drive because of io issues and that crashed OS. That too me wasn't acceptable. Given it was a hardware issue, but to lock up on removing device? Removing and adding device while system was on often had these results. So as you can see im ranting a bit.


But i have had many issues in getting this off ground and my zfs setup has so far cost me more than i bought my car for!! Hence the frustration.

Id like to know what sort of hardware you are using that it went so smoothly with for you and you truly didn't have any issues?
 
Ive destroyed pool TwoTB1 and am awaiting http://www.zapsonline.com/37547-lian-li-ib-01-5-port-sata-2-and-1x-external-e-sata-pci-express-card-interface-adapter-frac-lib01.html which ive ordered today.

The two 2TB Drives weir on a 2 port Sunnix PCI-E card and id like to maybe bring one of my WD15EADS or WD15EARS back into the picture and do another raidz for now and upgrade it later to a 2TB as-well. Now i know these drives are horrible because of the 4096 sector issue but to be honest it wasnt the cause of the slow write issues so shouldn't be an issue.
 
I dont used any harddisk with 4k blocks until now. The 2TB harddrives i have in use are Hitachi HDS722020ALA330 with native 512 byte blocks. Not the fastest harddrives but they doing there job.

Core2Quad Q6600 @ 2.40GHz
Supermicro X7SBE
4x2GB ECC 800

2x 3ware 9550sxu-8lp with bbu

--

2x Hitachi HTE545025B9A300 (for the system, gmirror on onboard SATA)

6x Hitachi HDT721010SLA360 1TB
6x Hitachi HDS722020ALA330 2TB

--

atm still on FreeBSD 8.0 without any zfs tuning

Code: 
zpool status
  pool: pool1
 state: ONLINE
 scrub: scrub in progress for 2h28m, 15.08% done, 13h54m to go
config:

	NAME        STATE     READ WRITE CKSUM
	pool1       ONLINE       0     0     0
	  raidz1    ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	  raidz1    ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	    da11    ONLINE       0     0     0

errors: No known data errors

zpool iostat -v
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
pool1       4.84T  11.5T  2.58K      6  69.5M  37.7K
  raidz1    2.83T  2.61T  2.32K      2  62.2M  13.4K
    da0         -      -    551      1  15.9M  3.02K
    da1         -      -    641      1  15.2M  2.88K
    da2         -      -    553      1  15.9M  3.01K
    da3         -      -    642      1  15.2M  2.88K
    da4         -      -    555      1  15.9M  3.01K
    da5         -      -    639      1  15.2M  2.87K
  raidz1    2.01T  8.87T    264      4  7.33M  24.3K
    da6         -      -     86      2  2.53M  5.35K
    da7         -      -     91      2  2.43M  5.08K
    da8         -      -     86      2  2.53M  5.34K
    da9         -      -     91      2  2.43M  5.07K
    da10        -      -     86      2  2.53M  5.34K
    da11        -      -     91      2  2.43M  5.06K
----------  -----  -----  -----  -----  -----  -----

--

I started with one 3ware contr. and only 6x 1TB disk with FreeBSD 7.2 amd64
The only big trouble i had after start using ZFS beside of UFS (because i dont trusted ZFS) was a system freeze every 3-5 day while rsync a large amount of files from UFS to ZFS every night. This was a bug i think it is fixed now.

--

An older Broadcom BCM5785 onboard sata controller on a tyan s3992 v1.0 produced a lot of chksum errors while i test it (on 8.0 amd i guess). Well i wasnt interested in finding the bug because this chipset is rare on mainboards and i only have one of them. But i could reproduce the errors on this.
 
Dear User23,

User23 said:
After configuring them (not as single drive pools) i stress test them for two weeks continuously with 16-32 single processes reading/writing at the same time. The load was high and the disks permanent busy. I saw no hardware problems and no zfs problems like checksum errors. After that i was sure that the zfs/freebsd with 2TB drives will work stable on my hardware.
Click to expand...

Could you please provide some details on the stress test? Is it a script or something?

Thank you,

M
 
Slightly off topic...
(Please consider the following not
definitive. I may have missed some
important something).
...
BTW those buggy sata controllers are not
entirely
useless on Freebsd. What I've done is
put a jgournal FS on the target sata disk
and use them for rsync backup with the
BWLIMIT parameter which limits data flow rate.
at the OS before the controller.
...
Also if you have fixed disks with high
reallocated_sector_counts which may soon
expire altogether, you can use them
as WRKDIRPREFIX targets to save space on
/usr during port builds, removing them only
when the builds fail due to filesystem
errors.
 
Update: This might have been memory as only happened late night when temp changed. Reseated Ram , no crash since yet......


Also at User23: Thank you for the extra info, i have a feeling that besides simply bad timing on some of the issues i might have inferior hardware on the controller side and memory. I also have a suspicion that these "corrupted" files might have been because of memory issues?


Also how does one check the checksum errors that the controller produces?
 
mefizto said:
Dear User23,

Could you please provide some details on the stress test? Is it a script or something?

Thank you,

M
Click to expand...

You will need screen but at least iozone.

Iam just using iozone and move the outout into a file, example:

Code: 
screen

Screen version 4.00.03 (FAU) 23-Oct-06

Copyright (c) 1993-2002 Juergen Weigert, Michael Schroeder
Copyright (c) 1987 Oliver Laumann

[...]

                        [Press Space or Return to end.]

Code: 
iozone -R -l 1 -u 8 -r 8k -s 4096m -F /backup1/f0 /backup1/f1 /backup1/f2 /backup1/f3 /backup1/f4 /backup1/f5 /backup1/f6 /backup1/f7 | tee -a /root/iozoneoutput.txt

Code: 
-R Generate Excel report
-l minimum threads at the same time
-u maximum threads at the same time
-r blocksize
-s size each thread will read, write, etc.
-F path and filenames destination of the test (must match -u)

With -l 1 and -u8 iozone will start benchmark with one thread read, write etc 4GB, after what 2x 4GB ... up to 8x 4GB
 
thavinci said:
Update: This might have been memory as only happened late night when temp changed. Reseated Ram , no crash since yet......

Also at User23: Thank you for the extra info, i have a feeling that besides simply bad timing on some of the issues i might have inferior hardware on the controller side and memory. I also have a suspicion that these "corrupted" files might have been because of memory issues?

Also how does one check the checksum errors that the controller produces?
Click to expand...

If the memory is corrupted everything could happend :(

Checksums are used to validate data during reads and writes. Checksumming and data recovery is done at a filesystem level; it is not visible to applications. If a block becomes corrupted on a pool protected by mirroring or RAID, ZFS will identify the correct data value and fix the corrupted value.
 
FreeBSD is an operating system that will push the hardware to it's limits. This is a good thing, because you paid for that hardware to perform. But, with flaky hardware (motherboard, controllers, memory, disks etc) things will just break from time to time --- badly. You will not notice such behavior under Windows and like, because these "operating systems" do not stress the hardware or use various tricks to slow down in hope flaky hardware will not break on you.
ZFS is yet another example of an file system that will stress your hardware. All they way from memory, CPU, controllers and disks.

Glad you found the source of the problem. The good thing is that ZFS actually saved your data! By providing all this "useless" on-disk checksums, cow etc. If you were using UFS or other similar file systems, chances are data would be totally unrecoverable.

I only wonder, why you not use pair of disks as mirror or at least copies>1?
 
You must log in or register to reply here.
Back
Top