ZFS data corruption problem

pkubaj · Feb 5, 2012

I have recently attached a new disk to my ZFS pool. Before, it was running only one disk, now it's two. (I added it as a storage, not a mirror) That's why I haven't got used to it, and, while dd'ing, I chose the new HDD for an output. dd has finished correctly, which obviously led to data corruption. Since I didn't use my whole firsst disk, the OS booted up correctly. But there are lots of errors. Output of zfs status -v while doing zfs scrub tank0:

Code:

 pool: tank0
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scan: scrub in progress since Sun Feb  5 19:28:05 2012
    9.63G scanned out of 106G at 12.7M/s, 2h9m to go
    332K repaired, 9.09% done
config:

	NAME        STATE     READ WRITE CKSUM
	tank0       ONLINE       0     0  380K
	  ada0s1d   ONLINE       0     0     0
	  ada1      ONLINE       0     0  760K  (repairing)

errors: Permanent errors have been detected in the following files:

        tank0:<0x946>
        tank0:<0x2ac1>
        /var/db/pkg/pixman-0.24.2/+CONTENTS
        /var/db/pkg/avahi-app-0.6.29_1/+CONTENTS
        /var/db/pkg/cairo-1.10.2_3,1/+CONTENTS
        /var/db/pkg/cairomm-1.10.0/+CONTENTS
        /var/db/pkg/conky-1.8.1_4/+CONTENTS
        /var/db/pkg/consolekit-0.4.3/+CONTENTS
        tank0/var:<0x883d>
        /var/db/pkg/gobject-introspection-0.10.8/+CONTENTS
        /var/db/pkg/gstreamer-0.10.35/+CONTENTS
        /var/db/pkg/gstreamer-plugins-0.10.35_1,3/+CONTENTS
        /var/db/pkg/gstreamer-plugins-bad-0.10.22,3/+CONTENTS
        /var/db/pkg/gstreamer-plugins-good-0.10.30,3/+CONTENTS
        /var/db/pkg/gstreamer-plugins-ogg-0.10.35,3/+CONTENTS
        /var/db/pkg/gstreamer-plugins-theora-0.10.35,3/+CONTENTS
        /var/db/pkg/gstreamer-plugins-vorbis-0.10.35,3/+CONTENTS
        /var/db/pkg/gstreamer-plugins-vp8-0.10.22_1,3/+CONTENTS
        /var/db/pkg/hal-0.5.14_18/+CONTENTS
        /var/db/pkg/xorg-macros-1.16.1/+CONTENTS
        /var/db/pkg/p5-Glib2-1.250/+CONTENTS
        /var/db/pkg/gmp-5.0.3/+CONTENTS
        /var/db/pkg/openbox-3.5.0/+CONTENTS
        /var/db/pkg/openbox-themes-1.0.1_5/+CONTENTS
        /var/db/pkg/p5-Cairo-1.090/+CONTENTS
        /var/db/pkg/p5-Pango-1.223/+CONTENTS
        /var/db/pkg/pango-1.28.4/+CONTENTS
        /var/db/pkg/pangomm-2.28.2/+CONTENTS
        /var/db/pkg/glib-networking-2.28.7_3/+CONTENTS
        /var/db/pkg/polkit-0.99/+CONTENTS
        /var/db/pkg/poppler-0.18.0/+CONTENTS
        /var/db/pkg/gnutls-2.12.16/+CONTENTS
        /var/db/pkg/poppler-glib-0.18.0/+CONTENTS
        tank0/var:<0x718f>
        /var/db/pkg/py27-cairo-1.8.10/+CONTENTS
        /var/db/pkg/libsoup-2.34.3/+CONTENTS
        /var/db/pkg/py27-gobject-2.28.6/+CONTENTS
        /var/db/pkg/mpfr-3.1.0_2/+CONTENTS
        /var/db/pkg/nettle-2.4/+CONTENTS
        /var/db/pkg/webcamd-3.2.0.2/+CONTENTS
        /var/db/pkg/bison-2.5,1/+CONTENTS
        tank0/var:<0x82e2>
        tank0/var:<0x82e5>
        tank0/var:<0x82e8>
        tank0/var:<0x82eb>
        tank0/var:<0x82ee>
        tank0/var:<0x83ee>
        tank0/var:<0x82f1>
        /usr/local/include/nss/nss/secder.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib.pm
        /usr/local/include/nss/nss/nssckfwc.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/GenPod.pm
        /usr/local/include/nss/nss/hasht.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/OptionContext.pod
        /usr/local/include/nss/nss/seccomon.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/BookmarkFile.pod
        /usr/local/include/nss/nss/preenc.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Error.pod
        /usr/local/include/nss/nss/sslproto.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/index.pod
        /usr/local/include/nss/nss/crmft.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/MakeHelper.pm
        /usr/local/include/nss/nss/nssilock.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/ParamSpec.pod
        /usr/local/include/nss/nss/pk11pqg.h
        tank0/usr:<0x122d08>
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/ParseXSDoc.pm
        /usr/local/include/nss/nss/secdert.h
        tank0/usr:<0x122d09>
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Markup.pod
        /usr/home/pkubaj/.opera/optrust.dat
        tank0/usr:<0x12200a>
        /usr/local/include/nss/nss/ecl-exp.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Flags.pod
        /usr/local/include/nss/nss/pkcs11t.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Object.pod
        /usr/local/include/nss/nss/jar.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/MainLoop.pod
        /usr/local/include/nss/nss/key.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Type.pod
        /usr/local/include/nss/nss/keythi.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Log.pod
        /usr/local/include/nss/nss/secerr.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/xsapi.pod
        /usr/local/include/nss/nss/blapit.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Signal.pod
        /usr/local/include/nss/nss/pkcs7t.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/devel.pod
        /usr/local/include/nss/nss/pk11priv.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/KeyFile.pod
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Boxed.pod
        /usr/local/include/nss/nss/certt.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/OptionGroup.pod
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/version.pod
        /usr/local/include/nss/nss/utilrename.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/CodeGen.pm
        /usr/local/include/nss/nss/keyt.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Utils.pod
        /usr/local/include/nss/nss/secasn1t.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Object/Subclass.pm
        /usr/local/include/nss/nss/jar-ds.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/UInt64.pod
        /usr/local/include/nss/nss/secoid.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/UInt.pod
        /usr/local/include/nss/nss/pkcs12t.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/GType.pod
        /usr/local/include/nss/nss/nssilckt.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/Enum.pod
        /usr/local/include/nss/nss/nssckfwt.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/Int64.pod
        /usr/local/include/nss/nss/pkcs11u.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/Double.pod
        /usr/local/include/nss/nss/nssrwlk.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/Flags.pod
        /usr/local/include/nss/nss/crmf.h
        /usr/local/lib/perl5/site_perl/5.14.2/mach/Glib/Param/Int.pod
        /usr/local/include/nss/nss/ciferfam.h
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::devel.3
        /usr/local/include/nss/nss/nssb64t.h
        /usr/local/lib/perl5/5.14.2/man/man3/Glib.3
        /usr/local/include/nss/nss/nssckbi.h
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::BookmarkFile.3
        tank0/usr:<0x104523>
        /usr/local/include/nss/nss/pk11pub.h
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::MainLoop.3
        /usr/local/include/nss/nss/cert.h
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::ParamSpec.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Utils.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Object::Subclass.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Param::GType.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Flags.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Object.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Log.3
        tank0/usr:<0x122d2c>
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Param::UInt.3
        tank0/usr:<0x98d2c>
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Param::UInt64.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Type.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::xsapi.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Param::Flags.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::ParseXSDoc.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Error.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::OptionContext.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::OptionGroup.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::index.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::CodeGen.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::MakeHelper.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::version.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Param::Int.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::KeyFile.3
        /usr/local/lib/perl5/5.14.2/man/man3/Glib::Signal.3

It's still running, of course, there are now 1740 data errors. I understand that recompiling all ports dependent on perl5, libxul etc. and deleting configs of opera and the others will help, but what about errors like:

Code:

tank0/var:<0x883d>
tank0/usr:<0x122d5e>

Is there anything I can do besides reinstalling?

phoenix · Feb 6, 2012

At this point, your best bet is to reinstall, or at least destroy/recreate the ZFS pool. And to configure the drives as a mirror vdev.

Since there's no redundancy in the pool, there's no way to fix the corrupted files short of recreating them from backups.

If you had used zpool attach instead of zpool add, then this would be fixable.

pkubaj · Feb 6, 2012

There's no zfs attach, so I guess you meant zpool attach. The problem is that, according to the man page, zpool attach creates a mirror. It's a great thing, and that's what I would prefer, but for some reason no OS can boot from the new HDD (I've tried even Windows). It has no bad blocks and is in a good shape. So I'm forced to use it only as a storage space.

gkontos · Feb 6, 2012

pkubaj said:
There's no zfs attach, so I guess you meant zpool attach. The problem is that, according to the man page, zpool attach creates a mirror. It's a great thing, and that's what I would prefer, but for some reason no OS can boot from the new HDD (I've tried even Windows). It has no bad blocks and is in a good shape. So I'm forced to use it only as a storage space.

If no OS can boot from this drive, then there is something wrong with that drive. So, don't use it for your data!

pkubaj · Feb 6, 2012

Well, Windows said that it was something wrong with computer's hardware (BIOS related). The thing is, any other drive boots up properly...

adri · Feb 7, 2012

If you create a mirror by adding a second HDD to the storage pool, you system will still boot from the original first drive.
The second HDD isn't used at all during the boot, unless you specify in the BIOS to boot from the second HDD.

pkubaj · Feb 8, 2012

adri said:
If you create a mirror by adding a second HDD to the storage pool, you system will still boot from the original first drive.
The second HDD isn't used at all during the boot, unless you specify in the BIOS to boot from the second HDD.

I know, but if the first HDD broke, I'd have to boot from the second one, which is not possible.

phoenix · Feb 8, 2012

Yes it is. Just install the ZFS boot blocks onto the second disk.

Since you are using MBR partitioning, you would install the zfsboot blocks into the MBR. I believe the following will do what you want (not tested as I've moved on to GPT partitioning):
# gpart bootcode -b /boot/zfsboot ada1

pkubaj · Feb 9, 2012

phoenix said:
Yes it is. Just install the ZFS boot blocks onto the second disk.

Since you are using MBR partitioning, you would install the zfsboot blocks into the MBR. I believe the following will do what you want (not tested as I've moved on to GPT partitioning):
# gpart bootcode -b /boot/zfsboot ada1

Thanks, but I doubt FreeBSD will boot, since neither Ubuntu nor Windows could. Besides, I have also tried a clean install of FreeBSD on this disk, which didn't work. As I wrote before, Windows installer said it was about the computer. Smartmontools also didn't detect any damage.