We had some zpool issues while running 8.2, basically every time we ran any zpool / zfs command the system would kernel panic and reboot, upgraded to 8.3 and this went away.
I'm running 8.3-RELEASE on VMware, with two pools, a 10TB and a 6TB.
After upgrading to 8.3 from 8.2, there were no issues, however we have NEVER scrubbed our pools because the ETA under 8.2 was 7,000,000,000 days.
I started a scrub and that was going nicely, then it detected data errors on the pools and has started resilvering them, It's been going for around 6 hours so far:
The server has 8 cores 18GB of RAM and ZFS is tuned as such:
What exactly is going on here, should I leave it for 300+ hours to do it's thing, or do I need to intervene and fix something?
I'm quite new to ZFS and certainly not a BSD expert, should resilvering be configured differently when using virtual disks rather than physical disks?
Any help would be greatly appreciated.
I'm running 8.3-RELEASE on VMware, with two pools, a 10TB and a 6TB.
After upgrading to 8.3 from 8.2, there were no issues, however we have NEVER scrubbed our pools because the ETA under 8.2 was 7,000,000,000 days.
I started a scrub and that was going nicely, then it detected data errors on the pools and has started resilvering them, It's been going for around 6 hours so far:
Code:
int-freebsd-backup# zpool status -v
pool: backup
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Dec 11 18:30:32 2012
71.4G scanned out of 9.23T at 11.8M/s, 225h23m to go
56K resilvered, 0.76% done
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 5.33K 8 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 3
da3 ONLINE 5.32K 351 2.04K (resilvering)
da4 ONLINE 0 0 8
da5 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198a7>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198ab>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198ad>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198b9>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198bd>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198be>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198bf>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198c3>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198c4>
backup/somecooldesktop@auto.2012-11-16-11-04:<0x198c6>
pool: backup2
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Dec 11 14:45:12 2012
26.8G scanned out of 5.45T at 1/s, (scan is slow, no estimated time)
0 resilvered, 0.48% done
config:
NAME STATE READ WRITE CKSUM
backup2 ONLINE 1.00K 9 0
da6 ONLINE 961 0 691
da7 ONLINE 64 0 0
da8 ONLINE 1002 35 538
errors: Permanent errors have been detected in the following files:
<metadata>:<0x0>
<metadata>:<0x1>
<metadata>:<0x12e>
<metadata>:<0x13a>
<metadata>:<0xdc>
<metadata>:<0xe2>
<metadata>:<0xe3>
backup2/db-somecoolapp@auto.2012-11-04-13-45:<0x0>
backup2/db-somecoolapp@auto.2012-11-04-13-45:<0x5078>
backup2/db-somecoolapp@auto.2012-11-04-13-45:<0x507f>
int-freebsd-backup#The server has 8 cores 18GB of RAM and ZFS is tuned as such:
Code:
cat /boot/loader.conf
# Beginning of the block added by the VMware software
vmxnet_load="YES"
# End of the block added by the VMware software
# Beginning of the block added by the VMware software
vmxnet3_load="YES"
# End of the block added by the VMware software
#ZFS Tweaks
#I have 16G of Ram
vfs.zfs.prefetch_disable=0
#If Ram = 4GB, set the value to 512M
#If Ram = 8GB, set the value to 1024M
vfs.zfs.arc_min="2048M"
#Ram x 0.5 - 512 MB
vfs.zfs.arc_max="15872M"
#Ram x 2
vm.kmem_size_max="32G"
#Ram x 1.5
vm.kmem_size="24G"What exactly is going on here, should I leave it for 300+ hours to do it's thing, or do I need to intervene and fix something?
I'm quite new to ZFS and certainly not a BSD expert, should resilvering be configured differently when using virtual disks rather than physical disks?
Any help would be greatly appreciated.
Let ZFS handle the redundancy, otherwise, it can't fix the errors the scrub detects. All those errors ... are non-recoverable, and not picked up by the hardware RAID.