ZFS System become slow, hard drive failure?

B

bxbzq

Ok, so it is FreeBSD 14.3 on a ZFS based single disk pool. Lately I accidentally noticed It's been very slow when logging in. I don't know since when it became this bad because it serves a backup VPS whose load is low and I don't pay attention to it.
Code: 
% sudo  zpool status -xv
  pool: zroot
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: [URL]https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A[/URL]
remove: Removal of vdev 1 copied 344K in 0h0m, completed on Thu Jun  8 14:02:16 2023
        240 memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0     0
          ada0p3      ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        //var/run/devd.pid

Luckily the OS still boots.

Not sure if it makes sense, but I check .zfs/snapshot/2025-05-07-15:01:10-0/var/run/devd.pid, and compare it with /var/run/devd.pid. Both devd.pid files contain very simple 3-digit numbers.

Is this a hard drive failure? Can I still fix it?
 
#pkg install smartmontools
# pkg info smartmontools
smartmontools-7.5_1
Name : smartmontools
Version : 7.5_1

You can use smartmontools ,
binary is /usr/local/sbin/smartctl

# smartctl -HA /dev/ada0

to dump the SATA drives error counters.

Look for reallocated sectors , number of failed but not relaocated sectors and UDMA errors
If those numbers are substantial , hundreds to thousands , the drive is failing and not safe to use.
 
drsnx60 said:
#pkg install smartmontools
# pkg info smartmontools
smartmontools-7.5_1
Name : smartmontools
Version : 7.5_1

You can use smartmontools ,
binary is /usr/local/sbin/smartctl

# smartctl -HA /dev/ada0

to dump the SATA drives error counters.

Look for reallocated sectors , number of failed but not relaocated sectors and UDMA errors
If those numbers are substantial , hundreds to thousands , the drive is failing and not safe to use.
Click to expand...

Like this?
# smartctl -HA /dev/ada0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 148 144 021 Pre-fail Always - 3575
4 Start_Stop_Count 0x0032 095 095 000 Old_age Always - 5319
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 033 033 000 Old_age Always - 49478
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 095 095 000 Old_age Always - 5316
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 148
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 4627290
194 Temperature_Celsius 0x0022 107 097 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 190 190 000 Old_age Always - 817
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
 
It's worth reading the TrueNAS Hard Drive Troubleshooting Guide. Current_Pending_Sector count (197) greater than zero indicates that
"the drive has failed for RMA purposes".

ZFS is indicating that you may be able to read files and save them, however ultimately you will probably have to replace the disk and restore from backup.

My experience of 197 > 0 is that the disk is toast. Rescue what you can as soon as you can, and consider the advantages of redundant storage.
 
bxbzq said:
Like this?
# smartctl -HA /dev/ada0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 148 144 021 Pre-fail Always - 3575
4 Start_Stop_Count 0x0032 095 095 000 Old_age Always - 5319
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 033 033 000 Old_age Always - 49478
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 095 095 000 Old_age Always - 5316
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 148
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 4627290
194 Temperature_Celsius 0x0022 107 097 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 190 190 000 Old_age Always - 817
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
Click to expand...
so we can see from this data that the HDD has been turned on for 6 years. and computer has been rebooted 2 to 3 times every day for 6 years , probably normal for a laptop/desktop machine, ofcourse if this machine is only used 8 hrs a day those 49478 hours of poweron turns into a very long time., 18 years ?? Reallocated event counter , reallocated sectors and offline uncorrectable sectors are all zero, which indicates that the disk platters are not worn out. You should not have a sectors pending for reallocation. (197) ! the HDD should realocate a failed sector and up the counter for reallocation events and number of reallocated sectors
If the HDD is no able to perform a reallocation of a failed sector its at risk of failing

If the HDD is really old you may have failures in the Logic board on the HDD as well.
 
You must log in or register to reply here.
Back
Top