Solved zpool repair input/output errors in /var/log

[myso]

Dear community, I'd like to kindly ask you for a little help in my situation:

I'm running a ZFS pool of 3 drives. What happened is - a power cable powering 2 drives had a bad contact, so I started loosing 2 out of 3 drives and having a lot of UDMA errors and CAM status timeouts etc.

Yesterday I fixed the power cable, zpool is ONLINE with no errors, but some permanent errors can be seen:

# zpool status -v
  pool: zroot
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0h0m with 3 errors on Tue Jan  9 10:26:40 2018
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0    29
          raidz1-0    ONLINE       0     0    68
            gpt/zfs0  ONLINE       0     0     0
            gpt/zfs1  ONLINE       0     0     7
            gpt/zfs2  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /var/log/messages
        /var/log/cron
        /var/log/auth.log

All three log files are returning an input/output error:

# cat /var/log/messages
cat: /var/log/messages: Input/output error

What is the correct way to fix this, so I don't mess up the system and all my data stored on the pool?

Many thanks

Mike

 

[SirDice]

I think you can get away easily if you just deleted those files. None of them are really important, they're just log files. Just remove them, run newsyslog (as root) and restart syslogd(8), that will create new log files.

 

[myso]

Many thanks SirDice,

I'll just add all the additional steps to this thread, so it is complete for future referrence:

Remove the broken files:

# rm /var/log/messages
# rm /var/log/cron
# rm /var/log/auth.log

Run newsyslog

# newsyslog

Restart syslogd() with the -C option (Create log files that do not exist (permission is set to 0600).)

# /etc/rc.d/syslogd stop
Stopping syslogd.
Waiting for PIDS: 9132.
# syslogd -C
# cat /var/log/messages
Jan  9 11:31:45 czkdsrv1 syslogd: kernel boot file is /boot/kernel/kernel
# /etc/rc.d/syslogd status
syslogd is running as pid 9170.

zpool status still showing errors, so run zpool scrub

# zpool status
  pool: zroot
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0h0m with 3 errors on Tue Jan  9 10:26:40 2018
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0    31
          raidz1-0    ONLINE       0     0    72
            gpt/zfs0  ONLINE       0     0     0
            gpt/zfs1  ONLINE       0     0     7
            gpt/zfs2  ONLINE       0     0     0

errors: 3 data errors, use '-v' for a list

# zpool scrub zroot
# zpool status
  pool: zroot
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Jan  9 11:21:49 2018
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0    31
          raidz1-0    ONLINE       0     0    72
            gpt/zfs0  ONLINE       0     0     0
            gpt/zfs1  ONLINE       0     0     7
            gpt/zfs2  ONLINE       0     0     0

errors: No known data errors

Errors are gone, but the status is still negative, so use zpool clear to clear the status message:

# zpool clear zroot
# zpool status
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Jan  9 11:21:49 2018
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            gpt/zfs0  ONLINE       0     0     0
            gpt/zfs1  ONLINE       0     0     0
            gpt/zfs2  ONLINE       0     0     0

errors: No known data errors

Everything looks OK now, thanks again for a very quick response!