Solved zfs "bad property list: invalid property 'checkpoint'"

Hi, I have posted a question on the freebsd list, but I have not received a reply yet. Can you please help me with fixing the ZFS filesystem on a machine that I administer?


On my FreeBSD XXXX 11.2-RELEASE-p9 FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5
15:30:36 UTC 2019
[hidden email]:/usr/obj/usr/src/sys/GENERIC amd64

every time I issue zpool list the following error is emitted as you can see
bellow.

According to #zpool history this pool was created on FreeBSD on
2010-08-18.18 and has been running on the same disk flawlessly 24/7 even
though M/B and cpu were occasionally upgraded. The tool is operational and
is heavily used. It is not a boot ZFS pool. Just a storage area. zfs scrub
reports no problems. This pool was always connected to the FreeBSD and never
to a Linux or other OS.
I have been always doing a freebsd-update and zpool upgrade after every
major FreeBSD version upgrade.
I am not sure why this has happened. I just noticed one day.

I cannot export/import this pool because this is on a remote server and I
fear that i may not be able to re-import it again. According to zfs history
the last upgrade was:
...
2017-02-05.12:19:35 zpool upgrade tank
....
after this there are zpool scrub tank entries.

The uptime is 76 days and I would not like to perform a reboot because it
may cause the pool not to be mounted. I would like to fix it without reboot.


The questions are:
1) Is the property 'checkpoint' supported on 11.2-RELEASE-p9
2) How can I remove this property and all the (null) properties as reported
by zpool upgrade.


Thank you very much for your help. Any suggestions are welcome.

=================================
# zpool list
bad property list: invalid property 'checkpoint'
usage:
list [-Hpv] [-o property[,...]] [-T d|u] [pool] ... [interval
[count]]

the following properties are supported:

PROPERTY EDIT VALUES

allocated NO <size>
capacity NO <size>
dedupratio NO <1.00x or higher if deduped>
expandsize NO <size>
fragmentation NO <percent>
free NO <size>
freeing NO <size>
guid NO <guid>
health NO <state>
leaked NO <size>
size NO <size>
altroot YES <path>
autoexpand YES on | off
autoreplace YES on | off
bootfs YES <filesystem>
cachefile YES <file> | none
comment YES <comment-string>
dedupditto YES <threshold (min 100)>
delegation YES on | off
failmode YES wait | continue | panic
listsnapshots YES on | off
readonly YES on | off
version YES <version>
feature@... YES disabled | enabled | active

The feature@ properties must be appended with a feature name.
See zpool-features(7).


Also zpool-upgrade does not operate:


=================================

# zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.

POOL FEATURE
---------------
tank
(null)
(null)
(null)

# zpool upgrade -a
This system supports ZFS pool feature flags.

cannot set property for 'tank': invalid feature '(null)'

=================================
 
Read zpool-features(). The checkpoint feature turns on automatically when you use the checkpoint subcommand of the zpool() command. It turns off when ... read the man pages.

I suspect that if you discard the checkpoint, the problem will go away. Now I do think that you may have found a latent bug, where the zpool list and upgrade subcommands don't play nice with the checkpoint subcommand. If that's really true (would take validation by testing it on a sacrificial pool), one should open a PR (a.k.a. bug report).

P.S. The man page links above automatically go to release 12; you can select release 11.2 on the web pages.
 
HI, thanks you for suggestion.
I just tried your suggestion, but I have not access to zpool checkpoint subcommand:

# zpool checkpoint
missing pool argument
usage:
checkpoint [--discard] <pool> ..

# zpool checkpoint tank
/sbin/zpool: Undefined symbol "zpool_checkpoint"


Also on the status command I have no checkpoint indication

# zpool status tank
pool: tank
state: ONLINE
scan: scrub repaired 0 in 45h5m with 0 errors on Wed May 1 09:49:21 2019
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
ada0s1g ONLINE 0 0 0

errors: No known data errors


while as I see from the output of other user (https://sdimitro.github.io/post/zpool-checkpoint/ ) if there is a checkpoint a line is added on the status
...
checkpoint: created Sat Apr 22 08:46:54 2017, consumes 88.5K
...




Also I tried to remove the checkpoint but this command failed.

# zpool checkpoint --discard tank
/sbin/zpool: Undefined symbol "zpool_discard_checkpoint"



Also, I tried
# ldd /sbin/zpool
/sbin/zpool:
libgeom.so.5 => /lib/libgeom.so.5 (0x80083b000)
libnvpair.so.2 => /lib/libnvpair.so.2 (0x800a40000)
libuutil.so.2 => /lib/libuutil.so.2 (0x800c57000)
libzfs.so.2 => /lib/libzfs.so.2 (0x800e61000)
libc.so.7 => /lib/libc.so.7 (0x8010aa000)
libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x801462000)
libsbuf.so.6 => /lib/libsbuf.so.6 (0x80168e000)
libmd.so.6 => /lib/libmd.so.6 (0x801891000)
libumem.so.2 => /lib/libumem.so.2 (0x801aa9000)
libutil.so.9 => /lib/libutil.so.9 (0x801caa000)
libm.so.5 => /lib/libm.so.5 (0x801ebe000)
libavl.so.2 => /lib/libavl.so.2 (0x8020e9000)
libz.so.6 => /lib/libz.so.6 (0x8022eb000)
libzfs_core.so.2 => /lib/libzfs_core.so.2 (0x802504000)
libthr.so.3 => /lib/libthr.so.3 (0x80270b000)




but on another box that has FreeBSD 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC amd64
the dynamic library dependenci is for libzfs.so.3 and not libzfs.so.2

# ldd /sbin/zpool
/sbin/zpool:
libgeom.so.5 => /lib/libgeom.so.5 (0x800260000)
libnvpair.so.2 => /lib/libnvpair.so.2 (0x800269000)
libuutil.so.2 => /lib/libuutil.so.2 (0x800282000)
libzfs.so.3 => /lib/libzfs.so.3 (0x80028f000)
libc.so.7 => /lib/libc.so.7 (0x8002df000)
libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x8006d2000)
libsbuf.so.6 => /lib/libsbuf.so.6 (0x8006fe000)
libmd.so.6 => /lib/libmd.so.6 (0x800704000)
libumem.so.2 => /lib/libumem.so.2 (0x800721000)
libutil.so.9 => /lib/libutil.so.9 (0x800726000)
libm.so.5 => /lib/libm.so.5 (0x80073d000)
libavl.so.2 => /lib/libavl.so.2 (0x80076f000)
libz.so.6 => /lib/libz.so.6 (0x800773000)
libzfs_core.so.2 => /lib/libzfs_core.so.2 (0x80078d000)
libthr.so.3 => /lib/libthr.so.3 (0x800798000)




Running freebsd-update zfs I have this error

/lib/libzfs.so.2 has SHA256 hash 8154a4b07b27893babb48b2546f04bcad1c8dd58ecca0699614fb7c3ebc2c7dc, but should have SHA256 hash a4e6f493c73c2613e75171a248ceb77ce867c2c4a20cc1cb92cdebb14d95b78a.
/lib/libzfs_core.so.2 has SHA256 hash 46c56e9337e5dad4c4b57d935054adf7b58833bacb7a7ef8d9fc140916bcef40, but should have SHA256 hash 779a111fb3a4a25d8526aa9a8db875cf76568330365dbc71f480cea905b82483.



So perhaps the libzfs has a problem, and I can assume that I have to boot in single user mode and copy from the other box these two libraries.

I will try this and let you know.
 
I replaced the two files that has mismatched SHA256 and now I had to perform a zpool upgrade



# zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.

POOL FEATURE
---------------
tank
device_removal
obsolete_counts
zpool_checkpoint




# zpool upgrade tank
This system supports ZFS pool feature flags.

Enabled the following features on 'tank':
device_removal
obsolete_counts
zpool_checkpoint




and problem solved. So the wrong version of the /lib/libzfs* was the culrpit.


Thank you for your help. Problem Solved.
 
You are welcome. When I saw this in your post ...

# zpool checkpoint tank
/sbin/zpool: Undefined symbol "zpool_checkpoint"
... it became obvious that your installation is screwed up. Fortunately, you also saw the obvious, and fixed it.

Ideally, if you had time and energy, you should do a post-mortem: How did your ZFS installation become screwed up? Just so you can learn to not repeat whatever mistake caused it. This is likely too much work for an individual user.
 
Back
Top