no longer fun with upgrading: file offline

PMc

PMc

Code: 
# pkg upgrade -f -r current
...
Number of packages to be upgraded: 2
Number of packages to be reinstalled: 37

Proceed with this action? [y/N]: y
[1/39] Reinstalling indexinfo-0.3.1...
[1/39] Extracting indexinfo-0.3.1: 100%
[2/39] Reinstalling readline-8.0.4...
[2/39] Extracting readline-8.0.4: 100%
[3/39] Reinstalling libffi-3.3_1...
[3/39] Extracting libffi-3.3_1: 100%
[4/39] Reinstalling gettext-runtime-0.21...
[4/39] Extracting gettext-runtime-0.21: 100%
[5/39] Reinstalling python37-3.7.9...
[5/39] Extracting python37-3.7.9:  12%
pkg: Fail to create temporary file: /usr/local/lib/python3.7/ctypes/test/__pycache__/.pkgtemp.__init__.cpython-37.opt-1.pyc.dAfs3PvwCtyF:Permission denied
[5/39] Extracting python37-3.7.9: 100%
#
# cd /usr/local/lib/python3.7/ctypes/test/__pycache__
/usr/local/lib/python3.7/ctypes/test/__pycache__: Permission denied.
# cd /usr/local/lib/python3.7/ctypes/test
# ls -la __pycache__
total 0
ls: __pycache__: Permission denied
# ls -lad __pycache__
drwxr-xr-x  2 root  wheel  161 Oct 17 15:10 __pycache__
# getfacl __pycache__
# file: __pycache__
# owner: root
# group: wheel
            owner@:rwxp--aARWcCos:-------:allow
            group@:r-x---a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow
# getfacl .
# file: .
# owner: root
# group: wheel
            owner@:rwxp--aARWcCos:-------:allow
            group@:r-x---a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow
# ls -laod __pycache__
drwxr-xr-x  2 root  wheel  offline,reparse,sparse 161 Oct 17 15:10 __pycache__
# ls -laod .
drwxr-xr-x  3 root  wheel  uarch 56 Dec  7 03:58 .

By all Gods, what is this??

Nothing except pkg has ever touched this, This jail's purpose is to run the lets-encrypt updater, once a week, that's all.
 
  • Thanks
Reactions: a6h
Code: 
# chflags nooffline __pycache__
# ls -load __pycache__
drwxr-xr-x  2 root  wheel  uarch,reparse,sparse 161 Oct 17 15:10 __pycache__
# chflags nosparse __pycache__
# ls -laod __pycache__
drwxr-xr-x  2 root  wheel  uarch,reparse 161 Oct 17 15:10 __pycache__
# chflags noreparse __pycache__
chflags: __pycache__: Operation not permitted
# cd __pycache__
__pycache__: Permission denied.

How come?
Daily log from the day before yesterday:

Code: 
Checking for packages with mismatched checksums:

-- End of security output --

Checking userland and kernel versions:
Userland and kernel are in sync.

Daily log from yesterday:

Code: 
Checking for packages with mismatched checksums:
python37-3.7.9: /usr/local/lib/python3.7/ctypes/test/__pycache__/__init__.cpython-37.opt-1.pyc
...

Checking userland and kernel versions:
Userland and kernel are not in sync
Userland version: 1104001
Kernel version: 1202000

Correct, the system was then upgraded to 12.2, while this jail was not yet upgraded.
It seems it is highly dangerous to run 11.4 jails under 12.2 base, even for a short time.

There is only Windows dokumentation about this "reparse" thing, and it says one needs to run a Windows program to remove it..

Worse, ZFS does not appear to support the clri(8) command. :(
 
Reparse points are a type of file system object. Like files, directories, soft links, devices, FIFOs, and a few other obscure ones that I forgot. Windows is the only OS that uses them; I think they are sorta-like a soft link, but not quite as soft, they can be used as mount points, and they can hook into special functionality (for example, I know that archival storage software uses them to pretend to be a file, but to read/write the file, the reparse point automatically runs software).

Lots of Unix-based file systems can store reparse points, because you need that if you want to be a full-function server for Windows machines (for example via a CIFS/SMB server such as Samba, or using cluster file systems that have dedicated windows clients). That's why ZFS supports them as a special object type.

Given that your ZFS file system has probably never been touched by windows, the existence of a reparse point in it indicates a bug in ZFS.

Now how to get rid of it? No idea. In utter emergencies, put your system into single-user mode, create a second ZFS file system on it, then copy all the files from the first to the second one, carefully skipping the reparse point. This sounds like a big pain.
 
ralphbsz said:
Reparse points are a type of file system object. Like files, directories, soft links, devices, FIFOs, and a few other obscure ones that I forgot. Windows is the only OS that uses them; I think they are sorta-like a soft link, but not quite as soft, they can be used as mount points, and they can hook into special functionality (for example, I know that archival storage software uses them to pretend to be a file, but to read/write the file, the reparse point automatically runs software).
Click to expand...

Thank You, that is about what I read on the various resources - and there are only windows resources. (I did dump what was before windows, already in 1990, and I didn't want to see it again.)

The question is: why do these even exist in FreeBSD, and why does not even the superuser at securelevel=-1 have the right to create or remove them? What kind of windows trojan is this?

ralphbsz said:
Lots of Unix-based file systems can store reparse points, because you need that if you want to be a full-function server for Windows machines (for example via a CIFS/SMB server such as Samba, or using cluster file systems that have dedicated windows clients). That's why ZFS supports them as a special object type.
Click to expand...
Okay, But it blocks any access to them. Accoding to the manpage chflags should be able to create and remove them - the option gets accepted, but gives just EPERM. And the directory carrying the flag cannot be read, EACCES. What's the point in this? Have windows users hide their stuff from the unix admin?

ralphbsz said:
Given that your ZFS file system has probably never been touched by windows, the existence of a reparse point in it indicates a bug in ZFS.
Click to expand...
Certainly not. That thing has not been touched by anything except periodic daily in the respective time.
ralphbsz said:
Now how to get rid of it? No idea. In utter emergencies, put your system into single-user mode, create a second ZFS file system on it, then copy all the files from the first to the second one, carefully skipping the reparse point. This sounds like a big pain.
Click to expand...
There is no problem with that, this is /usr/local and can be reinstalled from packages. I also could restore from backup.
But first I would like to know what this all means. A ZFS bug of such kind is not acceptable, anyway.

The pain is actually a lot bigger: 12.2 is entirely broken. When I start only the smallest of my jails, and stop it again, and do this two more times, I have a kernel panic, reproducibly, same backtrace.
 
ralphbsz said:
Given that your ZFS file system has probably never been touched by windows, the existence of a reparse point in it indicates a bug in ZFS.
Click to expand...

I checked this. The only place where ZFS does handle these reparse points (besides moving around the attributes) appears to be this:
Code: 
                if (XVA_ISSET_REQ(xvap, XAT_REPARSE)) {
                        ZFS_EXIT(zfsvfs);
                        return (SET_ERROR(EPERM));
                }

And this is quite what one can observe: no matter if trying to read, delete or change the attribute, one receives a "permission denied".
There is one other mention in the code:
Code: 
                 * XXX KDM
                 * We need to figure out whether it makes sense to allow
                 * UF_REPARSE through, since we don't really have other
                 * facilities to handle reparse points and zfs_setattr()
                 * doesn't currently allow setting that attribute anyway.

So how might it come this attribute suddenly appears?
It is harmless in this case, as I can just recreate the filesystem. But considering that this can suddenly appear (and could also appear on a filetree with important data), is disquieting.

So I am thinking, what could have happened? In this jail nothing has happened, the /usr/local is not even open - I was able to just unmount it for repair. But, in the timeframe where this did appear, was the base system, that carries the jail, upgraded to Rel. 12.

Normally, there is not much problem running programs that are compiled for an older release. Only a few that tend to grab into system tables, will encounter problems. And even less would a jail have problems running an older release.
This time a lot of ports didn't like the new situation and had to be upgraded to function - among them postgres, rsyslogd, ruby.

And there is a change that is nowhere mentioned in the release notes, only in UPDATING:

Code: 
20170523:
        The "ino64" 64-bit inode project has been committed, which extends
        a number of types to 64 bits.  Upgrading in place requires care and
        adherence to the documented upgrade procedure.

        If using a custom kernel configuration ensure that the
        COMPAT_FREEBSD11 option is included (as during the upgrade the
        system will be running the ino64 kernel with the existing world).

        For the safest in-place upgrade begin by removing previous build
        artifacts via "rm -rf /usr/obj/*".   Then, carefully follow the
        full procedure documented below under the heading "To rebuild
        everything and install it on the current system."  Specifically,
        a reboot is required after installing the new kernel before
        installing world.

Nothing I would not have adhered to. But then, the word "inode" rings a bell...

Conclusion: Do not run Rel. 11 jails on a Rel. 12 machine if they contain anything of value.
 
PMc said:
The question is: why do these even exist in FreeBSD,
Click to expand...
Imagine your FreeBSD system is a file server, and one of the clients is windows: ZFS has to be able to store reparse points. Now, once you have a reparse point stored (as a file system object, like a file or soft link), the ZFS implementation has to make it visible in some fashion to the Unix side. If /home/pmc/foo exists, and is a reparse point, then "ls ~/" needs to show that there is some object there. You can't just hide it from the Unix side, otherwise you violate an invariant, namely that a non-existing object can be created (someone might do an exclusive create of foo, and be told that it exists, but ls doesn't show it). That brings up the painful question: How do you expose it? Classic problem of multi-OS file systems, BTDT got the tee shirt. The classic answer is: when looking from Unix, pretend that it is something (say file or soft link), but do not allow reading or writing it, since the "content" of the reparse point just doesn't make any sense to Unixes (it's some kind of binary gobblygook that is meaningful to Windows). So far, so bad.

Now: Creating and reading/writing reparse points can not use the standard Unix API. There is no POSIX call for "create reparse point". That's why someone decided to hide this in chflags.

... and why does not even the superuser at securelevel=-1 have the right to create or remove them?
Click to expand...
Tough question. Ask the ZFS implementors. My hunch is that you're looking at a work in progress, and what you're seeing is unfinished code. The comment you copied seems to indicate that.

The really big question is: what bug caused them to be created in the first place? I have no idea.

Conclusion: Do not run Rel. 11 jails on a Rel. 12 machine if they contain anything of value.
Click to expand...
That might be it. It might also be a red herring. I think you need to file a PR, just so the developers get told about this.
 
You must log in or register to reply here.
Back
Top