corrupted file after upgrade to 14.0?

F

fwabi9

I think after upgrading to 14.0, I have this weird behavior where index.html, originally a normal HTML file, is rendered with (seemingly random) binary data:

sh: 
# cp -a /tmp/some/backup/index.html /usr/local/www/

# md5sum /usr/local/www/index.html
7be9d7c91bab0691478e105902317a86  /usr/local/www/foudil.fr/index.html

# xxd index.html
00000000: 3c21 444f 4354 5950 4520 6874 6d6c 3e0a  <!DOCTYPE html>.
00000010: 3c68 746d 6c3e 0a20 3c68 6561 643e 0a20  <html>. <head>.

# stat -x index.html
  File: "index.html"
  Size: 757          FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (    0/    root)  Gid: (   80/     www)
Device: 18446744072768100063,1174405355   Inode: 296400    Links: 1
Access: Thu Dec 14 00:55:57 2023
Modify: Thu Dec 14 02:30:58 2023
Change: Thu Dec 14 02:30:58 2023
 Birth: Thu Dec 14 00:55:57 2023

[…some minutes or hours later…]

# md5sum index.html
6cda6db21ce4ac5de8e13613dac19e5bd  index.html

# stat -x index.html
  File: "index.html"
  Size: 757          FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (    0/    root)  Gid: (   80/     www)
Device: 18446744072768100063,1174405355   Inode: 296400    Links: 1
Access: Thu Dec 14 00:55:57 2023
Modify: Thu Dec 14 02:30:58 2023
Change: Thu Dec 14 02:30:58 2023
 Birth: Thu Dec 14 00:55:57 2023
foudil has logged on pts/3 from tmux(4535).
foudil has logged on pts/4 from tmux(4535).

[…some more minutes or hours later…]

# md5sum index.html
64cb6596b5433b111b15932200b54ad8  index.html

# xxd index.html
00000000: 85d5 d597 8523 b5d9 2a9b 27df 142c a777  .....#..*.'..,.w
00000010: 5353 a99d 21a3 da7a 3159 e464 0739 4b6a  SS..!..z1Y.d.9Kj
[…]

/usr/loca/www is a ZFS pool on a geli-encrypted partition. Note between changes, inode and ctime remain unchanged!

What's more, I renamed these (index.html1, index.html2, etc.), and after some time, or after a reboot, they are again rendered as the original HTML file. I.e. md5 = 7be9d7c91bab0691478e105902317a86

I checked what processes or jails could have access to this file, but couldn't find any obvious ones. Only subdirs of /usr/loca/www are mounted inside jails. The file is served by nginx then haproxy.

At this point I have 2 hypotheses:
  1. I've been hacked and the hacker is playing with my nerves.
  2. this might somehow be related to recent ZFS bugs
So throwing it out there: has anyone else got similar issues?
 
Explanation 1 is extremely unlikely. And if that happened, most likely you would be seeing traces, such as logins.

Explanation 2 is even less likely. The recent ZFS bugs needed extremely bizarre ordering of operations, and should be very rare.

The likely explanation is some sort of confusion happening, for example involving renames, backups, soft-links, or such.
 
Thx everyone for your input. Could not find any clue so far. Things I've tried:
  • smartd: long test didn't reveal any error.
  • zfs scrub: no errors.
  • auditd: didn't reveal any writes to index.html, only read calls like openat(2) - read or fstatat(2).
  • links: find -L / -samefile /usr/local/www/foudil.fr/index.html or symlinks didn't reveal any link to index.html.
  • backups: monitoring its md5, change pattern looks like "random within 1h", but I couldn't find any related task. Sometimes the file is even reverted to its the correct original content! Backups would read the content, not modify it.
  • logs: no significant time-based matches in /var/log/**/*, no traces of suspicious auth attempts (auth.log or audit logs).

I noticed transient SSL errors from clients incidentally…

Before upgrading to 14.0, I re-installed the box and restored pools from zfs backups (zfs recv). I messed up in the process and had to manually reset the mountpoints and related properties. Just mentioning but I don't think this relates to the current issue.

So really my observations are just one single file consistently being rendered randomly by the OS. I insist: inode and ctime remain unchanged!

At this point I guess my next step is to re-install from scratch and restore from previous file-based backups.
 
fwabi9 said:
logs: no significant time-based matches in /var/log/**/*, no traces of suspicious auth attempts (auth.log or audit logs).
Click to expand...
IF (big IF) your system was compromised it wouldn't show up in those logs. If you have some sort of shell or SQL injection issue on the website then they're never actually logging in. It'll be the webserver or engine (PHP, Ruby, Python, etc) that modifies the file. These generally run on the www account and don't have write access to that file (you're not running them as root are you?!?). So it would require a second step to elevate privileges in order to overwrite it.

fwabi9 said:
So really my observations are just one single file consistently being rendered randomly by the OS
Click to expand...
You assume it's the OS that's doing it. All you can say for sure is that the file gets changed. You don't know how or what actually changes it.
 
Thx again for looking into this.

I get the point about a second step priviledge elevation. Here some additional details: /usr/local/www/foudil.fr/index.html is actually owned by www. But all web apps run in jails which do NOT have /usr/local/www/foudil.fr mounted, only subdirs. These web apps do run as www.

I think the main point is: how does any user or process modify the content of a file without changing atime, mtime, ctime?

Since I can't find any explanation, yes, my interpretation is that this file is rendered differently, especially since it keeps the same size (plus atime/mtime/ctime) and actually sometime "revert" to the original content. And what renders the file through say cat or xdd? Well libc and… the OS no?
 
Seems to happen to all static files HTML-accessible from my website… 🤔

zdb -r zroot/usr/local/www foudil.fr/index.html /usr/local/www/foudil.fr/index.html "restores" the file successfully. 🤷‍♀️
 
You must log in or register to reply here.
Back
Top