UFS root filesystem remount oops

freezmi

freezmi

I got this FreeBSD 14.3 instance that's using ufs. 1GB of ram, available between 60-40%.
it's got / mounted read-only and /var is r/w in order to protect the filesystem against the unexpected.
whenever I need to install new packages I mount -u -o rw /, do the install, sometimes tweak the packet filter, pfctl -f /etc/pf.conf and then mount -u -o ro /.

most of the time this works without issues. however sometimes the ro remount takes a lot of time and I repeat the command the kernel oops-es.
I need to force a restart of the instance from the ISP's control panel since reboot hangs.
Unfortunately the crash never seems to get logged, even if the /var partition remains rw.

What steps should I take to better debug this issue the next time it happens?

and this is not a new regression, it was also happening with 14.2.

thanks!
 
I never used this trick, while I find it interesting it is also binding in some aspect but I should probably try it one day, sounds funny.

freezmi said:
What steps should I take to better debug this issue the next time it happens?
Click to expand...
If I were you I'd start by digging the logs, launch a sysutils/tmux (or sysutils/screen) session where in one window you run your usual commands and in another window you watch the logs.
I'd probably also test few times with and without the pfctl command just to be sure that pf is not the culprit.
Good luck.
 
freezmi said:
What steps should I take to better debug this issue the next time it happens?
Click to expand...
You should configure a dump device which the kernel will use to write a core dump when it panics. Search for dumpdev in rc.conf(5).
You can verify with dumpon -l that there is a dump device active.

Remounting / read-only while /var remains read-write sounds to me like a weird configuration that might be unsupported.
 
JordanG said:
Remounting / read-only while /var remains read-write sounds to me like a weird configuration that might be unsupported.
Click to expand...

been using this trick in both Linux and OpenBSD for years. In these OS-es sometimes the kernel refuses to remount / ro because 'filesystem is in use', but it never crashes.

cracauer@ said:
A panic should certainly not happen. Please do as Jordan suggests and file a bug report with the kernel backtrace.
Click to expand...
thanks, will do.
 
today the system became unresponsive, so no crash:

# mount -u -o ro /
mount: /dev/da0s1a: Resource temporarily unavailable
Click to expand...

dmesg:
fsync: giving up on dirty (error = 35) 0xfffff80012103c40: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818ac9e8
usecount 1, writecount 0, refcount 152 seqc users 0 rdev 0xfffff800121b8400
hold count flags ()
flags ()
v_object 0xfffff8000304ec60 ref 0 pages 1506 cleanbuf 141 dirtybuf 9
lock type mntfs: EXCL by thread 0xfffff8000a095000 (pid 38742, mount, tid 100412)
fsync: giving up on dirty (error = 35) 0xfffff80012103c40: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818ac9e8
usecount 1, writecount 0, refcount 152 seqc users 0 rdev 0xfffff800121b8400
hold count flags ()
flags ()
v_object 0xfffff8000304ec60 ref 0 pages 1506 cleanbuf 141 dirtybuf 9
lock type mntfs: EXCL by thread 0xfffff8000a095000 (pid 38742, mount, tid 100412)
Click to expand...

/var/crash only has a minfree file in it, and I did previously set up a swap partition, dumpdev="AUTO" and dumpdir="/var/crash" based on your past feedback.

booted the install image, ran fsck_ufs and fixed all partitions, so they are all marked clean now.

dark.initr0 said:
I could be wrong but maybe try to use sync before going to RO mode?
Click to expand...

while I agree that sync helps in some instances - ie. prior to unplugging removable media, I feel it would be the kernel's job to schedule a sync when simply remounting a filesystem.

I forgot to mention, but I have the exact same / mounted readonly spiel on a bare metal FreeBSD 4.3 system with ZFS, I have seen zero filesystem crashes on that one.
so it could be UFS, linode's virtualization layer or limited memory (1GB 40-60 %free, 1GB swap 100% free) related.

can you knowledgeable gentlemen see any valuable clues in that dmesg ^ snippet?
 
You must log in or register to reply here.
Back
Top