12.0/STABLE: interactive input stalls after inactivity

[rowan194]

I'm having problems with FreeBSD 12 momentarily ignoring interactive input (via SSH) when nothing has been typed for a while.

If I don't type anything for a little while - maybe 20 or 30 seconds - then the first keypress takes at least a few seconds to be echo'd back. Once that happens, subsequent keypresses come back instantly as expected.

The delay doesn't just happen when sitting at a terminal prompt; it will also happen, for example, when editing a file in 'ee'

It's almost like a memory constrained system that is swapping hard, and so becomes less responsive... but there's plenty of free RAM, and the input echoing is fine once it revives.

More detail:

I'm running two very similar setups, with some differences:

- DX79TO vs DX79SR mainboard (same chipset, one board has a few extras)
- i7 3960X vs i7 3930k (same CPU family, different clock speeds)
- 32GB vs 16GB RAM
- FreeBSD 11.2-RELEASE vs FreeBSD 12-STABLE

Both systems run at a medium to high CPU load (4.0 to 10.0 on 6 cores/12 threads) and high HD load (consistently 95%+)

The problem does not happen on the FreeBSD 11.2 system, nor have I seen it happen on the other two FreeBSD 12-STABLE systems (with different hardware and applications) that I run.

FreeBSD xxx 11.2-RELEASE FreeBSD 11.2-RELEASE #0: Thu Jun 13 10:13:15 AEST 2019 xxx@xxx:/usr/src/sys/amd64/compile/XXX amd64
FreeBSD xxx 12.0-STABLE FreeBSD 12.0-STABLE r351024 XXX amd64

Here's a dump of the SSH TCP stream showing that the keypress packet from the client (C.C.C.C) is immediately acknowledged by the server (S.S.S.S), but the server takes 1.4 seconds to echo the keypress back:

23:08:38.382155 IP C.C.C.C.51663 > S.S.S.S.22: Flags [P.], seq 192:256, ack 481, win 1861, length 64
23:08:38.485670 IP S.S.S.S.22 > C.C.C.C.51663: Flags [.], ack 256, win 1025, length 0
23:08:39.864833 IP S.S.S.S.22 > C.C.C.C.51663: Flags [P.], seq 481:545, ack 256, win 1026, length 64
23:08:39.915896 IP C.C.C.C.51663 > S.S.S.S.22: Flags [.], ack 545, win 1861, length 0

Any ideas? Thanks.

 

[scottro]

Keep in mind that STABLE is actually a development release and can often break. You probably want FreeBSD-12-RELEASE, which is now available.

http://srobb.net/release.html was written while FreeBSD 4.x,5.x, and 6.x were the most recent releasese, but Fred Cash, who wrote it, gave an excellent explanation of the different types of release and it's still applicable.

-STABLE is the development tree for the latest release(s) of FreeBSD. After every release is branched from the -CURRENT tree, it becomes the -STABLE tree. As of mid 2006, there are three of these: 4-STABLE, 5-STABLE, and 6-STABLE. These are development trees, but they tend to be more usable than the -CURRENT tree, as there are more stringent tests and rules on what gets placed in these trees. Sometimes, it can be advantageous to run -STABLE, especially if you need specific driver support or a specific bug fix. It's not always guaranteed to work, though, so be prepared. cvsup tags are: RELENG_4, RELENG_5, and RELENG_6.

Note it was written when FreeBSD was using cvs intead of svn. There's a bit more explation in the article.

 

[ralphbsz]

If I don't type anything for a little while - maybe 20 or 30 seconds - then the first keypress takes at least a few seconds to be echo'd back. Once that happens, subsequent keypresses come back instantly as expected. ...

Both systems run at a medium to high CPU load (4.0 to 10.0 on 6 cores/12 threads) and high HD load (consistently 95%+)

Theory, based purely on the observed behavior: Your system is running out of memory, the swapper is trying to find idle processes to swap out, and the user shell is swapped out. When you press a key, it needs to be swapped in. The fact that some other process is running makes it more likely that some other process is consuming memory, causing something else to be swapped out.

How to check whether this theory is correct? In another window, keep top or vmstat running, and watch the memory behavior.

It could also be something else ... this is just one simple theory. Clearly, running a development version on a production system is a bad idea too.