System stops responding to network requests and cannot login

J

Jason_25

Lately one of my systems stops accepting network connections after about a month of running. It is a PC stick with a relatively recent install running FreeBSD 13.1-RELEASE.

I can not check on the problem because I do not have a prompt at the console after logging in and launching further graphical applications results in a hang. The problem is not memory consumption because even after ctrl-c on the first virtual terminal to stop xorg I still cannot log in. Also I do not think it is memory use because the system does not slow down or stutter. Furthermore I do not think it is memory related because while it has just 4GB I am using the UFS filesystem which uses less memory. Finally I do not think memory is a factor because at least at first the system uses minimal RAM to have 2 terminal windows, 2 browsers with 1 tab each, and a video playing. The device in question uses an EMMC for storage so the only thing I can think of is that the EMMC is faulty and stops responding after a while even though it is barely being accessed.

I have gotten used to my FreeBSD systems running for half decade increments so this failure after only a month is a show stopper. What can I do to fix this?
 
boot single user mode and check hardware. or press crtrl alt f1 at the display manager login ui to switch to console
also check power brick, fan
 
Jason_25 said:
a relatively recent install running FreeBSD 13.1-RELEASE.
Click to expand...
13.1-RELEASE is end-of-life since July 2023, not related to the issue (an existing installation should just keep running), but you need to upgrade to 13.2.

Jason_25 said:
I can not check on the problem because I do not have a prompt at the console after logging in and launching further graphical applications results in a hang.
Click to expand...
Sounds like a dead or dying disk. Anything that's already in memory keeps running fine but as soon as something needs to be loaded from disk it will just stall indefinitely.
 
Make sure its not going into suspend mode or some other power state.

Look at /var/log/messages for info.

I do not use swap disk on ssd/nvme/emmc.
 
I would look at overall system heat.

Since this is 'computer on HDMI fob' it may be overheating.

Not all these fobs are made by Intel and others may be of inferior quality.
 
I just got back to this because I had so many things come up all at once. The PC stick has a Z8350 Atom CPU. It only has 4GB RAM and I opted not to create a swap because I wanted to reduce wear on the EMMC. This ultimately was a bad idea because I had such ambitious goals for it being a display board with several things running at once. Even though the problem seemed disk related I think memory was so full that nothing could be loaded in instead. Right now free memory shows less than 100MB which does not seem too good. I may have to create a swap file regardless.
 
What filesystem are you using? (Wondering if ZFS)
Have you looked at top to determine the offending program?

You could try a memory based tmpfs instead of swap for testing.

tmpfs

man.freebsd.org man.freebsd.org
This is all that is needed:
Configure a tmpfs mount via fstab(5):
Click to expand...
 
SirDice said:
13.1-RELEASE is end-of-life since July 2023, not related to the issue (an existing installation should just keep running), but you need to upgrade to 13.2.


Sounds like a dead or dying disk. Anything that's already in memory keeps running fine but as soon as something needs to be loaded from disk it will just stall indefinitely.
Click to expand...
smartctl -t long is your friend in this regard. Do a smartctl -a against the disk first. Note the number of pending sectors. Reallocated sectors are OK, up to a point, but when all spare sectors have been assigned the disk will die.

If smartctl -t long results in an aborted attempt to read the disk, it's time to replace it and quickly.
 
I made sure to use UFS for this because I knew memory would be limited. Good tip on mmc-utils I will have to try that some time.

I created a small swap space because an mpv instance was killed because memory finally ran out. I will keep a close watch on it and expand if necessary. This was definitely the cause of my network scripts having trouble because the system itself could not ping out because ping said "send to: no buffer space available". As I write this memory is running down near to where it will start swapping so hopefully I can still get away with the 4GB plus swap. I was able to add a swap partition with the system running with just gpart and swapon and an edited fstab. I was suprised that there was no mkswap formatting step as gpart was smart enough to do that automatically.

When it last crashed it sat for about 3 weeks until I got to it. Somehow during that time the attached 32" 1080p screen got this grey checkerboard look to the screen that I thought was because of the crash. In fact the TV was completely damaged and even the menus were gray and nearly colorless. It was a 2009 TV so I replaced it with a spare 50" 4k TV. This has allowed me to fully realize a networking focused NORAD-style status board kind of thing. I am hoping that KDE window rules and session keeping will help here so I do not spend 45 minutes setting it all back up when UPS power runs out.

By the way mpv does an incredible job with the -vo x11 output and the switch --profile=sw-fast. I am running 2 HD ATSC streams at once on this 4k screen and this made all the difference because even 1 stream was jerky before. I also found that hardware acceleration works with mpv -vo gpu switch but is always in a dedicated full screen framebuffer that covers all my other windows. In contrast to the PC stick, my FreeBSD "workstation" laptop that I am writing this with has a Ryzen 2400GE processor and 32GB RAM and much lower resolution 1366x768 screen and needs no tweaks and even VLC plays all videos at perfect speed.

You can find these PC sticks for around $60. I added a microSD card which I was happy to see the PC stick internal reader works perfectly with FreeBSD and oh yeah the PC stick runs a small web server and is the network Samba server serving files from the 256GB microSD card. This PC stick is running a huge amount of tasks because it is the result of an effort of "network consolidation" I have been making.

I am really happy with the performance overall. Hopefully it lasts because I do not want to set all this up again but I am backing up my scripts and notes and things like that and this is another pretty good day in BSD land.
 
Jason_25 said:
You can find these PC sticks for around $60.
Click to expand...
Is it an Intel Compute Stick or Generic?

So in effect you turned a Compute Stick into a Kiosk style dashboard for an HDMI display.

What are you streaming to mpv? Is it RTSP stream? How are you producing its content?

How does the web server figure in here? lighttpd?

Serving files from the microSD Card too. You are working that thing.
 
Allow me to answer the first question with 2 paragraphs.

This PC stick is a "Terryza". The Intel sticks I have found either have older 32-bit atoms, a very powerful and expensive Core M, or limited USB ports. I do not necessarily recommend using cheaper, older, secondhand sticks like this because they have a BIOS battery that is soldered to the board and difficult to replace. I destroyed a couple of sticks trying to fix that but I am VERY bad at soldering even though I do it all the time.

This is the PC stick I would recommend:
It may or may not have a soldered battery but you will not need to worry about that for a while because it is actually brand new with a warranty. I have 2 of these for my Mobile Internet Device project and they are really great. They use higher voltage USB-C which is more efficient on battery life due to less current losses in the wiring and Peukert's Law with the batteries. They are fast and reliable but a little larger than a classic PC stick. I guarantee they will work with FreeBSD but have not tried them yet with this.

The closest official term to describe what I have built is a "video wall". But one more for information than one like on the news. I found a great picture of a modern version of what I am talking about. NASA's stuff looks ancient compared to my network. Their video wall looks like poverty. Here is a good one made by someone that actually knows what they are doing.
Long link incoming:

The streams are ATSC from an HDHomerun twin tuner. If you notice on the large boards at NOCs and things like that they often have news feeds going along with many other things so that is what I am doing. I also have "internally produced" streams that I have in a media hosting area on the website. The streams are of my physical workspace where I build and work on things, my gaming computer (without sound because VLC desktop streaming does not support that), and finally a view from inside the 3D printer.

The web server is the "backup" web server that runs on another port so I can maintain a web presence when the main server is down like it is right now. It shows network status information including mains voltage that it dutifully collects from a "network agent" script I made. Also it has a link to my old Github and very basic info. I would like to add a little comment section where I reassure people that the website is coming back up one day. I am thinking about moving the main web server from Proxmox to FreeBSD Jails. I use Apache since that is all I have ever used.
The Samba server is just for staging and temporary files before they are archived. Speed is only around 35MB/sec but does not bother me. The main storage server is still running FreeBSD 11 with KDE 3.5 and has no servers running for security and I manually sit at it and "pull" data from other computers to be archived from time to time.

Status report - The 1GB swap is 24% in use after several days of running at full strength.
 
You must log in or register to reply here.
Back
Top