FreeBSD 11 hangs on rpi-B

G

giorgiob

Hello,

I am running the official FreeBSD 11 image on a Raspberry pi B and most of the things seem to work but I had a problem twice recently.

My configuration: rpi-b with two externally powered USB disks, connected to my home LAN via
ethernet. I also have a screen and a keyboard attached (console).

I use the rpi as a backup system for all my computers. Today I was transferring files from my desktop to the disks on the rpi using rsync (via ssh) when the transfer just stopped. I tried to open a terminal using ssh but the rpi does not respond. I checked port 22 on the rpi with
nc -v -w 1 rpi-1 -z 22
and I get:
Connection to rpi-1 22 port [tcp/ssh] succeeded!
I also tried ping, and the rpi-1 is there.

So I tried logging in on the console. The console is responsive but after I entered my username, it does not prompt for the password. If I type enter multiple times, the console reacts, but still no login.

I thought there might be a problem with the external USB drives, so I unplugged them. The console registered this: da0: .... detached, da1: ... detached.

So the system seems to be running but there is no way I can log in. I do not understand where it gets stuck. The only solution I have is to pull the plug and reboot. This has happened twice in the last few days.

Do you have any hints on how to find out more about this problem?

UPDATE: I have found this https://unix.stackexchange.com/questions/242898/rsync-keeps-disconnecting-broken-pipe
Since I am mirroring a folder that contains more than 500 G of files, could it be that the raspberry runs out of memory and no new processes can be started?
 
The obvious question is: where there any error messages, in particular from the kernel, on the console? I presume not, otherwise you would have told us. You should review /var/log/messages and tell use whether there's anything suspicious there. But I think basically your "diagnosis" (actually guesswork) may be correct: Something (probably rsync) has used all the memory, and while the computer isn't down, it's working very badly. Here are a few observations:

Ping works, connection to port 22 works: Kernel is basically functioning, although parts of the kernel might be damaged.

The existing getty process still works well enough to accept your username (which means that the tty IO subsystem in the kernel works). But then when it tries to verify your password, it stalls. That probably requires file IO and fork'ing processes, so either complete death of file systems or exhausted memory is a sensible explanation.

You unplugged disks while the file systems on them were mounted. That was nasty of you. On the other hand, given that the machine was probably going down soon anyhow, it probably made no difference.

My only other suggestion for debugging this (other than trying to read log files): Next time you do this, be already logged in as root on the console. There, run tail -f /var/log/messages & and vmstat 1. That will give you memory statistics every second, and interspersed anything that goes to the log. Good luck!
 
giorgiob said:
UPDATE: I have found this https://unix.stackexchange.com/questions/242898/rsync-keeps-disconnecting-broken-pipe
Since I am mirroring a folder that contains more than 500 G of files, could it be that the raspberry runs out of memory and no new processes can be started?
Click to expand...
That's a very good possibility and would result in the behavior you're seeing, the TCP stack is still active (ping still works) but the system can't find the memory to open a full SSH session. It's probably desperately trying to free up some memory and swapping like mad. On a "normal" system you'd actually hear the HD thrashing around but SD cards are awfully quiet ;)
 
SirDice said:
That's a very good possibility and would result in the behavior you're seeing, the TCP stack is still active (ping still works) but the system can't find the memory to open a full SSH session. It's probably desperately trying to free up some memory and swapping like mad. On a "normal" system you'd actually hear the HD thrashing around but SD cards are awfully quiet ;)
Click to expand...
I do not have a swap space on the SD card (unless one is set up by default, but I could not find it), I have created two 256 M swap files, one on each external USB disk. I have observed the problem again today with htop running on the console. When the raspi hung also htop froze. I will have to retry following the indications of ralphbsz (I tried that before, but the problem did not appear).

I had the same problem also with unison, but maybe unison uses rsync(?)
 
I have a similar problem, and still haven't found a solution... I think the problem is tied to the SD card I am using, though. It seems the SD card has some internal buffer or something, and at some point it locks up, grinding the entire system to a halt.

I've put the card in a Windows machine before and tried a mass-copy of files to it... after a while, it freezes up there too, and the only remedy is to physically pop out the card and try again.

For now I just unplug and replug the device, and external monitoring that sends me an email when my site goes down.
 
Hornpipe2 said:
I have a similar problem, and still haven't found a solution... I think the problem is tied to the SD card I am using, though. It seems the SD card has some internal buffer or something, and at some point it locks up, grinding the entire system to a halt.

I've put the card in a Windows machine before and tried a mass-copy of files to it... after a while, it freezes up there too, and the only remedy is to physically pop out the card and try again.

For now I just unplug and replug the device, and external monitoring that sends me an email when my site goes down.
Click to expand...
Thanks for the hint. Interestingly, (1) my SD card is new and I did not have problems with a card from the same producer (but different model) under raspbian (2) the system hangs while running rsync, which writes to an external disk. I have just reproduced the problem right now: I cannot start any new command even though there is plenty of memory: vmstat 1, logging on the console, indicates only 362M are in use, while I have 512M RAM + 1G swap on the external disks.

Anyway, thanks for the hint. I'll try switching back to the other SD card.

P.S.: I was planning on using all my three raspberries on FreeBSD as servers, but if it keeps hanging I do not feel very comfortable with it.
 
Last edited:
I still haven't solved my problem, which is occurring from time to time while I am transferring files to / from the rpi using rsync or tar through a ssh
connection.
I have never managed to log anything before but today I finally could reproduce the problem with some log on the console.

So here's what happened. I had several ssh sessions opened on the rpi and from one of them I started to transfer files over ssh with the command
tar -cpf - <input directory> | ssh <username>@<host> tar -C <destination> -vxpf -
After a while the file transfer froze. I tried to open a new ssh connection to the rpi in another terminal but it hung too.
Then I noticed I had another open ssh session on the rpi where I had recently used sudo, so I entered:
sudo reboot
This hung too, but on the console the following error message appeared:
Code: 
vm_thread_new: kstack allocation failed
The message appeared once, then again twice after a few minutes, then again four more times after a few minutes.

In the blocked file transfer (which is started on the rpi) I type C-C. In that terminal the message
Code: 
^C Killed by signal 2.
appeared, but the process still remained blocked.

If I understand correctly, this could be a memory problem. The rpi has only 512 M RAM, but I have 512 M swap file on disk. Also, I would not understand why the
process should run out of memory: I thought the data would be streamed through tar and ssh in relatively small chunks.
 
You must log in or register to reply here.
Back
Top