Serial console blocks/hangs at speeds > 9600 baud

P

pvoigt

I am on 10.1-RELEASE (amd64) and I have configured the serial console for emergency access. At 9600 baud it is working reliable, e.g. I can navigate through the boot loader, can see kernel and services messages and can finally login.

When I increase the serial speed above 9600 baud, everything seems to be OK. However, after some time, e.g. after some login processes, suddenly the console blocks or hangs. There is no output anymore and I cannot login anymore.

When investigating the process list, nothing unusual can be found, e.g. getty seems to be still waiting for connections. There are no error messages in /var/log/messages or dmesg. If I restart init with kill -HUP 1 a new getty is started, but login is still not possible.

Moreover, if I restart the server, even shutdown hangs and I have to hard reset on most of these cases. Lowering speed to 9600 baud brings me back a reliable serial console again.

I am currently out of ideas, because I am running a second server with the same mainboard and the same serial null modem cable under FreeNAS 9.3. Under FreeNAS the serial console works at 115200 baud without any issues.

Maybe I should add that I am using minicom under Linux to connect.

I am currently out of ideas and I do appreciate any suggestions.

Regards,
Peter
 
Last edited by a moderator:
Try the simple stuff first: press ctrl-q. Would a shutdown be paused if it could not send to the root console? Maybe.
 
junovitch said:
Can you show your /boot/loader.conf console related items and the console related items in /etc/ttys?
Click to expand...

I have:
Code: 
# cat /boot/loader.conf
geom_eli_load="YES"
aesni_load="YES"
coretemp_load="YES"
console="comconsole,vidconsole"
boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
# Avoid openjdk core dumps - see PR187238.
vm.pmap.pcid_enabled="0"
# Enable NewCons vt instead of default syscons. See UPDATING 20141001
# entry.
kern.vty=vt
and:
Code: 
# grep -E "ttyu0|115200" /etc/ttys
ttyu0  "/usr/libexec/getty std.115200"  vt100  on  secure
Besides that I have experimented with:
Code: 
hint.uart.0.baud="115200"
in /boot/device.hints and with
Code: 
-P -S115200
in /boot.config without success.

Regards,
Peter
 
wblock@ said:
Try the simple stuff first: press ctrl-q. Would a shutdown be paused if it could not send to the root console? Maybe.
Click to expand...

Sorry, but what doess this magic ctrl-q do? Will it bring back the serial console or does it continue shutdown process?

One general question in this context: Should I be able to change serial speed for the getty alone, if I change /etc/ttys and issue a subsequent kill -HUP 1? My observation is, that getty indeed will be started with the new speed according to the process list, but I am not able to login with the new speed. Instead a minicom login is possible only with the old speed.

Regards,
Peter

EDIT: After some search I have stumbled accross this about the ctrl-q mystery - at least to me:
http://unix.stackexchange.com/quest...er-accidentally-pressing-ctrl-s-in-a-terminal
 
Ctrl-s and ctrl-q are the software flow control characters. When a terminal is receiving characters too rapidly to display, it sends a ctrl-s. When it can accept more characters, it sends a ctrl-q. These are also called XON/XOFF. These characters can also be typed manually to halt or continue output.

It's possible to use hardware flow control instead, provided the cable is wired correctly and the software is configured that way at both ends. There can be problems if the computer and terminal are not configured consistently.
 
wblock@ said:
Ctrl-s and ctrl-q are the software flow control characters. When a terminal is receiving characters too rapidly to display, it sends a ctrl-s. When it can accept more characters, it sends a ctrl-q. These are also called XON/XOFF. These characters can also be typed manually to halt or continue output.

It's possible to use hardware flow control instead, provided the cable is wired correctly and the software is configured that way at both ends. There can be problems if the computer and terminal are not configured consistently.
Click to expand...

Thanks for clarifying. I was not aware that you can manually control XON/XOFF. I would like to check this with higher baud rates, or more precisely: I would like to check, if my "hanging" console is stopped by a ctrl-s and can be "reactivated" with ctrl-q. However, I would like to avoid rebooting my server. Therefore I have asked in my last post, if the getty baud rate in /etc/ttys can be changed without a reboot. Could you please give me some advice on this?

Regards,
Peter
 
wblock@ said:
Restarting init(8) with kill -HUP 1 is probably enough. But why not just type ctrl-q and see if the output resumes?
Click to expand...

Because I have currently switched back to reliable 9600 baud. And according to my above described observations, changing /etc/ttys and kill -HUP 1 is not enough to change getty speed. My hope is, I have simply forgotten a step but I am afraid I will have to reboot.

Regards,
Peter
 
Well, time passed by and I would like to reactivate this thread.

I am on 10.2-RELEASE now and I have experimented a bit more with the serial console. For any speed above 9600 baud I am observing this unpredictable hang. Unfortunately, I cannot reactivate the serial console with ctrl-s or ctrl-q. Even kill -HUP 1 or killing the tty process does not bring back the serial console once it hangs. I am forced to reboot to get my serial console back.

I would appreciate any instructions on how to configure hardware flow control instead. I am using minicom on the client machine. The very same serial cable is working with FreeNAS at 115200 baud on another machine with the same motherboard.

Peter
 
It acts like you are already using hardware flow control, but without a correctly wired cable. I don't use serial consoles, and do not know how it is set up by default, but am investigating. How many wires are in the cable, and how are they connected?
 
Thank you for your reply. I have just found some time to messure the pin connections of my DB9-DB9 null modem cable. Unfortunately, the battery of my digital meter is empty. As soon as I have replaced it I will report to the forum.

Meanwhile I have experimented with different speeds. With 115200 baud I am still having the described hang mostlly within one minute. At 38400 baud the serial connection is working stable. I have done my tests both with a serial port on client side and a USB port on client side with same results.

Regards,
Peter
 
Well, finally I have replaced my battery and have determined the DB9/female-DB9/female pin connections of my cable:

1 - 8,7
2 - 3
3 - 2
4 - 6
5 - 5
6 - 4
7 - 1
8 - 1
9 - unused

Just for clarification: The pin numbers are determined when looking into the female DB9 connector:

5 4 3 2 1
9 8 7 6

I am not an expert on the correct pin connections of a null modem cable. I have been searching the web for the correct connections of a null modem cable with contradicting information. I therefore cannot judge, if my cable has proper null modem layout. My FreeNAS machines, my Soekris and my Alix2D13 are all working properly with 115200 baud with this cable. So far I am having just problems with FreeBSD and highest baud rates. After several days of testing 38400 baud turn out to be as stable as 9600 baud. 115200 baud definitely produces hangs and 57600 is so far untested with FreeBSD.

I would really like to know, if my cable layout is suited for hardware flow control. And if so: How can I use it with e.g. Minicom on a Linux box.

Thanks in advance for feedback
Peter
 
Based on that, I'd say no, it is not meant for hardware flow control. CTS and RTS (pins 7 and 8) are tied to CD (pin 1). It ought to work with software flow control, though, which would explain the situations where it does work.
 
wblock@ said:
Based on that, I'd say no, it is not meant for hardware flow control. CTS and RTS (pins 7 and 8) are tied to CD (pin 1). It ought to work with software flow control, though, which would explain the situations where it does work.
Click to expand...

How should pins 7 and 8 be tied in order to provide a hardware flow compatible null modem? I have searched for it and found somewhat contradicting answers. Is this the correct layout: http://lavalink.com/wp-content/uploads/2012/05/full_handshaking.jpg

Are there other settings required besides a correctly wired cable, e.g. on the FreeBSD machine or the client side, in order to profit from hardware flow control?

Do you have an idea why pure FreeBSD is obviously more sensitive to not ideally wired null modem cables like FreeNAS or pfSense? My FreeNAS 9.2 machines are even running the same motherboard and do not make any problem at 115200 baud.

Peter
 
That looks better. In that diagram, they show CTS connected to RTS, and vice versa.

The other systems have to be configured for software flow control. The question is where that is done. Many they have an added hint to set software flow control. You did now show your whole /etc/ttys earlier, particularly the console entry.
 
I am not aware that there are other settings besides the above in /etc/ttys that control serial console behavior and in particular usage of software flow control. Nevertheless, here is my current complete file content:
Code: 
console none  unknown off secure
#
ttyv0  "/usr/libexec/getty Pc"  xterm  on  secure
# Virtual terminals
ttyv1  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv2  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv3  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv4  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv5  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv6  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv7  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv8  "/usr/local/bin/xdm -nodaemon"  xterm  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0  "/usr/libexec/getty std.38400"  vt100  on  secure
ttyu1  "/usr/libexec/getty std.9600"  dialup  off secure
ttyu2  "/usr/libexec/getty std.9600"  dialup  off secure
ttyu3  "/usr/libexec/getty std.9600"  dialup  off secure
# Dumb console
dcons  "/usr/libexec/getty std.9600"  vt100  off secure

Peter
 
A software flow control cable really only needs ground, TX, and RX. Technically, signal ground also. For null modem, TX and RX are crossed over. That is the bare minimum necessary. This does not work on all systems because some require that certain signals on the connector are valid, like carrier detect. The extra connections on your cable might actually be preventing it from working.

Remember also that the systems on both ends of the cable have to be set for the same type of handshake.

I don't see anything weird in your /etc/ttys except the std.38400 entry, but that should not be a problem.

There are serial break-out boxes that give control over which pins are connected and where. Before molded cables, it was possible to remove connector pins non-destructively.
 
wblock@ said:
A software flow control cable really only needs ground, TX, and RX. Technically, signal ground also. For null modem, TX and RX are crossed over. That is the bare minimum necessary. This does not work on all systems because some require that certain signals on the connector are valid, like carrier detect. The extra connections on your cable might actually be preventing it from working.
Click to expand...
Ah, that might be the reason.

wblock@ said:
Remember also that the systems on both ends of the cable have to be set for the same type of handshake.
Click to expand...
I usually just start minicom on a Linux client where any kind of flow control is disabled. I am working this way for years without any issue - except the "native" FreeBSD machine. My Soekris, Alix and FreeNAS machines (with identical motherboard) working with this setting at 115200 baud and the same null modem cable.

Additionally, I have checked the serial port settings with stty on FreeBSD
Code: 
# stty -a -f /dev/ttyu0 |grep -E "xon|off|crtscts"
iflags: -istrip -icrnl -inlcr -igncr -ixon -ixoff -ixany -imaxbel -ignbrk
cflags: cread cs8 -parenb -parodd -hupcl clocal -cstopb -crtscts -dsrflow
and on Linux side:
Code: 
# stty -a -F /dev/ttyS0 |grep -E "xon|off|crtscts"
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff

wblock@ said:
I don't see anything weird in your /etc/ttys except the std.38400 entry, but that should not be a problem.
Click to expand...
38400 baud is still working reliably on my FreeBSD server.

There is obviously no simple answer why 115200 baud are not working reliably on my "native" FreeBSD machine compared to the others. But tests during this discussion revealed that increasing baud rate up to 38400 is working reliably on the FreeBSD machine.

Thank you very much for your feedback so far.

Peter
 
It would still be worth comparing with FreeNAS. Finding out what they have changed from the default FreeBSD settings should allow the use of that same cable.
 
wblock@ said:
It would still be worth comparing with FreeNAS. Finding out what they have changed from the default FreeBSD settings should allow the use of that same cable.
Click to expand...
You are right and that was what I tried before posting here. But to be honest I have not be successfull in analyzing the differences and in drawing conclusions for 115200 baud on my "native" FreeBSD machine.

I am running FreeNAS 9.3 stable (FreeNAS-9.3-STABLE-201503120630) based on FreeBSD-9.3-RELEASE-p10. It is some month old and I should upgrade soon to the 9.3.1 stable branch. FreeNAS has a ZFS root file system and is booted by Grub2. System is not directly comparable with my FreeBSD 10.2-RELEASE-p3. Nevertheless, serial console at 115200 baud is working correctly. I see no differences in
/etc/ttys, /boot.config, /boot/device.hints and stty output. /boot/loader.conf does not contain serial console configuration parameters because FreeNAS is booted by grub2.

My pfSense machine is based on FreeBSD 10.1-RELEASE-p15 and is working fine with 115200 baud as well. I can observe just differences in /etc/ttys:
Code: 
ttyv0  "/usr/libexec/getty Pc" cons25  on  secure
ttyu0  "/usr/libexec/getty std.115200" cons25  onifconsole  secure
Furthermore, /boot/device.hints does not contain a baud rate. That is all and does not help me. May be anybody else has some ideas.

Peter
 
ColdfireMC said:
From what are you hearing the serial console? This sounds like a DCE/DTE confusion
Click to expand...
I am not sure what you mean with "hearing". If you ask for the client: I am using minicom from a KDE konsole under Linux.
And could you please be a bit more specific with DCE (client machine with minicom)/DTE (getty on FreeBSD side) confusion?

Peter
 
If/when it hangs, check if the hardware still handles the characters.

iostat; echo foo > /dev/ttyu0; iostat. vmstat -ishould show an increase in interrupts as well. ps lax might show what the hung processes are waiting for.

I see a 3wire.115200 entry in /etc/gettytab, commented "Entries for 3-wire serial terminals".

Juha
 
You must log in or register to reply here.
Back
Top