Reading / Writing in serial port

[jjachuf]

Hi,

I am attempted to communicate in a bidirectional way with an Arduino device using the serial port, obviously in FreeBSD. I have to read what comes from the Arduino and also to write.

I tried with Python [1] and Rust [2] and in both cases I get errors, but the same code in Linux or Windows works fine.
I was wondering if it is necessary to touch some variable with sysctl or something similar in FreeBSD, o there is any limitation in FreeBSD

I can do it in Linux, but I want to do it in FreeBSD because all the rest of the app is ready to deploy as a jail.


[1] https://github.com/pyserial/pyserial/discussions/622
[2] https://gitlab.com/susurrus/serialport-rs/-/issues/114

 

[SirDice]

Check the permissions on /dev/cuaU0:

root@molly:~ # ll /dev/cuau0*
crw-rw----  1 uucp  dialer  0x43 Nov  9 21:48 /dev/cuau0
crw-rw----  1 uucp  dialer  0x44 Nov  9 21:48 /dev/cuau0.init
crw-rw----  1 uucp  dialer  0x45 Nov  9 21:48 /dev/cuau0.lock

As you can see only the uucp user and the dialer group are able to read/write to the device. Simplest solution is just to add your user to the dialer group. pw groupmod dialer -m <user>

 

[jjachuf]

Check the permissions on /dev/cuaU0:

root@molly:~ # ll /dev/cuau0*
crw-rw----  1 uucp  dialer  0x43 Nov  9 21:48 /dev/cuau0
crw-rw----  1 uucp  dialer  0x44 Nov  9 21:48 /dev/cuau0.init
crw-rw----  1 uucp  dialer  0x45 Nov  9 21:48 /dev/cuau0.lock

As you can see only the uucp user and the dialer group are able to read/write to the device. Simplest solution is just to add your user to the dialer group. pw groupmod dialer -m <user>

Hi SirDice

I'm running the app as root

 

[SirDice]

You mentioned you're running this in a jail? Did you set devfs.rules correctly? Many devices are hidden away in a jail.

 

[jjachuf]

You mentioned you're running this in a jail? Did you set devfs.rules correctly? Many devices are hidden away in a jail.

Thanks SirDice . Yes, that would be for the deployment, but now in development is running directly on the "host".
In the explanation I put in PySerial [1] I show that I can read and write in the first cycle, but when it is executed for the second time it gives error. Maybe it's a limitation of the PySerial itself
[1] https://github.com/pyserial/pyserial/discussions/622

 

[ralphbsz]

In general, serial port access works fine. I use it several places on my machines, both FreeBSD and Linux, both from root and from user accounts, both in C and in Python (with pyserial). Common problems include permissions (see SirDice's message), but you're running as root so that should be OK.

... it gives error.

We can't help debug "error". There are zillions of possible errors. You need to tell us exactly what the error is, and what causes the error.

I would also suggest beginning with a super simple test case: Take a wire, and bridge pins 2 and 3 of the serial port connector on the computer, leaving all other pins unconnected (doesn't matter whether this is a 9-pin or 25-pin connector). That bridges TxD and RxD, meaning any data you transmit will be immediately received. Set the port for any baud rate, disable hardware flow control, and run a simple software test that you get data transmitted back again. The pyserial installation comes with a little terminal emulator program (I think it's called minicom something) which you can use for testing.

 

[Tieks]

The second call to reset_input_buffer() causes the problem. I never used pyserial, but it occurs to me that you may not need that call at all. Just try to run without it.
If there is a need to explicitly empty that buffer at that point, you might take a look at flushInput() instead.

 

[jjachuf]

In general, serial port access works fine. I use it several places on my machines, both FreeBSD and Linux, both from root and from user accounts, both in C and in Python (with pyserial). Common problems include permissions (see SirDice's message), but you're running as root so that should be OK.


We can't help debug "error". There are zillions of possible errors. You need to tell us exactly what the error is, and what causes the error.

Hi ralphbsz
Yes, so as not to repeat I had put what I mentioned in the PySerial repository: https://github.com/pyserial/pyserial/discussions/622

ser = serial.Serial('/dev/cuaU0')
while True:   
    ser.reset_input_buffer()
    ser_bytes = ser.readline()
    decoded_bytes = ser_bytes[0:len(ser_bytes) - 2].decode("utf-8")

    print('Bytes:', ser_bytes)
    print('UTF-8:', decoded_bytes)
  
    resp = ser.write(b'v')
    print('Resp:', resp)
    sleep(2)

The error is given when cleaning the buffer. The first cycle in the loop works fine and when he tries the heavy failure.

File "/home/jose/desarrollos/fav/desarollo/venv/lib/python3.8/site-packages/serial/serialposix.py", line 677, in _reset_input_buffer    termios.tcflush(self.fd, termios.TCIFLUSH) termios.error: (6, 'Device not configured')

If I do not run the ser.reset_input_buffer() line the same, execute the first cycle and in the second one gives another error

raise SerialException(
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)

The next portion of code works fine. Every time a data is sent from Arduino, I can see it by console and it is not necessary run ser_bytes = ser.readline()

ser = serial.Serial('/dev/cuaU0')
while True:
    ser_bytes = ser.readline()
    decoded_bytes = ser_bytes[0:len(ser_bytes) - 2].decode("utf-8")
    print(decoded_bytes)   
    sleep(2)

I have the problem when I read data and, based on these data, I write in the Arduino

I would also suggest beginning with a super simple test case: Take a wire, and bridge pins 2 and 3 of the serial port connector on the computer, leaving all other pins unconnected (doesn't matter whether this is a 9-pin or 25-pin connector). That bridges TxD and RxD, meaning any data you transmit will be immediately received. Set the port for any baud rate, disable hardware flow control, and run a simple software test that you get data transmitted back again. The pyserial installation comes with a little terminal emulator program (I think it's called minicom something) which you can use for testing.

Ok I will see

Thanks for your answer

 

[jjachuf]

The second call to reset_input_buffer() causes the problem. I never used pyserial, but it occurs to me that you may not need that call at all. Just try to run without it.
If there is a need to explicitly empty that buffer at that point, you might take a look at flushInput() instead.

Hi Tieks

I answered this to ralphbsz , it is necessary to clean the buffer because (in Linux and Windows) there is always the last reading and does not stop to wait for the Arduino to send you data. And if I do not run it, it error:
And about flushInput():
reset_input_buffer()
Flush input buffer, discarding all its contents.
Changed in version 3.0: renamed from flushInput()

raise SerialException( serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)

 

[jjachuf]

Seeing the PySerial code has differences according to the OS. Then I tried with Linuxlator but I get the same result as native to FreeBSD

 

[ralphbsz]

Let's untangle this. You are getting two separate errors. If you call reset_input_buffer() on every iteration of the loop, then the second time around, it gives you "device not configured". That is nothing but the error string corresponding to errno ENXIO, which literally means: you are trying to do something to a device that does not exist, or you are asking a device to do something it is not capable of. Common examples include: You open a device by name (in your case /dev/cuaU0), but by the time the call comes around, the USB service has disconnected it, and it no longer actually exists. There are other causes, such as sending a device an ioctl that it can never execute, such as sending a SCSI disk the command to "eject page" (which is a good command for SCSI printers, but disks don't do that), or sending a device an ioctl that it temporarily impossible (like asking a tape drive to rewind the tape, but no tape is physically present). Now, I have no idea why calling reset_input_buffer() would cause that, other than that the device has been disconnected. Suggestion: Look in your log for disconnection events. Maybe add to your python code to do an "ls -l /dev/cuaU0" and "stty -a < /dev/cuaU0", to see that the device is in good shape.

Second error: If you don't use reset_input_buffer(), you get "device reports readiness to read but returned no data". That's not an OS error, it's a pyserial internal error. It happens when you have to programs using the same serial port, and both are reading from the same input (which happens regularly to me, if I forget to stop another program). And it can happen if the device is disconnected while you are using it.

My suspicion is that debugging device disconnects might help.

 

[Tieks]

it is necessary to clean the buffer because (in Linux and Windows) there is always the last reading and does not stop to wait for the Arduino to send you data.

So you need to run this code on Windows, Linux and FreeBSD, is that correct?
It seems to me that ser.readline() on FreeBSD behaves differently. On FreeBSD it stops executing until there is for data from the other side, whereas in Windows and Linux it keeps executing without returning data. Is that right?

 

[jjachuf]

So you need to run this code on Windows, Linux and FreeBSD, is that correct?

No, I need it in FreeBSD, but if it does not work I'll go for Linux. I tried in Linux and Windows to rule out that it is my code, both work fine.

It seems to me that ser.readline() on FreeBSD behaves differently. On FreeBSD it stops executing until there is for data from the other side, whereas in Windows and Linux it keeps executing without returning data. Is that right?

If it's like you say, in Linux and Windows, the last data is in the buffer and therefore considers it a new reading, as if it came from the arduino

 

[jjachuf]

Let's untangle this. You are getting two separate errors. If you call reset_input_buffer() on every iteration of the loop, then the second time around, it gives you "device not configured". That is nothing but the error string corresponding to errno ENXIO, which literally means: you are trying to do something to a device that does not exist, or you are asking a device to do something it is not capable of. Common examples include: You open a device by name (in your case /dev/cuaU0), but by the time the call comes around, the USB service has disconnected it, and it no longer actually exists. There are other causes, such as sending a device an ioctl that it can never execute, such as sending a SCSI disk the command to "eject page" (which is a good command for SCSI printers, but disks don't do that), or sending a device an ioctl that it temporarily impossible (like asking a tape drive to rewind the tape, but no tape is physically present). Now, I have no idea why calling reset_input_buffer() would cause that, other than that the device has been disconnected. Suggestion: Look in your log for disconnection events. Maybe add to your python code to do an "ls -l /dev/cuaU0" and "stty -a < /dev/cuaU0", to see that the device is in good shape.

Now I'm going to try what you say. Something I noticed is that after error the device /dev/cuaU0 does not exist and now has the name /dev/cuaU1. Then I have to disconnect from the USB and reconnect me so that it will be /dev/cuaU0 again

Second error: If you don't use reset_input_buffer(), you get "device reports readiness to read but returned no data". That's not an OS error, it's a pyserial internal error. It happens when you have to programs using the same serial port, and both are reading from the same input (which happens regularly to me, if I forget to stop another program). And it can happen if the device is disconnected while you are using it.

My suspicion is that debugging device disconnects might help.

 

[jjachuf]

without
ser.reset_input_buffer()

==================================================


iteration 1
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1bf 14 nov. 11:13 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1c0 14 nov. 11:13 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1c1 14 nov. 11:13 /dev/cuaU0.lock
--------------------------------------------------
==================================================
Waiting for data...
reading in UTF-8: usuario:2
--------------------------------------------------
Resp: 1
==================================================


iteration 2
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1bf 14 nov. 11:14 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1c0 14 nov. 11:13 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1c1 14 nov. 11:13 /dev/cuaU0.lock
crw-rw---- 1 uucp dialer 0x1d2 14 nov. 11:14 /dev/cuaU1
crw-rw---- 1 uucp dialer 0x1d3 14 nov. 11:14 /dev/cuaU1.init
crw-rw---- 1 uucp dialer 0x1d4 14 nov. 11:14 /dev/cuaU1.lock
--------------------------------------------------
==================================================
Waiting for data...
Traceback (most recent call last):
File "/home/jose/desarrollos/fav/desarollo/puerto/read_port.py", line 133, in <module>
ser_bytes = ser.readline()
File "/home/jose/desarrollos/fav/desarollo/venv/lib/python3.8/site-packages/serial/serialposix.py", line 595, in read
raise SerialException(
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)

with
ser.reset_input_buffer()

==================================================


iteration 1
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1c3 14 nov. 11:17 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1c4 14 nov. 11:17 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1c5 14 nov. 11:17 /dev/cuaU0.lock
--------------------------------------------------
==================================================
Waiting for data...
reading in UTF-8: usuario:2
--------------------------------------------------
Resp: 1
==================================================


iteration 2
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1c3 14 nov. 11:17 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1c4 14 nov. 11:17 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1c5 14 nov. 11:17 /dev/cuaU0.lock
crw-rw---- 1 uucp dialer 0x1cb 14 nov. 11:17 /dev/cuaU1
crw-rw---- 1 uucp dialer 0x1cc 14 nov. 11:17 /dev/cuaU1.init
crw-rw---- 1 uucp dialer 0x1d5 14 nov. 11:17 /dev/cuaU1.lock
--------------------------------------------------
==================================================
Waiting for data...
Traceback (most recent call last):
File "/home/jose/desarrollos/fav/desarollo/puerto/read_port.py", line 132, in <module>
ser.reset_input_buffer()
File "/home/jose/desarrollos/fav/desarollo/venv/lib/python3.8/site-packages/serial/serialposix.py", line 683, in reset_input_buffer
self._reset_input_buffer()
File "/home/jose/desarrollos/fav/desarollo/venv/lib/python3.8/site-packages/serial/serialposix.py", line 677, in _reset_input_buffer
termios.tcflush(self.fd, termios.TCIFLUSH)
termios.error: (6, 'Device not configured')

In both cases in the iteration 2 appears /dev/cuaU1

 

[jjachuf]

with
ser.reset_input_buffer()

and when I do not write about the Arduino
I do not run the line:
resp = ser.write('v'.encode('ascii'))

==================================================


iteration 1
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1c5 14 nov. 11:24 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1d6 14 nov. 11:24 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1d7 14 nov. 11:24 /dev/cuaU0.lock
--------------------------------------------------
==================================================
Waiting for data...
reading in UTF-8: usuario:2
--------------------------------------------------
==================================================


iteration 2
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1c5 14 nov. 11:24 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1d6 14 nov. 11:24 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1d7 14 nov. 11:24 /dev/cuaU0.lock
--------------------------------------------------
==================================================
Waiting for data...
reading in UTF-8: usuario:2
--------------------------------------------------
==================================================


iteration 3
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1c5 14 nov. 11:24 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1d6 14 nov. 11:24 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1d7 14 nov. 11:24 /dev/cuaU0.lock
--------------------------------------------------
==================================================
Waiting for data...
reading in UTF-8: usuario:1
--------------------------------------------------
==================================================


iteration 4
--------------------------------------------------
is_open True
--------------------------------------------------
ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1c5 14 nov. 11:24 /dev/cuaU0
crw-rw---- 1 uucp dialer 0x1d6 14 nov. 11:24 /dev/cuaU0.init
crw-rw---- 1 uucp dialer 0x1d7 14 nov. 11:24 /dev/cuaU0.lock
--------------------------------------------------
==================================================
Waiting for data...

The port /dev/cuaU1 is not created
for some reason when I write the second port is created
Jose

 

[ralphbsz]

Look at the dmesg output to maybe get an idea *why* the device is disconnected.

My guess would be: insufficient power connection.

 

[jjachuf]

Look at the dmesg output to maybe get an idea *why* the device is disconnected.

My guess would be: insufficient power connection.

This is before run [resp = ser.write('v'.encode('ascii'))]

jose@lenovo:~ % dmesg | grep uchcom
uchcom0 on uhub0
uchcom0: <vendor 0x1a86 USB2.0-Ser, rev 1.10/2.54, addr 4> on usbus0
uchcom0: CH340 detected

and this after run

jose@lenovo:~ % dmesg | grep uchcom
uchcom0 on uhub0
uchcom0: <vendor 0x1a86 USB2.0-Ser, rev 1.10/2.54, addr 4> on usbus0
uchcom0: CH340 detected
uchcom0: at uhub0, port 1, addr 4 (disconnected)
uchcom0: detached
uchcom0 on uhub0
uchcom0: <vendor 0x1a86 USB2.0-Ser, rev 1.10/2.54, addr 4> on usbus0
uchcom0: CH340 detected

I do not see the reason

 

[jjachuf]

I contribute something else, then that the app fails and stops, if I do an ls in dev obtain only cuaU1. cuaU0 disappeared

# ls -l /dev/cuaU*
crw-rw---- 1 uucp dialer 0x1d2 15 nov. 17:50 /dev/cuaU1
crw-rw---- 1 uucp dialer 0x1d3 15 nov. 17:50 /dev/cuaU1.init
crw-rw---- 1 uucp dialer 0x1d4 15 nov. 17:50 /dev/cuaU1.lock

# usbconfig
ugen0.1: <0x8086 XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <Logitech USB Receiver> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (98mA)
ugen0.3: <vendor 0x8087 product 0x0a2a> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.4: <Azurewave EasyCamera> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (500mA)
ugen0.5: <vendor 0x1a86 USB2.0-Ser> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (96mA)

 

[ralphbsz]

I do not see the reason

But it is clear what happens: When running your program, it causes cua0 to be disconnected, and then immediately reconnect (see below). Now, I agree with you: no idea why this happens, but it is clear that it does happen.

Debugging the why is going to be interesting. I would start by making sure you are not using flaky cables and connectors, nor cables that are too small. I've seen lots of weird USB problems, which are caused by bad connections, in particular on the power lines.

Also: Many USB to serial adapters use FTDI chips. Those chips are often cloned in China, usually badly. Cheap Chinese serial adapter clones cause all manners of funny results. What I do now is to throw any serial adaptor that doesn't work in the trash, and replace them with ones from reputable sources (such as Adafruit). This could be part of your problem, but it doesn't have to be.

I contribute something else, then that the app fails and stops, if I do an ls in dev obtain only cuaU1. cuaU0 disappeared

That makes perfect sense: Clearly the USB system can not create two devices with the same name; there can't be two /dev/cuaU0 that refer to different chips. To prevent confusion, the system will also not create a new /dev/cuaU0 for a small timeout period after the old one has been removed. So if a chip has been removed, and another chip inserted, it makes sense to give the second one a different name.

This is a symptom of a much more generic design problem: USB devices are not always discovered in the same order, so their device names can be unpredictable. You can't rely on one particular device always being /dev/cuaU0, if other USB serial things may get attached. What I do in my code is that when looking for a specific piece of hardware, I iterate over all /dev/cuaU* things, see whether they are already in use (if yes, leave them alone), if not ask them for some individual characteristics, and then use the one that matches.

 

[jjachuf]

Hi ralphbsz

But it is clear what happens: When running your program, it causes cua0 to be disconnected, and then immediately reconnect (see below). Now, I agree with you: no idea why this happens, but it is clear that it does happen.

I agree with you. I mean it's detached, but I do not see the reason.

Debugging the why is going to be interesting. I would start by making sure you are not using flaky cables and connectors, nor cables that are too small. I've seen lots of weird USB problems, which are caused by bad connections, in particular on the power lines.

Also: Many USB to serial adapters use FTDI chips. Those chips are often cloned in China, usually badly. Cheap Chinese serial adapter clones cause all manners of funny results. What I do now is to throw any serial adaptor that doesn't work in the trash, and replace them with ones from reputable sources (such as Adafruit). This could be part of your problem, but it doesn't have to be.

Some notes about this:
1) In the same laptop (and same usb port) with Linux works fine. I was wondering if the power handling in the USB port does SO or it is something of the Motherboard itself
2) This week my customer will bring me another device to replace the one I'm using

You have added or changed something with sysctl or in sysctl.conf?

 

[ralphbsz]

Some notes about this:
1) In the same laptop (and same usb port) with Linux works fine.

Some power management of USB ports can be done in software, but not on all USB ports. Old traditional USB2.0 ports had Gnd and +5V hardwired to the computer supply. Modern USB-C ports have very complex power management. And in the middle is a big grey zone.

You have added or changed something with sysctl or in sysctl.conf?

No, completely stock FreeBSD 12.2, and USB serial hasn't changed behavior for me since version 8 or 9.

 

[jjachuf]

ralphbsz In your app you do reading and writing on the device?

 

[msplsh]

So, you're getting ENXIO when termios uses ioctl to send TIOCFLUSH to the descriptor... Digging through the source, I think you need to take heart the comment in the source for this driver, uchcom.c

"Driver for WinChipHead CH341/340, the worst USB-serial chip in the world"

 

[ralphbsz]

ralphbsz In your app you do reading and writing on the device?

Yes, and there is more than one program. I talk to Omega industrial controllers (which have the USB chip built in), to Weeder data acquisition boards, to a HAI UPB PIM, and finally to a Dell modem. The last one is read only in production (I use it just to monitor caller ID), but I've written to it to configure it, by hand with a program like minicom.

The only problem I see once bad USB adapters are eliminated is "device reports readiness to read but returned no data". This happens if two programs are accessing the same serial device at the same time. In most cases, I have eliminated this possibility (by locking the devices so they can only be used by one reader/writer), but I haven't had time to clean up the UPB PIM software, so it occasionally happens.