Python Python 3 selectors.select() fails for named pipes

[jef]

After getting my code running "properly" under Mac OS X and Python 3.6, I moved it over to a jail on FreeBSD 11.1-RELEASE-p1 and python36-3.6.1_4 and found that selectors.select() would block as expected until the first data was written to and read from the pipe, but then no longer block (unexpected). If I use a regular file on FreeBSD, it behaves as expected, blocking until more data is written to the file.

Has anyone run into this before?



The goal of the code behind the stripped-down example here is to allow "any" other process/script to easily pass short bits data ("commands") to a running Python program. I'm open to any solutions, as well as "better" ways of accomplishing the goal.

Works:

$ rm commands-in
$ touch commands-in
(run the program, while running)
$ echo "Some data" >> commands-in​

Fails (be prepared to ^C the program!):

$ rm commands-in
$ mkfifo commands-in
(run the program, while running)
$ echo "Some data" >> commands-in # ">" also fails​

Looking at key_event_list in the debugger shows that it does not change once "triggered" when reading from a pipe.

selectors.DefaultSelector() returns a selectors.KqueueSelector object under FreeBSD 11.1. Under Mac OS X, it returns a selectors.PollSelector object.

Changing to select_cp = selectors.PollSelector() does not change the behavior for a named pipe under FreeBSD.

select(timeout=None) should block until one (or more, if there were any) monitored file objects are ready. None is the stated default, but was made explicit to ensure it was properly set.

If timeout is None, the call will block until a monitored file object becomes ready.​

edit: same behavior seen for the C-esque select.select([cp], [], []) with no timeout (blocking)

import logging
import selectors


def test():

    command_pipe = 'commands-in'

    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger()

    logger.info(f"Opening command pipe: '{command_pipe}'")
    with open(command_pipe, 'r') as cp:
        select_cp = selectors.DefaultSelector()
        select_cp.register(cp, selectors.EVENT_READ)
        while True:
            key_event_list = select_cp.select(timeout=None)
            line = cp.readline()
            logger.info(f"Read: '{line}'")


if __name__ == '__main__':
    test()

 

[tobik@]

The default state of a FIFO is to wait for at least 1 reader and 1 writer to appear.

The initial open() in your program will block until some other process appears and opens the FIFO for writing e.g. your echo. The first readline() succeeds because there is data queued in the FIFO, but once the FIFO has no writers and no data queued in it, the next readline() will always return 0 bytes (=> EOF). This is also why select() will not block and return immediately.

After opening the FIFO for reading, open it again for writing to prevent your program from ever seeing EOF at all (i.e. pretend there is always at least 1 writer). There is no need to use select() here; readline() will already block and wait for new data.

import logging

def test():
    command_pipe = 'commands-in'

    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger()

    logger.info(f"Opening command pipe: '{command_pipe}'")
    with open(command_pipe, "r") as cp, open(command_pipe, "w"):
            while True:
                line = cp.readline()
                logger.info(f"Read: '{line}'")


if __name__ == '__main__':
    test()

 

[jef]

Thanks!

I had guessed that the initial blocking being due to the pipe's FIFO not "existing" until there was a reader and a writer, but the approach of opening it for write as well is not one that I would have thought of myself. Clean solution!