C OpenSSL questions

[lost_in_c]

If I use SSL_read_ex on an open connection with a client, the program blocks until the client writes something. This means I sort of have to guess when there will be a request and use read only there. But sometimes I want to be able to tell whether the client has anything to ask before I send anything. SSL_peek blocks the same as SSL_read, and SSL_has_pending seems to give 0 every time, seeming to indicate that a read function has to already be deployed in order to know if there is any buffered data from the client to begin with. The underlying BIO object is nonblocking, but I don't think that matters, I think the blocking behaviour is particular to each function in this case.

Does this make any sense? Does anybody have experience with this? Did I write this all wrong, or is it expected behaviour and there is some other way to check if the client wants to write before blocking with SSL_read?

Google is not friendly to OpenSSL questions, always giving the same generic "how to check certificate chain" results, and my scouring of the library reference has yielded nothing on this yet.

 

[CShell]

The traditional way to perform an asynchronous read(2) on a socket is to use fnctl(2) to set OPTION F_SETFL with the option O_NONBLOCK on the socket. This will result in your read(2) call returning (-1) and errno will be set to (EAGAIN) - which means call read(2) again "some time in the future" to see if there is data available. Otherwise your read call will return ssize_t, which is the actual number of bytes read from the socket.

In the OpenSSL world - you can actually just "get" the socket descriptor from the OpenSSL library and use either poll(2) or older style select(2) to see if the socket descriptor is "ready-to-read". Use the OpenSSL SSL_get_fd() call to get the "raw" file descriptor so you can do this.

You can view the FreeBSD manual pages for (each) of the above FreeBSD system function calls with:

shell$ man 2 read

shell$ man 2 poll

Etc.. etc...

What you want to do is to write a poll() loop (or) a select() loop -- so your code can run inside a thread (pthread(3)?) or fork(2) process or even just your main() function -- and occasionally check for socket I/O. You can use the same poll(2) logic to check the file descriptor for "ready-to-write" and ERROR issues as well.

This logic can likely also be done with SSL_poll(2).

This same coding logic also works on Linux, Mac and Windows (unless you are in a WinSock world, then you will need to bend it slightly).

Hope that helps!

 

[lost_in_c]

It helps a bunch! Thank you!

I had seen the SSL_poll() reference in the OpenSSL reference (which is a word-for-word with the man pages) and was really hoping it wasn't that, as I was getting lost in the morass. You laying out the logic made it much clearer and it now feels feasable.

The OpenSSL specs constantly refer to the asynchronous mechanism you describe, but I believe I have set my BIO object to be non blocking, and I still can't get it not to block. I'm probably doing that part wrong. I create the listening BIO with BIO_set_nbio_accept(mybio, 1), and the "1" flag supposedly makes it non-blocking. But maybe when I pop it with BIO_pop and create the final client BIO object ( clientbio=BIO_pop(mybio)), it somehow loses that property. I'll have to look into that, it would be great if I could get it properly non-blocking instead of finding a work around.

This adventure did reveal a logical error in the preceeding code, which once fixed will allow me to continue in my hacky fashion of "guessing" what and when the client will ask.

I will iron out the basic logic, then figure out how to make it non blocking (on the SSL level, using SSL_get_fd if I can't), and then implement threads or forks as a final step. I feel this way it has the best chance of being a robust system. I was dead set on threads, but somebody sold me on forks because then you aren't tied to a single physical cpu and the program becomes more scaleable. We'll see about that, plenty of problems to solve in the interim.

Thanks again, very valuable help.

 

[CShell]

The OpenSSL specs constantly refer to the asynchronous mechanism you describe, but I believe I have set my BIO object to be non blocking, and I still can't get it not to block. I'm probably doing that part wrong. I create the listening BIO with BIO_set_nbio_accept(mybio, 1), and the "1" flag supposedly makes it non-blocking. But maybe when I pop it with BIO_pop and create the final client BIO object ( clientbio=BIO_pop(mybio)), it somehow loses that property. I'll have to look into that, it would be great if I could get it properly non-blocking instead of finding a work around.

I seem to remember (in the past) using the SSL_get_fd() call because I was facing a similar problem like you are describing (aka my OpenSSL file descriptor was not asynchronous/non-blocking or similar). I also felt like the OpenSSL library should "just support this" and tried for awhile to look for an "in OpenSSL library" solution to the problem - but ultimately I did not find a good solution "in library". There might be a better solution out there on a web page or similar?

I was dead set on threads, but somebody sold me on forks because then you aren't tied to a single physical cpu and the program becomes more scaleable.

The FreeBSD scheduler will schedule (any) program thread that is "ready-to-run" on to (any) available CPU/hyper-thread on your machine. So (by default) threads are not tied to a specific CPU/hyper-thread. Because... if it didn't work that way it would make your entire process "single threaded" or "time shared" . Of course you will need to use mutex locks to keep 2x (or many) threads from stepping on each other while they are simultaineously running at the same time in your program. But.. that's the fun of programming with threads!

VERY early implementations of "thread libraries" were in fact "time shared" ! But that was a long time ago during the Clone Wars era -- during The Old Republic.

Threads (pthread(3), etc) use multiple CPUs and hyper-threading - BUT you can actually "lock" a running thread in your program to run on a very specific CPU hyperthread using thread affinity -- which is (very cool !) for curtain coding situtations. This means that whenever the thread is "scheduled" to run on FreeBSD the thread will always run on the CPU/hyper-thread you scheduled it for. If you have like 24 CPUs with 2 hyper-threads per/CPU or similar you can divide all the "work" you want to do amoung the available CPUs/hyper-threads. The FreeBSD way to lock a thread to a specfic CPU/hyper-thead is to use affinity(3). Threads are very "light weight" meaning they don't take up a lot of resources when you create them and run them in your code - they are also called light weight processes (LWP). You can also schedule curtain threads in your code to have a higher or lower thread priority -- which means they can be scheduled to run relatively (on your FreeBSD machine - how BUSY the machine is) more or less frequently by the FreeBSD scheduler.

fork(2) is completely fine as well - but unless you apply an "affinity" to the forked process the forked process will be automatically "scheduled" by FreeBSD on to (any) available CPU/hyper-thread when the forked/child process is "ready to run". Because the fork(2) call used to be "heavy weight" when the fork(2) call was called (aka it copied the parent processes pointers, file descriptors, and even all of the parent processes "data" into the child process) - they are generally considered to be heavy weight. They have (since) optimized fork(2) more recently so it is better and less resource intensive when it is called. On FreeBSD systems you can also consider using vfork(2) which is MUCH more optimized when your source code is going to immediately call one of the execve(2) system calls. (aka execvp(2), execle(2), etc).

If you want to "really" learn threads I recommend the Orriely Ptheads book -- which seems "old" now!? But a quick check of Amazon shows 4+ stars.

For socket processing the books by Dr Stephen's are very good. Dr Stephens passed away many years ago, but everyone still reads his books even today to learn socket programming.

And if you want to REALLY understand FreeBSD and the *BSD's operating system in general - pickup Marshall Kirk McKusick's book - The Design and Implementation of the 4.4 BSD Operating System. This book may seem dated - but "a large chunk" of the things you read in the book you will find in "todays" FreeBSD implementation (and the other BSD's - Ghost, NET2, etc).

 

[kpedersen]

Note that using select() or poll() on underlying sockets behind SSL is a little tricky because SSL sometimes needs to write *as part* of the read or read *as part* of the write in order for the out of band encryption to take place.

Basically if you poll waiting for "can I read yet?" you find it will never come until you write. You will see this manifest as

SSL_ERROR_WANT_WRITE and SSL_ERROR_WANT_READ returned from SSL_get_error(3).

I would focus on non-blocking sockets (with SSL) first before thinking about adding threads. The Beej Networking guide is a good reference, in particular the part on Blocking. I typically start with a busy loop server processing the sockets (whilst glowing read hot). I then retroactively fit in the select() to wait for an actionable socket. Possibly not the best for design but allows for an incremental approach of implementation.

(Then once you are serving ~3000 clients nicely and need to support magnitudes more, then chuck in threads and design a good verification test suite )