C strerror_r best practices, POSIX vs GNU

[JanBeh]

In many C programs, I have the need to convert an errno into a printable string. Since strerror is not thread-safe and perror only works with stderr, I resort to using strerror_r, see strerror_r(). However, there are two different definitions of this function:

int strerror_r(int errnum, char *strerrbuf, size_t buflen); // POSIX
char *strerror_r(int errnum, char *strerrbuf, size_t buflen); // GNU

See also here.

FreeBSD uses the variant that returns an integer, while Linux (depending on compile options) uses the variant that returns a pointer (that doesn't necessarily point to the provided buffer).

What is the best practice when converting an errno to a string with regard to portability and simplicity in mind?

 

[zirias@]

Go with the POSIX version and define the appropriate feature test macros before including any system headers (which you should do in any case when writing POSIX-code).

E.g. if you just want the old POSIX.1-2001, add #define _POSIX_C_SOURCE 200112L. There are newer versions available, depending on which other POSIX APIs you want to use in your program, you might need a different value.

If you also want some BSD and SysV APIs, #define _DEFAULT_SOURCE is helpful, although I'm not sure how portable it actually is. It does work with glibc and FreeBSD's libc.

 

[JanBeh]

I'm afraid I don't know the differences between the various POSIX standards in detail. In the past, I didn't care that much. I'm fine with a more modern version of the standard (I guess?). Which feature test macros would you generally recommend? Unfortunately, I didn't find anything in the FreeBSD manpage feature_test_macros(), which does not exist. Also, FreeBSD's strerror_r() man page in particular does not give any recommendation on how to include the function.

The Linux manpage on feature test macros is quite long. And it recommends _GNU_SOURCE which got me into trouble in the first place a long time ago. So I'm not sure if following those recommendations is wise.

What should I do precisely, and where can I read about my options? Would I have to dig deep into reading standards (and are the respective standards published as open access?), or is there also some sort of good overview somewhere?

What kind of BSD ands SysV APIs are you referring to in particular? Something like bzero()? I guess it's not good to use that anyway? And what SysV APIs? Do you mean SysV IPC for example? Not sure what else is there. Again, I don't really have a good overview on all these things (yet). I'm not that familiar with the different Unix operating systems and their history and corresponding standards.

How careful must I be with different definitions of functions in general? So far, I only ran into issues with strerror_r. Are there a lot of other pitfalls that go beyond anything that would be caught at compile time?

 

[unitrunker]

Also, FreeBSD's strerror_r() man page in particular does not give any recommendation on how to include the function.

The link you gave seems clear to me:

#include <string.h>

char *
strerror(int errnum);

char *
strerror_l(int errnum, locale_t);

int
strerror_r(int errnum, char *strerrbuf, size_t buflen);

 

[zirias@]

I'm afraid I don't know the differences between the various POSIX standards in detail. In the past, I didn't care that much. I'm fine with a more modern version of the standard (I guess?). Which feature test macros would you generally recommend?

Here's what FreeBSD understands (from /usr/include/sys/cdefs.h):

 * Here's a quick run-down of the versions:
 *  defined(_POSIX_SOURCE)              1003.1-1988
 *  _POSIX_C_SOURCE == 1                1003.1-1990
 *  _POSIX_C_SOURCE == 2                1003.2-1992 C Language Binding Option
 *  _POSIX_C_SOURCE == 199309           1003.1b-1993
 *  _POSIX_C_SOURCE == 199506           1003.1c-1995, 1003.1i-1995,
 *                                      and the omnibus ISO/IEC 9945-1: 1996
 *  _POSIX_C_SOURCE == 200112           1003.1-2001
 *  _POSIX_C_SOURCE == 200809           1003.1-2008

As I said previously, I found defining _DEFAULT_SOURCE somewhat handy as well (giving you the latest POSIX plus BSD and SysV standard APIs, but not GNU-specific stuff with glibc), although I wouldn't expect it to be portable, so for really portable POSIX code, I'd generally recommend to explicitly define _POSIX_C_SOURCE to the version you need.

Unfortunately, I didn't find anything in the FreeBSD manpage feature_test_macros(), which does not exist. Also, FreeBSD's strerror_r() man page in particular does not give any recommendation on how to include the function.

As far as I know, FreeBSD's libc is "POSIX with extensions" (just like glibc) and doesn't have anything conflicting with POSIX (unlike glibc, the conflicting strerror_r() declaration is an example), so the feature test macros are less important when building on/for FreeBSD. But defining them is always important for portability of course.

And it recommends _GNU_SOURCE which got me into trouble in the first place a long time ago.

Sure.
They recommend that because, you know, GNU is the greatest. Portability to other POSIX-compliant systems, who the hell needs that?

 

[JanBeh]

The link you gave seems clear to me:

#include <string.h>

…

int
strerror_r(int errnum, char *strerrbuf, size_t buflen);

The problem is that following the recommendations of FreeBSD's manpage causes issues under Linux.

For demonstration, I wrote the following test program:

#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

int main(int argc, char **argv) {
    int status = close(-1);
    if (status == -1) {
        char buffer[1024] = "THIS SHOULD NOT BE DISPLAYED!";
    strerror_r(errno, buffer, 1024);
        printf("%s\n", buffer);
    }
    return 0;
}

If I compile this on FreeBSD, then I get:

$ uname
FreeBSD
$ cc -Wall test.c && ./a.out
Bad file descriptor

This also works when I add -std=c99:

$ uname
FreeBSD
$ cc -Wall -std=c99 test.c && ./a.out
Bad file descriptor

On Linux (I tried on Ubuntu), with default compile options, it also works:

$ uname
Linux
$ cc -Wall test.c && ./a.out 
Bad file descriptor

However, if I add -std=c99, then I run into trouble:

$ uname
Linux
$ cc -Wall -std=c99 test.c && ./a.out 
test.c: In function ‘main’:
test.c:10:9: warning: implicit declaration of function ‘strerror_r’; did you mean ‘strerror’? [-Wimplicit-function-declaration]
   10 |         strerror_r(errno, buffer, 1024);
      |         ^~~~~~~~~~
      |         strerror
THIS SHOULD NOT BE DISPLAYED!

I don't exactly understand why I get a warning in the last case (can someone explain that to me?), but the bigger problem is that "THIS SHOULD NOT BE DISPLAYED!" is displayed. Assuming the buffer wouldn't be initialized, this might even cause a read beyond the buffer, which is a security problem.

As I said previously, I found defining _DEFAULT_SOURCE somewhat handy as well (giving you the latest POSIX plus BSD and SysV standard APIs, but not GNU-specific stuff with glibc), although I wouldn't expect it to be portable, so for really portable POSIX code, I'd generally recommend to explicitly define _POSIX_C_SOURCE to the version you need.

I tried adding the following on top of my #includes:

#ifndef _DEFAULT_SOURCE
#define _DEFAULT_SOURCE
#endif
#ifdef _GNU_SOURCE
#error _GNU_SOURCE must not be defined
#endif

Would you consider that good practice? (Apart from _DEFAULT_SOURCE maybe having some portability issues.) It seems to do the job on both Linux and FreeBSD. Note the extra #ifdef _GNU_SOURCE to throw an error if _GNU_SOURCE is used. That's because if both _DEFAULT_SOURCE and _GNU_SOURCE are defined, GNU/Linux will still choose the non-standard-compliant function.

 

[zirias@]

I don't exactly understand why I get a warning in the last case (can someone explain that to me?),

You request the compiler to compile C according to the ISO standard (IOW without any extensions). To get the extensions anyways, you MUST define an appropriate feature test macro.

but the bigger problem is that "THIS SHOULD NOT BE DISPLAYED!" is displayed.

Without any proper prototype, you end up linking the GNU-specific function.

Would you consider that good practice?

No. There's no point in assuming feature test macros are defined from the commandline (outside your own build-system which might do that). Anyone doing that deserves the build failure for sure.

The correct portable way to solve your issue here is to just add a single #define _POSIX_C_SOURCE 200112L on top of your files, before including any system headers. Then you must get functions according to this version of the POSIX standard.

 

[JanBeh]

No. There's no point in assuming feature test macros are defined from the commandline (outside your own build-system which might do that). Anyone doing that deserves the build failure for sure.

Well, I had a case in one project where someone added -D_GNU_SOURCE to the Makefile of a project. Back then, I wasn't aware of the bad consequences it may have. I guess you could say that "anyone" deserves the build failure, but it still isn't good when it results in a buffer overrun at runtime rather than a build error. That was my reasoning when checking for _GNU_SOURCE, but I don't know. It is verbose, and I don't really like it either.

The correct portable way to solve your issue here is to just add a single #define _POSIX_C_SOURCE 200112L on top of your files, before including any system headers. Then you must get functions according to this version of the POSIX standard.

Which standard should I pick? 200112L or 200809L?

Bonus question, how to solve this:

#define _POSIX_C_SOURCE 200112L

// It's a trap:
//#define _GNU_SOURCE

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

int main(int argc, char **argv) {
    int status = close(-1);
    if (status == -1) {
        int err = errno;
        char buffer[1024] = "THIS SHOULD NOT BE DISPLAYED!";
        strerror_r(err, buffer, 1024);
        printf("%s\n", buffer);
        char *logentry = NULL;
        asprintf(&logentry, "Error %d", err);
    // do something with logentry here
    free(logentry);
    }
    return 0;
}

Apparently either asprintf isn't defined, or I get the wrong strerror_r function. Does this mean I should generally refrain from using asprintf?

 

[zirias@]

Well, I had a case in one project where someone added -D_GNU_SOURCE to the Makefile of a project.

So that is your own build system. Guess you need some guidelines and governance for your team projects?

Which standard should I pick? 200112L or 200809L?

I kind of keep repeating myself, so I'll try to be a bit more verbose now. Your goal is portability to POSIX-compliant systems, then you will request the minimum POSIX version you actually need (because that way you'll still be portable to systems implementing this version but not a newer one). For strerror_r(), #define _POSIX_C_SOURCE 200112L is enough. If you need APIs from a newer POSIX version somewhere else in your program, you'll define a newer version instead.

Does this mean I should generally refrain from using asprintf?

For portable code, definitely. asprintf() is an original GNU idea. It's not in POSIX, not in SUS, not in classic BSD or SysV APIs. Some BSDs (at least FreeBSD and OpenBSD) implemented it as well (probably to improve portability for GNU-specific code), but that's all about it.

Actually I would also question its practical value. Most implementations you find are just wrappers around "allocate something, try vsnprintf, if result is too large realloc and vsnprintf again", with some sensible error checking. You can add that to your own codebase in a few lines needing nothing but standard C functions.

It would theoretically be possible for an implementation to have the allocation feature included in its "core printf-like function" (typically something close to vsnprintf()). In that case, multiple passes could be avoided, but the overhead of reallocating the result buffer will always remain. I'm too lazy now to check whether glibc indeed does something like that

edit: mentioning this GNU-specific function just motivated me to improve my own recent printf-like function, handling all possible error conditions and edge cases. It's also an example where asprintf() would be much shorter code, but inefficient: In this case I don't have to return the buffer, so I can just use a fixed buffer on the stack for the vast majority of cases, avoiding both an allocation and a second printf-pass.

 

[JanBeh]

So that is your own build system.

Sure. It was a learning curve for me. But I wonder if other people run into the same problem.

Also, I wonder if the FreeBSD manpage should warn about different definitions of strerror_r being around. Even if I don't like those other (non-standard-compliant) definitions, Linux is widespread.

I kind of keep repeating myself, so I'll try to be a bit more verbose now.

Yes please, sorry if I didn't understand it yet.

Your goal is portability to POSIX-compliant systems, then you will request the minimum POSIX version you actually need (because that way you'll still be portable to systems implementing this version but not a newer one). For strerror_r(), #define _POSIX_C_SOURCE 200112L is enough. If you need APIs from a newer POSIX version somewhere else in your program, you'll define a newer version instead.

I see that the manpage for strerror_r() lists the POSIX standard versions for each function. I'll pay attention to those sections from now on.

Regarding asprintf, it's a bit tragic to not being able to use it in FreeBSD just because Linux doesn't let me include it without forcing me to define _GNU_SOURCE It's a Linux issue, not a FreeBSD issue.

While asprintf can be worked around with your own implementation of it, there are other functions that cause the same problem which may be more problematic. I ran into this problem when using accept4(). I'm not sure if I remember the exact problem, but I believe that when you use accept and when you are in a multi-threaded environment, you might leak the returned file descriptor to your child processes unless SOCK_CLOEXEC (see socket(2)) is set as soon as the socket is created by accept4. And accept doesn't allow you to set this flag atomically, I think, but I don't remember the situation exactly where I needed or wanted accept4.

So I do believe there are other scenarios where you run into needing (or wanting?) _GNU_SOURCE for code that runs at least on Linux in addition to FreeBSD. Any way to get out of this trap? Do I really have to refrain from using asprintf, accept4, and possibly others? Or do I "give in" but then have to create my own strerror_r implementation which checks the environment / feature test macros and then internally uses whichever function is provided by the OS? It all is such a hassle. What do you think is the best practice when I need to use strerror_r and other functions like accept4. Even assuming I don't want to be very portable but only support FreeBSD and Linux, this seems to be a challenge already.

edit: Apparently, if I define _POSIX_C_SOURCE, I lose all the non-standard functions under FreeBSD. What is the way under FreeBSD to get them, e.g. to use asprintf and strerror_r at the same time even if no portability is necessary? The following does not work:

#define _POSIX_C_SOURCE 200809L
#define _DEFAULT_SOURCE

But only using _DEFAULT_SOURCE (without _POSIX_C_SOURCE) seems to work.

 

[zirias@]

I believe that when you use accept and when you are in a multi-threaded environment, you might leak the returned file descriptor to your child processes unless SOCK_CLOEXEC (see socket(2)) is set as soon as the socket is created by accept4.

The portable solution to this issue is to design your application/service properly: Do your "event loop" (including accepting new connections) exclusively on your "main" thread, as well as forking child processes. Use worker threads only to delegate work.

So I do believe there are other scenarios where you run into needing (or wanting?) _GNU_SOURCE for code that runs at least on Linux in addition to FreeBSD.

If you want portable code, you don't want _GNU_SOURCE. It's either the one or the other.

Apparently, if I define _POSIX_C_SOURCE, I lose all the non-standard functions under FreeBSD.

That's (part of) the point...

What is the way under FreeBSD to get them, e.g. to use asprintf and strerror_r at the same time even if no portability is necessary?
[...] only using _DEFAULT_SOURCE (without _POSIX_C_SOURCE) seems to work.

AFAIK, FreeBSD's libc headers don't know _DEFAULT_SOURCE while glibc will give you latest POSIX, BSD/SysV, etc, but not GNU-specific extensions. I don't think it's a portable thing, but you can use it to just get "common" extensions on GNU/Linux and (most?) BSDs.

Still, if you want better portability, restrict yourself strictly to POSIX....

 

[JanBeh]

Thanks a lot for all the information. So the clean way would be to do

#define _POSIX_C_SOURCE 200112L

before I include any standard header files and to refrain from using functions like asprintf, accept4, etc. entirely.

A less portable alternative would be to

#define _DEFAULT_SOURCE

and find some work arounds in Linux for missing functions like asprintf or accept4.

Or I define _GNU_SOURCE conditionally on Linux and handle all the problems that arise from this with #ifdefs.

None of these solutions really feel good to me though, but I guess using the clean way, whenever possible, would be the best option.

 

[JanBeh]

One more question: Is there any reason why I should add the "L" suffix to "200112"? I checked /usr/include/sys/cdefs.h and I don't see the "L" suffix there. Is this for compatibility with systems where int is a 16 bit integer?

 

[JanBeh]

So the clean way would be to do

#define _POSIX_C_SOURCE 200112L

In theory nice. But after I added this, the following symbols aren't found anymore:

* TCP_NOPUSH (I intended to use TCP_NOPUSH on FreeBSD and TCP_CORK on Linux)
* SO_NOSIGPIPE
* AF_LOCAL (exists on both FreeBSD and Linux but apparently not available for POSIX)
* SOCK_CLOEXEC
* SOCK_NONBLOCK

I feel like my code would require major rework if I want to define _POSIX_C_SOURCE and not sure if it's worth the effort. Besides, does it mean I can't use local domain sockets at all and/or won't have control about TCP flushing?

Update:
I finally decided to take a flexible approach. Where possible, I go the _POSIX_C_SOURCE path. But when it comes to networking, I have to rely on BSD and GNU extensions, so I define _GNU_SOURCE on GNU/Linux and only there, and then define my own macros for converting an errno into a string (which just works differently for Linux). This allows me using accept4, SOCK_CLOEXEC etc.

While I don't like that solutions, it seems to be the best choice to me. It might not be very portable but seems to work fine on Linux and FreeBSD at least.