How to get sendfile() working? (EAGAIN error)

[PMc]

When I try to transfer a single webpage to the browser (via local 1GB Ethernet, it is not sending the CSS and icons, and I see lots of these errors in the webserver (www/uwsgi 2.0.20):

uwsgi_response_sendfile_do(): Resource temporarily unavailable [core/writer.c line 655] during GET /pgadmin/static/favicon.ico?ver=60400
uwsgi_response_sendfile_do(): Resource temporarily unavailable [core/writer.c line 655] during GET /pgadmin/static/js/generated/pgadmin.css?ver=60400
uwsgi_response_sendfile_do(): Resource temporarily unavailable [core/writer.c line 655] during GET /pgadmin/static/js/generated/pgadmin.style.css?ver=60400

When I disable sendfile() in the uwsgi config, the errors go away.

reading sendfile(2), it talks about tuning and the sysctl knobs kern.ipc.nsfbufs, kern.ipc.nsfbufsused, kern.ipc.nsfbufspeak. But I don't have these, and the only kernel source that uses them seems not to be included in amd64 kernels.

The system has lots of free memory and does not experience serious network traffic.

16753/17912/34665 mbufs in use (current/cache/total)
12166/7966/20132/2033500 mbuf clusters in use (current/cache/total/max)
10/6568 mbuf+clusters out of packet secondary zone in use (current/cache)
0/845/845/1016750 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/301259 9k jumbo clusters in use (current/cache/total/max)
0/0/0/169458 16k jumbo clusters in use (current/cache/total/max)
28521K/23790K/52311K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
977 sendfile syscalls
489 sendfile syscalls completed without I/O request
160 requests for I/O initiated by sendfile
621 pages read by sendfile as part of a request
3697 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
845 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

 

[zirias@]

If it's really EAGAIN, this looks like a bug in the software calling sendfile(2).

This will only happen if the socket was set to O_NONBLOCK (the application must do that explicitly) and the file can't be sent without blocking. An application should be able to handle that correctly (e.g. by learning how many bytes were transferred, which unfortunately isn't portable – the Linux sendfile() works differently, and then calling sendfile() again with a new offset parameter).

 

[PMc]

Is it legal for sendfile() to return EAGAIN with *zero* bytes sent?

The server treats this as an error:

ssize_t uwsgi_sendfile_do(int sockfd, int filefd, size_t pos, size_t len) {
#if defined(__FreeBSD__) || defined(__DragonFly__)
        off_t sf_len = len;
        int sf_ret = sendfile(filefd, sockfd, pos, len, NULL,  &sf_len, 0);
        if (sf_ret == 0 || (sf_ret == -1 && errno == EAGAIN)) return sf_len;
        return -1;
#elif ...
}

int uwsgi_proto_base_sendfile(struct wsgi_request * wsgi_req, int fd, size_t pos, size_t len) {
        ssize_t wlen = uwsgi_sendfile_do(wsgi_req->fd, fd, pos+wsgi_req->write_pos, len-wsgi_req->write_pos);
        if (wlen > 0) {
                wsgi_req->write_pos += wlen;
                if (wsgi_req->write_pos == len) {
                        return UWSGI_OK;
                }
                return UWSGI_AGAIN;
        }
-        if (wlen < 0) {
+        if (wlen <= 0) {
                if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINPROGRESS) {
                        return UWSGI_AGAIN;
                }
        }
        return -1;
}

Changing that as shown seems indeed to solve the matter.

 

[zirias@]

Is it legal for sendfile() to return EAGAIN with *zero* bytes sent?

Yes, if the socket is not ready to send any data. But it's somewhat unusual, I would expect some software to check that first (e.g. with classic select() or modern stuff like kqueue() on FreeBSD)...

edit: given that's not the case here (would need to see a lot more code to be sure), your fix looks correct to me, still I think it would be better to change the if (wlen > 0) above to if (wlen >= 0)...

edit2: Ok, it depends on what this internal uwsgi_sendfile_do() function is returning. If it's the return code of sendfile(), it must be <0 to set any errno. But if it's the number of bytes sent (as "returned" by FreeBSD's sendfile() in off_t *sbytes), then yes, this is a realistic scenario.

edit3: Damn, I should read ALL the code in your post. This looks like the problem to me:
if (sf_ret == 0 || (sf_ret == -1 && errno == EAGAIN)) return sf_len;
So, it returns "success" and a potential length of 0 specifically if errno was EAGAIN. This case isn't properly handled in the calling code.

 

[zirias@]

I'd restructure the whole function like this to be robust when called on a socket that would currently block on writing:

int uwsgi_proto_base_sendfile(struct wsgi_request * wsgi_req, int fd, size_t pos, size_t len) {
       ssize_t wlen = uwsgi_sendfile_do(wsgi_req->fd, fd, pos+wsgi_req->write_pos, len-wsgi_req->write_pos);
       if (wlen >= 0) {
               wsgi_req->write_pos += wlen;
               if (wsgi_req->write_pos == len) {
                       return UWSGI_OK;
               }
               return UWSGI_AGAIN;
       }
       if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINPROGRESS) {
               return UWSGI_AGAIN;
       }
       return -1;
}