(nginx 1.9) Enable reuseport then all load on 1 worker?

M

meteor8488

Hi all,

I just upgrade nginx to 1.9.2 and also enabled reuseport for my website.

Code: 
worker_processes  18;

...

listen  80 accept_filter=httpready reuseport;


But I found that all workloads are handled by the same worker. Even though this worker is pretty busy and almost 100% CPU usage. And if I remove reuseport and use accept_mutex , all my workers are working and each of them get about 10%-15% CPU usage.

Based on nginx document, it supposed to balance the work load on different workers.


With the SO_REUSEPORT option enabled, there are multiple socket listeners for each IP address and port combination, one for each worker process. The kernel determines which available socket listener (and by implication, which worker) gets the connection.
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/


And I check FB document it's said that freebsd is supporting SO_REUSEPORT.
So is this a bug or I need to enable SO_REUSEPORT for freebsd?

Thanks
 
Nginx 1.9 branch is not a production version! That is developmental version. If you are using Nginx in production like me you should stick to 1.8 branch. If you chose to upgrade you are doing that on your own risk.
 
Oko said:
Nginx 1.9 branch is not a production version! That is developmental version. If you are using Nginx in production like me you should stick to 1.8 branch. If you chose to upgrade you are doing that on your own risk.
Click to expand...

I understand that... But 1.9 is the only version which can support http/2
 
I don't know if this is related, but I toyed with something like this in UDP just a while ago. Unicast packets go all to one socket, the first one to bind, the other receivers succeed in all system calls, but never receive anything. Only when the first receiver exits, the second one takes over. REUSE-options were for multicast*, maybe it hasn't changed.

I cannot see anything in in_pcblookup() that would suggest "load balancing", but that doesn't amount to much.

Juha

Doodling... There does not seem to be magic in REUSEPORT for tcp. 4 processes each creating their own sockets with the option, leaves 3 totally idle (but happy). Creating a socket and then forking off into 4 acceptors behaves as expected, all take their share.

* half-truth or worse
 
Last edited:
Juha Nurmela said:
I don't know if this is related, but I toyed with something like this in UDP just a while ago. Unicast packets go all to one socket, the first one to bind, the other receivers succeed in all system calls, but never receive anything. Only when the first receiver exits, the second one takes over. REUSE-options were for multicast, maybe it hasn't changed.

I cannot see anything in in_pcblookup() that would suggest "load balancing", but that doesn't amount to much.

Juha

Doodling... There does not seem to be magic in REUSEPORT for tcp. 4 processes each creating their own sockets with the option, leaves 3 totally idle (but happy). Creating a socket and then forking off into 4 acceptors behaves as expected, all take their share.
Click to expand...


https://forum.nginx.org/read.php?2,264913,264929#msg-264929

I asked the same question on nginx forum, it seems that NGINX doesn't support reuseport on freebsd.
 
It would still benefit from it. If you must restart the server for upgrade or something, you could launch the new one and then politely kill the old ones.

Juha
 
meteor8488 said:
it seems that NGINX doesn't support reuseport on freebsd.
Click to expand...
That is bizarre. Until just a couple of months ago, nginx was developed on FreeBSD and only moved to Linux through market pressure (not technical reasons).

While SO_REUSEPORT socket option is available on FreeBSD, its
behaviour is different from one nginx relies on: instead of
balancing between all sockets as Linux and DragonFly do, it
preserves historic behaviour for TCP and delivers all connections
to one socket instead.
Click to expand...

So is this a failing of FreeBSD? Or is there a technical advantage reason FreeBSD does not do the same as Linux and DragonFly?

EDIT: This is an area I am not familiar with but, this thread from just over two years ago, seems to indicate Linux and Dragonfly are shoehorning this into their systems in a most inappropriate way, though I may be misunderstanding what is being said.

That is what multicast is for.
If you want the same data sent to all listeners, then
that is multicast behavior and you should be using
a multicast socket.
Click to expand...
 
Is state of the art really so fast, that 1 socket, N acceptors does not cut it anymore? That soinherit() looks really hairy to be worth it.

Juha
 
Juha Nurmela said:
I don't know if this is related, but I toyed with something like this in UDP just a while ago. Unicast packets go all to one socket, the first one to bind, the other receivers succeed in all system calls, but never receive anything. Only when the first receiver exits, the second one takes over. REUSE-options were for multicast*, maybe it hasn't changed.
Click to expand...

I've tried that on linux and there is a perfect RR load balancing.
 
Going on a tangent...

It's hard to stop listening cleanly. How to handle any remaining queued connections before exiting, but hide from new connection attempts. I could only think of shutdown(rendezvous_fd, ) or listen(rendezvous_fd, 0). Neither worked as one might hope, though both returned "No error".

shutdown with any SHUT_XX makes next accept return "Software caused connection abort", any or none pending connections has no effect.
listen 0 has no effect.

No practical value, as always,
Juha

bind to some other port, hiding, is EINVAL.
SO_RCVTIMEO {0,0} or [0,1000} has no effect (and an improper solution)
 
The problem, as I stated earlier, is using reuseport this way appears to be an inappropriate use of that method. Linux hacked the wrong thing to make this work that way instead of using what they should have. Asking FreeBSD to do the wrong thing is asking a bit much, though I know you just want to use this thing.

What you should do is ask this on the FreeBSD mailing list where the developers are and can properly answer this question.
 
  • Thanks
Reactions: Oko
drhowarddrfine said:
The problem, as I stated earlier, is using reuseport this way appears to be an inappropriate use of that method. Linux hacked the wrong thing to make this work that way instead of using what they should have. Asking FreeBSD to do the wrong thing is asking a bit much, though I know you just want to use this thing.
Click to expand...
I am glad I read your post. Frankly my instinctive reaction to somebody's request to do the wrong thing is to show middle finger. I hope that will be the reaction of FreeBSD developers.
 
You must log in or register to reply here.
Back
Top