www/nginx 1.10.x "service nginx upgrade" results in "Address already in use"

adams

adams

We use service nginx upgrade to do seamless restarting of NginX processes weekly due to high volume sites' having their NginX processes simply exit after a few weeks otherwise. I appreciate that this is a bandaid and shouldn't be needed, but it is. This spans dozens of machines and clients.

We recently just upgraded most of our clients to 1.10.x, and started seeing this in the log when we run the service nginx upgrade command:
Code: 
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [alert] 73753#0: worker process 73757 exited on signal 11
2016/06/27 00:18:56 [emerg] 74294#0: bind() to 1.2.3.4:443 failed (48: Address already in use)
nginx: [emerg] bind() to 1.2.3.4:443 failed (48: Address already in use)
2016/06/27 00:18:56 [emerg] 74294#0: still could not bind()
nginx: [emerg] still could not bind()
This is even on essentially idle servers (backups) with no traffic. We've turned off this weekly restart via cron for now, but still this command should work regardless.

The end result is that NginX exits entirely and doesn't come back up.

Anyone else seeing this?
 
I never even heard of the upgrade option till now and can't find any documentation, even in the man page for service. Can you point me to some? Even Googling only turned up a link to this thread.
 
Does a regular service nginx restart work? Looking at the errors it looks like it has issues shutting down correctly.
 
SirDice: Regular service nginx restart does work fine. I agree, something appears broken with the shutdown.

drhowarddrfine: It is a little known operation but very useful for high volume sites since it lets you restart NginX without losing traffic or down time. Upgrade sends the SIGUSR2 signal, per the man page:

SIGUSR2 Upgrade the nginx executable on the fly.

All the Linux servers we manage (CentOS/RHEL/Ubuntu) don't even reveal this operation via the service/systemctl controller.
 
All adding "one" in front of "restart" does is ignore the sysrc check. If you've got a site with thousands of hits per second you're going to lose a lot of traffic if you use "restart" and not "upgrade", so it's not an appropriate substitute.

Regardless, "upgrade" should work correctly. I was wondering if anyone else had run into this issue before I opened a PR.
 
You must log in or register to reply here.
Back
Top