Solved php56 memory leak?

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

Hi all,

Twice this morning my VPS running FreeBSD v11 became unresponsive. Logging into the console at my host service it appears to have run out of swap space and twice I was forced to reboot. The server has 1GB of RAM and 1.7GB swap.

I am pretty sure it is a www/nextcloud process, but obviously I can't be 100% sure.

I've never encountered this issue before in all my years running a FreeBSD server.

A couple of questions:

First, are there any techniques to contain www/apache24 (or lang/php56?) to prevent the entire server from being affected? Ie, a jail?

Second, are there any mechanisms for determining what process might be leaking memory? I mentioned the issue on the Nextcloud forum this morning, and a dev told me to open an issue on Github, but I feel that if I don't make some effort to determine whether the issue is in fact a Nextcloud process they may just shrug.

Thanks for any ideas.
 
OP
OP
cbrace

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

I hate to do this, but I am bumping this post, in the hope of some kind soul coming to my aid.

A few minutes ago, my server once again become unresponsive and via the admin console at the hosting company it appeared to have yet again run out of swap space.

Normally the server uses about 15% of its swap space, days and weeks on end -- until something suddenly goes wild. What could be going on here? I think it is a nextCloud process, but I can't be at all sure.

Any ideas?
 

masteroman

Member

Reaction score: 8
Messages: 30

You may want to look into Resource Limits section of the handbook to see if there's somehting useful in there, but basically yes, you could put it in jail and limit memory usage from there.

Some sort of graphing tool like Munin could give you insight into memory usage over time. On some of my Linux machines I use utility sysutils/atop. Haven't had the need to use it on FreeBSD yet so I don't know if there are some differences but is definitely worth a try.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,291
Messages: 29,742

I highly recommend getting some monitoring in place so you can detect issues before they become a problem. When the monitoring tells you things are getting out of hand you will be able to quickly restart the machine before it goes offline completely. Granted, it's not a solution but it does limit the impact of issues. With the monitoring in place you can also create additional data points, hopefully pointing you to the original problem.
 
OP
OP
cbrace

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

Thank you both. I am going to start with sysutils/atop and see how far I get with that.

I've also upgrade to the lastest release of www/nextcloud, v11.0.2, in the off chance that the memory leak was in that package and has been fixed
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,291
Messages: 29,742

That's always a good idea when you run into problems. Make sure you're running the latest available versions for everything. There's always a chance the issues you're having have already been fixed. If they're available also check the Changelogs of the applications. That might provide confirmation of the bug fix.
 
OP
OP
cbrace

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

Hi all,

My server was running fine for several weeks, and then suddenly this afternoon it once it is now running out of swap space, repeatedly.

I've identified that it is happening with httpd:

I reboot, ssh to it, start top, sort by SIZE. I can then watch a dozen or so httpd processes slowly grow and consume all swap. If I restart httpd, the process repeats itself.

Just now I have commented out nextcloud in my vhosts file, as I still think that is the culprit, but I have no way of knowing for sure.

:/
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,291
Messages: 29,742

Check your Apache access and error logs. They may provide some clues.
 
OP
OP
cbrace

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

FWIW, I'm sure now that www/nextcloud is the culprit.

I open top and watch the amount of swapspace being consumed. When it starts to grow, I tail /var/log/httpd-access.log and /nextcloud/remote.php is active. From the IP address, I can see that the request is coming from one of my desktop systems. I've now shut down the desktop clients while I investigate further.
Code:
*.*.*.* - root [24/Mar/2017:06:56:47 +0100] "PROPFIND /nextcloud/remote.php/webdav/esa/%23T2525%20ARTES%20Common%20Core%20Brief%202016 HTTP/1.1" 207 4224
*.*.*.* - root [24/Mar/2017:06:56:48 +0100] "PROPFIND /nextcloud/remote.php/webdav/esa/%23T2525%20ARTES%20Common%20Core%20Brief%202016/draft HTTP/1.1" 207 844
*.*.*.* - root [24/Mar/2017:06:56:49 +0100] "PROPFIND /nextcloud/remote.php/webdav/esa/%23T2525%20ARTES%20Common%20Core%20Brief%202016/round%204 HTTP/1.1" 207 848
*.*.*.* - root [24/Mar/2017:06:56:50 +0100] "PROPFIND /nextcloud/remote.php/webdav/esa/%23T2525%20ARTES%20Common%20Core%20Brief%202016/round_1 HTTP/1.1" 207 846
*.*.*.* - root [24/Mar/2017:06:56:50 +0100] "PROPFIND /nextcloud/remote.php/webdav/esa/%23T2525%20ARTES%20Common%20Core%20Brief%202016/round_2 HTTP/1.1" 207 846
*.*.*.* - root [24/Mar/2017:06:56:51 +0100] "PROPFIND /nextcloud/remote.php/webdav/esa/%23T2525%20ARTES%20Common%20Core%20Brief%202016/tasf%20flex%20converter HTTP/1.1" 207 862
 

masteroman

Member

Reaction score: 8
Messages: 30

I've now shut down the desktop clients while I investigate further.
Shutting down these clients resolves the immediate issue?

You can try looking at process by showing its opened files with fstat:

fstat -p PID

To see if there is some obvious anomaly there. You may additionally debug that specific process.

Since you've said you're seeing httpd is causing high memory usage I assume you're using PHP as apache module? Have you tried switching to PHP-FPM to see if that makes any difference?
 
OP
OP
cbrace

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

I finally bit the bullet and upgraded my VPS from 1G to 4G RAM. That seems to have solved the problem. It would seem www/nextcloud simply needs lots of memory.
 

masteroman

Member

Reaction score: 8
Messages: 30

Weird, as I used to run it normally with 1GB RAM + 1GB of swap, swap was rarely used but needed as I then used to compile every new PHP version.

Most likely it depends on number of users using nextcloud instance...
 
OP
OP
cbrace

cbrace

Well-Known Member

Reaction score: 8
Messages: 271

Well, in my case, it seemed to be not so much the number of users (not so many) but a rather large number of files (300k).

I see that the server in normal operation still appears to run well with around 460MB of active memory, but it seems to require a whole lotta extra RAM in periodic spurts.
 
Top