Solved www/webkit2-gtk3 broken or synth not liking it?

[PacketMan]

I never had an issue with www/webkit2-gtk3. My make.conf already has MAKE_JOBS_UNSAFE=yes, but that Watchdog killed runaway process! output got my attention. Is ports-mgmt/synth killing it? Prematurely? Why?


tail -25 www___webkit2-gtk3.log

cd /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5 && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5 /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5/Source/WebCore /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5 /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5/Source/WebCore /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5/Source/WebCore/CMakeFiles/WebCore.dir/DependInfo.cmake
###  Watchdog killed runaway process!  ###



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Saturday, 19 NOV 2016 at 12:05:46 UTC
Duration: 01:33:43
Scanning dependencies of target WebCore
gmake[3]: Leaving directory '/construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5'
gmake -f Source/WebCore/CMakeFiles/WebCore.dir/build.make Source/WebCore/CMakeFiles/WebCore.dir/build
gmake[2]: gmake: Command not found
gmake[2]: *** [CMakeFiles/Makefile2:718: Source/WebCore/CMakeFiles/WebCore.dir/all] Error 127
gmake[2]: Leaving directory '/construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5'
gmake[1]: *** [Makefile:153: all] Error 2
gmake[1]: Leaving directory '/construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make: stopped in /xports/www/webkit2-gtk3

less /etc/make.conf
#
#
# PORT COMPILE OPTIONS #
#
#


#MAKE_JOBS_NUMBER=1
MAKE_JOBS_UNSAFE=yes


#
#
# END OF FILE #
#

 

[ASX]

I have built webkit2-gtk3 yesterday, using synth, with no issues at all ...
-rw-r--r-- 1 root wheel 19325780 Nov 18 20:25 /var/synth/live_packages/All/webkit2-gtk3-2.8.5_6.txz

your log show:

gmake[2]: gmake: Command not found

so, there is your problem .. don't know where it come from ... synth always worked reliably for me.

 

[marino]

The watchdog looks for stalled builds. It determines if a build is stalled if the build log doesn't increment in X minutes, where X is variable based on the load of the machine. X can range from 25 minutes to like 4 hours. If the watchdog is kicking in, the build machine is either vastly underpowered or vastly overloaded.

 

[marino]

ah, or third option, as ASX mentions, the build could break without returning and then the watchdog faithfully reaps it.

 

[marino]

the gmake thing might be a red herring. The termination section is supposed to the last lines, so the extra output may be caused by the watchdog kicking in somehow.

 

[PacketMan]

If the watchdog is kicking in, the build machine is either vastly underpowered or vastly overloaded.

I would say it's neither for this machine. It's a home desktop machine and works wonderfully. When I do these synth upgrades I boot up without desktop (gnome3 and gdm) running, so the machine is idle. If synth can only work with high power machines then I would say that is a deficiency of synth and not the machine. I could tail -f the build log if you like, but I watched it for a brief moment of time and it wasn't stalled. I suppose I would have to watch it to the end. Better go buy some beer.

the gmake thing might be a red herring. The termination section is supposed to the last lines, so the extra output may be caused by the watchdog kicking in somehow.

So what are you thinking; boo boo in the watchdog code or vastly loaded machine that synth can't handle.

Other ports are building fine; no gmake errors.

 

[marino]

I told you *why* the watchdog kicks in. It's up to you figure out why the log stopped if you don't like my generalizations. If you are saying it kicked in why the log was progressing, I'll say that did not happen.

 

[PacketMan]

Does the watchdog time (count) apply to the whole port build process, meaning it allocates time for a whole building of the port, or, does it allocate time on a per line incremental basis? This port does take a long time to build I must say, but as far as I know its never stalled completely. Let me know please and I will try to monitor the build of the port.

Edit - I now see other www/webkit ports are also failing in synth build.

 

[marino]

it's per line.
Each time the log increments, the timer resets to zero. That check happens every second.

 

[PacketMan]

So I ran synth a 2nd time and one of the failed webkit ports built fine (took 8 hours). Another failed again. I ran synth a 3rd time for giggles, and synth said more packages needed to be downloaded, but the webkit port previously failed did not need to be rebuilt or downloaded. Note I did not do a portsnap fetch update for the 2nd and 3rd passes. When the ports were building I was watching the logs and never did I see them stall.

Mario, would be it be possible for you to include a watchdog time configurable option so the user can alter the base time (that gets multiplied by the load factor) so that we can accommodate for heavy or light loaded machines? Assuming watchdog was the issue here. I will have to keep my eye on this?

 

[marino]

please provide the output of synth version

 

[PacketMan]

# synth version

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 1.65
====================================================================
               Copyright (C) 2015-2016 John R. Marino


Usage: synth [zero-parameter-option]
-or-   synth [list-option] <list of port origins | filename>

zero-parameter-option includes 'help', 'configure', 'version' (this screen),
                      'status', 'upgrade-system', 'prepare-system',
                      'status-everything', 'everything', 'purge-distfiles',
                      'rebuild-repository'
list-option includes  'status', 'build', 'just-build', 'install', 'force'
                      'test'

# portmaster -L | grep synth

===>>> synth-1.65

I did another portsnap fetch update and sure shot that triggered another massive ports rebuild, 300 approx ports to be rebuilt. And sure shot guess what? And I bet if I rerun synth a few times (without ports tree updates) this will eventually build. I am betting the watchdog timer needd more tuning or more flexibility. But what do I know, I am not a coder, I just make hundreds of routers with dozens of 10 and 100 Gps connections work.

# tail -30 www___webkit2-gtk3.log | grep dog

###  Watchdog killed runaway process!  ###

I realize this machine ain't no datacenter speed demon, but the only load on this machine during synth upgrade-system or synth prepare-system is the load that synth and compiling itself creates. I boot up with all services including desktop not running.
# dmesg | grep CPU

CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4000+ (2109.64-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs

# dmesg | grep memory

real memory  = 2147483648 (2048 MB)
avail memory = 2034528256 (1940 MB)

....and load average of machine will vary between 3.xx to low 4.xx during builds.

 

[ASX]

load average of machine will vary between 3.xx to low 4.xx during builds.

Look like a slow cpu, with too much few RAM and overloaded, most likely going to swap heavily too.
How many "concurrent builders" and how many "job per builders" do you have configured in synth ?
You should use no more than 2 builders and max 1 job per builder, or better 1 builder and 2 jobs per builder.

EDIT: on a second though, may be 1 builder and 1 job could be required to avoid memory swapping.

 

[PacketMan]

I am using the defaults in synth - 2 builders and max 2 job per builder. Swap during the builds never exceeds 10%, there is usually always a few MB of ram available, and rarely does the cores get pinned; meaning 0% idle. Yeah they might say 0.5% idle, for some time but I don't see the symptoms of a pinned machine, and as I said previously my watching the log shows that it does continue building. The synth notes on Freshports say "....It will "drop-in" on any system". It doesn't say ".....It will "drop-in" on any light or moderately loaded system".

Yeah its an older machine, but it handles GNOME3 and a graphics intensive video game simultaneously pretty darn good, so I don't think I am being unreasonable in thinking that synth shouldn't abort because the box is loaded with synth work. Now that just said, I am running right now synth just-build www/webkit2-gtk3 for fun just to see what will happen.

EDIT:

last pid: 41421;  load averages:  2.10,  2.07,  1.80
CPU 0: 74.8% user,  0.0% nice, 24.4% system,  0.8% interrupt,  0.0% idle
CPU 1: 88.5% user,  0.0% nice, 11.0% system,  0.0% interrupt,  0.5% idle
Mem: 119M Active, 1274M Inact, 316M Wired, 21M Cache, 213M Buf, 226M Free
Swap: 4096M Total, 14M Used, 4082M Free

 

[marino]

it's the only known system to get hit by the watchdog. the algorithm has been working everywhere else regardless of hardware. There's been no evidence that it needs further tuning.

 

[marino]

I think ASX is on the right path. The amount of memory is shockingly low. Probably 1 builder x 2 (or 3 ) jobs would be more appropriate.


There's probably a spinning disk in there too slowing things far below norms.

 

[PacketMan]

The amount of memory is shockingly low.........There's probably a spinning disk in there too slowing things far below norms.

Maybe for Synth, but as I said before everything else works fine. Synth is the only issue I am having on this machine. I ain't going out to buy a new machine with dozens of gig of ram and more cpu just to satisfy synth. Portmaster didn't complain either. I don't see how more RAM will address the issue when all that is there now is not being used. With all due respect I think synth does need a little more tuning, but I am not asking you to do so. And so synth requires SSD too, is that what you are saying?

I'll lower the build values down from the defaults and report back later what I find. Might be a week given tomorrow is Monday.

CPU 0: 45.6% user,  0.0% nice, 53.0% system,  0.9% interrupt,  0.5% idle
CPU 1: 47.3% user,  0.0% nice, 51.7% system,  0.0% interrupt,  0.9% idle
Mem: 74M Active, 1309M Inact, 322M Wired, 21M Cache, 233M Buf, 230M Free
Swap: 4096M Total, 14M Used, 4082M Free

 

[marino]

The amount of memory + swap limits the number of builders and jobs. This is no secret.
If you build two huge c++ things simultaneously (e.g chromium + libreoffice) it takes gigabytes to build (think 16Gb minimum). We haven't been talking about the tmpfs setting. It should go without saying you can't use it.

So assuming you aren't using tmpfs, and the machine is swapping like crazy, that would explain how the log is stalling out even if the watchdog is set to 1 hour +.
It means synth is misconfigured. In that case, the solution is not kill the watchdog, it's lower the settings to match the capability of the machine. As I said, yours is the single complaint and it appears to be a configuration error.

Try 2 builders, 1 job or 1 builder, 3 jobs as a starting port.

SSD is not required but having one is like night and day, especially if you have little ram.

 

[PacketMan]

I'm not using tmpfs, I was using defaults up until now. I have lowered the settings to 1 builder x 2 jobs as a starting point. This machine will be replaced eventually, but not soon. I will report back as soon as I am sure I can give you a firm answer. With over a 100Mb ram free, and only 14MB in swap, I don't consider that "swapping like crazy". I'll say it one last time, every time I check the log it was incrementing.

Ironically, I have an even lesser powered machine (1GB ram, 1 cpu core), and no synth issues on it. But its defaults chose 1 builder 1 job. (This is the machine that is due for replacement. I've been talking to the folks at Cray, but their stuff is a little out of reach for me, so I guess it will be home built machine.)

Thanks again everyone.

 

[ASX]

Yeah its an older machine, but it handles GNOME3 and a graphics intensive video game simultaneously pretty darn good, so I don't think I am being unreasonable in thinking that synth shouldn't abort because the box is loaded with synth work.

These are two completely different workloads, graphics is offloaded to GPU, if it work well is because of your graphics card. Compiling software is a CPU bound type of load. They just don't compare.

Portmaster didn't complain either

Portmaster workload is probably equivalent to use 1 synth builder, assuming you used 2 jobs while building, that would be approximately half of the synth load when using 2 builders and 2 jobs x builder. (and half RAM requirements, ... and half disk I/O ...).

 

[marino]

I'm not using tmpfs

good.

I was using defaults up until now.

The defaults are normally fine for 2Gb but apparently the CPU and spinning disk are particularly slow, so some adjustment may be necessary.

With over a 100Mb ram free, and only 14MB in swap, I don't consider that "swapping like crazy".

I do. 100Mb is nothing. It could be swapping to disk with every compilation. It's not the amount of memory in swap, it's how often the memory is swapped. If it's happening continuously, that will increase build time tremendously.

I'll say it one last time, every time I check the log it was incrementing.

What are you saying?
There's exactly one reason that a watchdog kicks in, and that is a log that ceases to increment.
You seem be saying that either A) there's another reason for the watchdog to kick in (there isn't) or B) the watchdog is malfunctioning?
I don't really know what you're getting at, but yes, the log had to be frozen long enough to time out. I've never seen it not work and I can't imagine how it could be possible it doesn't.

 

[marino]

Portmaster workload is probably equivalent to use 1 synth builder, assuming you used 2 jobs while building, that would be approximately half of the synth load when using 2 builders and 2 jobs x builder. (and half RAM requirements, ... and half disk I/O ...).

all true, but there's also no watchdog. The synth task would eventually complete as well if it were off.

The watchdog is supposed to reap a stuck/frozen build but in this case there's a false positive on detecting that due to the machine being over tasked (mainly with bad luck on the ports that build simultaneously)

 

[PacketMan]

I do. 100Mb is nothing. It could be swapping to disk with every compilation. It's not the amount of memory in swap, it's how often the memory is swapped. If it's happening continuously, that will increase build time tremendously.

I will run vmstat that will tell me how much swapping is going on.

What are you saying?
There's exactly one reason that a watchdog kicks in, and that is a log that ceases to increment.
You seem be saying that either A) there's another reason for the watchdog to kick in (there isn't) or B) the watchdog is malfunctioning?
I don't really know what you're getting at, but yes, the log had to be frozen long enough to time out. I've never seen it not work and I can't imagine how it could be possible it doesn't.

I am saying the log is incrementing, its not stuck for 10 minutes, or an hour, etc. I don't know how else to say what I am observing. But maybe another reason (C) - Bug, or logic boo boo causing false positive?

We can rant this all night. But lets not bother. I'll do some testing and report back. Thanks again.

 

[marino]

I am saying the log is incrementing, its not stuck for 10 minutes, or an hour, etc. I don't know how else to say what I am observing. But maybe another reason (C) - Bug, or logic boo boo causing false positive?

It should be clear, but I'm saying there's no bug here at all. I'm positive on that. That means that either A) you're mistaken in the observation or B) there's a hardware malfunction regarding the computer's timing hardware. I assume the likelihood of the latter is extremely low.

I know things like GCC and chromium and gitweb have linking steps that take a ridiculous amount of time even on lightly loaded machines. linking large c++ programs can cause a log to stop incrementing for a while.

 

[marino]

This was from your earlier post:

cd /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5 && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5 /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5/Source/WebCore /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5 /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5/Source/WebCore /construction/xports/www/webkit2-gtk3/work/webkitgtk-2.8.5/Source/WebCore/CMakeFiles/WebCore.dir/DependInfo.cmake

It's possible that the process locked up (kernel malfunction?) because that's not a line that would lead to a long pause. So maybe the machine has some hardware issues ...