Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

tobik said:
What's the reason it aborts and which ports?
Click to expand...
It was not an "abort", the package was webkit2-gtk3, and while running on an old slow machine the watchdog that is implemented in Synth to reap stalled builds, killed that build. There is a whole thread about that: Thread 58614
 
ShelLuser said:
So here I am in /usr/ports/devel/apache-ant and I want to check which ports Synth found to be new. Picture me surprised...

Code: 
peter@macron:/usr/ports/devel/apache-ant# synth status
Please change the current directory; Synth is unable to launch from here.
As more experimentation showed me: Synth refuses to run inside the /usr/ports structure. I'm sure there are good reasons for that, but it's not always very desirable nor user friendly. Especially when using commands which do very little but show me the current status.
Click to expand...

A very good reason. If the cwd is null mounted inside the slave builders, the run will fail in spectular and completely unpredictable ways. solution: (cd / && synth status). In other words, run in a subshell to change directory first. While portmaster is designed to assume it's being run in ports tree, synth requires the opposite never happen for technical reasons, but it's not hard to work around.

But more so because I sometimes use Portmaster for specific tasks. Including, if needed, a careful upgrade of one specific port (-o). Which means that I'll have to specify the origin. Doable by utilizing #pkg info -ox <name> but also by being in the physical directory itself. That of course doesn't work.
Click to expand...

confirmed, you have to specify the origin. it's an intentional requirement given it's impossible to run synth in the ports tree due to null mount issue. It's a minor feature of portmaster that doesn't translate.

Why in the world would Synth rely on a (memory wasting) temporary file system while /tmp and /var/tmp have been carefully set up for this very specific reason? This machine I'm using isn't the most modern one, but it can maintain a package repository just fine.
Click to expand...

All current releases of FreeBSD comes with tmpfs by default. It can only be turned off by rebuilding the kernel that way I think. Confirmed, tmpfs is required although the big mounts (localbase and the work directory) can use a disk optionally but not all uses of tmpfs have alternatives. But no, I don't think it's specifically documented that the kernel has to support tmpfs.

So I learned of # synth configure, used that and confirmed my suspicions: The options K and L which provide me with a way to make Synth use (or not use) tmpfs for the work area and localbase (or both of course).

Just too bad that these options apparently get totally ignored. It's the only conclusion I can come up with.
Click to expand...

which as I just said is a faulty conclusion. K and L work just fine, but tmpfs is used in many other places.

So yeah, each to their own but this is definitely not for me.
Click to expand...

Okay, well. thanks for looking and good luck.
 
ASX said:
It was not an "abort", the package was webkit2-gtk3, and while running on an old slow machine the watchdog that is implemented in Synth to reap stalled builds, killed that build. There is a whole thread about that: Thread 58614
Click to expand...

I guess that depends on your definition of abort.

"Abort - bring to a premature end because of a problem or fault." Hmmm what's the definition of problem or fault? :D
 
PacketMan said:
guess that depends on your definition of abort.
Click to expand...

Among developers "abort" is usually intended as premature end of the program itself (Synth), not as stopping to produce its output (webkit2-gtk3 pkg), more precisely "abort" refer to events that will generate a SIGABRT signal or more in general a different signal, one of those cases where a core file will be dumped on your filesystem.
signal(3) abort(3)
 
I try to install net/wireshark-lite with synth, however looks like I can't do it
Code: 
abishai@sphinx:~ % doas synth status net/wireshark-lite
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
  N => net/wireshark-lite
Total packages that would be built: 1

abishai@sphinx:~ % doas synth just-build net/wireshark-lite
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.

abishai@sphinx:~ % doas synth force net/wireshark-lite
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
This command should work, what makes wireshark-lite special?
 
step 1 is understanding the problem.
how does one understand what synth does? They look at the logs.

There are two explanations for what you've seen:
  • the package has already been built and is current
  • The package can't be built because the port is setting IGNORE
The 00* log will tell you which one. So check the logs.


added: The "force" command means the first bullet can't be happening. You've got an IGNORE set somewhere.
 
Right, KRB_BASE is incompatible with libressl and I forgot to make config. I thought synth will print it. This tool is not newbie friendly! :p
 
Synth using memory till swap is full

(Not sure if this is the best place for this--perhaps give it its own thread or link to the thread below, that ended in May? )

I see that someone had this issue back with FreeBSD-10.x, where synth started using swap at an unexpected rate. https://forums.freebsd.org/threads/56171/ Lately, this has started happening to me on 2 FreeBSD-11 systems, both with reasonably good processors, zfs, and 8 GB of RAM. Adding a second swap partition didn't help. This seems to happen when building large packages, such as libreoffice, however, using portmaster or just doing make install from a port doesn't do this--it will use a lot of CPU, but swap stays at unused or just a little bit used. This happens when I run synth prepare-system. I haven't seen any mention of it save for that one thread I linked, so it may be one of those Just Me(TM) things, but I do see it on two different machines. I see swap gradually climbing till it reaches 100 percent and the machine becomes unresponsive. Logs show things like
Code: 
 09:28:47 s kernel: swap_pager: out of swap space
Dec 26 09:28:47 s kernel: swap_pager_getswapspace(5): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s last message repeated 6 times
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
 
it's definitely not a good thread for this. the thread is already 35 pages long and nobody reads (good) information contained anymore.
There are 2 possibilities:
  1. you've set too many builders + jobs for the resources you have for every case
  2. you've got a leak somewhere. Something is using but not releasing swap space. grep log directory for watchdog failures to see if those are happening (they really should never happen)
 
Thanks, (as always) for the super quick answers. I will try tonight, and see if I can find something useful. If so, I'll open a new thread (or add to the old one, as it will show up in new posts)

For what it's worth, I used defaults, and looking through logs I don't see any watchdog errors. This may mean more work in tracking down a memory leak than I have the ability (or time) to do right now.
 
scottro said:
Synth using memory till swap is full

(Not sure if this is the best place for this--perhaps give it its own thread or link to the thread below, that ended in May? )

I see that someone had this issue back with FreeBSD-10.x, where synth started using swap at an unexpected rate. https://forums.freebsd.org/threads/56171/ Lately, this has started happening to me on 2 FreeBSD-11 systems, both with reasonably good processors, zfs, and 8 GB of RAM.
[/code]
Click to expand...

I had the same problem still, now on FreeBSD 11-RELEASE. Not many times but it happened.
Not so long I use Synth and at the same time it built LibreOffice and Blender. SWAP raise to 36% and after LibreOffice was done dropped to 13.6%. And when was everything done swap stayed on 13%??
Thank you.
 
fernandel said:
Not so long I use Synth and at the same time it built LibreOffice and Blender. SWAP raise to 36% and after LibreOffice was done dropped to 13.6%. And when was everything done swap stayed on 13%??
Click to expand...

Is this is a question?
There's nothing wrong with swap staying high if that's the only fact available.
In other words, just because memory becomes available doesn't mean that contents stored in swap move back to memory (and consequently swap goes back to zero). It's OS-specific, but I don't think FreeBSD works that way either.
 
8Gb of ram is not enough for synth on my system. When 2 builders start to build Firefox/Thunderbird it starts swapping heavily and eventually system crashes. I disabled tempfs for workarea in Synth config and everything runs smoothly after that.
 
how much swap do you have? maybe you find a cheap 64gb or higher SSD and use that (assuming we're not talking about a space constrained machine like a laptop here).

The original problem was likely your swap partition was way too small, but if the swap is a spinning disk, it's not much better than just turning off tmpfs for the workspace.

alternatively, you can shift the synth build root to an SSD partition so that building without tmpfs won't be such a big performance hit.
 
Today I started a synth update, but first tried limiting vfs.zfs.arc_max from its default of RAM minus 1 GB to about 4 GB. I think something is leaking memory, but don't have the expertise to track it down. In my case (on a ZFS mirror in one case, on a single disk pool in another--haven't tried on that machine yet), it seems to have fixed the issue. (The build is not yet complete, but yesterday, buidling llvm37 and 39 and rust it swapped out, this time swap use is staying, so far, at around 12 percent. I'll update in the other thread if this solution fails)

marino@, I realize that at this point you don't consider this a synth issue, so I'm copying these few posts, (with thanks for their input to abishai and fernandel) to a fresh thread.

https://forums.freebsd.org/threads/59107/
 
Thanks scottro@, can you copy my last response there too?
reading your new thread, I realize there was an obvious 3rd explanation that nobody mentioned: The swap partition is simply too small (thus running out of swap is a legitimate message)
 
For me a uneventful synth run starts like this:
  • service cron onestop
  • disable screensaver and sleep functions in my KDE system settings
  • set scrollback in Konsole to 'none'. Just imagine how much memory your consuming in 8 hrs if your using top 'vmstat -m'
  • remove/dont use troublesome usb devices during build. This is obvious but included for completeness.
 
I would say continue using the previous discussion thread you posted.

Edit: Whoops, logged onto the forum and saw this alert. Carry on.
 
I used synth today to upgrade my packages and noticed something new. Normally I have 2 build processes running constantly. This time, one or the other slot would periodically go idle. I haven't watched synth work with rapt attention in the past but this seems to be a new behavior. Is it expected for a build slot to remain idle with work left to be done? If that doesn't sound right what data should I collect for a bug report?
 
lebarondemerde said:
Probably, all the ports waiting was the single port that was being done as dependency. So, they need to wait that one to be done before get a slot.
Click to expand...

Yep, that is what is happening. They will change to "shutdown" rather than "idle" when #builders > #packages yet to build (or something similar -- not sure if it has look-ahead planning to see how many might yet run in parallel.)
 
You must log in or register to reply here.
Back
Top