Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

But I still don't get it. by it's nature, building is greedy. (jobs/builders can be modified with different profiles, so launch a "nicer" profile instead, right?)
 
I'll stick to nice(1) synth itself. But the default config just hogs all resources, which is good when you want to have it done faster. If you want to have anything else done while building, this is not so good.
 
protocelt
Try this: create a second profile, call it, say, "Nice"
Set the builders and jobs low.

Then launch synth with env SYNTHPROFILE=Nice synth upgrade-system or something like that.

Crivens can do the same.
 
I bring up the configure screen to check that things are in order after each minor update you push out before building so I just change it from there since I'm already at that screen. I won't likely do that after Synth is tagged feature complete/bug free. SYNTHPROFILE is mentioned in the ENVIRONMENT section of the man page so I was aware of that, but thanks for the pointer. :)

I do have a second profile and have tried that already. No problems.
 
it's been feature complete since 0.99 --- at least as far as the features I think 1.00 should have that is.
I'm not aware of any bugs right now.
The only thing preventing 1.00 from being tagged is time. If a few days goes by and nobody finds any new problems, I'll do that.
In any case, it's damn close.
 
marino@ said:
it's been feature complete since 0.99 --- at least as far as the features I think 1.00 should have that is.
I'm not aware of any bugs right now.
The only thing preventing 1.00 from being tagged is time. If a few days goes by and nobody finds any new problems, I'll do that.
In any case, it's damn close.
Click to expand...

Out of curiosity, what are some of the future features you're considering adding?
 
On the short list is caching tree scans (only scan tree after it's changed) because those take minutes. 4.5 minutes for me, but I just heard somebody quote 20 minutes to 60 minutes. Maybe ability to delete profiles. Then we'll see what other people request (that can't be handled in other ways)
 
I didn't realize the scan was taking so long for some people. It takes about 45 seconds or so on my main machine and I'd say about 3 minutes max on the other one so I never gave it a second thought. Anyway, I'm sure a speed increase of any kind would be appreciated.

One thing that might be nice is adding a way to add/remove make(1) options for ports in the configure screen instead of manually editing Livesystem-make.conf(or whatever profile your using). Just a thought.
 
/usr/bin/time synth status
82.11 real 58.41 user 42.57 sys

It's scanning 930 installed packages on a pretty old core2duo laptop. I'd say ZFS on SSD can make up for some old age.
 
One thing that came to my attention might be to block the different builders from staging/building ports in parallel which are huge and have a big set of dependencies. Especially when you use tmpfs for this. When for example libreoffice and firefox try to get staged at the same time, this will hit the swap pretty hard and performance will tank completely. Maybe as a rule of tumb you can count all the dependencies and allow only for a maximum to be active at any time, then schedule the builds accordingly? But that might be something for maybe 3.0 ;)
 
it was in the original design but I did not implement it because I never hit it myself. If that is happening the builders/jobs settings are wrong. You should always increase jobs over builders when memory is in play. My suggestion is reduce number of builders and increase number of jobs (e.g. 4 builders x 3 jobs => 3 builders x 4 jobs)
 
in other words, synth won't prevent somebody from shooting their foot off.
Did you keep the suggested builders/jobs from the initial configuration, or did you modify those settings?
 
talsamon what are you implying? Are you trying to say this is a bug? Why do you feel qualified to discuss what is "necessary" and "not necessary"? You've made several inaccurate technical statements already. If you believe those statements, fine, but I know Synth build logic has no flaws in it.
 
half of this 16 page thread has been dedicated to these observations by you and I'd just as soon not continue doing that (it's noise that can confuse others and give them the wrong idea). If you aren't confident in synth logic or if you prefer another approach, then just don't use the tool.
 
marino@ said:
in other words, synth won't prevent somebody from shooting their foot off.
Did you keep the suggested builders/jobs from the initial configuration, or did you modify those settings?
Click to expand...
I kept the initial settings. That was two builders with two jobs each, for a core2duo with 4GBytes of memory. I switched it to "tmpfs" where possible to keep the wear of the SSD down a bit. Maybe that was a mistake as the swapping will have done some wear on it's own. And I added devel/ccache to the settings.

Maybe it is that ZFS is holding on to it's memory with more strength than anticipated? This might mess heuristics up somewhat.
 
If you can run tmpfs(5), then do that. Only turn if off if you have no choice.
Personally I've not seen loads higher than 10 or swap higher than 5% on my machine.
2x2 is pretty low. maybe 2x1 would be better? I recommend at least 2 so at least you can fetch and build at the same time (unless you have the bad luck of both builders fetching).
Santa Claus didn't bring a modern machine this year? :)
Maybe building and doing something other than building isn't going to work well on that machine, no matter what.
 
marino@ said:
Santa Claus didn't bring a modern machine this year? :)
Click to expand...
Not this year, I have been a bad boy. For some time. ;) Maybe this year, or next. I usually use any hardware untill it breaks down completely, for environmental reasons among others. Also, there is currently no laptop which is affordable, worth it's money and trustworthy (sorry Lenovo, you are out).
 
I noticed that the log files under /var/log/synth could use some more organization, poudriere for example uses subdirectories with timestamps in their names to separate logs from each build run to their own subdirectories.
 
kpa said:
I noticed that the log files under /var/log/synth could use some more organization, poudriere for example uses subdirectories with timestamps in their names to separate logs from each build run to their own subdirectories.
Click to expand...

And poudriere's choice causes a ton a problems. each full bulk run costs 2Gb of disk space. Poudriere has no inbuilt way to clean logs and boom: suddently you have 20G of logs.

I made a conscious and informed decision to handle logs differentfy and intentionally only retain the last log and not every log that's every been attempted. It's also a conscious decision to base log names on port origins and not package names. So this is not an oversight. Poudriere has a different purpose and timestamped logs make sense for that purpose. I'm not interesting in following poudriere lead on logs as its been working quite well in my opinion (with the only real criticism that apache doesn't like to display 20,000 logs in one virtual directory, but that's an apache problem)
 
kpa said:
It's not really needed but # pkg upgrade -f is safe to run (if it wasn't we'd have big problems) and usually doesn't take too long. It's needed when you switch from one repository to another or there is a major change in the repository you're using such as a change in the default version of perl.
Click to expand...
# pkg upgrade -f and everything was fast and no errors (1689 packages).
 
marino@ It is difficult to follow up this thread. However, I wanted to congratulate you for your excellent job. I really hope that more developers can get inspired by your patience and innovation.
 
You must log in or register to reply here.
Back
Top