Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

Is it possible to use synth for speedier buildkernel/buildworld, too?

(Sorry if this has been asked before... this thread is just too long to read all)
 
are you using synth prepare-system or something like that? (as opposed to a static build list?)
If so, it could be that an old package relies on python27, even if the newer versions don't.

Also, you state they are being built, but not if they are being installed. Python27 can easily be a build dependency for any number of ports.
 
Yes, I am using synth prepare-system and that is what triggered it.

I was just curious on what package depended on it, because the only reason I installed lang/python27 to begin with is in order to use those 3 ports mentioned above.

It was never a build dependency or installed until I installed those 3 ports.
 
when in doubt, use the procedure to remove all packages from the system and reinstall them from a local repository. That gets rid of all the cruft that builds up. You may have manually installed python27 in which case pkg(8) is telling synth to rebuild it because it's a primary package.
 
When synth is regenerating the flavor index, does it use tmpfs or does it write directly to disk?

I have tmpfs enabled in the synth config, but my HD shoots up to 40%-50% load when the index is regenerating. I have plenty of free ram and swap is not being used.
 
it reads the mounted ports tree (every port) with all available CPUs and writes directly to disk. The writing is not an issue. If it takes 10 minutes to scan the ports directory with 4 CPUs, it would probably take an hour to do it with 1 CPU. With recent (long overdue) improves in compiler features testing, the scan time might reduce considerably though.

In any case, the server is very busy during portstree scans and a huge load jump is normal.
 
To reduce the disk io, would it not be possible to build/store the flavor index in tmpfs until it's complete and then dump it to disk?

I don't notice any load on my SSD's on my production server, but my home machine (which has a standard HDD) is hovering right around 50% for that 15 minutes or so.
 
as I said, *building* is not the issue. The actual creation of the index takes less than 1 second. Dozens of instances of make are trying to read various directories, each one spawning hundreds of forks. You can thank the ports collection makefile design for that. tmpfs isn't going to help any of that.
 
They are already in the tree I think. You should already be seeing improvements in the scan time (I'm guessing; all I saw was the code, I don't know for a fact that it significantly helps FreeBSD, but I hope so)
 
then i would guess that it's code that makes poudriere faster that isn't active for any other system. If poudriere is using tricks to store the compiler features, it would be nice if that trick was exposed for all systems. The change has been in place for about a week now.
 
marino said:
then i would guess that it's code that makes poudriere faster that isn't active for any other system
Click to expand...

I really hope that isn't the case!

I'm definitely not seeing any improvement over here. I just timed it and it took 21 minutes to regenerate the flavor index on my home machine. It's only used to receive backups from my dedicated server.

It's an Intel i5 CPU with 8 GB ram. Here is the "Top" stats when synth is regenerating the index.
Code: 
last pid: 14470;  load averages:  1.16,  0.93,  0.54
31 processes:  2 running, 25 sleeping, 4 zombie
CPU 0: 28.0% user,  0.0% nice,  6.5% system,  0.5% interrupt, 65.0% idle
CPU 1: 12.6% user,  0.0% nice,  4.7% system,  0.0% interrupt, 82.7% idle
CPU 2: 15.0% user,  0.0% nice,  6.1% system,  0.0% interrupt, 79.0% idle
CPU 3: 22.4% user,  0.0% nice,  6.5% system,  0.0% interrupt, 71.0% idle
Mem: 32M Active, 368M Inact, 945M Wired, 536M Buf, 6471M Free
Swap: 3852M Total, 3852M Free

And my config :
Code: 
   [I] Num. concurrent builders   4
   [J] Max. jobs per builder      4
   [K] Use tmpfs for work area    true
   [L] Use tmpfs for localbase    true
 
here's the commit:
https://svnweb.freebsd.org/ports?view=revision&revision=463259

maybe you can find a reference to where _CCVERSION_hash and _CXXINTERNAL_hash are getting defined. I was guessing that poudriere outputs these.

However, it means it's possible to add these caches to synth's [profile]-make.conf file to leverage this new code. I just don't know what the values are for FreeBSD release 11, 12, etc. Somebody could figure it out and post it.

The idea is that the values are determined ONCE and written down in [profile]-make.conf, rather than be determined hundreds of times a second, all with the same identical result.
 
Out of curiosity, when you say the flavor index takes 21 minutes to generate, how many ports do you have installed? Or is this literally recursing through every single port in the port tree? I'm building around 200 ports.

The reason I'm curious is because I switched to Poudriere when Synth temporarily broke because of gcc-aux not building and I've not tried Synth again since. But Poudriere doesn't spend any time generating long indexes so I'm wondering why Synth needs to take that long. I don't even use ZFS so it spends time copying files around on UFS but a bulk run with four jails still only takes a single minute. If Synth takes 20 times longer than that on every run then that's not a good improvement on when I was using it before.
 
I don't have many ports installed at all. This machine is only used to receive backups every night from my dedicated server.
Code: 
[backup@localhost ~]$ pkg stats
Local package database:
        Installed packages: 15
        Disk space occupied: 37 MiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 32
        Unique packages: 32
        Total size of packages: 101 MiB


This is the log I saved from last night so I have something to compare to if / when things get sorted out with Synth. As you can see, it took 21 minutes to rebuild the flavor index.
Code: 
[backup@localhost ~]$ date && sudo synth status && date                     
Mon Mar  5 17:52:58 PST 2018
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
Mon Mar  5 18:14:00 PST 2018
[backup@localhost ~]$
 
xtaz said:
But Poudriere doesn't spend any time generating long indexes so I'm wondering why Synth needs to take that long.
Click to expand...

Because the ports tree design sucks. And this is a misleading question. The same exact operation on DragonFly takes about 2-3 minutes. Yes, DragonFly Dports is 10x faster than FreeBSD ports because most of the inefficient crap that is in the ports tree was removed/fixed on dports.

poudriere doesn't use indices, but it's not like it's not doing the same calculations. Synth does it ONCE per tree update, poudriere does it hundreds or thousands of times repetitively. It determines via a make query each and every time. The main difference seen by a local user is that even thought it's inefficient, they might scan only a fraction of the tree for any given run, but synth does the entire tree because it doesn't know what will be required between then and the next tree update.
 
Ahhh that's a good answer. Thanks. OK. But I would question the wisdom of this. As Synth is kind of being used as a supposedly less resource intensive alternative to poudriere for people to build local ports with. But for that use it's now recursing the entire port tree every time you use it which negates this purpose.

If you're building the entire ports tree like the dragonfly or freebsd pkg builders would be then fair enough, I totally see your point. But if like most users you're only building a 100 ports or so it seems intensive.

Every 24 hours I update my ports tree and run a bulk build on the ports I have listed. Assuming no new ports get built poudriere is taking just over a minute. Lets say Synth takes 20 minutes I don't want to be doing that every 24 hours.
 
But for that use it's now recursing the entire port tree every time you use it which negates this purpose.
Click to expand...
False. It's scanning the tree when the tree changes. People don't update the tree, build one port, update the tree again, build another port.
I assume most people have the port upgrade and index creation on a cron job and do it unattended.

The real issue is that ports doesn't provide this information in the INDEX provided by portsnap.
It all goes back to a poor design. You can blame synth for how it works around this obvious deficiency, but I wasn't going to completely rewrite synth to work in the hacky way poudriere works.

edit: and obviously, if ports manager would fix the real issues rather than obscuring performance tweaks inside poudriere, then every tool could benefit from a more efficient tree. If I had time, I could post a make.conf that would speed up scan times by 75% easily, which is basically doing the same tricks poudriere is doing. Oh, and maybe they could arrange the index to contains flavors (or a second index) so the index generation is done by the freebsd cluster once and doesn't have to be recreated by every user.
 
I have portsnap fetch update & synth status on cron, but I still run them manually before I upgrade any ports to ensure I have the latest versions. So I'm seeing that 21 minute delay each time.

That's why I am hoping someone can figure out what we need to do with Synth to work with the new changes to the ports system and speed things up.
 
You must log in or register to reply here.
Back
Top