Synth v1.10 has been released (fast repo generation)

marino

marino

I had planned release 1.1 even before the 1.0 release to address the noticibly long repository generation times, especially on older hardware with mechanical disks. I think the latest version should make people happy but definitely feedback is welcome.

Code: 
ports-mgmt/synth: Upgrade version 1.03 => 1.10

This release addresses unacceptably long repository rebuild times for the
worst cases (FreeBSD [1], slow CPU, slow mechanical disk).  Until now,
rebuilding the repository required a full tree scan (nearly 26k ports).
While I only saw around 4 minutes on a 4-year DragonFly machine equipped
with a SSD, others reported times exceeding 20 minutes.

This new method scans existing packages twice -- first to eliminate those
packages where the port was removed and also those with a mismatching
version (parallel).  This sets up a second pass to serially and
recursively scan the ports of the remaining packages.  That leads to the
final validation (same as previously done) and the actual repository
generation.  Now the repository generation time is much shorter, but
corresponds to the number of build packages in the packages directory.

The long repository generation times were identified prior to the 1.0
release, but I targetted 1.1 for the enhancement.

I forgot to put it in the commit message, but [1] was going to refer to the horrible USES=compiler:features implementation that makes freebsd slow at scanning the ports tree.
 
I running synth-upgrade system now about 20 minutes and building is going very slow. It is still on package for print/texlive-texmf (16 minutes). Before I had in 20 minutes at least 10 port or more done.
Now is done: 17 minutes.
And I didn't change anything in the settings.
 
irrelevant.
  • texlive-texmf is a huge port that takes a long time to build
  • nothing changed with building
it's a coincidence at best.
 
I am wondering if you misunderstand what the enhancement is.
To be clear, it makes the command synth rebuild-repository generate the repository from a set of already-built packages in seconds rather than many minutes.

You seem to be talking about time to build individual packages, hence I'm wondering if you are thinking it's supposed to build packages faster. That's not what the change was about.
 
Code: 
ports-mgmt/synth: Upgrade version 1.10 => 1.11

This fixes a regression in building ports that have dependences that
install kernel modules.  When DTrace support was added by providing a
read-only mount of /boot to the builder, the kernel modules could no
longer be installed at /boot/modules by pkg(8).

Previously, although successful, module installs would have caused a file
system violation on test mode checks.  Since /boot is now excluded from
checks (since DTrace support), leftovers in /boot/modules will not be
detected in test mode.  The fix is too elaborate and FreeBSD-specific
to worry about (plus there's the philosophy question about why the ports
framework is even allowed to modify the base but that's out of scope).
 
I've just timed how long it took to rebuild-repository on my system. It took 31 minutes before. With this version it takes 1 minute 31 seconds. So I'm a very happy bunny :) Thanks!
 
Something screwy is going on actually. I just tried to prepare-system after changing a port option in openssl. It rebuilt 67 packages, rebuilt the repository, and then all the ports it rebuilt failed a dependancy check and it deleted all the existing packages from the repo.

Code: 
The task is complete.  Final tally:
Initial queue size: 67
  packages built: 67
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:50:44
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 212 ports serially.
Scanning existing packages.
py27-setuptools27-20.0.txz failed dependency check.
py27-pytz-2015.7,1.txz failed dependency check.
py27-Babel-2.2.0_1.txz failed dependency check.
... and so on

I tried it again, and same thing, it built 67, then deleted them again. From running ls in the repo directory I can see that all the built packages exist right up until the final "Scanning existing packages." part, then they vanish.
 
it smells like a port error somewhere.
However, it's usually the first port on the list that causes the cascade, so that would mean py27-setuptools27 is the culprit. (or the first victim of your change), but that port will wipe out a lot of others.

What *exactly* did you change?
 
I changed the MAN3 option to false in the openssl port. But even changing it back to true does exactly the same thing so I don't tihnk it's related. My port tree was out of date though as portsnap appears to have not had any updates all day, I guess there was some issue. It appears to be updated again now though so I've just managed to upgrade it to synth 1.11. I'm just giving it another run with the latest port tree, see what happens.
 
seems that lang/python27 is the only dependency of py-setuptools27.

You might try dumping the build list, using "just-build" option, and then "status" so it doesn't actually detect the packages next time. (and set WHYFAIL=yes in the environment before running status command)
 
Last edited by a moderator:
Heh OK. Either updating to synth 1.11 or updating the ports tree has fixed it. It's not deleted anything this time. Portsnap updated between Thu Mar 3 06:45:00 GMT 2016 and Thu Mar 3 22:08:51 GMT 2016. So might have been something between that time. Oh well working now :)
 
Just adding this update is working well for me and is quite fast across the board again. No problems yet to speak of. :)
 
marino@ said:
irrelevant.
  • texlive-texmf is a huge port that takes a long time to build
  • nothing changed with building
it's a coincidence at best.
Click to expand...

Hi marino@

I don't think so...I have four hours in "checksuming" the texlive-texmf ports, and nothing. Synth stuck in this task, right now I'm reboot my computer, delete all distfiles and rebuild again this port, more information incoming.

My CPU is a Pentium G645 and 4 Gb RAM, two hard disk 320 GB in stripe zpool.
 
how is it relevant?
The release didn't change building at all, and fernandal was talking about building issues (and latter retracted it).

You realize that one of the texlive-texmf tarballs is close to 2Gb is size, right? I'd say that would give a pentium some exercise to checksum.
 
(note that it also takes a long time to fetch a 2Gb file, and it's possible the checksum phase is including the fetch)
 
Fetch file isn't a issue here, but checksumming is other history, in this Pentium (sorry for my "old" hardware) I checksumming files for 4 GB and more using SHA1, without issues and don't take long time.
 
Run the fetch, checksum, extract and patch phases manually on the port and measure the time taken by them with time(1). This takes the package builder out of the picture. You will be surprised by the results.
 
okay, then given that this machine is the only known machine that fails to build texlive-texmf, what is it you would like me to do?
Nobody else is reporting this. Did you check the build logs?
 
also, I would look at your memory and swap situation. Maybe you saturated swap? you aren't using tmpfs with low memory are you?
 
I have already said, I am restarting the reconstruction of the package, I deleted everything again, distfiles, waiting is a failure by some strange and random reason, I'll leave it running long enough until it is built, and then'll post the result. Then do the same using the time option, obviously erasing the distfile and see what happened.

On the log, I have not seen, nor package actually came to build the process stopped about 5 hours of checksumming, the rest, synth has worked perfectly.

Perhaps because of what you have told me marino@, the checksummig maybe that package is a good exercise for my Pentium,
 
okay, well, you didn't address memory (starting with how much the machine has and the tmpfs settings) or what the "swap" measurement was. I'd look at memory in this case.
 
It happened to me today:
security/gcr: package Lines: 2043
, time 00:01.047
www/webkit2-gtk3 configure, lines 1076, time 00:00:49
Code: 
swapinfo
Device  1K-blocks  Used  Avail Capacity
/dev/ada0p6  3773548  312736  3460812  8%

top shows:
Code: 
PID USERNAME  THR PRI NICE  SIZE  RES STATE  C  TIME  WCPU COMMAND
15021 root  1  84  0  155M  135M CPU1  1  0:05  41.36% c++
15014 root  1  84  0  167M  145M CPU7  7  0:05  39.60% c++
15024 root  1  82  0  156M  132M CPU3  3  0:04  36.28% c++
15030 root  1  80  0  123M  103M CPU4  4  0:03  26.95% c++
15026 root  1  52  0 16820K  3432K wait  2  0:00  2.39% ccache
15029 root  1  52  0 36272K 14140K wait  3  0:00  2.39% c++
10662 fernandel  59  27  0  934M  416M uwait  5  1:45  1.76% firefox
15023 root  1  52  0 36272K 14140K wait  6  0:00  1.76% c++
15010 root  1  52  0 16820K  3228K wait  0  0:00  1.66% ccache
15018 root  1  52  0 16820K  2972K wait  7  0:00  1.66% ccache
15019 root  1  52  0 36272K 14140K wait  1  0:00  1.66% c++
15013 root  1  52  0 36272K 14140K wait  5  0:00  1.46% c++
 1896 root  1  52  0  338M  330M piperd  6  0:58  1.37% gmake
15006 root  1  52  0 16820K  3332K wait  3  0:00  1.37% ccache
52875 fernandel  8  20  0  657M  146M kqread  0  14:21  0.98% gnome-she
 1274 fernandel  2  20  0  210M 37820K uwait  0  32:02  0.00% Xorg
It looks building is going (gcr is build and is in/var/synth/live_packages/All). Maybe is something wrong with ncurses?
Thank you.
 
"It happened to me today:"
What happened today? I don't understand your description of what the problem is.

Are you saying something didn't build?
 
You must log in or register to reply here.
Back
Top