Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

talsamon, you've been making a lot of technical comments like "this is not necessary to do". I've not been responding to them because they are not correct, but I don't want to get into the hows and why they aren't correct. The failure cascades are absolutely required, not doing them would result in an unusable repository. Your "one level deep" theory will fail. If you give it a lot of thought you can probably deduce why.

I'm working on some polish problems (synth hasn't reached release yet) but one area that has been bulletproof is the build logic. It's correct.

And yes, if you aren't using ccache, you absolutely should. (And the reason I don't trust your machine is because you have several issues that cannot be reproduced on multiple machines and multiple OS, so I'm really leary of your machine)
 
talsamon said:
No, if you say above you need 55 minutes for lang/ghc my machine need 68 minutes, not much slower.
And the system is running good since last february without bigger problems (only not ptoprtrly plugged cable, in july
destroyed the system).

Maybe portmaster don't recompile all depencies that needed and don't check the portstree right, but this is too much.
Why must firefox recompiled? Alsa-libs sure, but firefox? Why must libxul recopmiled, gstreamer yes, but libxul? And why virtualbox-ose has only one audiooption (pulseaudio) it is off. Which option has any dependency to audio or multimeida=

During the compile you can't use the machine (compiler take all resources it could get). And if I limit it, it needs an endless time. If it's one or one and a half hour (or sometimes longer) it is ok, in summary, but nor for only one small package like audio/lame. And why it compiles all the other packages store it in the repo - but don't use (install) it?? But you can't wait e.g. three hours for them, and maybe each day (or as I mentioned above, for a day, if you update once in the week.
Don't misunderstand me, I honor your work - it seems a good tool for testing, but it seems not for updating And I wanted to have a better tool as portmaster for long time - but this is nearly "overkill".

kpa I see it with the options in lang/gcc (this was not synth, ok). I did not try to remove ghc. (and I see compat9x has no libutil file, so I think we can remove this LIB_DEPENDS line in Makefile, I see this after the last try). Maybe that is the reason for synth rejected config, if I try to set the option Boot to on. Somethings wrong with the port. Next try I will remove ghc before I try it with synth.
Click to expand...
For example:
Code: 
pkg info -r lame
lame-3.99.5_3:
   gstreamer-plugins-lame-0.10.19_1,3
   ffmpeg-2.8.5,1
   transcode-1.1.7_25
   gstreamer1-plugins-lame-1.6.3
   sox-14.4.2_1
   normalize-0.7.7_10
   aqualung-1.0
   mencoder-1.2.r20151219_2
and than:
Code: 
pkg info -r ffmpeg
ffmpeg-2.8.5,1:
   youtube_dl-2016.01.15
   libxul-38.5.2
   gstreamer1-libav-1.6.3
   alsa-plugins-1.1.0
   vlc-2.2.1_6,4
   opal-3.10.10_9
   mediastreamer-2.12.1
   transcode-1.1.7_25
   mlt-0.9.6
   chromaprint-1.1
   libquicktime-1.2.4_11
   firefox-43.0.4_1,1
   gimp-gmic-plugin-1.6.0.0_4
   opencv-2.4.9_7
   aqualung-1.0
   blender-2.76b

It is what I think why synth rebuild more ports not just "lame" in this case.
Maybe I am wrong.
 
marino@ said:
Yes, it is. The code is designed to pick "FreeBSD" repo (and then eject). If "Synth" was encountered, it would just keep going. However, it seems like the external repository check is happening on Synth, not FreeBSD. Strange. Seems that it should be working. (It works for me, but I'd like to see it work on FreeBSD)
Click to expand...

Well I really hope you get this working, and I will be willing to work with you to provide any help I can. I intend on using the [N] fetch prebuilt option heavily on my XFCE desktop machine, and maybe other desktop machines someday. I think for my NAS machines I will not use [N] option since number of ports installed is small. That just said I will be at the mercy of waiting for ports to get updated before we can 'try again'. (All we need is to wait for a Firefox or Chromium update. :D)

Lastly, (maybe for a version 2.0 of synth) I would really appreciate seeing the fetching of pre-built packages, and associated counter, after the ncurses display, not before. I know you got the smarts; you can do it! :)......but I realize beggars can't be choosers. :beer:
 
Hmmm, okay so my initial run is done. Rebooted, updated the ports tree again, and ran synth again. Does this mean the fetch packages option is working?

Code: 
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
These are the ports that would be built:
  => audio/jack
  => audio/fluidsynth
  => editors/openoffice-4
  => multimedia/vlc
Total packages that would be built: 4
The complete build list can also be found at:
/tmp/synth_status_results.txt

# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.

That just said, for these 4 ports, manual compiling from source is being done, fetching a prebuilt package was not done.

I'll keep watching for it, but I did not see these lines this 2nd time around:
Code: 
Stand by, updating external repository catalogs ... pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Failed!
The external repository could not be updated.
No prebuilt packages will be used as a result.
 
And no end:
I tested ccache with firefox.
First time it compiles firefox.
Second time it compiles firefox and 3 other packages (why the 3 others).
After this:
Code: 
subversion-1.9.3_1.txz failed option check.
openjdk8-8.66.17_3.txz failed option check.
p5-subversion-1.9.3.txz failed dependency check.
py27-mako-1.0.3.txz failed dependency check.
portdowngrade-1.5.txz failed dependency check.
porttools-1.06.txz failed dependency check.
truthtable-1.2.2_2.txz failed dependency check.
freecol-0.11.6.txz failed dependency check.
tvbrowser-3.4.3.txz failed dependency check.
icedtea-web-1.5.2.txz failed dependency check.
jedit-5.3.0,1.txz failed dependency check.
deluge-1.3.12,1.txz failed dependency check.

what the hell, I have nothing changed in the ports.
 
P.S: I have removed the firefox-package. Synth does not want to compiles, as it was there. Causes the removing of the packages such things. The same package the same version, and what has firefox to do with portdowngrade and so on.....?? That is irrational.
 
There is no way out. If I start with ccache, without swapfile, it runs out of space. If I start with both, ccache does not work.

Edit: Is not true, both work in thier ports. synth only works one time with ccache. After reboot it works no more. I have changed nothing in the configuration.
 
Last edited by a moderator:
Dear All,
please start with a minimalistic collection of ports first. I have run my very first tests starting with ports-mgmt/poudriere and also with devel/ccache just with mail/abook and www/lynx with almost no dependencies. I am pretty sure that for ports-mgmt/synth the procedure makes sense, too.

If a tool is new to me it is very likely that the mistakes are on my side. Sorting that out is by far less time consuming using small ports. In 99.99% it is not sorting out but finding out that the issue is https://en.wikipedia.org/wiki/User_error#PEBKAC.
 
talsamon said:
fernandel Num, concurrent builders: 2. Max. jobs per builder 3. Swapfile 12 GB. (8 GB RAM).
Click to expand...
Do you have swap partition or swap file?
I have 2 as you and 4 per jobs, swap partition is 3 GB and I never had a problem...usually is used about 1% to 4% and I have 8 GB of RAM too.
 
Last edited by a moderator:
A swapfile. I know it is too big, the system only use 1200 MB. But experimented, cause I had a kernel-panic warning at shutdown (But till now, it causes nothing - Last reboot it was no warning).
 
PacketMan said:
Hmmm, okay so my initial run is done. Rebooted, updated the ports tree again, and ran synth again. Does this mean the fetch packages option is working?

I'll keep watching for it, but I did not see these lines this 2nd time around:
Click to expand...

Maybe but not necessarily. Your synth repo exists now, so it could be reading it (and thus never downloading anything). You got the lines the first time because there was no repository generated yet, but now it has been.
 
fernandel said:
Do you have swap partition or swap file?
I have 2 as you and 4 per jobs, swap partition is 3 GB and I never had a problem...usually is used about 1% to 4% and I have 8 GB of RAM too.
Click to expand...

I think you are on the right track. Something is seriously suspicious about his machine. He keeps talking about how "irrational" everything is, and how nothing is reproducible, but he's the only one this these issues. Chances are there is a serious misconfiguration somewhere. Maybe it's this swapfile thing. If not, it's got to be something unorthordox.
 
marino@ said:
Maybe but not necessarily.
Click to expand...

Just did another ports tree update. 26 updates available, and all are being built, meaning zero pre-built packages were fetched. If you need me to post anything let me know, otherwise I'll just carrying on using synth and waiting for an update that says fetching of pre-built packages is fixed! :)
 
marino@ said:
Maybe but not necessarily.
Click to expand...

This sure looks like it fetched (and then installed) some pre-built packages. But note this came after the ncurses screen, not before. NetSpades and libslang2 were built in the ncurses screen first.

Code: 
# synth install games/netspades
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
The following packages will be fetched:

New packages to be FETCHED:
   glib12-1.2.10_15 (11.04% of 1 MiB: 145 KiB)
   gtk12-1.2.10_24 (88.96% of 1 MiB: 1 MiB)

The process will require 1 MiB more space.
1 MiB to be downloaded.
Fetching glib12-1.2.10_15.txz: 100%  145 KiB 148.4kB/s  00:01  
Fetching gtk12-1.2.10_24.txz: 100%  1 MiB  1.2MB/s  00:01  
Scanning entire ports tree.
Scanning existing packages.  
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01  
Fetching packagesite.txz: 100%  150 KiB 153.4kB/s  00:01  
Processing entries: 100%
Synth repository update completed. 658 packages processed.
Checking integrity... done (0 conflicting)
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   NetSpades: 4.2.0_9 [Synth]
   glib12: 1.2.10_15 [Synth]
   libslang2: 2.3.0 [Synth]
   gtk12: 1.2.10_24 [Synth]

The process will require 13 MiB more space.
[1/4] Installing glib12-1.2.10_15...
[1/4] Extracting glib12-1.2.10_15: 100%
[2/4] Installing libslang2-2.3.0...
[2/4] Extracting libslang2-2.3.0: 100%
[3/4] Installing gtk12-1.2.10_24...
[3/4] Extracting gtk12-1.2.10_24: 100%
[4/4] Installing NetSpades-4.2.0_9...
[4/4] Extracting NetSpades-4.2.0_9: 100%

But last night I did another ports tree update and Open Office was updated. But regretfully the pre-built package was not fetch, instead it downloaded the source, and took about 6 hours to compile. :(

Thoughts? Comments? And why does it scan existing packages twice?
 
I don't think it's technically possible that fetching came come after ncurses screen.
Those are difference package scans. The first is limited to the build (very small number) and the second is every package that has been built (potentially a very big number). That last scan only happens on installs.
As for skipping OpenOffice, it apparently decided it was unsuitable (did not meet requirements).
 
Is there actually a problem with the pre-built packages? I think you are all talking past each other unless I have missed something.

Now, for build time dependencies it is of course beneficial to use existing packages as much as possible and as far as I have gathered it is working already? This is situation where a port has an update but the update will not affect other installed ports because none of them is depending on the updated port. The build time dependencies can be installed from existing packages before launching the port build.

The other thing is then what happens when there are dependent ports on the updated ports. I don't see how you could use pre-built packages in this case. The knowledge that a dependent port did not actually require a rebuild is not available until you actually have finished rebuilding of the dependent port. There is nothing you can do before the build to do any clever guessing if the rebuild will result in a changed package or not, that would require some very advanced code analysis tools that just not available for FreeBSD ports.
 
as far as I have gathered it is working already?
Click to expand...
It might be buggy. There's at least one suspicious thing that needs investigation. It is "experimental" given how new the feature is and how little testing it's had.

For the rest, synth queries several factors of the remote packages in order to determine if it can be used or not. If it can't be used it just ignores it and rebuilds it.
 
kpa said:
Is there actually a problem with the pre-built packages? I think you are all talking past each other unless I have missed something.
Click to expand...

Yeah I think you missed something. Based on earlier log files we thought there was a problem, and a 2nd person had informed of similar complaint if I understood Marino right. I was posting in response to him. Now this morning I am seeing behaviour that suggest to me that fetching of pre-built packages is working. Would you rather I not update him and leave him wasting his time? I'd rather be courteous and let him know what I have found. And since this is brand new software and this feature even newer with very little testing, I thought I would contribute. As simple as that. I would say unless Marino ask me of something, or I see something else, or whatever, my commentary in this thread will come to a stop very soon.
 
marino@ said:
i don't think it's techincally possible that fetching came come after ncurses screen.
Click to expand...

I am sure it did. NetSpades and libslang2 were built in the ncurses screen first, and then I got the output shown above where the other pre-built packages were downloaded and installed. I'll keep watching for that behaviour if you want me to.

Really liking this synth. Thank you so much Marino. :beer:
 
marino@ said:
It might be buggy. There's at least one suspicious thing that needs investigation. It is "experimental" given how new the feature is and how little testing it's had.
Click to expand...

It might be buggy but it sure does seem to be working Marino. I did this just for fun. Synth didn't even download the package. It just reused what was already in 'store'. Sweet.

Code: 
/usr/ports/x11/xeyes # make deinstall clean
/usr/ports/x11/xeyes # make rmconfig

# synth install x11/xeyes
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Scanning entire ports tree.
Scanning existing packages.   
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01   
Fetching packagesite.txz: 100%  150 KiB 153.4kB/s  00:01   
Processing entries: 100%
Synth repository update completed. 658 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   xeyes: 1.1.1 [Synth]

The process will require 22 KiB more space.
[1/1] Installing xeyes-1.1.1...
[1/1] Extracting xeyes-1.1.1: 100%
 
PacketMan said:
I am sure it did. NetSpades and libslang2 were built in the ncurses screen first, and then I got the output shown above where the other pre-built packages were downloaded and installed. I'll keep watching for that behaviour if you want me to.
Click to expand...

Here's what happened:
1) it fetched it before the screen
2) you missed it, it might have been fast
3) the ncurses screen came up
4) when it was done, it restored the terminal to what it was before it came up
5) you saw the old text (pre-build) and though it was new because it was the first itme you saw it.
 
I for one, would take the moment to thank you for some great software!
This thread is turning into rope, sort of thing. Hopefully it will turn into some chapter of the handbook.

Good work!
 
You must log in or register to reply here.
Back
Top