Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

No, I don't see what you mean because fetching happens before the ncurses screen comes up.
Every increment is built; none are fetched.
 
marino@ said:
somehow you mouse is emitting escape key code. I think there's enough weird stuff happening to switch this to a capital "Q" which requires a two-key combination and will be hard to do accidently (unless some ansi code happens to trigger it)
Click to expand...

Dunno,.....my mouse wheel works perfectly fine in all other apps and XTerm sessions. How about CTRL-Q since its a control signal we are sending after all.

But at least you got the main guts of the synth program working really well it seems. You have a beautiful mind. :)
 
Thanks!
Maybe control-q would work (or control-shift-q) if it's a single character. I don't want to look for multiple consecutive keypresses; too much trouble.
 
Sorry I am confused too I guess. I am using the [N] option. I thought some sort of counter that increments for every prebuilt package downloaded and used (instead of source code being downloaded and compiled) could be made.

For example.
Built 500
Compile 400
PreBuilt Fetched 100

Anyway I didn't intend to muddy up a good thread so I will stop this topic... bigger problems in life to fix. ;)
 
If it's working correctly:

  1. starts with 875 (for example)
  2. scans existing packages
  3. finds 375 suitable packages, total down to 500
  4. scans remote packages
  5. finds 100 suitable remote packages
  6. downloads the 100, total down to 400
  7. starts building 400 ports from source.
If the totals reflect something else, it's not working as intended.
 
Hmmm, I don't think I am seeing that, my Total counter does not decrement. My Left counter decrements, my Built counter increments, serially one port at a time. (Right now its an average of 18 per hour). In other words, in my opinion, the Built counter is an accumulation of ports compiled from source PLUS ports that used/fetched prebuilt packages.

But remember I am on my first run if that matters.
 
Total does not decrement; it's constant.
Remaining column starts as same as total, and decrements.
Same thing.
 
The build counter is 4 counters. All reflect the outcome of the build.
There is no package fetching during this phase at all. Only building.
 
Let me ask it this way. These are my numbers currently:

Total 568
Left 385
Built 183

How many so far have used prebuilt packages?
 
Zero.
Any fetching would have been before that screen appears, and "TOTAL" reflects what is left to build (as demonstrated in my previous example).
 
marino@ said:
zero.
any fetching would have been before that screen appears, and "TOTAL" reflects what is left to build (as demonstrated in my previous example)
Click to expand...

marino@ said:
you should have seen the fetching. it takes some time. it should have been obvious.
Click to expand...

Okay I will look into this, but I'm gonna say no prefetching is happening. I'll restart using my mouse roll wheel, errr ahhh I mean ESC key. :)
 
Clue?

Code: 
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Failed!
The external repository could not be updated.
No prebuilt packages will be used as a result.
Scanning existing packages.
 
PacketMan: it's the second time it's been reported (and I asked for the output of pkg -vv after "REPOSITORIES {" but the person didn't comply)

My guess is that it's finding it's own repository and trying to update itself and that's not supposed to happen. If it finds the Synth repository, it's supposed to skip it. That's why I need to see what pkg(8) thinks the repositories are.
 
marino@ said:
PacketMan: it's the second time it's been reported (and I asked for the output of pkg -vv after "REPOSITORIES {" but the person didn't comply)
Click to expand...

This what you looking for?

Code: 
REPOSITORIES {
}
VALID_URL_SCHEME [
  "pkg+http",
  "pkg+https",
  "https",
  "http",
  "ftp",
  "file",
  "ssh",
]


Repositories:
  FreeBSD: {
  url  : "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly",
  enabled  : yes,
  priority  : 0,
  mirror_type  : "SRV",
  signature_type  : "FINGERPRINTS",
  fingerprints  : "/usr/share/keys/pkg"
  }
  Synth: {
  url  : "file:///var/synth/live_packages",
  enabled  : yes,
  priority  : 0
  }
$
 
Yes, it is. The code is designed to pick "FreeBSD" repo (and then eject). If "Synth" was encountered, it would just keep going. However, it seems like the external repository check is happening on Synth, not FreeBSD. Strange. Seems that it should be working. (It works for me, but I'd like to see it work on FreeBSD)
 
marino@ said:
I don't think anything is wrong with lang/ghc as a port, because it builds fine for me in about 55 minutes.
Click to expand...

I hit a small problem with lang/ghc which mystified me at first. Synth reported that the options for that port were out of date and that I needed to make -C /usr/ports/lang/ghc config. After running the make config I got the same error and so ran make rmconfig && make config. Still the options seemed to be out of date according to Synth. Looking at the Makefile for lang/ghc I found that it adds "hidden" options if /usr/local/bin/ghc or /usr/local/bin/HsColour exist:
Code: 
.if exists(${LOCALBASE}/bin/ghc)
OPTIONS_DEFINE+=  BOOT
.endif

.if exists(${LOCALBASE}/bin/HsColour)
OPTIONS_DEFINE+=  BOOTH
.endif
allowing GHC to bootstrap using the installed ghc and hscolour. I got around the impasse by removing lang/ghc, running make rmconfig && make config one more time, and then building my packages. Maybe you want to consider this to be a (somewhat general) problem in how Synth handles the environment when analyzing options. Or you've already fixed it?? [Personally I think the lang/ghc port does the wrong thing in checking the live environment that way. I would prefer that it simply check for the availability of a local GHC package, but I may be wrong. It's a bootstrap problem. Always ugly.]

I used synth-0.99_2 to build a new repository, reproducing what I had previously built with ports-mgmt/poudriere using the same general setup (3 jails, 3 jobs, tmpfs for the workdir and data). Synth built my 1800 packages in about 18 hours of wall time with no problems. It had taken more than 24 hours of wall time to build that repository with poudriere(8). It seems that Synth is very efficient in setting up and tearing down builders (chroot(8)s / jail(8)s). I noticed that small packages can be done in 20 seconds by Synth but take a minimum of two minutes with poudriere(8).

Synth is very pleasant to use. Thanks again.
 
garry said:
Looking at the Makefile for ghc I found that it adds "hidden" options if /usr/local/bin/ghc or /usr/local/bin/HsColour exist:
Code: 
.if exists(${LOCALBASE}/bin/ghc)
OPTIONS_DEFINE+=  BOOT
.endif

.if exists(${LOCALBASE}/bin/HsColour)
OPTIONS_DEFINE+=  BOOTH
.endif
allowing ghc to bootstrap using the installed ghc and hscolour. I got around the impasse by removing ghc, running make rmconfig && make config one more time, and then building my packages. Maybe you want to consider this to be a (somewhat general) problem in how synth handles the environment when analyzing options. Or you've already fixed it?? [Personally I think the ghc port does the wrong thing in checking the live environment that way. I would prefer that it simply check for the availability of a local ghc package, but I may be wrong. It's a bootstrap problem. Always ugly.]
Click to expand...

That is very hackish and is never going to work under a package builder that starts from a clean state for each port. The binaries mentioned are never going to exist in the jail/chroot unless they come from an explicitly declared build dependency that gets installed before building of the port starts.
 
kpa is correct. Unfortunately it's not as uncommon as I would like. When Synth highlights issue like that, I usually see if we can get the port fixed.
For something like this, we could probably wrap the conditional statements with a PACKAGE_BUILDING check. It's fixable but we need the maintainer to buy in.
 
Sorry, I don't understand the sense of this tool. Today one update (audio/lame). It compiles 28 packages (2 and a half hours), some of them indirect dependent, and installs only the lame package. For what it compiles the other packages if it does not install it.
If I had 20 packages, I guess there were 120 packages or more to compile, or for once in a week it could be 400 or more. Who had the time to compile this?
 
1) it compiles what is necessary. If it says audio/lame is necessary for any reason, it will build it if has to. It could be a build dependency.
2) It sounds like you have a very weak machine (on top of already being unreliable). In that case you probably should only build what you customized and let the rest fetch from official binaries. If it's too painful to build, then don't build. Nobody is forcing you.
 
No, if you say above you need 55 minutes for lang/ghc my machine need 68 minutes, not much slower.
And the system is running good since last February without bigger problems (only not properly plugged cable, in July destroyed the system).

Maybe ports-mgmt/portmaster don't recompile all dependencies that needed and don't check the portstree right, but this is too much.
Why must www/firefox recompiled? audio/alsa-lib sure, but Firefox? Why must www/libxul be recompiled, multimedia/gstreamer yes, but www/libxul? And why emulators/virtualbox-ose has only one audio option (PULSEAUDIO) it is off. Which option has any dependency to audio or multimedia=

During the compile you can't use the machine (compiler take all resources it could get). And if I limit it, it needs an endless time. If it's one or one and a half hour (or sometimes longer) it is ok, in summary, but nor for only one small package like audio/lame. And why it compiles all the other packages store it in the repo - but don't use (install) it?? But you can't wait e.g. three hours for them, and maybe each day (or as I mentioned above, for a day, if you update once in the week.
Don't misunderstand me, I honor your work - it seems a good tool for testing, but it seems not for updating And I wanted to have a better tool as ports-mgmt/portmaster for long time - but this is nearly "overkill".

kpa I see it with the options in lang/gcc (this was not synth, ok). I did not try to remove ghc. (and I see misc/compat9x has no libutil file, so I think we can remove this LIB_DEPENDS line in Makefile, I see this after the last try). Maybe that is the reason for synth rejected config, if I try to set the option boot to on. Something is wrong with the port. Next try I will remove ghc before I try it with synth.
 
You must log in or register to reply here.
Back
Top