Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

[talsamon]

You have add it to the devel/Makefile to silence this message.
(Better make a own category for your ports. The category (here devel)-Makefile could be overweitten next update. You have then write your category in /usr/ports/Makefile. This will not overwritten.).

 

[marino]

talsamon is correct; that's the problem (on FreeBSD, each category has a makefile and for the port to be considered eligible, it has to be listed in that makefile. On DragonFly, there is no category makefile, by definition all ports are considered eligible)

 

[talsamon]

Another question: Is in synth something similar to portmster -o neworigin oldorigin.

 

[marino]

No, because it's not applicable
Look for examples of that command in /usr/ports/UPDATING. Now look at the pkg(8) equivalent. Most of the time a regular upgrade fixes it. In any case, it would be the pkg(8) command that would be run.

portmaster installs ports.
synth does not install ports. It tells pkg(8) to install packages from the local repository it builds (and then only on the localhost). So you can see it's not applicable.

 

[kpa]

Both do install ports in a way because built port ends up as an installed package either way. The real difference is that portmaster builds and installs one port at a time taking no notice of the previously built ones, this effectively disables the dependency solver in pkg(8). Synth on the other hand uses a set of built ports/packages and uses pkg-upgrade(8) to perform the upgrade and that allows pkg to use its dependency solver to figure out if the requested upgrade is possible and "walk" the dependency tree to figure out all packages that need to be replaced to fulfill the dependency conditions.

 

[marino]

i don't really see it that way. If synth had no "install" command or "upgrade-system" command and simply stopped at building the repository, then the user would have to run something like pkg upgrade -r Synth manually and then it's clear Synth is not involved with installing ports. The install/upgrade commands are just for convenience; they just invoke pkg.

 

[fernandel]

i'm not sure why you quoted it and underlined it. Synth did exactly what the man page said it would do. I am very confused about what you are trying to tell me.

I am sorry. It was mistake. I sent quoted text to someone and it happened

 

[talsamon]

Would anybody stop this incredible nonsense. Nobody can tell me that the update of gettext causes 608 packages to rebuild and that is technically correct (including e.g. compilers, never need gettext).

If this is true, why could anybody live only one day with portmaster, and have no unresponsive system?

To rebuild all is the most minimal and simplest idea. This can never be the truth.

 

[protocelt]

talsamon, it is technically correct and I really just don't think you understand how a package builder works. Poudriere works the same way in that respect. It's already been explained how this works earlier in this monster thread.

If you don't like the way Synth works, you're certainly free to use Portmaster or any other tool you like if that works better for you, but continuing to make posts complaining about Synth rebuilding too many packages all the time isn't adding anything constructive to the discussion here.

 

[talsamon]

With the compiler I was wrong (depend via indexinfo from gettext-runtime), but

pkg info -r gettext-runtime | wc -l
  231

that's the truth.
I stop the (senseless) experiment with synth an leave the thread.

 

[kpa]

I think this illustrates what the whole issue is about. The mail/mutt port requires the following ports (at runtime and also at build time):

% pkg info -d mutt         
mutt-1.5.24_4:
        openssl-1.0.2_8
        urlview-0.9.20131021
        mime-support-3.58
        db5-5.3.28_3
        libiconv-1.14_9
        libidn-1.31
        gettext-runtime-0.19.6

Now, devel/gettext-runtime in turn requires:

% pkg info -d gettext-runtime-0.19.6
gettext-runtime-0.19.6:
        indexinfo-0.2.4

However, print/indexinfo is not directly required by mail/mutt but indirectly.

% pkg info -r indexinfo-0.2.4     
indexinfo-0.2.4:
        gettext-tools-0.19.6
        libffi-3.2.1
        libidn-1.31
        gettext-runtime-0.19.6

Even so when print/indexinfo has an update you must rebuild mail/mutt, absolutely and no way to avoid it. This is because the change in print/indexinfo triggers the deletion of the package for the dependent devel/gettext-runtime. Now if we first build print/indexinfo and then the dependent devel/gettext-runtime we have to ask a question, "did the recompilation of devel/gettext-runtime with the updated print/indexinfo cause any changes to it that could affect mail/mutt?". Unfortunately that's not something you can test by automated programmed means. You could manually inspect everything in devel/gettext-runtime and decide it didn't change but requires human ingenuity and that's something a computer program does not have. So the only way to answer the question is to play it safe and delete the package for mail/mutt and rebuild the port.

 

[marcpearsonti]

What am I suppose to do with a situation like this one

Scanning entire ports tree.
progress: 33.12%             
culprit: games/libretro-cores
Scan aborted because dependency is malformed.
FETCH: 0.20151110 (games/libretro-cores)

Synth can't build my repository anymore. How I can ignore this ports ?

 

[talsamon]

Write in port Makefile

BROKEN=<TAB><TAB>some_text

, so synth ignores it.

(marked this thread as unwatch, but still got a mail...).

 

[free-and-bsd]

1. That's not the definition of downtime.
...
7. 300 ports is NOTHING. My home machine can hit 1000 ports an hour, our blade machine can hit 2800 ports per hour. If you use ccache, it will rebuild huge ports in minutes.

Hi, marino@, first of all let me thank you greatly for this port.
Would you please share how you've set up your machine to rebuild packages with such speeds?

Once again, thank you for this port, I like it very much and I also like the idea after having read all these discussions and complaints etc. I HAVE had problems upgrading with portmaster created packages, so I can see now why this happened.

 

[free-and-bsd]

Yes, I definitely like this ports-mgmt/synth. I've never used poudriere, only portmaster. But the latter doesn't make parallel builds, does it? And many other things.

But here I have a question. I usually build editors/libreoffice with -j6 for speed. How could I define the same with synth?

I also want to confirm that the time synth takes to build all my packages seems to be shorter than with portmaster. Neither does it make my system (Intel Celeron G2030 3.00GHz RAM 16G) "unresponsive" with 4 builders x 2 jobs per builder. Except for that unfortunate webkit-gtk package which takes forever to build either way, but still it's only time and not an "unresponsive system".

BTW, which is the best configuration for speed with synth? Is it to have more builders, or have less builders but more jobs per builder?

 

[marino]

But here I have a question. I usually build editors/libreoffice with -j6 for speed. How could I define the same with synth?

The jobs number are global; they affect every port that hasn't specifically disabled multijob builds. The only setting is shown on synth configure

BTW, which is the best configuration for speed with synth? Is it to have more builders, or have less builders but more jobs per builder?

It is usually limited by memory. When memory is the limiting factor, it's better to have less builders with more jobs per builder. If the machine has a ridiculous amount of memory, you could get away with more builders with more jobs per builder.

 

[free-and-bsd]

...
Better make a own category for your ports. ... You have then write your category in /usr/ports/Makefile. This will not overwritten.).

Right, but the port will not be built for the reason that "category XXX is not in list of valid categories". And these are defined in /usr/ports/Mk/bsd.port.mk, so I guess it must be defined there as well?

But I have run into another problem here. I'm trying to build a customized port which builds well with both portmaster and manual make install command. But with synth this problem appears (from synth log):

===>  Configuring for grub2-dev-2.02.b2.d
Importing unicode...
autogen.sh: python: not found
*** Error code 127

Stop.

Looks like /usr/local/bin/python symlink is not installed in the build environment? How can that be fixed?

 

[marino]

Right, but the port will not be built for the reason that "category XXX is not in list of valid categories". And these are defined in /usr/ports/Mk/bsd.port.mk, so I guess it must be defined there as well?

No. Just add "VALID_CATEGORIES=mycategory" to /etc/make.conf (or the synth profile version of make.conf)

But I have run into another problem here. I'm trying to build a customized port which builds well with both portmaster and manual make install command.

Forget "manual". It has to build in poudriere testport or synth test and pass. if it does, it works in manual, and it's required to work in the former.

But with synth this problem appears (from synth log):

===>  Configuring for grub2-dev-2.02.b2.d
Importing unicode...
autogen.sh: python: not found
*** Error code 127

Stop.

Looks like /usr/local/bin/python symlink is not installed in the build environment? How can that be fixed?

It looks like grub2-dev has a problem. It should be reproducible in poudriere using the same exact options.

 

[free-and-bsd]

...
It looks like grub2-dev has a problem. It should be reproducible in poudriere using the same exact options.

Thank you very much for your answers.
Now this one is my custom port based on GIT version of GRUB2 (not submitted officially) and I'm not using poudriere. Having fixed the python problem (replaced python with /usr/local/bin/python2.7 in autogen.sh), I've run into another similar one: /usr/local/bin/autoreconf symlink is not installed in build environment, so it can't find autoreconf in the same way it failed to find python.OK, that symlink belong to package autoconf-wrapper, but /usr/local/bin/python symlink seems to be part of python package.

 

[marino]

well, that's kind of my point. You pretty much *have* to check new ports with either poudriere or synth. If you don't, the person that tests it will and as soon as it fails, they'll kick it back to you. so doing those checks aren't optional (and I recommend it so so you can save at least one kickback).

autoreconf is another setting. it's probably something like "USES+= autoreconf". check /usr/ports/Mk/Uses/autoreconf.mk or grep other ports to see how it's used.

 

[free-and-bsd]

well, that's kind of my point. You pretty much *have* to check new ports with either poudriere or synth. If you don't, the person that tests it will and as soon as it fails, they'll kick it back to you. so doing those checks aren't optional (and I recommend it so so you can save at least one kickback).

autoreconf is another setting. it's probably something like "USES+= autoreconf". check /usr/ports/Mk/Uses/autoreconf.mk or grep other ports to see how it's used.

Right, I've got the message now, thank you. Yes, I just missed it about autoreconf in porter's handbook. It works now.

 

[kpa]

Thank you very much for your answers.
Now this one is my custom port based on GIT version of GRUB2 (not submitted officially) and I'm not using poudriere. Having fixed the python problem (replaced python with /usr/local/bin/python2.7 in autogen.sh), I've run into another similar one: /usr/local/bin/autoreconf symlink is not installed in build environment, so it can't find autoreconf in the same way it failed to find python.OK, that symlink belong to package autoconf-wrapper, but /usr/local/bin/python symlink seems to be part of python package.

I think the port should depend (build time dependency only probably) on lang/python if it wants to use the unqualified python interpreter without a version number. That port will provide the /usr/local/bin/python symlink. This should be cleaner solution that hacking the configure script.

 

[free-and-bsd]

I think the port should depend (build time dependency only probably) on lang/python if it wants to use the unqualified python interpreter without a version number. That port will provide the /usr/local/bin/python symlink. This should be cleaner solution that hacking the configure script.

Yes, I should think so, too.

BUILD_DEPENDS= ${LOCALBASE}/bin/python:${PORTSDIR}/lang/python

works fine.
Before it had only

USES= ... python ...

Well, actually I stole this file from another grub port, that's why. Borrowed it, modified accordingly, but didn't know yet which tool is the right one to work with ports.

So, thank you marino@ again for your amazing tool. My impressions are VERY positive. Rebuild of ALL my packages took considerably shorter time than with portmaster, which was originally used. Yea, system was perfectly workable during the time of the build, no overloads or something. Sure, firefox and libreoffice were a bit slower during the build time, but not so much so as to make working a torture. So, having "ridiculous amount of memory" is not entirely useless after all , and I'm glad it isn't. When building that machine I figured that 16G of RAM would cost much less than quadcore CPU, which would in turn require much RAM to do its thing anyway.

In addition to that, synth has helped me to learn how to create ports properly, and a good deal of information that goes along with that. Great! Thank you all for help.

 

[kpa]

I couldn't find out if this is provided by Mk/Uses/python.mk so the BUILD_DEPENDS method has to be used I guess, the PORTSDIR is now an optional prefix in the path because it is filled in automatically so it becomes:

BUILD_DEPENDS+= ${LOCALBASE}/bin/python:lang/python

 

[free-and-bsd]

I couldn't find out if this is provided by Mk/Uses/python.mk so the BUILD_DEPENDS method has to be used I guess, the PORTSDIR is now an optional prefix in the path because it is filled in automatically so it becomes:

BUILD_DEPENDS+= ${LOCALBASE}/bin/python:lang/python

Right, I just added another line to this effect to other BUILD_DEPENDS mentioned in Makefile. Thank you for your help, it works all right.
Maybe you could advise on debugging a similar problem with this same port?

The script grub-core/genmod.sh uses symbol @TARGET_STRIP@ which causes build to fail for some reason:

if test xpc != xemu; then
@TARGET_STRIP@ --strip-unneeded \
  -K grub_mod_init -K grub_mod_fini \
  -K _grub_mod_init -K _grub_mod_fini \
  -R .note.gnu.gold-version -R .note.GNU-stack \
  -R .note -R .comment -R .ARM.exidx $tmpfile || exit 1
fi

So I had to create a patch to replace it with /usr/local/bin/strip to make it work. But like the previous example, I should prefer a cleaner way of managing it.