Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

So that's it? We need to switch to poudriere if we want to have the performance tweaks?

Is no one even going to attempt to see what is needed to be done with Synth (or make.conf file) in order to bring it up to speed?
 
Mayhem30 said:
So that's it? We need to switch to poudriere if we want to have the performance tweaks?

Is no one even going to attempt to see what is needed to be done with Synth (or make.conf file) in order to bring it up to speed?
Click to expand...

I've been reluctant to point this out, but you've already said that you update ports via cron on a daily basis and update the flavor index at that time, but then invalidate that work by doing it manually because you *must* have the very latest ports tree and not one 6 hours old. If you "settled" for a resolution of a daily tree update, you personally wouldn't have the issue.

There is ZERO quality-assurance on the ports tree. Any given commit can break the index or cause many upstream ports not to build (because the port maintainer didn't bother to check for upstream breakage and just upgraded without regard, for example). So in reality, the "up to the last second" update may very well be worse than one a few hours old.

The best system would be a ports tree updated every 3 days after an integrity test (with possible exceptions for major security fixes) but that would never happen unfortunately.
 
marino said:
I've been reluctant to point this out, but you've already said that you update ports via cron on a daily basis and update the flavor index at that time, but then invalidate that work by doing it manually because you *must* have the very latest ports tree and not one 6 hours old.
Click to expand...

I have good reason to do this though. Almost *always* after a port has been updated, a patch is added an hour or 2 later tagged "fixed issue with ..." or something important. These updates are being done on a production server, so I need to make sure the port I'm updating is up to date.

The issue I'm having is finding time to do the updates, and when I do - I have to wait 20 minutes for the flavor index to be regenerated before I can get started.
 
Mayhem30 said:
Almost *always* after a port has been updated, a patch is added an hour or 2 later tagged "fixed issue with ..." or something important
Click to expand...

Yes, but unless you’re a fortune teller, you have no way of knowing if there is an upcoming “fixed issue” patch, or a new version, etc. Put another way, what about the patch that will come an hour or two from “now”? Are you going to go through it all again then? I’d guess not - with the potential exception for a security fix, depending on your environment and exposure.

And if it’s a production system, you should consider a build and test on another system (or jail/VM) before applying the latest and “greatest” anyway.
 
Eric A. Borisch said:
Yes, but unless you’re a fortune teller, you have no way of knowing if there is an upcoming “fixed issue” patch, or a new version, etc. Put another way, what about the patch that will come an hour or two from “now”? Are you going to go through it all again then? I’d guess not - with the potential exception for a security fix, depending on your environment and exposure.

And if it’s a production system, you should consider a build and test on another system (or jail/VM) before applying the latest and “greatest” anyway.
Click to expand...

I was going to say pretty much exactly this, but you did it better.
 
This has gone completely off topic. All I wanted to know was :

Is anything going to be done with Synth (or make.conf file) in order to bring it up to speed with poudriere?
 
Nothing will be done to synth. The "workaround" is strictly a [profile]-make.conf adjustment. I personally don't have time to work out the adjustment and apparently nobody else does either. If you want to take the initiative, ask the poudriere maintainer nicely what poudriere is doing to activate that compiler cache code.
 
I have poudriere build logs and can see stuff like this in them. Is this what you mean?

Code: 
_CCVERSION_921dbbb2=FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin

_ALTCCVERSION_921dbbb2=none

_CXXINTERNAL_acaad9ca=FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin "/usr/bin/ld" "--eh-frame-hdr" "-dynamic-linker" "/libexec/ld-elf.so.1" "--hash-style=both" "--enable-new-dtags" "-o" "a.out" "/usr/lib/crt1.o" "/usr/lib/crti.o" "/usr/lib/crtbegin.o" "-L/usr/lib" "/dev/null" "-lc++" "-lm" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/crtend.o" "/usr/lib/crtn.o"

_OBJC_CCVERSION_921dbbb2=FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin

_OBJC_ALTCCVERSION_921dbbb2=none
 
marino said:
Nothing will be done to synth. The "workaround" is strictly a [profile]-make.conf adjustment.
Click to expand...

The Poudriere maintainer had this to say about it :

While it's true that it is something that needs to be piped to make.conf, it should not be done by the user as the values may change in every execution. It needs to be done by the tool. You could wrap synth with a script that sets up a make.conf like this though until the tool learns about it.

https://github.com/freebsd/poudriere/blob/master/src/share/poudriere/common.sh#L2251-L2275
 
The hash might change but the values themselves should not.
I'd bet taking the output xtaz provided (as an example) and defining the hash as "921dbbb2" (in that example) in the make.conf would work just fine.

The issue you'd see with that approach is when freebsd system is updated. Then the old values continue (as they are now hardcoded).

Maybe the linked tool will provide output for the profile make.conf, dunno. I never used it.
 
marino said:
Maybe the linked tool will provide output for the profile make.conf, dunno. I never used it.
Click to expand...
Yes, synth will need to be programmed to run /Mk/Scripts/ports_env.sh and export it in to the make.conf file.

Here is the output that code produces on my machine :
Code: 
_CCVERSION_921dbbb2=FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin
_ALTCCVERSION_921dbbb2=none
_CXXINTERNAL_acaad9ca=FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin "/usr/bin/ld" "--eh-frame-hdr" "-dynamic-linker" "/libexec/ld-elf.so.1" "--hash-style=both" "--enable-new-dtags" "-o" "a.out" "/usr/lib/crt1.o" "/usr/lib/crti.o" "/usr/lib/crtbegin.o" "-L/usr/lib" "/dev/null" "-lc++" "-lm" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/crtend.o" "/usr/lib/crtn.o"
CC_OUTPUT_921dbbb2_58173849=yes
CC_OUTPUT_921dbbb2_9bdba57c=yes
CC_OUTPUT_921dbbb2_6a4fe7f5=yes
CC_OUTPUT_921dbbb2_6bcac02b=yes
CC_OUTPUT_921dbbb2_67d20829=yes
CC_OUTPUT_921dbbb2_bfa62e83=yes
CC_OUTPUT_921dbbb2_f0b4d593=yes
CC_OUTPUT_921dbbb2_308abb44=yes
CC_OUTPUT_921dbbb2_f00456e5=yes
CC_OUTPUT_921dbbb2_65ad290d=yes
CC_OUTPUT_921dbbb2_b2657cc3=yes
CC_OUTPUT_921dbbb2_380987f7=yes
_OBJC_CCVERSION_921dbbb2=FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin
_OBJC_ALTCCVERSION_921dbbb2=none
ARCH=amd64
OPSYS=FreeBSD
_OSRELEASE=11.1-RELEASE-p8
OSREL=11.1
OSVERSION=1101001
PYTHONBASE=/usr/local
HAVE_COMPAT_IA32_KERN=YES
CONFIGURE_MAX_CMD_LEN=262144
HAVE_PORTS_ENV=1

Here is the bash script I used to get the information :
Code: 
#!/usr/local/bin/bash

export SCRIPTSDIR="/usr/ports/Mk/Scripts"
export PORTSDIR="/usr/ports"
export MAKE="/usr/bin/make"
/bin/sh ${PORTSDIR}/Mk/Scripts/ports_env.sh |
grep '^export [^;&]*' |
sed -e 's,^export ,,' -e 's,=",=,' -e 's,"$,,'
 
I rebooted the machine, threw the output in to the LiveSystem-make.conf file. It took 4 mins and 35 seconds to rebuild the flavor index.

This is from my machine running with a single non-ssd 1TB drive :
Code: 
$ date && sudo synth status && date
Thu Mar 15 09:00:14 PDT 2018
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
Thu Mar 15 09:04:49 PDT 2018


This is from my machine running dual SSD's in raid 0 :
Code: 
$ date && sudo synth status && date
Thu Mar 15 09:12:02 PDT 2018
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
Thu Mar 15 09:15:44 PDT 2018

It only took 3 mins and 42 seconds. This is a huge performance boost.
 
man, talk about obscure. I'm pretty knowable in bmake and I didn't even recognize that as a builtin make function.

P.S. I bet there's another 2 or 3 simple make.conf tricks that would cut off another 50+% of the scan time for freebsd ports still left to find.
 
maybe with some more research, maybe these tricks could be incorporated into synth, but I'd like to see the requirements laid out about what is needed to happen.
 
A) I know for a fact there are more of these. I fixed them for dports, remember?
B) I don't have the exact requirements, the step-by-step "do A, B, C". To get that requires a lot of study on my part and right now I'm personally swamped, so I'm look for hand-holding here. If people wait for me to take the initiative when I'm not personally affected, I don't know how long they'll wait.
 
Hello Marino,
I am having an issue with synth, trying to install python27 and python36. I can build both ports directly, but when trying a synth just-build lang/python36, it is failing with the following error:
Code: 
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
===>  Script "configure" failed unexpectedly.
Please report the problem to python@FreeBSD.org [maintainer] and attach the
"/construction/xports/lang/python36/work/Python-3.6.5/config.log" including
the output of the failure of your make command. Also, it might be a good idea
to provide an overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /xports/lang/python36

I am not sure why in the jail synth is having issue building the port.

Does it ring a bell for you?

The issue came recently after I deleted all packages and tried to reinstall from scratch all packages on my system.
 
you have to use ENTERAFTER=stage environment variable and go find the config.log when you get the prompt back. view the config.log file with vi and look at the error that says the compiler doesn't work. The log will contain the answer.

obviously synth builds python for everyone else. There would be a lot of anger if everyone had this issue.
 
Thanks Marino. As I said, python used to compile fine with synth on my system. It is only recently I have some troubles. I could spot three errors in the config.log file. I am not very familiar with compilation, but here it is:

Code: 
configure:3900: cc -V >&5
cc: error: argument to '-V' is missing (expected 1 value)
cc: error: no input files
configure:3911: $? = 1
configure:3900: cc -qversion >&5
cc: error: unknown argument: '-qversion'
cc: error: no input files

and

Code: 
configure:4083: $? = 0
configure:4090: ./conftest
Shared object "libintl.so.8" not found, required by "conftest"
configure:4094: $? = 1
configure:4101: error: in `/construction/xports/lang/python36/work/Python-3.6.5':
configure:4105: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

I checked that /xports/devel/gettext-runtime/work/gettext-0.19.8.1/gettext-runtime/intl/.libs/libintl.so.8 exists.
So it looks libraries are not configured correctly in the jail? What could I try?
 
You must log in or register to reply here.
Back
Top