Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

fernandel said:
No, it doesn't work for sysutils/fusefs-exfat on my system.
Click to expand...

On my system without the option :
Code: 
====>> Ignoring sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined
build of sysutils/fusefs-exfat ended at Tue Feb  9 23:47:47 CET 2016
build time: 00:00:01
With the option:
Code: 
===========================================================================
=======================<phase: package  >============================
===>  Building package for fusefs-exfat-1.0.1
===========================================================================
====>> Cleaning up wrkdir
===>  Cleaning for fusefs-exfat-1.0.1
build of sysutils/fusefs-exfat ended at Tue Feb  9 23:48:50 CET 2016
build time: 00:00:04
So I am sure this option should work. IMHO, if the option is written properly then you put it in the wrong file. I do not use ports-mgmt/synth but from what I understood, if you kept the default profile, you should put this option in the /usr/local/etc/synth/LiveSystem-make.conf file.
 
marino@ said:
it doesn't seem to be able to read /usr/ports/ports-mgmt/pkg
Do basic queries like make -C /usr/ports/ports-mgmt/pkg -V PKGNAME work?
Click to expand...

I have just completely deleted and checked out the ports tree so I'm sure it isn't corrupt now. Running the command you said:

Code: 
# make -C /usr/ports/ports-mgmt/pkg -V PKGNAME
pkg-1.6.3

Despite deleting and recreating the ports tree the error with synth is exactly the same where it mentions pkg.
 
i guess you'll have to show the contents of synth configure screen and tell me if there is anything unconventional about any of the directories. I've never seen an error like that before. the bad value errors shouldn't even happen anymore. Something is seriously different about your configuration.
 
marino@ said:
i guess you'll have to show the contents of synth configure screen and tell me if there is anything unconventional about any of the directories. I've never seen an error like that before. the bad value errors shouldn't even happen anymore. Something is seriously different about your configuration.
Click to expand...
I have the same issue. I'm running 10.2 on i386:

Code: 
root@wintermute-pts/0(9:28pm)~/:synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Encountered issue with ports-mgmt/pkg or its dependencies
  => bad input for 'Value: ""
Unexpected pkg(8) scan failure!
Unfortunately, the system upgrade failed.
Just re-extracted the ports tree with portsnap extract, so it shouldn't be corrupted.

Here is my synth configure screen -- nothing unconventional:

Code: 
Synth configuration profile: LiveSystem
===============================================================================
  [A] Ports directory  /usr/ports
  [noparse][B][/noparse] Packages directory  /var/synth/live_packages
  [C] Distfiles directory  /usr/ports/distfiles
  [D] Port options directory  /var/db/ports
  [E] Build logs directory  /var/log/synth
  [F] Build base directory  /usr/obj/synth-live
  [G] System root directory  /
  [H] Compiler cache directory  disabled
  [noparse][I][/noparse] Num. concurrent builders  2
  [J] Max. jobs per builder  2
  [K] Use tmpfs for work area  true
  [L] Use tmpfs for /usr/local  true
  [M] Display using ncurses  true
  [N] Fetch prebuilt packages  false

  [>]  Switch/create profiles
  [RET] Exit

Press key of selection:

And this command works fine:
Code: 
root@wintermute-pts/0(9:35pm)~/:make -C /usr/ports/ports-mgmt/pkg -V PKGNAME
pkg-1.6.3
 
From that output, I know exactly which routine fails but not why. There are only two possible causes:
1) the builder (jail area) didn't get constructed so the internal chroot command fails
2) There's something wrong with the Makefile, at least in that enviroment, that causes bmake to return with an error

I would think 1) would show an error earlier.

this is the command it's trying to run on the chroot:
Code: 
/usr/bin/make -C /usr/ports/ports-mgmt/pkg \
-VPKGVERSION -VPKGFILE:T -VMAKE_JOBS_NUMBER -VIGNORE \
-VFETCH_DEPENDS -VEXTRACT_DEPENDS -VPATCH_DEPENDS \
-VBUILD_DEPENDS -VLIB_DEPENDS -VRUN_DEPENDS \
-VSELECTED_OPTIONS -VDESELECTED_OPTIONS \
-V_INCLUDE_USES_SCONS_MK -VUSE_LINUX

I'd ask if you can see mounts on /usr/obj/synth-live but it would construct and deconstruct so quickly you'd probably miss it.

FWIW, I can't reproduce on FreeBSD 11 i386 VM
 
Last edited by a moderator:
If one of you two want to try this, it could help understand what's going on. You'll need gcc6-aux installed ( pkg ins gcc6-aux).
  1. cd /usr/ports/ports-mgmt/synth
  2. make clean ; make patch
  3. cd <WRKDIR>/synth-4417017/src/
  4. edit portscan.adb
  5. go to line 562 and insert "TIO.Put_Line (content);". it's right before "for k in result_range loop"
  6. cd /usr/ports/ports-mgmt/synth
  7. make deinstall (if you already have synth installed)
  8. make install
  9. run synth status command again and show output
 
Last edited by a moderator:
My configuration is as follows:

Code: 
Synth configuration profile: LiveSystem
===============================================================================
  [ A ] Ports directory  /usr/ports
  [ B ] Packages directory  /var/synth/live_packages
  [ C ] Distfiles directory  /usr/ports/distfiles
  [ D ] Port options directory  /var/db/ports
  [ E ] Build logs directory  /var/log/synth
  [ F ] Build base directory  /usr/obj/synth-live
  [ G ] System root directory  /
  [ H ] Compiler cache directory  /var/db/ccache
  [ I ] Num. concurrent builders  3
  [ J ] Max. jobs per builder  3
  [ K ] Use tmpfs for work area  true
  [ L ] Use tmpfs for /usr/local  true
  [ M ] Display using ncurses  true
  [ N ] Fetch prebuilt packages  true

I have some things in /etc/make.conf that could maybe affect it?

Code: 
KERNCONF=TAO
SVN_UPDATE=yes
SVN=/usr/local/bin/svn
BATCH_DELETE_OLD_FILES=yes
WRKDIRPREFIX=/usr/obj
FORCE_PKG_REGISTER=yes
DISABLE_VULNERABILITIES=yes
OPTIONS_UNSET+=X11
WITH_OPENSSL_PORT=yes

DEFAULT_VERSIONS=apache=2.4 gcc=5 perl5=5.22 pgsql=9.5 php=5.6 python=2.7 python2=2.7 python3=3.5
WITH_BDB6_PERMITTED=yes

FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=4

.if defined(WITH_GCC)
CC=gcc52
CXX=g++52
CPP=cpp52
.endif

WITH_CCACHE_BUILD=yes

.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*))
.if !defined(NOCCACHE) && exists(/usr/local/libexec/ccache/world/cc)
CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1}
CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1}
.endif
.endif

I'm using 10.3-PRERELEASE amd64 dated from Jan 30th taken from the stable/10 branch. I'll try what you suggested in the post above, but it will take me a few hours to reinstall gcc6-aux again. Unfortunately it bootstraps itself so isn't cached with ccache. It's only an Intel Atom 1.8GHz so isn't massively powerful and for some reason if I try and install it using pkg it wants to install postgresql 9.4 despite the fact I have 9.5 already so that would break and I have no idea what gcc6-aux is linked to postgresql in the first place.
 
/etc/make.conf is not read at all.

gcc6-aux is not connected to pgsql in any way. Check again.
 
Last edited by a moderator:
Code: 
]# pkg install gcc6-aux
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Updating database digests format: 100%
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  gcc6-aux: 20160124 [FreeBSD]
  postgresql94-client: 9.4.5_1 [FreeBSD]

The process will require 261 MiB more space.
53 MiB to be downloaded.

Proceed with this action? [y/N]:

Go figure? Also in steps 1 and 6 above, did you mean /synth rather than /pkg ?
 
I can't explain why pkg wants to upgrade pgsql but I swear to you it's not related to gcc6-aux.

yes, 1 and 6 should be synth. I will edit.
 
Last edited by a moderator:
OK. This time I'll leave gcc6-aux installed! However I can't get it to compile. Pasting exactly what you said with the " marks around it gives this:

Code: 
ada -c -g -O2 -gnat12 -gnatyaAbBcdefhiklnOprsSxt -I- -gnatA /usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb
portscan.adb:562:06: statement expected
gnatmake: "/usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb" compilation error
*** Error code 4
If I remove the quotes then it gives this:

Code: 
ada -c -g -O2 -gnat12 -gnatyaAbBcdefhiklnOprsSxt -I- -gnatA /usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb
portscan.adb:562:10: no candidate interpretations match the actuals:
portscan.adb:562:10: missing argument for parameter "Item" in call to "PUT_LINE" declared at a-textio.ads:259
portscan.adb:562:21: expected type "Standard.String"
portscan.adb:562:21: found private type "Ada.Strings.Unbounded.Unbounded_String"
portscan.adb:562:21:  ==> in call to "Put_Line" at a-textio.ads:263
gnatmake: "/usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb" compilation error
*** Error code 4
You'll have to excuse me in that I have never seen ada before so don't know if the quotes were you quoting what needs to be added or part of the code!
 
it's my fault, I didn't test that line.
It should be
Code: 
TIO.Put_Line (JT.USS (content));
I think. (I didn't test this either. :) )
 
Getting somewhere now.

Code: 
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
1.6.3
pkg-1.6.3.txz
(huge amount of whitespace here)
Encountered issue with ports-mgmt/pkg or its dependencies
  => bad input for 'Value: ""
Unexpected pkg(8) scan failure!
Unfortunately, the system upgrade failed.
 
Strange. There's no problem where you modified the file. That's the expected output. Let me think about this a minute.
 
Code: 
CPU: Intel(R) Atom(TM) CPU D525  @ 1.80GHz (1795.74-MHz K8-class CPU)
avail memory = 4103008256 (3912 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
 
xtaz you should have had a third line of output with "4" indicating 4 cpus. It's not there. My guess is that it's a parsing error because it's not expecting a blank value.

e.g.
make -C /usr/ports/ports-mgmt/pkg -VMAKE_JOBS_NUMBER
should return "4", not nothing.
 
Last edited by a moderator:
marino@ said:
signal-handling: Use the "Escape" key. I had to abandon SIGINT capture because it propagates to the builders and breaks when conftests segfaults. You should almost never use control-C. However, if you do, run any major synth command, like synth configure and it will clean up the dirty mounts automatically.
Click to expand...
There may be a way around this. When you fork a new process, this is the child of a parent process. Signals may propagate to that one. Now, if you do not fork a builder directly but a trampoline process, which then in turn forks a builder, passes that builders PID to its parent (the main process) and then exits, your builder has no living parent process but may still be controlled by the main process via the PID. Could this solve the Ctrl-C problem?
 
marino@ said:
xtaz@ you should have had a third line of output with "4" indicating 4 cpus. It's not there. My guess is that it's a parsing error because it's not expecting a blank value.

e.g.
Code: 
make -C /usr/ports/ports-mgmt/pkg -VMAKE_JOBS_NUMBER
should return "4", not nothing.
Click to expand...

OK, so the problem *was* /etc/make.conf then. If you look at where I pasted that file in my previous post you can see I set these two variables:

Code: 
FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=4

Commenting those out solves the problem and synth is working now. I guess that is overriding something important then?

Although interestingly it shows 3, not 4:

Code: 
Stand by, comparing installed packages against the ports tree.
1.6.3
pkg-1.6.3.txz
3
Stand by, building pkg(8) first ...
 
Crivens, unfortunately no. I spent 2 solid days on it. There's no workaround. I could fork 3 generations, the parent always retains control of the signal.
 
Last edited by a moderator:
xtaz if you're correct and /etc/make.conf is considered, that's a problem. It might be a case where I intentionally scan the pkg(8) package (only) using the host ports tree and /etc/make.conf gets sucked in (and I forgot about consequences). I'll have to think about this because that's two reports that stem from that decision.
 
Last edited by a moderator:
xtaz by the way, you can just delete both lines because they don't do anything anyway.
 
Last edited by a moderator:
marino@ said:
Crivens, unfortunately no. I spent 2 solid days on it. There's no workaround. I could fork 3 generations, the parent always retains control of the signal.
Click to expand...
Maybe setsid(2) is the key then? Place the builders in their own sessions?
 
Last edited by a moderator:
You must log in or register to reply here.
Back
Top