Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

look at the build log that been saved to disk.
ports do fail through no fault of a builder. It's pretty common actually.
 
marino@ said:
look at the build log that been saved to disk.
ports do fail through no fault of a builder. It's pretty common actually.
Click to expand...
Yes, I know this happens. I've checked the log, it says:
Code: 
...
[build LNK] CppunitTest/libtest_svtools_graphic.so
[build CUT] sw_uwriter
[build LNK] CppunitTest/libtest_sw_htmlexport.so
Abort trap (core dumped)
OK (1)
Fatal error 'mutex is on list' at line 409 in file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 0)
The build output of libreoffice is not very enlightening, though ...
 
Dear marino@, I thank you once again for all your help and assistance in using this great app of yours and related things.

There's one question related to the idea of repositories I can't decide yet. How can one use a synth created repo for a machine with a different video configuration? Because on my synth machine I have nvidia card, so the created repo is built for this particular hardware. But then I have a laptop with Intel HD video, where most of those packages will work except those nvidia related.

How is this usually handled? Should I run synth on my laptop and make it use the existing repository, so it would only have to rebuild the video-related software? My guess is, if I copy the .ccache dir to my laptop, it might speed up the build considerably ;) ? Finally, I can insert the laptop's HDD into my synth machine, copy the .ccache dir over to it, boot from it and run synth locally in a normal way (yes, and also copy the repository there, too).

Or is it possible to build such packages on my synth machine, which just has good enough specs for repository building? Something like a separate repo, I should expect?

Thanks again for your king help :)
 
free-and-bsd said:
There's one question related to the idea of repositories I can't decide yet. How can one use a synth created repo for a machine with a different video configuration? Because on my synth machine I have nvidia card, so the created repo is built for this particular hardware. But then I have a laptop with Intel HD video, where most of those packages will work except those nvidia related.
Click to expand...

You making a mental relationship between what is installed on the synth system and what you want in the repository. This is your issue.

For people with exactly one computer, and they use synth to keep that computer up to date, the above relationship is valid.

However, for people with multiple computers that use synth on one computer to keep all of them up to date, the relationship is invalid. In this case, you would maintain a combined list of packages to populate the repository.

e.g.
on machine A: pkg query -e '%a=0' '%o'> prime.list.A
on machine B: pkg query -e '%a=0' '%o' > prime.list.B
on machine C: pkg query -e '%a=0' '%o' > prime.list.C

on synth machine after transferring the above files: cat prime.list.[ABC] | sort -u > repo.contents
synth just-build repo.contents
synth rebuild-repository

You'd need to manually set up pkg repo conf files for Machine A, B, and C that point to the newly created repository.

The above assumes machine A, B, and C are all the same architecture, thus only one repository is needed

edited: pkg prime-list doesn't show origins, so the raw query is needed
 
marino@ said:
synth just-build repo.contents
synth rebuild-repository
Click to expand...

I'm not sure that works that well. As you know I am running synth on multiple machines. However one machine is a designated synth builder. There are no other ports installed. I run synth status on the target machine, and take its output and feed it to the designated builder. After the building is done, I copy the packages over to the target machine and run synth upgrade-system. What happens is a whole bunch of dependency checks fail, so synth ends up rebuilding them anyway where they get installed afterwards. (My guess is in a key port or two I have changed an option.) Back to free-and-bsd's post, where he is running different video cards, my guess is he may have issues there, and will have to run synth locally. I personally am quite okay with this. Yesterday my designated builder rebuilt 300+ ports for my GNOME3 machine in a few hours. And now on my old cpu GNOME3 machine synth only has to rebuild 75 ports to fix up the dependency boo boos.

I've been meaning to experiment with synth profiles for just this sort of scenario but haven't had the time yet.
 
PacketMan said:
I'm not sure that works that well.
Click to expand...

Unless free-and-bsd wants the same port with different options on different machines, it works wonderfully. This is the exact concept of operations: One repository, many clients. The upgrade-system command is only a convenience command for the common case where the sysadmin wants to create a repository based one what's installed on the synth machine. It, and it's companion prepare-system command are not necessary and can be omitted. They are just there to create ports lists quickly (pkg(8) generates the list based on what is installed). You can just maintain the same list manually.
 
marino@ said:
... upgrade-system command is only a convenience command for the common case where the sysadmin wants to create a repository based one what's installed on the synth machine. It, and it's companion prepare-system command are not necessary and can be omitted. They are just there to create ports lists quickly (pkg(8) generates the list based on what is installed). You can just maintain the same list manually.
Click to expand...

Just to elaborate:

On my machine root has a shell script called printworld:
Code: 
#!/bin/sh
pkg query -e '%a = 0' %o | sort -d
and the routine for updating the local machine via synth is
Code: 
./printworld >world
synth just-build world
synth rebuild-repository
pkg upgrade
 
That one assumes there's only one repo, "Synth". Slightly better, the last command would be
pkg upgrade -r Synth , just to be explicit and avoid the possibility of upgrading from FreeBSD repository.
 
garry: Just thought I'd chime in and mention that you can create an alias for that in /usr/local/etc/pkg.conf. No need for a shell script. ;) There's already a similar alias called "leaf" available, but it shows [package]-[version], not origin/port.
 
PacketMan said:
...Back to free-and-bsd's post, where he is running different video cards, my guess is he may have issues there, and will have to run synth locally...
Click to expand...
Logically, big repo building machines don't ALL have to have all sorts of video-cards etc to maintain repo(s) suited to different video-drivers. The difference being, for example, in some packages being build differently depending on whether you have a nVidia provided GL or OpenGL.
You know, if synth (or poudriere for that matter) rebuilds ~200 packages over a slight possibility of changes in relation to actual changes in less than 30 of them, as was discussed earlier in this thread, then why should I be afraid that it would overlook such evident change of dependencies in video-driver and GL related stuff?

So, thanks for your concern, but it worked perfectly well for me. I built the repo on one machine, then uploaded it to my laptop and pkg(ng) upgraded. And pkgng knows perfectly well when dependency changes happen and doesn't miss the opportunity to complain whenever there is any slightest problem of this sort. I've had that multiple times when upgrading with portmaster created packages.
 
Hi marino@,

I have a non-working ports-mgmt/synth, version 1.33_1, on FreeBSD 10.3-RELEASE #0.

I was wondering, since after I created the profile, the /usr/local/etc/synth/[profile-name]-make.conf file didn't exist. But it doesn't matter if the file is empty or configured, the debugging output stays the same. Also the /usr/local/etc/synth/LiveSystem-make.conf file is not there.

Code: 
# synth status
Builder mounts detected; attempting to remove them automatically ...
Dismounting successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Illegal instruction (core dumped)

Code: 
# gdb synth
...
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
(gdb) core synth.core
Core was generated by `synth'.
Program terminated with signal 4, Illegal instruction.
Reading symbols from /usr/local/lib/libncurses.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libncurses.so.6
Reading symbols from /usr/local/lib/libtinfo.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libtinfo.so.6
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000000462369 in ?? ()
[New Thread 801806400 (LWP 101383/<unknown>)]
(gdb) bt
#0  0x0000000000462369 in ?? ()
#1  0x0000000000000060 in ?? ()
#2  0x000000000047a6bc in ?? ()
#3  0x3470bb414db74db7 in ?? ()
#4  0xdffff010ba490048 in ?? ()
#5  0x90e3ff4900007fff in ?? ()
#6  0x05f0bb41004623b9 in ?? ()
#7  0xdffff010ba490047 in ?? ()
#8  0x90e3ff4900007fff in ?? ()
#9  0x00007fffdffff130 in ?? ()
#10 0x00007fffdffff060 in ?? ()
#11 0x0000000000004db7 in ?? ()
#12 0x00000000004834e7 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb)

Since I am kind of a Luser I don't really know what's going on,
nor how to solve this. But I am good in following instructions
and eager to learn.

EDIT:

Using ports-mgmt/pkg works, but I get following info:

Code: 
pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
repository Synth has no meta file, using default settings
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Unable to update repository Synth
 
how did you obtain synth?
A) you built it from ports
B) you got it from freebsd official packages (if so, where packages built on 10.2 or 10.3?)

I am guessing you didn't built it and there's some kind of incompatibility between your new 10.3 system and whatever built the synth package.
 
marino@ said:
how did you obtain synth?
A) you built it from ports
B) you got it from freebsd official packages (if so, where packages built on 10.2 or 10.3?)

I am guessing you didn't built it and there's some kind of incompatibility between your new 10.3 system and whatever built the synth package.
Click to expand...

It's a fresh 10.3, no upgrade. I wanted to get the laptop being ready quickly, so I used pkg and the official packages to install everything.

Then I decided to try synth upgrade-system. But synth was not available from the freebsd official packages. So I did portsnap fetch extract and build it with make. All dependencies for synth, all but gcc6, were installed using official packages. gcc and synth were build using make.

Upgrade-system first worked, but then, after having build some ports, I gave it a graceful shutdown, since I wanted to let it finish over night. When trying to continue upgrade-system later on, above error message came up.

:)
 
it's weird that synth isn't available in packages. According to portsmon, it's been built (on 10.1 machine).

I'm shooting in dark here, but you might try:
  1. synth just-build ports-mgmt/synth
  2. pkg delete synth
  3. pkg add /path/to/synth/packages/All/synth-133_1.txz
And see if your synth-built synth works better.
 
marino@ said:
it's weird that synth isn't available in packages. According to portsmon, it's been built (on 10.1 machine).
Click to expand...
Now it's available to me also. I don't know what went wrong, but I am embarrassed. My mistake, maybe.

marino@ said:
I'm shooting in dark here, but you might try:
  1. synth just-build ports-mgmt/synth
  2. pkg delete synth
  3. pkg add /path/to/synth/packages/All/synth-133_1.txz
And see if your synth-built synth works better.
Click to expand...
Sorry, no good news. Same same, none different.
synth status and synth upgrade-system are still aborting with message "Illegal instruction".

Worth mentioning is, that synth just-building of synth did hang up(?) after finishing gcc. I recognized it, because the load went down. Graceful shutdown (Ctrl-Q) was not possible, so I Ctrl-C'd and started again: now it built synth and put the txz into place. From the logs in /var/log/synth I learned, that the built of gcc6-aux actually was fine. It just didn't went any further.
 
mirco said:
...
I was wondering, since after I created the profile, the /usr/local/etc/synth/[profile-name]-make.conf file didn't exist. But it doesn't matter if the file is empty or configured, the debugging output stays the same. Also the /usr/local/etc/synth/LiveSystem-make.conf file is not there....
Click to expand...
Neither was it there for me until I created it. Running synth configure doesn't seem to create that file.

Hey, mirco, you can at least do this as root:
Code: 
cd /usr/ports/ports-mgmt/synth
make reinstall clean (that assuming you have synth installed, else you `make install clean`)

Then try the resulting synth. My idea is, if something's wrong with your system, it may show up during the port building process. If nothing comes up and it builds fine, then see how it works.
I have had this on my system that packages didn't work, but ports built from sources did.
Anyway, don't worry: whatever is wrong it will be found out.

Oh, yes: debugging is no use unless the app is built with debugging symbols. Which by default they are not.
 
BTW, what kind of hardware do you have? Not an old machine one is about to trash but then decides to install FreeBSD instead? Just asking, you know ;).
 
I have an upgraded 10.3 and Synth works just fine on it.

Code: 
% uname -a
FreeBSD freebsd10 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297391: Tue Mar 29 19:52:26 EEST 2016     root@freebsd10:/usr/obj/usr/src/sys/GENERIC  amd64

Code: 
% pkg -v
1.7.1

Code: 
% synth version 

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 1.33
====================================================================
               Copyright (C) 2015-2016 John R. Marino
 
You must log in or register to reply here.
Back
Top