Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

hrenznaet said:
Oh, so I'm doomed to either
- not upgrade the ports in my system at all;
or
- upgrade selected ports hoping they won't cause a break in firefox again via [firefox dependencies, firefox dependencies dependencies, firefox dependencies dependencies dependencies, ...] and so on.

Neither of options is optimal for me.
Click to expand...

This means you didn't comprehend what I was telling you. If you lock a binary package, you're going to have issues with dependencies updating underneath it. If you build from source (by controlling the source tree, e.g. with svn) then you never have the issue of the dependencies being mismatched. The issue you have is the locked source port falls out of sync with the tree. Two totally different consequences to two totally different attempts of freezing a port.

For the latter, you can update the frozen port to keep working. But you're doing port maintenance at that point.

And nobody said "don't update ports at all". At best you could interpret this as "don't use firefox if you insist on an obsolete version".
or alternatively, use ravenports that will have firefox 52 for a long time.
 
hrenznaet said:
to either
-
Click to expand...
Effectively, you're talking about a container where all app dependencies are shipped together with the app. Like Ubuntu snaps or Windows side-by-side. With the app package being completely isolated from the rest of the system. Which is doable but requires one to maintain a custom port builds tree / deployment location. In other systems, that custom deps tree is implicitly maintained by somebody else, like the app developers themselves, but it does exist.
On FBSD you'll have to maintain it yourself. Then you can deploy it in a jail, as one of the options.
 
I'm having an issue and not sure how to move forward with it.
Code: 
$ sudo synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 35.51%             
culprit: emulators/xen
  Scan aborted because 'make' encounted an error in the Makefile.
  emulators/xen (return code = 1)
Flavor index generation failed: ports scan
The /usr/ports/emulators/xen directory does not exist. What should I do now?
 
Hello,

Would it be ok if the version of FreeBSD where Synth is installed is higher than the FreeBSD version of the servers I'm building pkg's for? For example the server where Synth is used runs 11.2-RELEASE and all other servers run 11.1-RELEASE.

Thank you.
 
it should work at least 99% of the time. Ideally the builder would be the older machine though. It's guaranteed that packages from older minor releases work on newer minor releases, but not the other way around.
 
Probably an old topic but I couldn't find an answer: every time I synth upgrade-system, it ignores sysutils/lsof and sysutils/htop. There are no logs to look into - that's what "ignored" really means I guess -, so I move to a manual install with synth install sysutils/lsof. This results in "The most recent version of packages are already installed", which again, is true.

With sysutils/htop however, there is something else: synth install sysutils/htop or any of "build, force" gives me "pkg: No packages available to install matching 'htop-2.2.0' have been found in the repositories". It seems to make sense since htop requires lsof and nothing was built, so synth is happy, but I am confused...

I had no issues installing both of the packages it with pkg, portmaster or make install.

And that begs the question: if the package is already installed with the latest available version why is synth selecting it for an upgrade? and since lsof it's a dependency from another package - also already installed and up to date, shouldn't synth ignore them both completely?
 
the log starting with 00_* will explain why its getting ignored. go to logs directory, find it, open it, and look for for lines involving lsof and htop.
 
marino said:
the log starting with 00_* will explain why its getting ignored. go to logs directory, find it, open it, and look for for lines involving lsof and htop.
Click to expand...

Hmm, yes that makes sense: lsof requires the kernel sources and I've installed the kernel sources on a non-standard location. Lesson learned.
 
Hello, I'm having this funny message upon the completion of a synth install command:
Code: 
The task is complete.  Final tally:
Initial queue size: 2
    packages built: 2
           ignored: 0
           skipped: 0
            failed: 0
Duration: 00:01:35
The build logs can be found at: /var/log/synth
Stand by, recursively scanning 2 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
pkg: Unknown repository: Synth
Unfortunately, the system upgraded failed
Interestingly, this last pkg error prevents pkg from installing whatever has been built by synth.
Interesting, too, that when I serve the same repository via httpd to other computers, the local pkg find no problems updating from it.
It's pkg-1.10.5_1 and synth-2.05.

Here is my repo.conf which pkg pretends not to see (although it did see it until now):
Code: 
# Automatically generated.
Synth: {
  url      : "file:///var/synth/11.1-packages",
  priority : 0,
  enabled  : yes
}
 
free-and-bsd said:
Hello, I'm having this funny message upon the completion of a synth install command:
Code: 
The task is complete.  Final tally:
Initial queue size: 2
    packages built: 2
           ignored: 0
           skipped: 0
            failed: 0
Duration: 00:01:35
The build logs can be found at: /var/log/synth
Stand by, recursively scanning 2 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
pkg: Unknown repository: Synth
Unfortunately, the system upgraded failed
Interestingly, this last pkg error prevents pkg from installing whatever has been built by synth.
Interesting, too, that when I serve the same repository via httpd to other computers, the local pkg find no problems updating from it.
It's pkg-1.10.5_1 and synth-2.05.

Here is my repo.conf which pkg pretends not to see (although it did see it until now):
Code: 
# Automatically generated.
Synth: {
  url      : "file:///var/synth/11.1-packages",
  priority : 0,
  enabled  : yes
}
Click to expand...
OK, I've fixed it :) Just mixed up a bit the repo.conf files. Now they look fine and everything is running again.
Sorry for the false alarm... just when I get nervous I become distracted and can mess up things a bit.
 
Hi Marino, I've run in to a weird problem.

I've updated my make.conf file to use the security/openssl111 port and Synth only want to build the new OpenSSL version, and not any of the 70 ports that depend on it.

Any ideas what I am doing wrong?

/usr/local/etc/synth/LiveSystem-make.conf :
Code: 
OPTIONS_UNSET = X11 CUPS

DEFAULT_VERSIONS+=ssl=openssl111 php=7.1 mysql=5.6
Code: 
$ sudo synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => security/openssl111
Total packages that would be built: 1
 
I think these 70 ports may need adjustments because OpenSSL internals changed and just forcing a new version on ports would not just work. So they are still hardwired to use older openssl version. Just wait until port maintainers perform their magic.
 
openssl111 is not a valid option:
Code: 
# Possible values: base, openssl, openssl-devel, libressl, libressl-devel
See /usr/ports/Mk/bsd.default-versions.mk
 
It's not listed as a valid option there but it does work as a custom setting for selecting an exact OpenSSL port to be used. Lot of ports seem to break though at the moment with that setting, I was lucky to have only one broken port and I don't really need that one.
 
Right, I had to dig through /usr/ports/Mk/Uses/ssl.mk to see what's happening. The code itself doesn't actually check what's put in, it simply uses security/${SSL_PORT}. So it automagically works for anything you put in. It actually allows bad input too, setting SSL_DEFAULT=gnutls for example, will actually install security/gnutls but then fail later during dependency checks. Interesting.
 
Another one: synth is failing to install databases/mongodb36.

/usr/bin/ld: final link failed: No space left on device
c++: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [build/opt/mongo/mongoperf] Error 1
scons: building terminated because of errors.
build/opt/mongo/mongoperf failed: Error 1
*** Error code 2

Stop.
make: stopped in /xports/databases/mongodb36

--------------------------------------------------
-- Termination
--------------------------------------------------
Finished: Thursday, 20 SEP 2018 at 18:13:42 UTC
Duration: 01:00:57


At some point I see synth using 99.9% of the 9G swap available, but only roughly 10% of RAM.
The machine has 12 cores, 20G RAM and fast enough disks so I guess my synth configuration is not optimal.
I currently have 6 builders, 3 jobs per builder and I could, according to the man pages, go up to 12*0.75=9 builders and 4 jobs per builder.
What else can I do? disable tempfs?
 
The rule of thumb is swap should be ideally be set at 4 times the ram, so 80G in your case.
If you build mongo by itself, you should be okay. one builder, 3 jobs, 20G ram, I would be surprised if you touched swap at all.
Don't disable tmpfs.
 
mfoacs - if you run "top", what does your memory usage show when Synth is not building?

I have 6 builders, 4 jobs per builder and it only uses ~4GB ram (no swap).
 
Mayhem30 said:
mfoacs - if you run "top", what does your memory usage show when Synth is not building?

I have 6 builders, 4 jobs per builder and it only uses ~4GB ram (no swap).
Click to expand...

Here you have it:


Mem: 6160K Active, 2536M Inact, 2315M Wired, 1526M Buf, 14G Free
Swap: 9670M Total, 9670M Free

 
Forgive me if this is the wrong place for my question, but it appears Synth allows you to have different profiles. Would you be able to have a profile for server and another for desktop (same arch)?

Also, this tool builds the entire ports tree, correct? If so, what size are we talking about here? 3TB, 6TB, 12TB? I don't suppose you can select part of the ports tree to build? IE: Say you want to omit games/xxxxxx ports.

I apologize if these were asked earlier. I tried to lookup size + synth on search and couldn't find much context aside from people asking how this tool is different from pkg or portmaster over and over.

edit: I think I found my answers at: http://www.kaba1ah.org/2017/07/31/managing-ports-for-multiple-freebsd-servers-2/ I understand I can just feed synth a list of ports to keep track of. I would be curious if anyone knew the size of every compiled package. I'm also trying to understand if synth is dynamic enough to build a package for a client that requests a port to be installed or if I have to go to synth first.
 
You must log in or register to reply here.
Back
Top