Solved Tracking latest packages ports branch

Minbari

Minbari

I know that for the quarterly packages exist a quarterly ports branch, which only receive security updates. So my question is: is there a branch of ports for the latest packages or the latest packages are built from the current ports ("rolling release")? The first reason for my question is: as a "ports user" (I need some custom ports) rebuilding big ports like llvm, rust, gcc, node (like 4-5 times in a few days), etc. is frustrating and sometimes I can't even use my main machine. For those who are going to recommend me ports-mgmt/synth or ports-mgmt/poudriere please keep your advice for someone else because this machine is the only one which run FreeBSD at the moment and if I'd build my own repository of packages or using portmaster would be the same thing. The second reason is that if I could track the "latest ports branch" I could download those big ports as packages and install them with pkg. I know mixing ports with pkgs is a bad thing to do but the fact that I'm not able to use my pc for 4-5 hours also sucks.
PS. The machine does not have bad specs: i7 processor, 16GB, ssd, etc. still when times comes to rebuild ports I have to switch to my similar specs GNU/Linux machine but "the sensation" is not the same.
 
Minbari said:
..... if I could track the "latest ports branch" I could download those big ports as packages and install them with pkg.....
Click to expand...

Buried in the manpage for pkg() is this example /etc/pkg/repos/FreeBSD.conf
Code: 
       FreeBSD: {
           url: "pkg+http://pkg.freebsd.org/${ABI}/latest",
           enabled:    true,
           signature_type: "fingerprints",
           fingerprints: "/usr/share/keys/pkg",
           mirror_type: "srv"
       }
Make yours look like that (i.e. change quarterly to latest) and you will be accessing the packages built from the latest ports. You can now install using pkg or from ports() -- I suppose the binary packages may sometimes lag a day behind updates in the ports tree so watch what pkg is installing. (I build evrything with poudriere and have little experience mixing ports and packages but as far as I know it should work well). Does that help?
 
Minbari said:
if I could track the "latest ports branch" I could download those big ports as packages and install them with pkg.
Click to expand...
First ports branch is the latest of all and latest packages branches are created in bulk in certain intervals from all ports. So big ports are also there in pkg branch.
If I correctly understand OP, I also would like to use port in same way but don't know the implications. During building of ports some big ports from devel, lang etc. are required as build dependency, except devel/llvm80 which is not deleted. So these ports can be built from package repo. (which is 2-5 days old) during building of ports if required. Thus reduce overall building time and hassle. This is my IMHO.
 
garry said:
Buried in the manpage for pkg() is this example /etc/pkg/repos/FreeBSD.conf
Code: 
       FreeBSD: {
           url: "pkg+http://pkg.freebsd.org/${ABI}/latest",
           enabled:    true,
           signature_type: "fingerprints",
           fingerprints: "/usr/share/keys/pkg",
           mirror_type: "srv"
       }
Make yours look like that (i.e. change quarterly to latest) and you will be accessing the packages built from the latest ports. You can now install using pkg or from ports() -- I suppose the binary packages may sometimes lag a day behind updates in the ports tree so watch what pkg is installing. (I build evrything with poudriere and have little experience mixing ports and packages but as far as I know it should work well). Does that help?
Click to expand...
I have the latest packages config file on /usr/local/etc/pkg/repos/FreeBSD.conf.
The only solution I've found till now is to verify if the local ports differ from FreeBSD remote pkgs repository and if they differ to manually download those big ports and perform the upgrade and then let portmaster do the rest. We'll see what will happen.
 
Minbari said:
I have the latest packages config file on /usr/local/etc/pkg/repos/FreeBSD.conf.
The only solution I've found till now is to verify if the local ports differ from FreeBSD remote pkgs repository and if they differ to manually download those big ports and perform the upgrade and then let portmaster do the rest. We'll see what will happen.
Click to expand...
Perhaps you are looking for something like the solution which I explained in one of my BLog posts:

FreeBSD: Dual Mode updating of installed ports and packages

obsigna.com obsigna.com
 
Minbari said:
You are correctly. That is the solution I'm looking for. Thank you for pointing me the right direction.
Click to expand...
A simpler alternative is to synchronise your ports tree with the package repository. I use a method based on https://github.com/reedacartwright/sync-ports/blob/master/sync-ports. This script hasn't been updated since 2016 but is quite easy to modify for present day versions of FreeBSD. In my case I just need to set PKG_SERVER to beefy6.nyi.freebsd.org for FreeBSD 12.1-RELEASE amd64. This script determines the SVN revision number of the ports tree that was used to build the current version of the package repository and then uses svnlite to update my ports tree to the same version.

The whole process could be even simpler if pkg could be made to display the ports revision which was used when the repository was built.
 
Minbari said:
I have the latest packages config file on /usr/local/etc/pkg/repos/FreeBSD.conf.
The only solution I've found till now is to verify if the local ports differ from FreeBSD remote pkgs repository and if they differ to manually download those big ports and perform the upgrade and then let portmaster do the rest. We'll see what will happen.
Click to expand...

Yes, I understand now. That is a problem (build only some ports from source). I think I could do it using poudriere to build a local repository for all of the packages that are to be built from source. I would add that repository to pkg/repos alongside FreeBSD.conf but with higher priority.

You need a list of all the ports to be built locally, LocalPorts (e.g. everything you now have installed except a few big packages that you don't want to build). Use poudriere to build those ports: e.g.
Code: 
poudriere bulk -j FreeBSD_12_1  -p HEAD  -f LocalPorts
Install/update those ports with pkg. Install the big packages with pkg (which since they aren't in the local repo will be pulled from the FreeBSD latest repo, and since you just built and installed the port's HEAD for everything else nothing else will be pulled in. Every update requires two steps: (1) update the ports tree and use poudriere to build LocalPorts. (2) update the system using pkg update. LocalPorts updates will be pulled from the local repo automatically and "big" packages will be updated from the FreeBSD binary repo online.
 
garry said:
Yes, I understand now. That is a problem (build only some ports from source). I think I could do it using poudriere to build a local repository for all of the packages that are to be built from source. I would add that repository to pkg/repos alongside FreeBSD.conf but with higher priority.

You need a list of all the ports to be built locally, LocalPorts (e.g. everything you now have installed except a few big packages that you don't want to build). Use poudriere to build those ports: e.g.
Code: 
poudriere bulk -j FreeBSD_12_1  -p HEAD  -f LocalPorts
Install/update those ports with pkg. Install the big packages with pkg (which since they aren't in the local repo will be pulled from the FreeBSD latest repo, and since you just built and installed the port's HEAD for everything else nothing else will be pulled in. Every update requires two steps: (1) update the ports tree and use poudriere to build LocalPorts. (2) update the system using pkg update. LocalPorts updates will be pulled from the local repo automatically and "big" packages will be updated from the FreeBSD binary repo online.
Click to expand...
The method you suggested was the first I've tried, but it's not working and also it's a wasted time. For eg. to build a custom firefox the poudriere jail needs llvm, rust, etc. instead of using the one on the system it need to build its own llvm, rust and so one and that is like using portmaster. poudiere is good for peoples with many machines/server not for lonely wolfs which use FreeBSD as a desktop. At least this is my impression about poudriere or I did not correctly configure it.
 
Minbari said:
PS. The machine does not have bad specs: i7 processor, 16GB, ssd, etc. still when times comes to rebuild ports I have to switch to my similar specs GNU/Linux machine but "the sensation" is not the same.
Click to expand...

There may be a completely different approach: adjust the machine configuration so that the port build would not bother your other work.
There is a couple of ways to achieve this, e.g.:
  • run the portbuild with idprio(1). That way it will only compute when nothing else wants to compute.
  • give it a limited number of CPUs with cpuset(1).
  • limit the number of parallel make jobs with make -jN - this should also limit memory consumption.
  • limit the effective compute ressources with rctl(8) - this would require a jail for the builds to separate them from the rest. It should also be possible to limit the amount of disk I/O that way.
Obviousely the port build will then take longer, but that doesn't seem to be Your major issue.

Minbari said:
For eg. to build a custom firefox the poudriere jail needs llvm, rust, etc. instead of using the one on the system it need to build its own llvm, rust and so one and that is like using portmaster. poudiere is good for peoples with many machines/server not for lonely wolfs which use FreeBSD as a desktop. At least this is my impression about poudriere or I did not correctly configure it.
Click to expand...

You're probably right, and there is no really good solution for single desktop, except just installing the pkgs as they are and taking what one gets. But there are two good reasons why poudriere et.al. do it that way: one may want to change arbitraty build options, and since these options can change the dependency tree (among other things), one would want to have that tree consistent and independent from the running installation. And second, one might want to have all ports built from the same tree with the same options - I never had any library issues since I'm doing that.
 
PMc said:
.....
But there are two good reasons why poudriere et.al. do it that way: one may want to change arbitraty build options, and since these options can change the dependency tree (among other things), one would want to have that tree consistent and independent from the running installation. And second, one might want to have all ports built from the same tree with the same options - I never had any library issues since I'm doing that.
Click to expand...

I consider it important to build ALL packages from one ports tree and to install ALL packages from that self-consistent repo. This means that I update ports, rebuild everything with poudriere, drop to single-user and remove all packages and then install all packages from that nice new self-consistent repository. The clean install step only takes a few minutes. This eliminates the occasional strange behavior for which the solution would be "try rebuilding x". This seems to me to be the power of open-source; the power to actually use the source (I change quite a few ports' options). Otherwise, and for most persons, the wise thing to do is to trust the FreeBSD continuous build system and utilize their powerful build computers to make such a good set of packages for you and use pkg
 
Hi guys sorry for late response but none of the above suggestions worked for me. The only working solution is to move to the quarterly repo and update ports with devel/subversion instead of portsnap, which from my knowledge only update the HEAD ports tree.

Solved.
 
You must log in or register to reply here.
Back
Top