Ports Build Time

[Deleted member 9563]

Have you ever seen the number of dependencies Xfce has?

Good point. In my (little) experience, the very large GTK system can be time consuming to compile.

 

[SirDice]

If you can do without it's GNOME compatibility you can turn off GVFS in x11-fm/thunar. You might want to turn off mousepad too. That seems to depend on a lot of GNOME stuff too nowadays.

 

[markbsd]

Well, since you already have X installed, you might want to leave it as is and consider a more lightweight window manager instead of a full blown desktop like x11-wm/xfce.

You may be interested in the following thread from the Howtos Section of the forum which walks you through building a minimal desktop environment Minimal FreeBSD Desktop.

Thanks for that link! I'm going to do that.

We still don't know how fast your computer is, or what the situation was. Did you start with no ports installed? Were there outdated packages installed? We know you had deleted build dependencies, so those have to be built.

So: stop. Take a deep breath. Don't obsess over the number of ports, it's not really indicative of anything. Many of the standard dependencies can take a while to build but then are mostly static. For example, gettext takes a while to build, but does not change often. So once it's installed, it does not need to be installed again.

Switch to another console and try ps axww | grep portmaster to see what it is doing. Or look in the compile output. Scroll Lock can be used to pause the output. When paused, the arrow keys and Page Up/Page Down can be used to scroll around.

It's an old notebook, about 10 years old. I can't even tell you the hardware off the top of my head; 2 GB RAM, 50 GB HDD, Px

I started with nothing. Fresh install >> PKGNG >> Portmaster >> Xorg >> Xfce4

Thanks, @wblock@. ps axww reveals that portmaster is currently working on:

libxfce4menu
glade3
atlas
py-gtk2
py-numpy
xfce4
thunar
squeeze
xfce4-panel
libexo

Do you have any option set in your portmaster's configuration files?

I do:

#Always delete stale distfiles without prompting (-d)
ALWAYS_SCRUB_DISTFILES=dopt
# Be verbose (-v)
PM_VERBOSE=vopt
# Install packages for build-only dependencies (--packages-build)
PM_PACKAGES_BUILD=pmp_build
# Delete build-only dependencies when finished (--delete-build-only)
PM_DEL_BUILD_ONLY=pm_dbo
#Suppress the build confirmation message (--no-confirm)
PM_NO_CONFIRM=pm_no_confirm

Maybe I've enabled some option that is making this process excruciatingly long?

Have you ever seen the number of dependencies Xfce has?

DON'T!!! This is totally unnecessary. What's wonderful about FreeBSD is that the base system and third-party ports are entirely separate entities.
You can get back to a clean install with 2 simple commands:
# rm -R /usr/local
# rm -R /var/db/pkg


"Experience is one thing you can't get for nothing." (Oscar Wilde)
"Experience is what you get when you didn't get what you wanted." (Randy Pausch)

Stay positive

Apparently, it only has 13 run dependencies. Not sure how many build.

Thanks for the advice (and words of encouragement). So, if I CTRL+C this process, I can revert to a clean base system with those two commands?

The short version is pkg_delete -a.

I think this needs to be done...

The Xfce4 port build is STILL GOING! That makes run-time 20 hours and counting! LOL

Something has to be wrong, this is absurd!

 

[trh411]

It's an old notebook, about 10 years old.

I have a 5 year old Intel Core2Duo system. A state-of-the-art Intel i7 quad core system is nearly 5 times faster (according to popular CPU benchmarks) and uses memory that is twice as fast as mine. It takes me about 2 hours to install the entire x11/xorg meta port.

Extrapolating backwards, your system is probably 1/5th as fast as mine, which means about 10 hours to install the entire x11/xorg meta port.

Certainly not scientific, but I have a better understanding now why your builds run so long.

Maybe you should install packages?

 

[markbsd]

I think I will. I'm going to abort this failure, delete everything, and use pkg install.

What is the pkg command equivalency of pkg_delete -a.

Alternatively, I might give FreeBSD a chance on one of my newer systems. Last time I tried it refused to cooperate.

 

[Juanitou]

The Xfce4 port build is STILL GOING??! That makes run-time 20 hours and counting! LOL

Something has to be wrong, this is absurd!

The first time I built a complete Xfce4 desktop with Gimp, Inkscape, Firefox, Java, etc. it took at least three full days on a salvaged old computer. There’s nothing wrong or absurd, it’s just slow… :e

 

[markbsd]

I can't believe this is normal! It's only up to 118/210! At this rate, just Xfce4 -- never mind a browser and Java -- will take over two days! Wow

 

[trh411]

I think I will. I'm going to abort this failure, delete everything, and use pkg install.

What is the pkg command equivalency of pkg_delete -a.

Alternatively, I might give FreeBSD a chance on one of my newer systems. Last time I tried it refused to cooperate.

Just drop the underscore ... pkg delete -a. This command will delete all installed packages from the system and clear the database.

 

[wblock@]

#Always delete stale distfiles without prompting (-d)
ALWAYS_SCRUB_DISTFILES=dopt
# Be verbose (-v)
PM_VERBOSE=vopt
# Install packages for build-only dependencies (--packages-build)
PM_PACKAGES_BUILD=pmp_build
# Delete build-only dependencies when finished (--delete-build-only)
PM_DEL_BUILD_ONLY=pm_dbo
#Suppress the build confirmation message (--no-confirm)
PM_NO_CONFIRM=pm_no_confirm

Maybe I've enabled some option that is making this process excruciatingly long?

Yes, deleting build dependencies is a mistake. But that won't have an effect until the next time.

Apparently, it only has 13 run dependencies. Not sure how many build.

Sure. However, those ports also have build and run dependencies. A typical system with X and xfce will have something like 400-500 ports installed.

If you were to stop and then restart the build, all of the ports it had already installed would not need to be installed again. But your settings to delete build dependencies will defeat that, forcing them to be rebuilt. That includes big things that are slow to compile, like gcc-6.4 at present. Deleting all ports magnifies that, discarding all the work that has been done.

If you stop throwing away all the work each time, it will eventually complete and then take much less time to update or add.

 

[kpa]

When I was still using the plain ports(7) and ports-mgmt/portmaster to build my ports I found it very helpful to figure out the most common run/build time dependencies and install them separately before launching bigger builds. One very helpful tool is make all-depends-list for a big port like x11/xorg. Also make missing is very helpful. Intersecting outputs from those when run on different ports should give you an idea what those very common ports are. From memory I can list at least a couple that are used all the time: lang/perl5.16, lang/python27, devel/autoconf, devel/automake, devel/libtool, devel/gmake, devel/gettext and many others.

 

[xibo]

I have a 5 year old Intel Core2Duo system. A state-of-the-art Intel i7 quad core system is nearly 5 times faster (according to popular CPU benchmarks) and uses memory that is twice as fast as mine. It takes me about 2 hours to install the entire x11/xorg meta port.

Extrapolating backwards, your system is probably 1/5th as fast as mine, which means about 10 hours to install the entire x11/xorg meta port.

Extrapolation is no good here, as ports installation times are affected by numerous sources. My dual Xeon E5-2650v2s (512GB memory) can build KDE4 with all options enabled and all dependencies in less than two hours if it's installing from/to it's zfs raid, but it takes more than a day to install them on the USB media it normally uses only for the bootloader and kernel.

On a modern HDD or SSD, the GEOM buffer drains that finally write data on the storage media after the kernel had taken responsibility of them will likely occur while make already is working at the next port, which eventually even builds on a filesystem backed by another GEOM, i.e. make is not at all affected by the ${PREFIX} media being slow.
On slow storage media (such as old HDDs or my USB flash media), however, the draining can take place while the data is still actively being written, blocking the install or cp proccess and slowing not only the installation process but the whole system, as every access on the draining GEOM that cannot be proccessed by the file cache - which is filled with port's data to be written - will also be blocked until the draining completes.

Once a port is large enough (libreoffice, some elements of KDE, ...) not to fully install into the buffers without being forced to drain them while doing so, the installation process is slowed by an order of magnitude. Additionally to the limit imposed by the amount of memory in the system that can be used for buffers, FreeBSD will keep "dirty" buffers (buffers that need to be written to the storage) for no more than 30 seconds before commiting them to the storage media even if there's enough buffers free, so a system with slower instructions and/or instruction scheduling, worse branch prediction, less logical processors, worse memory caches, slower memory, less memory bandwith, ..., or in general, a dated system, is more likely to run into this 30 seconds timeout than a current one.

 

[kisscool-fr]

I do:

#Always delete stale distfiles without prompting (-d)
ALWAYS_SCRUB_DISTFILES=dopt
# Be verbose (-v)
PM_VERBOSE=vopt
# Install packages for build-only dependencies (--packages-build)
PM_PACKAGES_BUILD=pmp_build
# Delete build-only dependencies when finished (--delete-build-only)
PM_DEL_BUILD_ONLY=pm_dbo
#Suppress the build confirmation message (--no-confirm)
PM_NO_CONFIRM=pm_no_confirm

Maybe I've enabled some option that is making this process excruciatingly long?

Yes, deleting build dependencies is a mistake. But that won't have an effect until the next time.

If you were to stop and then restart the build, all of the ports it had already installed would not need to be installed again. But your settings to delete build dependencies will defeat that, forcing them to be rebuilt. That includes big things that are slow to compile, like gcc-6.4 at present. Deleting all ports magnifies that, discarding all the work that has been done.

If you stop throwing away all the work each time, it will eventually complete and then take much less time to update or add.

From my experience, I stopped using options for portmaster. Specifically the one that said delete build dependecies. How I understand it is that for every port you want to install, portmaster will do the job for the build dependecies, then the port itself and at the end of the port installation it will delete build depencies. And that for all the ports. Imagine if for example all your ports need the same build dependecies, it will rebuild them for each port and this time consuming. You see ?

You can keep other options if you want, but the "delete build dependencies", I recommend to comment it.

 

[kpa]

There's a problem with portmaster when used with PKGNG and you'd like to use packages for build only dependencies to save time, it's not possible to do so at the moment. This means portmaster will build all build time dependencies as ports if they are not installed already. This can be very time consuming.

 

[jb_fvwm2]

Missing parameter (not yet exists)... "build from port but use older packages for build time dependencies... if they exist." Perhaps someone can start a [Sticky] portmaster thread with 'wanted upgrades' 'optimal switch configurations' 'FAQ use cases' to lessen the explanations scattered throughout multiple threads, quite often duplicating content, as least it seems.

There does not seem to be a wiki page for it; it seems to me that the information would be more useful here as a sticky.

 

[markbsd]

Thanks, everyone.

I've just decided to avoid ports on this system and stick to pkg installs. If I build up the courage to install FreeBSD on one of my newer computers I might return to ports, but the reward doesn't justify the cost (time) invested in this instance. In fact, I've been reading up a lot on ports vis a vis pkg installs and it seems ports don't really offer much more of a benefit at all. So, I may just remain with pkg installs notwithstanding what architecture I'm running.

 

[Juanitou]

A real-life example (far more serious then that any professional story) of using ports to obtain a great reward: My old box has spent ~20 hours updating the base system to FreeBSD 10.0-BETA3 and compiling the experimental ports providing KMS for its ATI Radeon video card, which is quite slow or totally unusable with the current ports. It was my wife’s birthday and I wanted to offer her a better experience with this computer she mainly uses to watch internet streams. Thanks to the dedication in compiling these ports, Firefox navigation and video performance have significantly improved and we could also have some fun with SDL games that were unplayable before the update.

A happy wife for her birthday, what better reward could I ask for?

 

[markbsd]

So, ports provided a significantly faster operation of x,y,z programs than what pkg install provided?

 

[Beastie]

The short version is pkg_delete -a.

Yes, but it often leaves traces everywhere in /usr/local and is slower since it has to navigate through the entire dependency>root package hierarchy instead of just deleting files indiscriminately.



--8<--

I've been reading up a lot on ports vis a vis pkg installs and it seems ports don't really offer much more of a benefit at all.

The only advantage that ports have over pre-built packages is that you can use non-default build options, including 1) the removal of some parts/dependencies you don't need and that make the application bigger (and sometimes slower) and 2) the inclusion of experimental features you may need.

If you don't really care about either, then definitely stick with packages. At least for the time being.

In the future you may want to build ports on your newer machines, create packages from these and install them on the older machines.

 

[Juanitou]

So, ports provided a significantly faster operation of x,y,z programs than what pkg install provided?

That’s it, since no packages are provided for these recently ported versions of libGL, dri, libdrm… For SDL games, it was not faster operation what they provided, but plain usability.

Anyway, my message was rather tongue-in-cheek, don’t take too seriously such a particular example. On the other hand, only recently the packages infrastructure has been reinstated after the famous security incident. People relying only on packages have had to find alternatives for some months.

 

[markbsd]

I take your point. I haven't noticed any performance decline now that I'm only using packages, but my experience is very limited. I'm not familiar with the security incident; FreeBSD, to me, is synonymous for secure!