• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

[Guide] A closer look at Portmaster

ShelLuser

Daemon

Thanks: 1,090
Best answers: 2
Messages: 2,398

#1
Hi gang!

Editorial: I've been using ports-mgmt/portmaster for a long time already and I figured it could use a little bit of extra love and attention ;) Also because there is a lot which Portmaster can do, so I'm hoping that briefly showcasing some of those features might be helpful for some of you out there.

The ports collection

As you probably know FreeBSD provides tons of 3rd party software through the so called ports collection. The ports collection is basically what I like to describe as a collection of blue prints which can tell the system how to obtain a certain piece of software (usually the source code, but binaries are also possible), then in case of a source tree it'll showcase how the application should be build (compiled) and finally a package is made which then gets installed using pkg.

Managing your ports collection is usually done using portsnap(8) but you can also manually keep track of things using Subversion. But that's beyond the scope of this guide, we're going to look at the process of building a port, and Portmaster in particular.

The vanilla way of installing a port

Most people who build ports manually will probably know about the # make config and # make install clean commands (also referred to as build targets). Basically you configure a ports options using the first command, and then use the latter to actually build and install the port after which the source tree gets cleaned up (provided the build was a success). But there's much more to it than that, in case you haven't already I definitely advice you to take a glimpse at the ports(7) manual page sometime, it will list most commonly available build targets, and some can be very useful.

Now, the vanilla routine definitely gets the job done but it's not perfect. For example: if you're compiling a larger port such as, for example, www/apache24 then you're looking at a pretty large amount of dependencies, some will be required to actually use the port, whereas others will be required for the building process:

Code:
peter@macron:/usr/ports/www/apache24# make all-depends-list | wc -l
     143
.... And some of the problems.

Now, this is all fine and well but it creates a bit of a hassle: plenty of those dependencies will have their own configuration to deal with, which then could lead up to the system presenting you with a configuration screen 35 minutes after having started the build process. While you were already busy getting some tea and perhaps do the dishes ;) Not quite useful, and basically a waste of time.

Of course there are ways to overcome this, for example by using the # make config-recursive target, this will present you with all required options at the same time. But even so, it is a little bit prone to issues sometimes.

Also: have you ever thought about using pkg-create(8) to make a backup of whatever it is you're installing before you replace it with a newer version?

And what about all those dependencies? As you might know a port has 2 types of dependencies: build and run dependencies. You can check for these yourself using the run-depends-list and build-depends-list build targets respectfully. What if you wanted to get rid of those build dependencies to save disk space?

Well, for all that and a whole lot more there's Portmaster. An rather small utility which can automate several of these tasks for you.

Portmaster

Portmaster is a small (well..., sort off ;)) shell script which uses all the default standard commands to automate the building of a port. You can get it by running # make install clean in the /usr/ports/ports-mgmt/portmaster directory.

An example... If we once again look at the previously mentioned www/apache24 port then all it takes to install that would be: # portmaster www/apache24. This can be executed from any random directory.

After issuing this command Portmaster will check for all the required dependencies and go over all of them. If some of them haven't been configured yet then you can just sit back and relax because Portmaster will show you all the required configuration for all those ports which are being processed, long before the actual building starts.

If Portmaster needs to do more tasks other than building the port which you specified (for example: installing build or run dependencies) then you will first be presented with a list of everything Portmaster is about to do, otherwise the building will be done straight away.

Of course Portmaster can do more than that...

However, always keep in mind that Portmaster basically uses the default set of tools. For example, one of its features is the --list-origins option, this will produce a list of all your installed ports, and that list can be directly fed to Portmaster again so that it can (re)install all the ports when needed. It's a useful option for sure, but... nothing which pkg info -qao couldn't handle.

Configuration & logging

Now, Portmaster has several options and I'll go over some of them in a moment, but before I do I'd like to specifically point your attention to Portmasters configuration file, by default this can be found in /usr/local/etc/portmaster.rc. Do note that this doesn't get automatically installed, look for portmaster.rc.sample when in doubt.

The best thing to do is to go over all the available options yourself, but there are a few which I'd like to specifically mention...

Code:
# Always save the backup packages of the old port (-b)
BACKUP=bopt
This option can be a life safer. What it does is ensure that Portmaster will create a backup copy of the currently installed port before applying an upgrade. This ensures that you'll be able to go back to the previous version no matter what.

So if something goes wrong simply go to /usr/ports/packages/portmaster-backup and use pkg-add(8) to install it (after you removed the previous version of course).

Code:
# Always delete stale distfiles without prompting (-d)
ALWAYS_SCRUB_DISTFILES=dopt
When you're building (new) software then the system will start by downloading the original (vanilla) source code package, extract that and then optionally apply some FreeBSD specific patches to it. But what about all those distribution files? Eventually they'll slowly fill up your system.

Portmaster counters for this and it will always ask you if it's ok to remove the unused (existing) version. However, that can sometimes become quite annoying because this basically means that after every successful package installation you'd have to tell Portmaster if it can remove the old distifles. So basically: instead of getting configuration options presented to you somewhere during the build process, you now get Yes/No questions ;)

Fortunately this option can help you counter that.

Code:
# Log actions taken by portmaster with a date/time stamp
PM_LOG=/var/log/portmaster.log
This will allow you to always keep track of what has been done on your system. I personally consider this very useful because I can always check back at what has happened on my system in the past. If you are going to use this approach then I also recommend that you look into newsyslog.conf(5) in order to ensure that the sizes of your logfile(s) doesn't grow out of hand (see also /usr/local/etc/newsyslog.conf.d).

Some (maybe) interesting command options

portmaster -l | less (of course feel free to use more if preferred).

(fun fact: did you know that /usr/bin/more and /usr/bin/less are more or less the same program? :cool:)

Portmaster divides the installed ports into 4 main categories: Root, Trunk, Branch and Leaf ports.

Root ports are ports which are purely stand-alone. They don't have any dependencies, and other ports don't depend on them.

Trunk ports are depended on by other ports but have no further dependencies themselves.

Branch ports are what their name implies: branches. They are depended on by other ports and also have dependencies themselves.

Leaf ports finally have dependencies on other ports, but are not being depended on themselves.

Well, you may have guessed as much: this command will show you all the installed ports as well as the category they belong to.

# portmaster -f security/gnupg

Normally you can simply install a port by mentioning it on the Portmaster command line. But sometimes that might not be enough. By default Portmaster will check the dependencies and if they're already present (so: a needed package is already installed) then that's that.

But what if you need for the dependencies to get rebuild as well? Well, in that case the -f parameter may come in handy. It will force Portmaster to check the dependencies of the port you're (re)installing and it will rebuild all those other dependency ports as well.

Keep in mind though that this command recurses. So it will also "chain-check" all the dependencies, and apply the required rebuilding on those as well. Thorough for sure, but depending on the situation not always needed.

If you're trying to fix problems (for example building problems) and you think the cause could be with a mis-installed dependency then a "1 tier" rebuild (as I tend to call it) might be more useful (quicker too). So, in the ports directory run something like this: # portmaster `make build-depends-list | sed -e 's/\/usr\/ports\///g'`. This would tell Portmaster to only reinstall those ports which are directly required for the rebuild process, but without looking at any of their dependencies.

Of course, your mileage may vary as is the saying ;)

# portmaster -r apr-1

In some way you could argue that -r is a little bit of the opposite of -f. Instead of rebuilding all the dependencies of the target port it tells Portmaster to rebuild all the ports which depend ('rely') on the port you're processing. This option can be especially useful if you're upgrading a library port and you want to ensure that all programs which use that get rebuild.

For example devel/apr1. If you look at all its requirements (= ports which depend on it) then you'll see yourself:

Code:
$ pkg info -rx apr
apr-1.6.2.1.6.0:
        serf-1.3.9_1
        subversion-1.9.5
        p5-subversion-1.9.5_2
        apache24-2.4.27
So using the above command would ensure that other than apr all these mentioned ports get rebuild also, even if they're up to date:

Code:
===>>> The following actions will be taken if you choose to proceed:
        Re-install apr-1.6.2.1.6.0
        Upgrade mysql56-client-5.6.36 to mysql56-client-5.6.37
        Upgrade postgresql95-client-9.5.7_1 to postgresql95-client-9.5.7_2
        Re-install serf-1.3.9_1
        Upgrade subversion-1.9.5 to subversion-1.9.6
        Upgrade p5-subversion-1.9.5_2 to p5-subversion-1.9.6_2
        Re-install apache24-2.4.27
# portmaster --clean-distfiles

As mentioned a few times now the ports collection depends on official source packages. This is in my opinion also the beauty of the system: all it does is grab the source code from a 3rd party (official) repository, then process it.

But as you keep installing and upgrading ports your collection of distribution files ('distfiles') will slowly but steadily grow. That's where this command can come in handy: it will check all the downloaded distfiles and if it finds one which isn't associated with a currently installed port Portmaster will ask you if you want to remove it.

Of course you can also chose to simply remove all the distfiles (see /usr/ports/distfiles), but I'd advice against that. It won't break anything, no worries, but if you need to reinstall a port it also means you'll have to download the source again.

# portmaster --check-port-dbdir

What gobbles up more disk space: one file of 10kb or four files of 10b? Sounds easy right? 10.000b > 40b, and thus the four files require less disk space. This may sound bizarre to you but you're probably wrong. However, it heavily depends on how your file system was set up. A file system is usually divided into data blocks, meaning that even a small 1kb file will take up at least a minimum amount of space:

Code:
peter@macron:/var/db/ports/archivers_rar# file -s /dev/ada0p2
/dev/ada0p2: Unix Fast File system [v2] (little-endian) last mounted on /, last written at Mon Jul 31 15:21:12 2017, clean flag 0, readonly flag 0, number of blocks 262144, number of data blocks 253831, number of cylinder groups 4, block size 32768, fragment size 4096, average file size 16384, average number of files in dir 64, pending blocks to free 0, pending inodes to free 0, system-wide uuid 0, minimum percentage of free blocks 8, TIME optimization
Here you can see that the block size on this filesystem is 32768, divided over several groups this would boil down to an 8kb minimum size. So one file will at least gobble up approx. 8kb. So going back to my question above; four files of 8kb account for 32kb and will therefor take up more space.

Why all this dry theory you ask?

Every time a port gets configured the system will store the build options in /var/db/ports. These options are stored in a matching directory using the file options. As you can imagine these files aren't very large. For example, these are the stored options of my favorite archiver (rar):

Code:
# This file is auto-generated by 'make config'.
# Options for rar-5.4.0,3
_OPTIONS_READ=rar-5.4.0,3
_FILE_COMPLETE_OPTIONS_LIST=DOCS SFX
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_SET+=SFX
This file is approx. 184 bytes in size. But effectively it gobbles up 8kb no matter what. Its regular size may be 184 bytes but its effective size is 8kb.

Now, for one single file this is obviously peanuts. And well, what is an extra 8kb exactly? But, uhm, what if you had 75 unused option files in your ports directory? 75 * 8 = 600kb, almost 1Mb.

Now this may not seem like much to you, but keep well in mind that block sizes can vary. On this system a file has a minimum size of 8kb. But larger file systems usually use larger block sizes, and that will result in larger minimum file sizes.

Fortunately Portmaster has you covered. By running the --check-port-dbdir command it will check the /var/db/ports directory and if it finds any unused option files it will offer to remove those.

So this command can save you some precious disk space!

# portmaster --delete-build-only.

Remember my comments about how some ports have build dependencies, so ports which they need to have installed before the port itself can be build? Although it's usually a good idea to leave these ports in place they do take up precious disk space. So... this option can help you remove those ports which were only installed to help build other ports.

And finally...

When in doubt: use -n!

If you're not sure about using a specific command then always keep the -n option in mind. This tells Portmaster to do a dry run; so it will simulate whatever you told it to do but without actually doing it. This can help you to see for yourself what would actually happen so that you can be sure that it is what you wanted.

And there you have it, a brief look at some of the Portmaster options.
 

puretone

New Member

Thanks: 7
Messages: 11

#2
(flame suit on, in case raising this aging thread begets me some anger)

How does one obtain the make -j4 mehfoobarblah functionality with portmaster?
I was thinking # portmaster -m '-j4' -dy --no-confirm mehfoobarblah *or* # portmaster -m "-j4" -dy --no-confirm mehfoobarblah would do the trick, but it don't.

I suspect it is something very obvious & right in front of my nose, but am too stupid to recognize...
 

ShelLuser

Daemon

Thanks: 1,090
Best answers: 2
Messages: 2,398

#3
(flame suit on, in case raising this aging thread begets me some anger)
What's that? You think my tutorial isn't deserving of a small bump? :D

Just kidding there!

Don't worry though; it is true that bumping very old threads can sometimes be counter productive, for example: you have a problem which looks like the OP, but after 3+ years many things have probably changed making the problem most likely totally unrelated. But this is simply asking something related to the tutorial, so that should always be ok IMO.

But since you expected a possible flame I obviously couldn't resist poking a little fun ;)

How does one obtain the make -j4 mehfoobarblah functionality with portmaster?
The right command is: portmaster -m -j4 section/port. But do keep in mind that it's not advised to use this casually because lots of ports were not set up with proper support for parallel building.

Take for example: portmaster -m -j2 ports-mgmt/pkg; I'm sure that this is going to fail on you:

Code:
===> Fetching all distfiles required by pkg-1.10.5_1 for building
--- /usr/ports/ports-mgmt/pkg/work/.extract_done.pkg._usr_local ---
`check-build-conflicts' was not built (made 1, flags 2009, type 2010001)!
    `check-build-conflicts' has .ORDER dependency against fetch (made 0, flags 9, type 3010001)
`extract-message' was not built (made 1, flags 2009, type a010001)!
    `extract-message' has .ORDER dependency against check-build-conflicts (made 1, flags 2009, type 2010001)
`check-categories' was not built (made 0, flags 2009, type 3010001)!
`identify-install-conflicts' was not built (made 0, flags 2009, type 3010001)!
    `identify-install-conflicts' has .ORDER dependency against check-categories (made 0, flags 2009, type 3010001)
`check-deprecated' was not built (made 0, flags 2009, type 3010001)!
    `check-deprecated' has .ORDER dependency against identify-install-conflicts (made 0, flags 2009, type 3010001)

---CUT---

    `/usr/ports/ports-mgmt/pkg/work' has .ORDER dependency against clean-wrkdir (made 1, flags 2009, type a010001)
`extract-fixup-modes' was not built (made 1, flags 2009, type a010001)!
    `extract-fixup-modes' has .ORDER dependency against do-extract (made 1, flags 3009, type a010001)
*** [/usr/ports/ports-mgmt/pkg/work/.extract_done.pkg._usr_local] Error code 1

make: stopped in /usr/ports/ports-mgmt/pkg
1 error
And it's not a problem with the port itself, because this command works without any hassle: portmaster ports-mgmt/pkg.

(edit): Also: you'll get the same error results when using make -j2 build within the /usr/ports/ports-mgmt/pkg directory, which basically proves my point regarding ports & parallel building.

So keep this well in mind, in most cases you're likely to run into problems instead of speeding things up. Another aspect to consider is that several ports don't use /usr/bin/make but instead rely on something else, for example devel/gmake is often used with Linux related (or based) software. In which cases Portmaster's -m parameter won't be used.

If you're looking to speed up your building process then my suggestion would be to look into devel/ccache instead. This can be used for both the ports (it can seemingly integrate with the process; just configure /etc/make.conf accordingly) and can even be applied to building the base system and/or kernel (see the WITH_CCACHE_BUILD target for src.conf(5)).

Hope this can help.
 

Zirias

Well-Known Member

Thanks: 118
Messages: 349

#4
First of all, it's a pity that there are still Makefiles breaking on parallel make. The only way this can happen is when dependencies are missing. Developers nowadays should really test their Makefiles for parallel builds.

But then, yes, the problem is still real. If you're building a lot and looking to speed up things, you might want to have a look at ports-mgmt/poudriere. On a ZFS system, and given you have enough RAM, this will be really fast, because it just builds different ports at the same time (which is entirely safe). It can also be configured to use parallel make on selected ports, that's a good idea for really large ports if you are sure they are safe to build in parallel -- I use it for example with www/chromium.

Of course it adds a whole lot of complexity over portmaster, so it might not be the right choice for you -- just wanted to mention :)
 
Top