Portmaster: skip already built ports & check everything first

[Caliante]

Hi all,

After my debacle of upgrading FreeBSD from 8.0 -> 8.1 I found out I have around a zillion ports that needed upgrading. There is this extremely kind person (an understatement, and this person even refuses the beer I am offering as a way of saying 'thank you':beer ) that advised me to use portmaster for it, more specifically the last section of the portmaster man: delete all ports and have them reinstalled all.

This proves to be a (cu)cumbersome e) path; it seems Mr. Murphy decided to come into my box and demonstrate his law. So, for the last 7 days, I have started portmaster every night before going to bed (portmaster -D `cat ~/installed-port-list`)(see man) only to find it to have halted every morning with yet another error. The extremely kind person has helped me solve each error, yet right now it is starting to get annoying; not only for me, but I could suspect for him also. However, just inserting the 8.1 DVD is the easy way and you don't learn from it, so I'll try to sit it out 'till the end.

I have two questions:

1. What surprises me is that on every new run I start at night I see portmaster starting to rebuild all the ports that it had built already the night before. I was suspecting portmaster -R to be avoiding this, but this seems not to be the case. The man isn't too clear about this: on the last page it says you need to use -R, on the first pages it is not too clear what -R is supposed to do.

Would anybody know what the correct syntax is?

2. I thought it better to have it run through everything first without making/compiling (as to see upfront what will go wrong later so I can fix that first), and thought portmaster -n would do this, but this also isn't doing it.

Again, would anybody know what the correct syntax is?

Thank you in advance,

Bye,

 

[Crivens]

Hello Caliante,

the -R option is, if I get the manpage right, only for updates and not for new installations.
So all ports you list on the command line will be build. However, I may also be missing something here.

Usually I split these new installs in smaller steps, like, first download all needed files (-F), then run the config for all ports (--force-config). Only then i start the build process. Little is more annoying than to look at the progress in the morning and to see it stopped with some "unable to fetch" error or a config screen right when you turned around to go to bed. When the build terminates, i mark all ports which were built with a marker character in the text file and use a "grep -v" in the pipeline to skip these in later runs.

 

[jb_fvwm2]

I'd try something like:

 cd /usr/ports/audio/yell && make install

then

 portmaster -d -B /var/db/pkg/this-# /var/db/pkg/this-# /var/db/pkg/that-# /var/db/pkg/this-# && yell

(maybe?? do the largest ones first)... when using portmaster. But I do not try to update a lot of ports at once. Why not just the ones you use then the remainder really slowly...

 

[Caliante]

Thanks to the both of you :e

Since the list of ports to update was huge, I was doing a full reinstall, which according to the portmaster man should work. Like this:

Using portmaster to do a complete reinstallation of all your ports:
1. portmaster --list-origins > ~/installed-port-list
2. Update your ports tree
3. portmaster --clean-distfiles-all
4. portmaster --check-port-dbdir
5. portmaster -Faf
6. pkg_delete '*'
7. rm -rf /usr/local/lib/compat/pkg
8. Manually check /usr/local and /var/db/pkg
to make sure that they are really empty
9. Re-install portmaster
10. portmaster `cat ~/installed-port-list`

You probably want to use the -D option for the installation and then run
--clean-distfiles[-all] again when you are done. You might also want to
consider using the --force-config option when installing the new ports.

Alternatively you could use portmaster -a -f -D to do an ``in place''
update of your ports. If that process is interrupted for any reason you
can use portmaster -a -f -D -R to avoid rebuilding ports already rebuilt
on previous runs. However the first method (delete everything and rein-
stall) is preferred.

Going through the ports one by one according to the above text shouldn't be necessary. And the process does work to a certain extend. According to pkg_info > 560 ports have been installed. It is just that on every error the process halts, and on fixing that error and restarting the `cat`-list (see above) the process starts rebuilding everything from port # 1, although that has been built already.

 

[Caliante]

I'd try something like:

 cd /usr/ports/audio/yell && make install

then

 portmaster -d -B /var/db/pkg/this-# /var/db/pkg/this-# /var/db/pkg/that-# /var/db/pkg/this-# && yell

(maybe?? do the largest ones first)... when using portmaster. But I do not try to update a lot of ports at once. Why not just the ones you use then the remainder really slowly...

Why would you want to clean the distfiles in a multi-port upgrade?

 

[Caliante]

Hello Caliante,

the -R option is, if I get the manpage right, only for updates and not for new installations.

Could you perhaps point out where you've read that? Because all I am seeing is the explanation at the bottom, in the big example I am following, where -R is supposed to skip already installed (by portmaster) ports. Then again, the man page isn't consistent in this aspect as far as I my small brain can comprehense :e

So all ports you list on the command line will be build. However, I may also be missing something here.

Usually I split these new installs in smaller steps, like, first download all needed files (-F), then run the config for all ports (--force-config). Only then i start the build process. Little is more annoying than to look at the progress in the morning and to see it stopped with some "unable to fetch" error or a config screen right when you turned around to go to bed. When the build terminates, i mark all ports which were built with a marker character in the text file and use a "grep -v" in the pipeline to skip these in later runs.

The -F is a usefull option, thanks for the tip

From my experience, using the example from the man page that I am following all the config happens upfront.

Isn't marking ports in a textfile manually a huge job when, like in my case, it concerns more than 560 ports (the ones currently already installed)? From my simple noob-perception this is FreeBSD, this is about automation; it should be possible to do it without having to mark 560 lines on a paper manually?

 

[jb_fvwm2]

The -d option only cleans older versions of distfiles afaik. The most recent one used is kept for each port. /edit/ Howsoever, it probably cleans all distfiles from dependencies or maybe even globbed-similar ports, that were or were not or both rebuilt, (I suspect an unintended consequence of the coding somewhere). /end edit/

 

[wblock@]

 portmaster -d -B /var/db/pkg/this-# /var/db/pkg/this-# /var/db/pkg/that-# /var/db/pkg/this-# && yell

/var/db/pkg is not needed, just use the name of the port, or the origin (category/portname):

# portmaster firefox ghostscript editors/nano

 

[jb_fvwm2]

Correct, however in the past if I run without the "/var/dg/pkg/" prefixed, it will (would) (did) (sometimes without asking) rebuild multiple ports: "di" can become dillo, diffuse, dirdiff... quicker sometimes to prepend /var/db/pkg/ than lookup the port category.

 

[Crivens]

Could you perhaps point out where you've read that?

Sorry, currently I am faced with something that is not FreeBSD (you may guess as to what OS it is), so this has to be answered later. If I remember correctly, I only found the -R option mentioned when updates were the topic, but not when reinstalls were mentioned. From that I cleverly deducted that the option has no sense in new installs.

The -F is a usefull option, thanks for the tip

You are welcome.

Isn't marking ports in a textfile manually a huge job when, like in my case, it concerns more than 560 ports (the ones currently already installed)?

You do not need to tag or remember all ports you have. When listing the ports with portmaster, you can see some of these are leaf ports, root ports, ...
The ports which have dependencies upon them, no matter if they depend on something else, then you do not need them in that text file. Installing the XFCE main port will pull in all the dependencies so there is no need to list them again. These dependencies may also vary, so you might end up installing ports which were used in the past, but due to changes is the application or different config settings, you no longer would need them.

My laptop has some 700+ installed packages but the file I track all things before I start the build has only a fraction of these. As long as there is a backup of that list I can reinstall all ports pretty quick.

 

[dougb@]

The portmaster -R option

Hello Caliante,

the -R option is, if I get the manpage right, only for updates and not for new installations.

Quoting from the man page:

-R used with the -r or -f options to skip ports updated on a previous run.

I fail to understand how that could be hard to understand.

 

[dougb@]

Hi all,

After my debacle of upgrading FreeBSD from 8.0 -> 8.1 I found out I have around a zillion ports that needed upgrading. There is this extremely kind person (an understatement, and this person even refuses the beer I am offering as a way of saying 'thank you':beer ) that advised me to use portmaster for it, more specifically the last section of the portmaster man: delete all ports and have them reinstalled all.

This process is only needed when you are doing a major upgrade of the FreeBSD version or architecture. Such as 8.x -> 9.x, or i386 -> amd64. You would have been better off with [cmd=]portmaster -a[/cmd]

This proves to be a (cu)cumbersome e) path; it seems Mr. Murphy decided to come into my box and demonstrate his law. So, for the last 7 days, I have started portmaster every night before going to bed (portmaster -D `cat ~/installed-port-list`)(see man) only to find it to have halted every morning with yet another error. The extremely kind person has helped me solve each error, yet right now it is starting to get annoying; not only for me, but I could suspect for him also. However, just inserting the 8.1 DVD is the easy way and you don't learn from it, so I'll try to sit it out 'till the end.

I have two questions:

1. What surprises me is that on every new run I start at night I see portmaster starting to rebuild all the ports that it had built already the night before.

I don't know why that's surprising, it's doing exactly what you told it to do. Portmaster has no way to know why you want to do what you're telling it to do, it just does it. Therefore, you have to be careful to tell it to do only what you actually want it to do. If there are ports on your list that you've already reinstalled and you don't wish to re-re-install them, remove them from the list.


hth,

Doug

 

[Crivens]

Quoting from the man page:


I fail to understand how that could be hard to understand.

I have learnt to be careful with documentation. Since it is pretty clear what these lines say, they still do not tell you when/if exceptions apply. Or if they simply are not connected with reality at all.
So without trying it and confirming it myself I have only the man page to rely on - and those have lied to me often enough. But I must say the man pages of FreeBSD are right almost all the time, keeping out the "user experience" with other documentations (work related) is the hard part...