Solved [SOLVED] Portmaster / upgrade issue(s)

[Chris_H]

Greetings,

I'm struggling with an upgrade on a RELENG_8 server. I haven't made any attempt to use anything but defaults, as used with a regular install (defaults to package-tools pkg_, etc.). Anyway, I'm performing (attempting) the upgrade with ports-mgmt/portmaster. I do want to take a moment to thank @wblock@ for his short writeup on it. Thus far, it has failed to complete twice; once, because misc/kde4-shared-mime-info has been removed from ports. Which left me having to simply delete /var/db/pkg/kde4/shared-mime-info/. I was then able to get somewhat farther. But now it's failing at: x11/kdelibs4, which fails to finish building (exits with fatal error -- non-descript).

Has anyone thoughts on how to get around this? Or fix x11/kdelibs4? I've filed a PR.

Thank you for all your time, and consideration.

--Chris

 

[ShelLuser]

Re: Portmaster / upgrade issue(s)

First of all: which version of FreeBSD are we talking about exactly? Because you simply say "8" but there are several versions. stable/8, releng/8.3 (which is nearing its EOL) and releng/8.4. The reason I mention this is because if you're actually using releng/8(.0) then you're using a horribly outdated environment. Then it also wouldn't be very surprising that several ports no longer build properly.

It sounds to me as if your problems are caused by some lack of maintenance. For example; when looking at misc/kde4-shared-mime-info it was removed because it was no longer needed. Since there are no specific instructions in /usr/ports/UPDATING I somewhat assume that this gets set up by merely performing the required updates.

I also shudder at the idea of manually removing /var/db/pkg/kde4/shared-mime-info/. That's a recipe for disaster; just let the package manager handle all that.

So what exactly goes wrong? Please share the exact error you got, without it all we can do is guess. But even so; I can't help think that your best solution here might end up with a clean install. Note that I'm not saying this is the case; but I get the feeling that you made many changes to the system which will probably only cause more problems than solutions.

 

[wblock@]

Re: Portmaster / upgrade issue(s)

There are hardly any situations that need a clean install. Any really severe problems can usually be cured by deleting all installed packages/ports and reinstalling them. That's shown at the end of the portmaster(8) page.

But usually that is not necessary. If you're going to upgrade to 9 or 10 (10-stable has been very good for me), do that first. I recommend using source. After that, you'll have to rebuild all ports anyway, so do the procedure at the end of portmaster(8). I have an update for that which uses the new package tools:

1. portmaster --list-origins > ~/installed-port-list
2. Update your ports tree
3. portmaster -ty --clean-distfiles
4. (skip this step, portmaster --check-port-dbdir)
5. portmaster -Faf
6. pkg_delete -afy
7. rm -rf /usr/local/lib/compat/pkg
8. Back up any files in /usr/local you wish to save,
   such as configuration files in /usr/local/etc
9. Manually check /usr/local and /var/db/pkg
   to make sure that they are really empty (optional, I leave it alone)
10. Install ports-mgmt/pkg and then ports-mgmt/portmaster, remove both from ~/installed-port-list
11. portmaster `cat ~/installed-port-list`

 

[Chris_H]

Re: Portmaster / upgrade issue(s)

Greetings @ShelLuser, and thank you for the response.

First of all: which version of FreeBSD are we talking about exactly? Because you simply say "8" but there are several versions. stable/8, releng/8.3 (which is nearing its EOL) and releng/8.4. The reason I mention this is because if you're actually using releng/8(.0) then you're using a horribly outdated environment. Then it also wouldn't be very surprising that several ports no longer build properly.

No, I said RELENG_8 -- it's 8.4 - I made a special point of putting it in my SIG, so everyone would know.
RELENG_8, in fact means 8-STABLE, which today, is 8.4

It sounds to me as if your problems are caused by some lack of maintenance. For example; when looking at misc/kde4-shared-mime-info it was removed because it was no longer needed. Since there are no specific instructions in /usr/ports/UPDATING I somewhat assume that this gets set up by merely performing the required updates.

I also shudder at the idea of manually removing /var/db/pkg/kde4/shared-mime-info/. That's a recipe for disaster; just let the package manager handle all that.

It's reasonably maintained, in fact this particular server was installed only a couple of months ago. Was updated then also.
As to the "no mention of it in UPDATING". I couldn't say. I have no control over what is put into /usr/ports/UPDATING. I only know that portmaster said there wasn't any reference in my newly up'd ports tree. So I let portmaster reconcile it, by removing it as a dependency, and rm -r'ing it from /var/db/pkg.

So what exactly goes wrong? Please share the exact error you got, without it all we can do is guess. But even so; I can't help think that your best solution here might end up with a clean install. Note that I'm not saying this is the case; but I get the feeling that you made many changes to the system which will probably only cause more problems than solutions.

Your typical make failure. Given that it stems from Trolltech's QT, it isn't as "chatty/verbose". So, what I already shared in the OP, is all I was given to offer. I simply ran portmaster -a, and also portmaster -a --check-depends. Both fail on kdelibs4, in about the same place, and with equally terse output.
This has a great of development on it. I shouldn't be required to be psychic, or have clairvoyance, to use FreeBSD. I've been building, managing, maintaing BSD, and FreeBSD servers for over 20yrs. now. All I can say is that the increasingly "kludgy" requirements mandated in UPDATING, along with their increasing frequency, to manage an update on one's system. Aren't doing it's users, or those developing with it, any favors. It begins to look less reliable, as recent years go by.

Thank you again, @ShelLuser, for taking the time to respond. I appreciate it.

--Chris

 

[Chris_H]

Re: Portmaster / upgrade issue(s)

There are hardly any situations that need a clean install. Any really severe problems can usually be cured by deleting all installed packages/ports and reinstalling them. That's shown at the end of the portmaster(8) page.

But usually that is not necessary. If you're going to upgrade to 9 or 10 (10-stable has been very good for me), do that first. I recommend using source. After that, you'll have to rebuild all ports anyway, so do the procedure at the end of portmaster(8). I have an update for that which uses the new package tools:

1. portmaster --list-origins > ~/installed-port-list
2. Update your ports tree
3. portmaster -ty --clean-distfiles
4. (skip this step, portmaster --check-port-dbdir)
5. portmaster -Faf
6. pkg_delete -afy
7. rm -rf /usr/local/lib/compat/pkg
8. Back up any files in /usr/local you wish to save,
   such as configuration files in /usr/local/etc
9. Manually check /usr/local and /var/db/pkg
   to make sure that they are really empty (optional, I leave it alone)
10. Install ports-mgmt/pkg and then ports-mgmt/portmaster, remove both from ~/installed-port-list
11. portmaster `cat ~/installed-port-list`

Greetings, @wblock@, and thank you for your reply.
I'm not looking for 9, or 10, any time soon. In fact, I'm running ~60 RELENG_8 (8.4) servers now. I've got to take a l-o-o-n-g look at things before making all the changes required for that "leap". I am contemplating 11, tho. But I've been cobbling/honing a toolchain that has (until recently) worked/served me very well, for some 20 years. I'm uncomfortable with the frequency of changes occurring in FreeBSD. I'm going to need to either change my whole toolchain, and stay with FreeBSD, hoping it doesn't continue changing so much, and so frequently. Or fork a release, and "wind" it back, to say; a couple of years ago. Then work with that. Or build an additional server(farm) that is a sort of "pointy hat" that continuously builds versions of the ports tree, until it manages to create a STABLE, working copy of all the ports required on my servers. Sigh... I get tired, just thinking about it.

Thank you again, @wblock@, for taking the time to respond.

--Chris

 

[wblock@]

Re: Portmaster / upgrade issue(s)

The frequency of entries in UPDATING has actually decreased. Which is kind of amazing, considering that the long-stagnant ports tree is finally getting some very overdue work.

See the new Handbook section on Poudriere. It does just what you want, building various versions of ports in jails: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-poudriere.html

 

[Chris_H]

Re: Portmaster / upgrade issue(s)

The frequency of entries in UPDATING has actually decreased. Which is kind of amazing, considering that the long-stagnant ports tree is finally getting some very overdue work.

See the new Handbook section on Poudriere. It does just what you want, building various versions of ports in jails: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-poudriere.html

Greetings, @wblock@,
Recently, yes. But the recent ones, have been real killers, for me, anyway, and others that have shared their experiences with me. I don't want to come off disagreeable, or anything. It's not that at all. It's just that the changes are big ones, and they don't lend themselves to what the future lies, or if there will be more "radical" changes, away from those that have already been done.
I greatly appreciate your suggestion, regarding ports-mgmt/poudriere. That really could be a "game changer", and would work, for the here-and-now. But I fear other changes down the road, that either change it, or ones ability to use it to accomplish the same goals it currently satisfies.
On another note;
Well, after a few different "incantations" with ports-mgmt/portmaster, I was able to get kdelibs4, and several others installed (upgraded). But now I'm stuck on textproc/redland. It fails with "paraphrasing":

/usr/bin/ld : warning : libuuid.so.1, needed for /usr/local/lib/librasql.so, not found
(try using -rpath or -rpath-link)
/usr/local/lib/librasql.so: undefined reference to `uuid_generate'
...

Any thoughts?

Thanks again, @wblock@. I really appreciate, and value your input.
I'll have a look at the article you referenced.

--Chris

P.S. FWIW I used portmaster -a --check-depends
followed by
portmaster -a

 

[wblock@]

Re: Portmaster / upgrade issue(s)

Don't use portmaster -a unless you are up to date with all the entries in /usr/ports/UPDATING.

Please install sysutils/bsdadminscripts. Then run pkg_libchk -o. This will tell the name of installed ports that need to be rebuilt.

From the error, you need to rebuild textproc/rasqal and then the things that depend on it. portmaster(8) should do that automatically with portmaster textproc/rasqal. If it does not, add -r: portmaster -r textproc/rasqal.

I don't know what to say about changes to the ports system. It's a lot of big changes that have needed to be done for a long time. Why not set up a 10.0 system as a Poudriere test? I think it can run an 8.4 jail to create packages for your other systems. Using those packages would require switching to ports-mgmt/pkg on the 8.4 systems, but if you could do that and install all the packages in a few minutes, that would kind of reduce the pain.

 

[Chris_H]

Re: Portmaster / upgrade issue(s)

Greetings, @wblock@,
WooHoo. Whew, I think it's finally upgraded. Time alone, will tell whether the system's stable, or not. But, for sure, the ports tree is in sync with what's installed. So, the worst of the battle's over. Am only 2 days behind. It's weird to say, but it's a relief. I had feared, given the start, that it would end up much longer.
In the end, simply repeating the process with portmaster(), got them all updated. I had to tweak options, every once-in-awhile. I also had to keep track of some of the failed upgrades, so I could make portmaster() treat them as an upgrade, even tho, because of where the fail occurred, portmaster() had no record of them having already been installed. As to the errors I received, and spoke to here. It merely took persistence, and in a couple of cases, forcing a re-upgrade of dependencies to finally overcome the error(s). It appears to me, that sometimes the jobs portmaster initiates, can get ahead of the tasks they've delegated. ld(), and libtool(), for example. Many times, because of the insistence of pkg() to create a (pre)package, prior to install. ld(), and libtool() have to work double-duty to keep up with everything, when using portmaster. I was forced to insert NO_STAGE= yes in a couple of the ports, before they were ever able to actually complete successfully. I have to say, I really dislike the new pkg() system. So very often, the packages it insits on creating, are really just an exercise in futility, because you have no intent on using, or keeping the package it creates. So it's really a waste of resources, and CPU cycles, that could be better spent, even if you do have an over abundance of them. It's get's tedious having to assert -DNO_STAGE to overcome it. I haven't checked yet, but I'm hoping an assertion of NO_STAGE=true in make.conf(), sets it globally. Then it becomes a handy "switch", (en|dis)abling, as needed.

On another note. Given I finished last night, instead of much later, as I had feared. I took a chance, and dragged an old spare of the shelf, with the intent to experiment with ports-mgmt/poudriere. it's an old AMD 3Ghz X2 dual CPU (2 cores). I chose it, because it has reasonable throughput across the platters - sata @3-5 gb/s. So it seemed like it might make for a good "workhorse", for this type of application. Anyway, in my haste to get FreeBSD on it, so I could get to sleep. I mis-sliced the platter. I used my usual "server" slice configuration. But given that ports-mgmt/poudriere best utilizes /var for it's work. I'm gonna need to re-size the slice scheme. I often spin up a gparted CD to wipe platters prior to a BSD install. Altho (at least my copy) it doesn't support FreeBSD's UFS (ffs), it does know about it. Any reason I couldn't use it to re-size the slices? Is there a better application that will accomplish this? And yes, I will, of course drop to single user mode, and run fsck prior to setting them "live" in multi-mode.

As always, thank you very much, @wblock@. Your input has made all the difference.

--Chris

 

[wblock@]

Re: Portmaster / upgrade issue(s)

GParted has been very frustrating for me. It almost can do what I want, but then veers off in some other, unwanted direction. I rarely use it any more.

Use gpart(8). It mostly just does what it should.

If you are setting up a new system anyway, ZFS is reported useful for Poudriere.

I don't understand the complaint about unnecessary packages. If they really are unnecessary, they should not be installed. If they are build dependencies, install them once and forget about them. (We've seen people who insist on removing build dependencies and then complain that they must be rebuilt when ports need them.)

Do not use NO_STAGE like that. Ports that still need it should already have it set. If they don't have it set but will not build, they are broken, and a PR should be entered.

 

[Chris_H]

Re: Portmaster / upgrade issue(s)

GParted has been very frustrating for me. It almost can do what I want, but then veers off in some other, unwanted direction. I rarely use it any more.

Fair enough. Thanks. I have to admit, I've been frustrated by it, as well. But my frustration stems more from it's lack of support for FreeBSD. One day soon, I'm going to add the necessary support, so that it'll properly support FreeBSD. Then make a really lightweight desktop CD (LXD), that sports BSDparted, and everything else a FreeBSD recovery CD/DVD should have.

Use gpart(8). It mostly just does what it should.

A quick look at it seems like it's the plan.[/quote]

If you are setting up a new system anyway, ZFS is reported useful for Poudriere.

Yea, that's my understanding, as well. But I'm (as yet) only experimenting. So using UFS/FFS, will make for a shorter "test phase" (less overhead, and resources required).

I don't understand the complaint about unnecessary packages. If they really are unnecessary, they should not be installed. If they are build dependencies, install them once and forget about them. (We've seen people who insist on removing build dependencies and then complain that they must be rebuilt when ports need them.)

I may not have been clear enough here. I meant, that every time a port is installed, pkg() builds a "pre-stage" package, prior to actually installing the port; eg make install. I don't get it. Sure, it's nice to have a package of the port you've just made/installed, should you need a copy of it later, or when you want to deploy it on another system. But why can't I decide when, and when not, to make one?

Do not use NO_STAGE like that. Ports that still need it should already have it set. If they don't have it set but will not build, they are broken, and a PR should be entered.

Thanks. As to pr()'s; Indeed, I've sent quite a few. But some of the developers have insisted it's my fault, because I'm not using the new pkg() system. Forget the fact that RELENG_8 uses pkg_ by default, and that many administrators have chosen that release, so as to have have time to evaluate their needs, to "tool up", as time permits, to start using the new system in the future -- if they even decide to continue. Really. Not everyone has unlimited resources, and enormous development teams. So why are those without those resources treated like dirt, or second-class FreeBSD users, because we don't make the switch immediately. RELENG_8 is EOL June 30, 2015. RELENG_8 is distributed with the pkg_ system. Why aren't those who use RELENG_, supported?

Anyway, thanks again, for all your time, and effort(s), @wblock@.
Greatly appreciated.

--Chris

 

[kpa]

Re: Portmaster / upgrade issue(s)

Staging isn't tied to the new PKGNG system in any way. It's an independent part of the ports(7) mechanism that works the same with the new PKGNG and the old style packages. As said above, do not play tricks with NO_STAGE. It's not a user settable variable and it's only meant to be set in a Makefile of a port that hasn't been yet converted to staging. Once a port has been converted to staging, setting NO_STAGE will not work anymore because the Makefile is then written with staging in mind and many things are done differently.

 

[wblock@]

Re: Portmaster / upgrade issue(s)

Then make a really lightweight desktop CD (LXD), that sports BSDparted, and everything else a FreeBSD recovery CD/DVD should have.

http://mfsbsd.vx.sk/ would need a desktop, but otherwise is extremely light.

I meant, that every time a port is installed, pkg() builds a "pre-stage" package, prior to actually installing the port; eg make install. I don't get it. Sure, it's nice to have a package of the port you've just made/installed, should you need a copy of it later, or when you want to deploy it on another system. But why can't I decide when, and when not, to make one?

The old package system worked that way also, although it wasn't overt. It makes sense if you consider that there is only one system tracking installed packages and ports.

Forget the fact that RELENG_8 uses pkg_ by default, and that many administrators have chosen that release, so as to have have time to evaluate their needs, to "tool up", as time permits, to start using the new system in the future -- if they even decide to continue. Really. Not everyone has unlimited resources, and enormous development teams. So why are those without those resources treated like dirt, or second-class FreeBSD users, because we don't make the switch immediately. RELENG_8 is EOL June 30, 2015. RELENG_8 is distributed with the pkg_ system. Why aren't those who use RELENG_, supported?

The only answer I have is that the ports system needs to change to be able to meet the needs people have today, and there is an upgrade path by switching to ports-mgmt/pkg. It's an inconvenience, but less of one than a lot of other routine maintenance jobs (like upgrading the operating system for security, for example):

1. Install ports-mgmt/pkg
2. Run pkg2ng
3. Profit!

 

[Chris_H]

Re: Portmaster / upgrade issue(s)

Then make a really lightweight desktop CD (LXD), that sports BSDparted, and everything else a FreeBSD recovery CD/DVD should have.

http://mfsbsd.vx.sk/ would need a desktop, but otherwise is extremely light.

I had completely forgotten about mfsbsd. Thanks for the reminder!

I meant, that every time a port is installed, pkg() builds a "pre-stage" package, prior to actually installing the port; eg make install. I don't get it. Sure, it's nice to have a package of the port you've just made/installed, should you need a copy of it later, or when you want to deploy it on another system. But why can't I decide when, and when not, to make one?

The old package system worked that way also, although it wasn't overt. It makes sense if you consider that there is only one system tracking installed packages and ports.

"It makes sense if you consider that there is only one system tracking installed packages and ports."
I can't argue that. But that wasn't quite my point.

Forget the fact that RELENG_8 uses pkg_ by default, and that many administrators have chosen that release, so as to have have time to evaluate their needs, to "tool up", as time permits, to start using the new system in the future -- if they even decide to continue. Really. Not everyone has unlimited resources, and enormous development teams. So why are those without those resources treated like dirt, or second-class FreeBSD users, because we don't make the switch immediately. RELENG_8 is EOL June 30, 2015. RELENG_8 is distributed with the pkg_ system. Why aren't those who use RELENG_, supported?

The only answer I have is that the ports system needs to change to be able to meet the needs people have today, and there is an upgrade path by switching to ports-mgmt/pkg. It's an inconvenience, but less of one than a lot of other routine maintenance jobs (like upgrading the operating system for security, for example):

1. Install ports-mgmt/pkg
2. Run pkg2ng
3. Profit!

I'm not going to belabor this point any further. I understand that some believe the new pkg() system is the best thing since sliced bread. It may, in fact be great. But my issue is probably more with the way the "new" changes are implimented, see; forced, asserted; than anything else, I guess.

Anyway, I'm always grateful for your help, and input, @wblock@.
All the best, to you! Thanks.

--Chris