Very painful upgrade

I'm fighting with this machine I want to seriously upgrade from 8.2 to 9. We can call it a multi-purpose machine I do use for very many private stuff. It is not only a file server, it is also a desktop for some jails and some video editing, it also have a SSH server becouse I need to administer it from other locations, often very far away locations, sometimes it serve some web pages in emergency cases.

It is perfectly fit for my needs, it runs on ridicolous hardware (this is the reason becouse I choose FreeBSD over other OS: it runs smooth on affordable, old, cheap hardware) and it is solid.

Said this, my main desktop for multimedia purposing with latest hardware and technology is a completely and continuosly hardware updated machine running a Windows 7 Ultimate OS. IMHO, great OS, the best Microsoft did, also if I also used to administer NT4 servers like 20 years ago totally solid.

Now, my main Desktop updates and upgrades are so simple and painless. Windows Update do a perfect job and when I had to Upgrade from Vista to Win 7, the DVD served the purpose very very well.

Now I want to update this important FreeBSD machine it served and it still serve me very well, I want to do a clean job then... and here is the problem, the path to a easy upgrade it don't exist on the FreeBSD world.

After posting something here. I decided the only way to go for package updating was the one explained in the Portmaster man page. With some modifications of the command line strings, I made a package list with the --list-origins option and then deleted all the packages and reinstalled them with this modifiles string:

[cmd=]portmaster -D --packages-if-newer `cat /root/installed-port-list`[/cmd]

This string seems to work quietly well, it download packages when same or new then ports and then it compile the others.

Considering I do have 1102 packages installed in this system, the time for the upgrade is totally not acceptable. I think I have like a 2 days down time. Too much.

Considering my main file server is a Archlinux machine I do update in minutes using my 100 Mbit Internet connection, I'm not sure if it's just me or seriously updating a FreeBSD machine is a titanic job.

I talk about packages obviously becouse the base system can be upgraded quietly fast using freebsd-update.

Considering I do have other three FreeBSD waiting for an upgrade, do you have any idea for a faster path?

Honest, the Vermaden Howto, very good and very completed, it is not what I call a rolling upgrade, it is even painfull compared to the path I choose.

piggy said:

Considering I do have 1102 packages installed in this system, the time for the upgrade is totally not acceptable. I think I have like a 2 days down time. Too much.

Click to expand...

It only takes 10 minutes here.

Considering I do have other three FreeBSD waiting for an upgrade, do you have any idea for a faster path?

Click to expand...

Download the packages and store them locally or build your own packages from ports. Export those packages using NFS and you're good to go.

As I said, it only takes about 10 minutes. I do a pkg_delete -a and install the packages I need from my local repository.

He does have a point. Rebuilding all the ports even on a powerful desktop is a long procedure.

Building the packages on a different box means that you need a box dedicated for this. You also need to remember which options you chose in the first place. Maybe, copying /var/db/ports can do the trick.

@Piggy, if you don't have a different box, consider installing misc/compat8x. That way your system will still be functional during the rebuild procedure. If rebuilding ports is too much then you can always install packages.

gkontos said:

Building the packages on a different box means that you need a box dedicated for this.

Click to expand...

You can build them in a jail. No need for a physically different box. And building in a clean jail also doesn't interfere with your current installations.

You also need to remember which options you chose in the first place. Maybe, copying /var/db/ports can do the trick.

Click to expand...

Do it often enough and it'll come automatically. You can also add all sorts of building options to /etc/make.conf. Copying /var/db/ports would definitely do the trick too.

SirDice said:

You can build them in a jail. No need for a physically different box. And building in a clean jail also doesn't interfere with your current installations.

Click to expand...

True, however a slow machine will still take its time for this!

Anyway, for me it is not a problem. The only time I ever had to install a package in FreeBSD was net/cvsup-without-gui and that's long gone!
I can understand the frustration of some people though.

gkontos said:

He does have a point. Rebuilding all the ports even on a powerful desktop is a long procedure.

Building the packages on a different box means that you need a box dedicated for this. You also need to remember which options you chose in the first place. Maybe, copying /var/db/ports can do the trick.

@Piggy, if you don't have a different box, consider installing misc/compat8x. That way your system will still be functional during the rebuild procedure. If rebuilding ports is too much then you can always install packages.

Click to expand...

Thankx gkontos for understanding what I want to say. When Dice say he can upgrade a 1.102 packages machine in 10 minutes, it is like a funny joke, then it is not funny for me try to do a good "upgrade" job in "modern OS" attitude.

It makes me think, if someone want to use pure FreeBSD, could be better to use the classic, old Unix attitude: save the home directories and reinstall.

Becouse if you want to do a proper upgrade, like you also said, it is quietly different compared to what Dice said and the Vermaden guide confirm the superficial Dice attitude is not the correct way to do things like the way I want to do them.

The "portmaster way", like explained in the Man page is probably the way to go. Obviously save the configuration files, save the list of packages, cleanup every bits of /usr/local folder, and "upgrade" everything rebuilding all the stuff with the new world. To remove any bits of old system, it is also suggested to delete everything related with the old packages databases, but the list of installed packages.

This is time consuming: if I just install packages with pkg_add, I still need to recompile everything in a next step; if I rebuilt everything the machine will be down for a lot of use also if - like you correctly suggested - I do install compat8. The desktop will be down for long time, before it is ready for my use.

And BTW, it is not even straightforward to automate the install of precompiled packages admitting you want to install them and think you have to go to another step to update everything. For what I know, I should install them manually, I don't think pkg_add accept some batch job like Portmaster do.

SirDice said:

It only takes 10 minutes here.


Download the packages and store them locally or build your own packages from ports. Export those packages using NFS and you're good to go.

As I said, it only takes about 10 minutes. I do a pkg_delete -a and install the packages I need from my local repository.

Click to expand...

And you think all this require 10 minutes? Come on... And how can you properly automate the "upgrade" of all the packages without some batch job like the one explained in Portmaster Man, honestly the only solution with some "sense" I read related with the subject?

Everything, and I say EVERYTHING, is time consuming and your solution needs days also if you own another machine dedicated to it (or a jail, but be serious: how can you run a jail in a machine u are upgrading).

@piggy ,

Like I said I understand your frustration. What SirDice suggested is a good alternative. It is something that I might do in the future, a good idea.

However, it is definitely not a 10 minute process and certainly not for someone who is not that experienced with FreeBSD. Sometimes, we tend to forget how difficult some things look to other people because we simply feel comfortable doing them.

I really think that FreeBSD is not for someone who just wants to plug and play literally. I also like this philosophy although I believe that there is still room for progress.

Having said that, I have to share my experiences with other OS that support a full turn key upgrade like Windows and some Linux distros. Well, it has not been that good. I have very traumatic experiences. Not so many with desktops but a lot with servers.

So, if I have to choose. I will choose the FreeBSD way. I might have to strangle and get involved, get my hands dirty, but in the end of the day the job will be done.

Best Regards,
George

My guide is not a cure for all these problems, it just tried to make it less PITA than it actually is, but it's still far from perfect. It's unbelievable for me that you will have to use a dedicated machine or virtualization with several daemons running 24/7 just to have up to date packages for your laptop. Such an environment may be ok for the enterprise, you have many FreeBSD servers, you want to have the same environment on all of them so it's quite reasonable to have your own package server, but not for a casual desktop/workstation/laptop/server ...

Older, when I 'tried' to use *-RELEASE, I used an even different method, keeping info what packages I have had installed, removing all of them and adding them again from *-RELEASE packages + building the missing ones (lame for example), like that:

Little update on the work in progress. Refer to message 1 of this thread for general info.

It is now 36 hours the machine is down compiling packages for the upgrade Portmaster way.

The packages to be compiled are quietly half of the 1.102 total becouse the option I specify for Portmaster "to use the packages when available" saved a lot of recompile time, I think. So half of the job seems done (I still have to wait to know if it works at the end), there are still quietly 500 packages to recompile and it is not so straightforward, even if the base system was just base 9.0-RELEASE system (upgraded from 8.2 using freebsd-update and that went upgraded in like 15 minutes or so) and I was supposed it was like a joke reinstall programs in a situation like this. I was wrong.

Before running the Portmaster command indicated in the first post, I had to install sources. Installing from dvd, for some reasons (?) I still have to understand, do not worked. I had to install from cvsup, it was the less painfull way, then I still wanted, on that slow machine, like half an hour.

The system is a quietly old 32 bit AMD System. Processor is a 1800+, so not that bad. Installed RAM is 4 GB.

After satisfy this prerequisites, I started Portmaster with the string you can read in the first post.

I was wondering everything was in the right way, quietly done, just the time needed to wait for completion. I was wondering the machine was completely clean, so no problem during the install/compile. WRONG! I had to solve some dependencies problems even if the system was totally virgin. Definitely, you can't automate nothing with such a complex system like a FreeBSD system is.

Also in the proceeds there were (and maybe still are) some dependency problems. How that can be is behond me considering this is a "virgin" system, then there are reported problems.

I saved all the logs in the process for later analysis (I piped the output on a txt file). For now I can see, during the compile/package install work, the package database become more and more fragmented and Portmaster suggest to run the --check-depends, even if it still continue to install and compile fine (for now, after like five stops I had to manually resolve).

In the same time I upgraded another machine the Unix way: reinstalled everything and restored the homes and the old configurations files. It takes a lot less time, then the system are not exactly what it was before, some packages are missing, some are installed and I don't want them. So I have to loose some time for that machine too, removing and adding stuff, and the system is at DVD updates level, so I should rebuild ports if I want it up to date (actually, after all this pain, I DON'T WANT it at up to date level).

Conclusion: FreeBSD is definitely not a rolling release and you ALWAYS have to be prepared to loose time to tweak it and if you want to be productive you have to schedule just one update like in a year, maybe just fixing kernel and security problems in the meantime.

I think this is the way I go for the future. And, like I did quietly 17 years ago, I will minimize the use of FreeBSD just for the job it do better, and definitely use more rolling Linux: quietly same stability, just new kernels start to become less compatible with old hardware. Yes, maybe I should put my old hardware in the garbage and buy new systems, then I'm not sure if I can afford this. Like I can't afford to buy Microsoft or Apple licenses for my small businness and for my personal use.

I share with FreeBSD developers the laptop: like them, I do use a cool Macbook. Just curious if they run MacOS like I do, or they like FreeBSD on it.

PS: I saw pictures of FreeBSD conventions where all the developers show shiny Macbooks

Maybe it is just me but I never had to deal with dependencies when I had to recompile all my ports. I always read /ports/UPDATING before I update my ports. I also always try to keep my ports up to date. Usually, all it takes is a:

[CMD=""] #portupgrade -af[/CMD]

Unless of course I decide to upgrade perl while I am at it. Having said that, I am curious about your definition of rolling releases.

Examples:

Hardcore:
Windows NT -> Windows Server 2008

Softporn:
CentOS 5 -> Centos 6

gkontos said:

Maybe it is just me but I never had to deal with dependencies when I had to recompile all my ports. I always read /ports/UPDATING before I update my ports. I also always try to keep my ports up to date. Usually, all it takes is a:

[CMD=""] #portupgrade -af[/CMD]

Unless of course I decide to upgrade perl while I am at it. Having said that, I am curious about your definition of rolling releases.

Examples:

Hardcore:
Windows NT -> Windows Server 2008

Softporn:

CentOS 5 -> Centos 6

Click to expand...

About rolling releases: http://en.wikipedia.org/wiki/Rolling_release

Believe it or not, my actual main desktop system main hard disk (actually a SSD) with Windows 7 Ultimate was formatted... in 1996! It used to be a NT4 upgraded to 2000, then XP, then Vista and now Ultimate, after a "conversion" between 32 bit and 64 bit version. And memory don't serve me well in this case, becouse it is too much time, then I wondering there were 3.11 on that disk before NT4, and NT4 was the first upgrade of the list. I still can find files dated 1994/1995 on the disk! Yes, I'm old :-(((

Well, obviously I cloned that partition like 20 times to change hard drives, I also changed any type of hardware but the core of the OS is still a NT4 upgraded!

My actual file/multimedia server is a Archlinux machine totally up to date using pacman.

My multimedia box (not the server) is a dedicated Sabayon (Gentoo derived) machine binary updated, totally "rolling", my jolly - as explained in the first post - is this venerable FreeBSD machine I try to update.

I have for my business other FreeBSD: a web server (with LXDE as interface and Apache for web serving) and a file server/backup customized for that and two PC-BSD desktops for office work (to save on licenses and becouse they work beautifully . My Macbook is proudly (and expensive) Macbook.

NT4 --->>> Server 2008 I didn't try it.

Centos is not a Rolling release.

About my problems: maybe Portupgrade is better, I don't know what to say, with Portmaster that is my actual experience. This machine was formatted on FreeBSD 7 and always binary upgraded with time consuming compile and never had a cleanup. Then, now, was completely clean, considering I deleted EVERYTHING but base and source system.

PS: there was no Pearl and no Compat when I started the Portmaster command: just as I said base system, source and Portmaster hisself installed via pkg_add.

This may help in some situations, maybe in the case, maybe not.

(Pardon the typos of course). Then,
. You should be prepared for package-install errors maybe if all dependencies are not upto date.

Or, you have a file (filtered result from which has ports that
could be upgraded.
That method may be of use; one could even install from a thumbdrive relatively quickly to an entire server farm if planned in advance a little bit.

Update related with this painfull upgrade.

After sixty (60) hours (not your 10 minutes, Dice!), the work is quietly done. Like you can read in previous post, I adopted the Portmaster man advise: saved the list of installed packages and deleted everything. I decided to change the Portmaster behaviour mixing packages with ports to speed up things (and still it needed 60 hours even if half of the packages were installed by packages repos) to be done.

After six or seven stops (on a totally cleaned system!) to sort dependencies problems, after 60 hours the system completed the run.

Not everything seems done the right way by Portmaster: after the first reboot I saw mouse and keyboard not working. I had to reset the system and recompile keyboard, mouse drivers and Xserver in hisself using SSH from a working system. That fixed this first problem. The problem was probably generated by the fact, to speed up things, I mixed packages with ports, then it didn't happen in a correctly maintained OS.

After another reboot I restored the configuration files backup. Home directories were always there, also if I had a backup for security reasons.

Now the system, with brand new KDE loaded as expected retaining all my previous settings, wallpaper and icons configuration but... when I click Exit and I choose between Reboot, Shutdown or Terminate the Session nothing happen :-(

Before the "upgrade" everything was working totally fine. So now I have to spend some time looking for this problem. :-( Before the "upgrade" (between quotation marks becouse this is not something you can call "upgrade", too painfull) it used to work perfect and my "upgrade" - like I remember again - was totally clean.

Other problem: Dolphin, the file manager I do use especially for browsing my 26 machines on the network, do not allow me to browse my remote computers anymore :-( I want to be clear: Samba is up and running, I can do everything from command line, but... clicking on the Network option in Dolphin do not produce any results, it is like if I didn't click on it, it is dead. Same happen to the external drive connected to the machine: Dolphin see them, load the icon, then when I click for open nothing happen :-((( I have to sort out this one too.

So, I try my alternative, the good Gnome Nautilus file manager. I do use some Gnome programs on my KDE desktop and Nautilus is one of this. I clicked on the icon and nothing happened. I fired on a Terminal window and I started Nautilus from command line just to get this error:


A preliminary check report the libxcb* packages seems pretty installed, then there is no trace of libxcb-so-0 in both /usr/lib and /usr/local/lib on this system. Now I need to investigate where is that library and why it wasn't installed.

Related with Samba. I used Samba36 before the upgrade. Samba34 is too old, absolutely out of time and it can't browse and authenticate shiny Windows 7/Server 2011 machines :-((( So I had to install Samba36. Before the upgrade was all ok and yes, it seems still all ok now too but... I have TWO samba packages installed now, 34 and 36! Why 34? It wasn't on the list of packages to reinstall and I never said to Portmaster to load Samba34 too. So why it happened?

Overall, a very painfull experience: I know, I'm not the 10 minute man mr. Dice, then honest, I'm not the least of the bunch, I'm with FreeBSD from version 4 (on and off) I think, so I know the system and I can say it is absolutely not ready for a not painfull desktop experience "upgrade".

I said the problem related with the need of a binary packages updated repositories some times ago and I was slapped with a: "if you want that, do it yourself". Ok, I wont pretend nothing, I want this be clear too, FreeBSD is developed on voluntari basis so nothing to pretend, but... they should think to have serious binary updates and upgrades for maybe RELEASE. It is my opinion.

The reason becouse I mix some FreeBSD to Windows and Linux machines is explained in previous posts, and I also enjoy FreeBSD and the ability to craft a system exactly for my needs.

Thats all. I will report if and how I sort this problems arise after this painfull "upgrade".

* After some googleing I saw this is a known problem, a bug in Portmaster and Portupgrade too. To fix it I should run "ldconfig -R" and then "portmaster -R -r xcb-util-0". Guess, this last command will rebuilt... pretty much everything! It is just crazy.

** UPDATE: dumb and boring package management? Dirty trick! I fixed the xcb-util missing libxcb-aux.so.0 simply getting this file from another FreeBSD (clean install, upgrade old Unix way) machine and copy it to /usr/local/lib. FIXED! Nautilus and other packages requiring it works just fine. Now, when I have some spare time, I will try to hack the Dolphin problem too and I will forget to "upgrade" FreeBSD until 2013 come

You made a mistake by going with portmaster. You can't compare speed of binary updates with compiling from source. Of course if will take days on slow machine. Instead you could've went with something like Vermaden suggested. Get a list of all packages, delete all packages and empty relevant directories, and just do binary update.

I had to do something similar and I had 1120 packages - KDE, Gnome, Xfce, every other possible DE, lots of other programs.
Not counting pre-downloading packages from ftp, which I don't count as process of upgrade as it was done separately, it took less than an hour to finish it all with pkg_add.

I didn't encounter any major problems with this major upgrade (which really you only do when going to new X.0-Release). The minor ones were dealt with swiftly and with style.

In a nutshell, you are the one who made a wrong choice. Not trying to offend you just saying what it is. The only barrier in making this work is your knowledge of what you need to do.

bbzz said:

You made a mistake by going with portmaster. You can't compare speed of binary updates with compiling from source. Of course if will take days on slow machine. Instead you could've went with something like Vermaden suggested. Get a list of all packages, delete all packages and empty relevant directories, and just do binary update.

I had to do something similar and I had 1120 packages - KDE, Gnome, Xfce, every other possible DE, lots of other programs.
Not counting pre-downloading packages from ftp, which I don't count as process of upgrade as it was done separately, it took less than an hour to finish it all with pkg_add.

I didn't encounter any major problems with this major upgrade (which really you only do when going to new X.0-Release). The minor ones were dealt with swiftly and with style.

In a nutshell, you are the one who made a wrong choice. Not trying to offend you just saying what it is. The only barrier in making this work is your knowledge of what you need to do.

Click to expand...

The problem is not me, the problem is the loosy FreeBSD Package System and build system.

Both Portmaster and Portupgrade are buggy and they do things in different way, and they take not month, ages to do things, and often they do it wrong, they always need operator intervention, they are unable, often, to make convenient choices, no one is realiable, IMHO.

If you read all the thread, you can read even on a clean system with no packages like mine, Portmaster was unable to automatically sort all the dependencies problem automatically.

On another box I did a simple binary upgrade using Package Add in Unix style, just saving the conf files manually and the home dirs. It was fast, then now the system is, and it will stay, at the binary package repository DVD level. You should know binary repository in FreeBSD are updated very rarely, often just when a major releasead come out.

So, you don't try to justify actual FreeBSD faults saying it is a user fault: actually it is a OS fault, IMHO, a big one like explained, on this and other threads. It make a shiny OS look like a geeky painfull experience.

So, you want the speed and simplicity installation of binary packages, yet instantiations freshness of ports the hour they are updated. Over 1000 packages at that.

Looks like OS issue to me.

On a side note, these days I compile pretty much everything, but while I was using -STABLE repository, packages were rebuilt every month or so.

more feedback

If I can chime in here, I've been using FreeBSD for ten years now and whether it's a headless box or a user workstation with KDE, there will come a time when the systems down for a week or so while I work through an upgrade. KDE3/4 is usually the worst ... errorless portmaster -ad with kde is rare and there's no way around the amount of time to rebuild these ports. A headless server system doesn't take nearly as long to upgrade via building ports because it doesn't have as much software to update.

Using packages seems to work for a while but in my experience over time they break down ... as long as I can get portmaster -ad to work through without errors, I can keep things working using compiled ports not packages.

I think if you were going to provide clients with FreeBSD based workstations in a professional/production environment, the only way to do it without large downtimes is to compile on a separate box and do binary upgrades via copying disc images. This process would be nice to see in documentation as I have trouble with simple concepts like obtaining and using a list of the make config options I have selected for each of the ports I've complied so I don't always have to remember what the options I selected were when wiping and reinstalling large collections of ports (which is sometimes necessary).

There always seems to be a chance of getting a long down time situation when doing an upgrade with portmaster -a but overall, It seems better to do your portsnap fetch update portmaster -a every week. Usually this can be done overnight or in a few hours. If you wait to long, there will be too many ports to update overnight or in off hours. Also, if there are problems, it's easy to locate the problem because there are fewer ports to work with.

User ports Updating has never been a good guide for me as to whether a port update process is going to have complications or take a long time. I don't know bash enough to make a script that would search /usr/ports/UPDATING for each of the ports I have and then display only the information relevant to the ports I have installed. UPDATING is a primary reference for fixing problems when a port doesn't build correctly.

In contrast to many ports, I currently just do ninety-five percent, noting failed builds on paper, and the reason. One can use one quicker CPU to generate the .tbz, often not too slowly with portmaster, then copy them to a thumbdrive
I've posted in other threads how to use the thumbdrive to installworld from it...

[Sorry for any incompleteness or typos...]

piggy said:

The problem is not me, the problem is the loosy FreeBSD Package System and build system.

Click to expand...

The world of FreeBSD is changing, please check pkgng. PC-BSD has been directed to rolling release in future, please check PC-BSD Status Update and Future Plans

piggy said:

Both Portmaster and Portupgrade are buggy and they do things in different way, and they take not month, ages to do things, and often they do it wrong, they always need operator intervention, they are unable, often, to make convenient choices, no one is realiable, IMHO.

Click to expand...

Both Portmaster and Portupgrade are not binary update tools

piggy said:

If you read all the thread, you can read even on a clean system with no packages like mine, Portmaster was unable to automatically sort all the dependencies problem automatically.

Click to expand...

May be it is the time for you to try ports-mgmt/tinderbox or ports-mgmt/poudriere for building ports

piggy said:

On another box I did a simple binary upgrade using Package Add in Unix style, just saving the conf files manually and the home dirs. It was fast, then now the system is, and it will stay, at the binary package repository DVD level. You should know binary repository in FreeBSD are updated very rarely, often just when a major releasead come out.

Click to expand...

The future of rolling-release style PC-BSD by pkgng packages manager is your answer. PC-BSD Status Update and Future Plans

piggy said:

So, you don't try to justify actual FreeBSD faults saying it is a user fault: actually it is a OS fault, IMHO, a big one like explained, on this and other threads. It make a shiny OS look like a geeky painfull experience.

Click to expand...

Both Ports and Package are not a part of FreeBSD OS, they are just tools for you to install 3rd parties software into FreeBSD OS.

I use ports-mgmt/poudriere to compile all 990 packages for my desktop laptops by an AMD PII X4 2.5G machine including xorg, qt4, kde, firefox, apache-openoffice. It took about 11 hours to complete.

Topic was 14 months old, you may be doing CPR on a very dead horse.