Ports need stability

[gpw928]

Hi,

I thought of telling a tale of woe about how I waited two years to upgrade to 8.0, and then libpng smashed the ports tree in the first week, and hurt me so bad... Instead, I'd like to appeal to the core team, because the problem is so obvious.

A naked o/s can't do much without applications. Ports are therefore critical to the success of FreeBSD. For a stable production environment, ports also need to be stable.

Ports are not stable. On the contrary, the ports tree is simply out of control.

Letting the hackers (traditional meaning) into the production code base is like letting the fox into the hen house.

I mean no disrespect to the people who labour there, but I can't think of a time in the last few years (probably since KDE and Gnome started) where the ports tree has not been causing me (semi-continuous) grief.

I don't want the latest features. I do want my systems to run reliably!

May we PLEASE have a stable version of ports that is not broken almost daily?!

--
Phil

[cajunman4life]

Hi Phil,

Please, don't take me as being combative or anything, that is not my intent.

I have a production server I manage with FreeBSD (right now 7.1, but we're evaluating an upgrade path to 7.3 before January when 7.1 loses security support). 3rd party apps are built/installed from ports system.

I've only been bitten once by something, and it was when I blindly upgraded dovecot. Turns out I needed to make some changes in my config file before it would start back up again. Other than that, I've had no problems with the ports tree.

I routinely go a few weeks between upgrading any of the software to it's "later version" (unless of course there is some security issue which requires an immediate upgrade). This policy has worked fairly well. Nothing says you have to update a port every time someone commits a change to the ports tree.

I quite like the fact that ports and base are so separated. For example, if I were using Debian, and let's say I was running an older version (let's say 4.0 for the sake of argument), all the 3rd party apps would be tied to that release. Now, while I understand there is a good reason to do something like this, I much prefer to be able to run the latest (if I prefer) version of package X while keeping my base a few versions back from the latest and greatest. This is just my personal preference.

Here I've gone an rambled on, I'm sorry. Needless to say, the ports tree can be as stable or as broken as you want it. Using csup, you can even get an older ports tree if you wish. You don't have to have the latest and greatest, which may or may not be broken.

[crsd]

Why do you update installed ports then if you don't want new features and only want stability (except for security fixes)?

[gpw928]

Hi,

I don't care to justify my need to occasionally install new ports on a production system. It happens. That is enough.

The current ports tree has been compromised for weeks because of just one update (to libpng). Make no mistake, the ports have been seriously broken. Yes, the ports maintainers are labouring valiantly to fix it. However...

I am suggesting that FreeBSD needs to embrace the principle of having a STABLE version of the ports tree guaranteed to be free of incompatible dependencies. This is hardly a revolutionary suggestion, as the principle is already applied to the o/s and core utilities.

Cheers,

--
Phil

[phoenix]

I thought of telling a tale of woe about how I waited two years to upgrade to 8.0, and then libpng smashed the ports tree in the first week, and hurt me so bad... Instead, I'd like to appeal to the core team, because the problem is so obvious.

A naked o/s can't do much without applications. Ports are therefore critical to the success of FreeBSD. For a stable production environment, ports also need to be stable.

Ports are not stable. On the contrary, the ports tree is simply out of control.

Letting the hackers (traditional meaning) into the production code base is like letting the fox into the hen house.

I mean no disrespect to the people who labour there, but I can't think of a time in the last few years (probably since KDE and Gnome started) where the ports tree has not been causing me (semi-continuous) grief.

I don't want the latest features. I do want my systems to run reliably!

May we PLEASE have a stable version of ports that is not broken almost daily?!

If you don't update the ports tree, you have a perfectly stable and unchanging tree to install apps from. ;)

And, if you don't want the hassle of installing from an updated ports tree, then use packages.

Switching to "a stable ports tree" means we end up with the "debian way" where you are stuck the same versions of software for 3 years, while the rest of the world uses the fun stuff.

The ports tree is what you make of it. Just because there are updates to it daily, doesn't mean you should be updating everything on a daily basis.

[phoenix]

I am suggesting that FreeBSD needs to embrace the principle of having a STABLE version of the ports tree guaranteed to be free of incompatible dependencies. This is hardly a revolutionary suggestion, as the principle is already applied to the o/s and core utilities.

;) And, what are you offering to help bring that to fruition? ;)

[gpw928]

Hi Freddie,

If you don't update the ports tree, then the tarballs you need will eventually become unavailable. BTDT.

I suggested a "stable *version* of ports". This does not preclude the current arangement. It is an additional option which eliminates the bleeding edge breakages where different ports have incompatible dependencies.

--
Phil

[gpw928]

;) And, what are you offering to help bring that to fruition? ;)

OpenSUSE looks good :e

--
Phil (FreeBSD since 2.0)

[cajunman4life]

OpenSUSE looks good :e

--
Phil (FreeBSD since 2.0)

Heh... good luck with that one :OOO

[vermaden]

And, if you don't want the hassle of installing from an updated ports tree, then use packages.


... and stay with unsecure/valuable system becase pacakges were built on 'RELEASE day' and after that time some security holes have been found.

You can of course rebuilt them, but You will have to also rebuilt dependencies, which means broken other packages that now wants to use jpeg-8 instead of jpeg-7 (en example), so in the end You will have to rebuild ALMOST ALL installed applitacions, just to fix one or two programs that have holes:

% portaudit -a
Affected package: curl-7.19.6_1
Type of problem: curl -- libcurl buffer overflow vulnerability.
Reference: <http://portaudit.FreeBSD.org/c8c31c41-49ed-11df-83fb-0015587e2cc1.html>

Affected package: gd-2.0.35_1,1
Type of problem: gd -- '_gdGetColors' remote buffer overflow vulnerability.
Reference: <http://portaudit.FreeBSD.org/4e8344a3-ca52-11de-8ee8-00215c6a37bb.html>

Affected package: linux-f10-pango-1.22.3
Type of problem: pango -- integer overflow.
Reference: <http://portaudit.FreeBSD.org/4b172278-3f46-11de-becb-001cc0377035.html>

3 problem(s) in your installed packages found.

You are advised to update or deinstall the affected package(s) immediately.



@gpw928

+1

[harishankar]

There is definitely a problem with the ports approach which is why I think there needs to be a more comprehensive approach to packages which keeps packages up to date with no dependencies on ports at all. Otherwise I agree with the OP that you are forced to eventually upgrade everything to prevent dependency breakage and stay on the bleeding edge ports tree. Also bear in mind that compiling an upgrade for a heavily laden desktop system from ports alone can take several days.

I wrote a little about this problem myself. Mind you, I am not dissing FreeBSD but merely stating my opinion. In my own experience the ports tree is TOO bleeding edge for some people while packages are quickly outdated where they end up being at least one major version behind the upstream versions. There is space for a happy medium (a somewhat similar approach to Debian -testing for instance)

http://harishankar.org/blog/entry.php/why-i-039-ve-gone-back-to-debian

On Debian -testing I find many packages more up to date than on -STABLE for instance. I know a lot of people love ports but the gap between outdated (and quite incomplete selection) of packages on -RELEASE and even -STABLE and too-bleeding edge ports needs to be filled.

It's not an easy problem to fix, but I think manpower required for maintenance of so many branches is what is the issue here.

[harishankar]

Also regarding the advise on using an "older ports tree". What about the handbook that says this?

Warning: Before installing any port, you should be sure to have an up-to-date Ports Collection and you should check http://vuxml.freebsd.org/ for security issues related to your port.

A security vulnerabilities check can be automatically done by portaudit before any new application installation. This tool can be found in the Ports Collection (ports-mgmt/portaudit). Consider running portaudit -F before installing a new port, to fetch the current vulnerabilities database. A security audit and an update of the database will be performed during the daily security system check. For more information read the portaudit(1) and periodic(8) manual pages.

Eventually things will definitely break if the ports tree is not updated.

[kpedersen]

As far as I see it,

If I am on an offline pc, I use RELEASE packages (because security is pretty low on my agenda when disconnected from the internet)

If I am on an online pc, I use STABLE packages which AFAIK usually have similar versions to Debian. (btw, Debian is not renowned for its up to date packages. That is one of the many reasons for ubuntu.)

If I need the latest stuff, then of course I need to compile from source. At least Ports makes it quite trouble free. The reason why I do not see the need for a "stable" ports tree is that it is unlikely you would want to be compiling old versions of applications.

Also, does the ports tree included with a RELEASE dvd, not allow you to use the dist files hosted on the freebsd ftp? Just copy them into the dist folder or set the respective environment variable to the distfiles folder on the ftp.

Although admittedly, I do also like to compile source outside of the ports tree. For example the latest mplayer, SDL, Glut because it gives me a bit more freedom.

[oliverh]

@vermaden And Debian, Ubuntu, Suse, Fedora etc. pp. don't have lots of problems with broken packages, wrong configurations etc.? Sometimes you have to wait several weeks if you don't fix it yourself. I'm using Linux too, but I don't know a single package management system that has got less problems than ports. There are even some more problems,which I can easily fix whith ports. The only problem is time.

The community can easily change this problem: contributing money or contributing code and similar knowhow. I have great respect of the port commiters, it's a plethora of work they're doing.

[harishankar]

@vermaden And Debian, Ubuntu, Suse, Fedora etc. pp. don't have lots of problems with broken packages, wrong configurations etc.? Sometimes you have to wait several weeks if you don't fix it yourself. I'm using Linux too, but I don't know a single package management system that has got less problems than ports. There are even some more problems,which I can easily fix whith ports. The only problem is time.

The community can easily change this problem: contributing money or contributing code and similar knowhow. I have great respect of the port commiters, it's a plethora of work they're doing.

I've used Debian for more than 7 years and never had even a single issue with broken packages except when I occasionally played with "unstable" or installed third party packages without proper dependency checking. So I think you're not accurate there by clubbing several Linux distributions together. Debian's packaging policy is one of the best in the industry and worth emulating even if you happen to dislike Linux in general.

And to add to that, binary package selection in FreeBSD is nowhere near as comprehensive as in Linux.

Second issue is that I've found by personal experience that Debian -testing is more up-to-date than FreeBSD -STABLE, but I do realize that porting stuff does that time.

I think acknowledgement of certain issues will help the FreeBSD community grow. I am not saying Linux is perfect, but in the above particular aspect Linux is better than FreeBSD.

The other issue is that using packages only in FreeBSD is a bit of a pain because of lack of comprehensive package managers (pkg_add and other pkg_* tools pkg_upgrade are not good enough, I'm afraid).

[jb_fvwm2]

More upstream (community) hardware, more packages
readily available. Howsoever I usually use ports.
....
Say you have 1000 ports installed but actually use
20 weekly, and 100 ports are dependencies. Then
the problem becomes downtime in those 20, while
major .so. bumps in dependencies occur.
(png, jpeg, gettext, gtk20, ...)
If you plan ahead, you can delay upgrades til you
have a few days to do it all from the console. I
use /yell/ and multiple tty's (and can use
lynx, w3m etc if X is not avail.).
....
With practice it is not so bad. (I have recently
cobbled together a few .sh to make the rebuilds
take about half as many hours as it would usually.
Unfortunately not production-ready, still beta
those .sh)
....
Furthermore, one can
"upgrade everything in March"
"ignore all upgrades 5 months"
"upgrade everything in September"
and repeat as necc.
..........
ending this particular post in case I am
not paying enough attn. to the previous
posts.
....

[achix]

The issue with the ports IMHO is there.
But i think FreeBSD tries to do the best it can given the lack of testers, contributors, programmers, maintainers, etc...
There is no (big) money behind FreeBSD, hence all these hassles.
Acknowledging is one thing (and most people deep inside agree on that) but correcting it is another.

[harishankar]

The issue with the ports IMHO is there.
But i think FreeBSD tries to do the best it can given the lack of testers, contributors, programmers, maintainers, etc...
There is no (big) money behind FreeBSD, hence all these hassles.
Acknowledging is one thing (and most people deep inside agree on that) but correcting it is another.

Absolutely. I think you've hit it. From the time I started using FreeBSD, I realized that the kernel+userland and ports are two different entities. Kernel+userland has a strong and stable record because of the core team, while ports seem to be a little bit flakier entirely because of lack of manpower and not because of lack of ability or knowledge on the part of contributors.

Under the circumstances I think FreeBSD does a tremendous job of being on top of all the issues as it has been.

But it doesn't take away from the fact that there is a gap that needs filling. If I were less busy with my other work, I would have gladly joined as a contributor. But I frankly acknowledge that with my current level of knowledge it would take a lot of work to get there and I just haven't the time to learn the ins and outs deeper.

[graudeejs]

I rarely have problems with ports on my desktop PC.
For last 5 months I've been using Linux on laptop (ubuntu, sabayon), and you know what?
both of them were broken once during this time, (in the end I reinstalled ubuntu again, because it was fastest way to get laptop up and running)

during these 5 months I haven't had any serious problem with ports, that I would remember.
And I use FreeBSD ports on my Desktop, Server and it's jails

[aragon]

I've used Debian for more than 7 years and never had even a single issue with broken packages except when I occasionally played with "unstable" or installed third party packages without proper dependency checking. So I think you're not accurate there by clubbing several Linux distributions together. Debian's packaging policy is one of the best in the industry and worth emulating even if you happen to dislike Linux in general.
It may be the best for you, but it is certainly not the best for everyone. Being binary based it is very fixed, and that in itself is a big problem IMHO. I've never had a debian system on which I didn't run into a case of packages being compiled with or without components I don't or do need respectively, which ultimately leads one down the path of having to manually modify source packages or compile from distribution tarballs. Both a messy headache.

And debian's idea of "stable" is misleading - in fact it really just means "old". How stable is a package with bugs that are only fixed in later versions?


And to add to that, binary package selection in FreeBSD is nowhere near as comprehensive as in Linux.

The other issue is that using packages only in FreeBSD is a bit of a pain because of lack of comprehensive package managers (pkg_add and other pkg_* tools pkg_upgrade are not good enough, I'm afraid).
Agreed.

If the FreeBSD project could create ports branches in the same way there are src branches (RELENG_8, RELENG_7_0, etc.) then we might be able to get the best of both worlds. However, doing so would be of limited value if port maintainers put zero effort into maintaining the individual branches of their respective ports. Not an easy problem to solve.

[harishankar]

It may be the best for you, but it is certainly not the best for everyone. Being binary based it is very fixed, and that in itself is a big problem IMHO. I've never had a debian system on which I didn't run into a case of packages being compiled with or without components I don't or do need respectively, which ultimately leads one down the path of having to manually modify source packages or compile from distribution tarballs. Both a messy headache.

And debian's idea of "stable" is misleading - in fact it really just means "old". How stable is a package with bugs that are only fixed in later versions?



Agreed.

If the FreeBSD project could create ports branches in the same way there are src branches (RELENG_8, RELENG_7_0, etc.) then we might be able to get the best of both worlds. However, doing so would be of limited value if port maintainers put zero effort into maintaining the individual branches of their respective ports. Not an easy problem to solve.

Stable packages also get security updates in Debian. http://www.debian.org/security/faq. That's why security.debian.org exists. ;)

And Backports are a great solution for getting more updated packages in stable.

As for "configuring" source packages, 99% of the time I don't need it. And besides, some important Debian packages tend to have different versions with different sane defaults so you can choose the package you want for your particular purpose.

E.g. Apache has different packages for "mpm-prefork" and "mpm-worker" etc.

Also the debian packaging methodology being highly modularized, the most common "source configuration" options are usually packaged as separate binary modules which fit neatly into the system like a jigsaw puzzle.

Eg. every PHP module is packaged as php-<feature> where feature is a PHP module like mysql, pdo, sqlite etc. and once it is installed it just works.

[oliverh]

I've used Debian for more than 7 years and never had even a single issue with broken packages except when I occasionally played with "unstable" or installed third party packages without proper dependency checking. So I think you're not accurate there by clubbing several Linux distributions together. Debian's packaging policy is one of the best in the industry and worth emulating even if you happen to dislike Linux in general.[...]

Well, I have my experience with Debian (ejabberd in Sarge for example) and once in a while I even have to use Ubuntu. It's not a Linux-Vs-BSD-thingy I'm talking of facts. I certainly will not throw Debian in the same pit as Ubuntu, but Debian has got its problems in different areas mentioned above. Finally I'm talking of testing, not stable. The latter is old as a rock and even on some servers hardly usable anymore.

http://www.suspekt.org/2010/02/27/debian-breaks-suhosin-security-feature/

Or the OpenSSL-thingy in Debian and so on. I don't not want to badmouth Debian, but it has got its problems.

>even if you happen to dislike Linux in general.

To make it short: I don't like the Debian package policy, it's better than rpm-hell but an epic fail for people adoring K.I.S.S. If I'm looking for a nice binary package management I would certainly choose pacman (Archlinux). That said, I'm using ports because they rock :-)

[oliverh]

@harishankar

>And Backports are a great solution for getting more updated packages in stable.

Or more bugs, problems and security breaches.

>Also the debian packaging methodology being highly modularized, the most common "source configuration" options are usually packaged as separate binary modules which fit neatly into the system like a jigsaw puzzle.

Highly modularized? Yes in terms of Debian maybe, but compared to Gentoo, ArchLinux or FreeBSD (choosing the proper options while compiling), it's rather fat.

[harishankar]

> Highly modularized? Yes in terms of Debian maybe, but compared to Gentoo, ArchLinux or FreeBSD (choosing the proper options while compiling), it's rather fat.

All I know is that I don't mind a bit of "fatness" when compared to waiting several days to compile stuff on my system. Any which way I look at it, I cannot get past the bit where you compile and it takes hours for a non-trivial package and days for an entire system upgrade.

Debian in my view offers the perfect balance between convenience and flexibility which is very rare even among Linux distributions. Nothing is flawless however and I recognize your needs might not match what it offers.

> Or more bugs, problems and security breaches.

This is pure conjecture and argument for argument's sake. Well which way do you want it? You cannot have it both ways. Do you prefer that older bugs remain unfixed then? What are you trying to prove here? That FreeBSD has a better "security" policy? Because quite frankly I'm not going down that route. All my initial intention was to correct a few wrong notions of Debian -stable never getting security updates.

Finally any proof that Debian is unique in this regard? I'd say it's way better than most Linux distributions in that regard.


I'm immensely experienced, I've used a wide variety of operating systems and personally keep going back to Debian.

And with your kind permission folk, I'll stop this debate here. Obviously different strokes for different folks. :)

[anomie]

Ports are not stable. On the contrary, the ports tree is simply out of control.

...

I don't want the latest features. I do want my systems to run reliably!

May we PLEASE have a stable version of ports that is not broken almost daily?!

I happen to agree with you on this. But there are clearly political (and probably massive logistical) problems to implementing such a request: you can see from this thread, for instance, that there is not a consensus among end-users.

Whatever the outcome, I can make a suggestion to substantially reduce pain on your end. Set up a tinderbox, i.e. a separate system where you build ports for the purposes of 1) testing that they build; 2) confirming that they don't break existing services; 3) producing packages that you can deploy to your production system(s).

I've read about different strategies on this forum (from SirDice, primarily), but if you don't have a separate physical system for this purpose, then you can turn a FreeBSD jail into a tinderbox. In my case, I have a host with several jails under it. I use the host for the tinderbox, and I deploy only packages to each of the jails.

[Carpetsmoker]

There are quite a few issues IMO. I've seen quite a few ports which compile fine but do not run, the maintainer (if any) or submiter or the last patch simply didn't test that. I've also seen people submit broken patches and then given maintainership of a port with no questions asked.
I think the concept of "Quality assurance" needs to be introduced ...

[ckester]

The impression I've got after becoming a port maintainer and interacting with several committers is that the intention is already to make the portstree as stable as possible. I.e., I don't think anyone involved sees it as "bleeding edge" like CURRENT.

It's simply untrue that there is no QA effort being made.

The problem isn't the intent, it's the available resources. Manpower, machines, test harnesses. If you don't think the maintainers and committers are doing an adequate job testing, what are you doing to help? Can you offer some of your own time and machines for testing? Can you help write test suites? Can you help improve tinderbox or other testing tools?

I don't see how the proposal to introduce a STABLE portstree solves these resource problems. If anything, it makes them worse, by adding one more thing to an already-overloaded plate.

[achix]

The impression I've got after becoming a port maintainer and interacting with several committers is that the intention is already to make the portstree as stable as possible. I.e., I don't think anyone involved sees it as "bleeding edge" like CURRENT.

It's simply untrue that there is no QA effort being made.

The problem isn't the intent, it's the available resources. Manpower, machines, test harnesses. If you don't think the maintainers and committers are doing an adequate job testing, what are you doing to help? Can you offer some of your own time and machines for testing? Can you help write test suites? Can you help improve tinderbox or other testing tools?

I don't see how the proposal to introduce a STABLE portstree solves these resource problems. If anything, it makes them worse, by adding one more thing to an already-overloaded plate.

well said.
Until the time when coordinated massive manpower is dedicated to FreeBSD,the general concept will be as of today. There is no substitute for people.

I guess, however, that this fact, cannot be openly admitted by FreeBSD advocate activities, since it would look harmful to the very concept of promotion.

(currently writing from a firefox depending on

libxcb.so.1 => /usr/local/lib/compat/pkg/libxcb.so.1 (0x28c20000)

:P )

[gpw928]

Set up a tinderbox, i.e. a separate system where you build ports for the purposes of 1) testing that they build; 2) confirming that they don't break existing services; 3) producing packages that you can deploy to your production system(s).


Hi,

Thanks, good suggestion. I'll do that, starting with a scratch install of 8.1.

--
Phil

[dennylin93]

Well, it's quite true that Ports can be unstable sometimes, but they've worked pretty well for me. Devs also send head ups before major changes.

A "stable" version of Ports would require a lot of work. A part of the infrastructure would have to be changed, so to bring this to reality, someone will be to work on this. FreeBSD is after all a volunteer project.

[Oxyd]

I'd also like to see a stable version of the ports tree.

I understand that lack of people and resources is a problem. But I think such a split could help reduce some of the maintainers'/committers'/developers' workload. Imagine there are two versions of the Ports tree: CURRENT and STABLE. New ports, updates to old ports and all that goes to CURRENT -- it's basically the same as it is now. Some people out in the wild -- presumably adventurous desktop users or really some people who want to devote a machine or two they own to help FreeBSD -- use the CURRENT tree. They hit problems like most of us does now and they report them and help fix them. When these problems are resolved, the now-nonproblematic ports are commited to STABLE.

This way, STABLE users would have their stable tree, and CURRENT users could easily help the project simply by testing new ports on their own machines and reporting bugs.

Of course, it would still mean that someone would have to devote their time to think out the details, do the infrastructure changes and basically get it going. But I think it would be nice to have, no?

[gilinko]

Splitting the ports tree in any way will increase the workload on the ports maintainers, and it might come to a situation where patches has to be back ported between two trees. As already noted there is a QA effort, but they need resources.

By supporting the tinderbox effort either locally or the main QA tinderbox is the quickest way to increase the stability and quality of the ports tree as a whole. And if you can adopt a port that you are willing to maintain.

One thing that almost always is overlooked is that the ports tree is there as a tool to make it easier to install software, it will not do the work for you. Using ports does require you to know what you are doing with each of the softwares that you have chosen to deploy. As it is (in most cases) a compilation from source, the help you get is if they need special patches to work on freebsd.

[eric81]

manual to survive a ports disaster:

1. make a backup from all your installed ports with "pkg_info -Ea | xargs -L1 pkg_create -b" (creates packages in the current directory)
2. store /usr/ports in a ZFS dataset and make a snapshot, then "portsnap fetch update"
(alternatively you can backup them away, but this will take much more time)
3. read /usr/ports/UPDATING
4. now update your ports (for example "portmaster -a")
5. if something fails terribly:
- you can deinstall everything with "pkg_delete -a" (takes time)
- change directory to that from 1. (the backup)
- "pkg_add *.tbz" reinstalls all the packages (takes time)
- of course you could try to only deinstall/reinstall the problematic ports
- rollback the snapshot from /usr/ports
- now your system should be as before

good luck for the next time

Eric

[graudeejs]

manual to survive a ports disaster:

1. make a backup from all your installed ports with "pkg_info -Ea | xargs -L1 pkg_create -b" (creates packages in the current directory)
2. store /usr/ports in a ZFS dataset and make a snapshot, then "portsnap fetch update"
(alternatively you can backup them away, but this will take much more time)
3. read /usr/ports/UPDATING
4. now update your ports (for example "portmaster -a")
5. if something fails terribly:
- you can deinstall everything with "pkg_delete -a" (takes time)
- change directory to that from 1. (the backup)
- "pkg_add *.tbz" reinstalls all the packages (takes time)
- of course you could try to only deinstall/reinstall the problematic ports
- rollback the snapshot from /usr/ports
- now your system should be as before

good luck for the next time

Eric


Why to make backups of all installed packages, when you can simply make zfs snapshot (from your post I assume, you are using it), and if anything goes wrong, simply rollback

[eric81]

Why to make backups of all installed packages, when you can simply make zfs snapshot (from your post I assume, you are using it), and if anything goes wrong, simply rollback

You mean /usr/local? Do you trust this procedure? Modules are stored in /boot/modules, for example from emulators/virtualbox-ose. But would be my preferred way in future.

[phoenix]

If /usr/local, /usr/ports, and /var are all ZFS filesystems, then you can just snapshot all three at the same time. If things fail, just roll-back.

If you have any kernel-module ports installed, though, you will also need / on ZFS, or do a backup of / in order to restore if things fail.

[graudeejs]

Why to make backups of all installed packages, when you can simply make zfs snapshot (from your post I assume, you are using it), and if anything goes wrong, simply rollback

I have everything on zfs, so I absolutely trust this, and in fact I do this quite a lot, never fails.

[phoenix]

However, using -b with portmaster/portupgrade is very handy, in case you need to roll-back just a single application.

[jalla]

I have everything on zfs, so I absolutely trust this, and in fact I do this quite a lot, never fails.

Don't forget you can snapshot UFS as well (even UFS1). It's been in FreeBSD since 5.

gong:/h/tl# snapshot -v list /
Filesystem Type User User% Snap Snap% Snapshot Name Snapshot Time
/ ufs 316MB 16.0% 42MB 2.1% weekly.0 2010-05-09T00:00
/ ufs 316MB 16.0% 7MB 0.4% daily.2 2010-05-13T00:00
/ ufs 316MB 16.0% 1MB 0.1% daily.1 2010-05-14T00:00
/ ufs 316MB 16.0% 1MB 0.1% hourly.1 2010-05-14T16:00
/ ufs 316MB 16.0% 1MB 0.1% daily.0 2010-05-15T00:00
/ ufs 316MB 16.0% 1MB 0.1% hourly.0 2010-05-15T16:00
/ ufs 316MB 16.0% 1MB 0.1% oneoff.0 2010-05-15T17:27

[phoenix]

But can you roll back a UFS snapshot, such that the filesystem returns to the state it was in before the snapshot was taken?

I though UFS snapshots were more geared toward backups, where you would have a never-changing source tree to read from even if the backups take hours.

Similar to how LVM snapshots work on Linux (although, those snapshots are completely pointless as you have to reserve space in your volume group for the snapshots, and you have to specify a size for the snapshot, and and and and).

[jalla]

That's right, you cannot rollback. You'd have to mount the snapshot and copy the files back.

[rden]

I'd also like to see a stable version of the ports tree.

I understand that lack of people and resources is a problem. But I think such a split could help reduce some of the maintainers'/committers'/developers' workload. Imagine there are two versions of the Ports tree: CURRENT and STABLE. New ports, updates to old ports and all that goes to CURRENT -- it's basically the same as it is now. Some people out in the wild -- presumably adventurous desktop users or really some people who want to devote a machine or two they own to help FreeBSD -- use the CURRENT tree. They hit problems like most of us does now and they report them and help fix them. When these problems are resolved, the now-nonproblematic ports are commited to STABLE.

This way, STABLE users would have their stable tree, and CURRENT users could easily help the project simply by testing new ports on their own machines and reporting bugs.

Of course, it would still mean that someone would have to devote their time to think out the details, do the infrastructure changes and basically get it going. But I think it would be nice to have, no?

As you say package goes into CURRENT, people test (or just use) it, report problems, problems get fixed, no new bugs reported (for a while), package moved to STABLE . . . then new problems are reported (can happen, particularly with libs and also more people using stable than current). The only result is the already hardworking port owners now have to maintain 2 versions: a CURRENT, and a [so called] STABLE version.

I'm one of those that has little trouble with the current ports system and don't see the need to change it. If I need a port (or package) I install it, and if it works fine I keep using it as-is. Unless there is some new feature in a later release that I absolutely must have why upgrade? Sure when installing some new package I've had to add the occasional symlink (i.e. quite common link libjpeg.7 --> libjpeg.8), but so what? Everything I use is still working.

[darehanl]

As for "configuring" source packages, 99% of the time I don't need it. And besides, some important Debian packages tend to have different versions with different sane defaults so you can choose the package you want for your particular purpose.

E.g. Apache has different packages for "mpm-prefork" and "mpm-worker" etc.

I agree.

One thing that I believe will lighten the load on port maintainers is reducing the number of knobs (either "make config" or WITH_*). Doesn't a port maintainer have to at least confirm that each knob combination is buildable? And I just can't imagine how a dependency tree can be maintained when flipping a knob can change large parts of it. We already have a few ports that do this (i.e. vim vs. vim-lite); I don't see why not.

I appreciate the hard work porters are doing (esp. the Xorg), but I do feel our ports system is more labor intensive than others'.

[oliverh]

>Doesn't a port maintainer have to at least confirm that each knob combination is buildable? And I just can't imagine how a dependency tree can be maintained when flipping a knob can change large parts of it.

@darehanl, no. The default knobs are the recommendation. If you play with it, then you should know what you're doing. Finally FreeBSD isn't Ubuntu or Fedora, it's more like Gentoo or Slackware, it's an operating system for the seasoned user.

>I appreciate the hard work porters are doing (esp. the Xorg), but I do feel our ports system is more labor intensive than others'.

Did you ever build rpm or deb packages? Did you ever experience cyclic dependencies in Debian? Well, there are usually more problems with binary packages then sourcecode.

>One thing that I believe will lighten the load on port maintainers is reducing the number of knobs

This is one of the huge advantages of using ports. If you want to reduce the load, test ports, file problem reports, file bug fixes, file updates etc. pp. It's that easy, anyway it's of almost no use to lament in some forum.

[jdereus]

How can a messed up port hurt so bad?

did you do it

- on a live production server
- at peek times
- without proper backup measures ?


I think this thread has no real direction, and doesn't add much to the rest of the awful this-versus-that-OS debate.

[z662]

I hate to open up an old thread, but I just read this and would like to add to the conversation in hopes of learning something. Basically I experienced a rather similar issue just a few weeks ago. First off, I always have an up-to-date ports tree, and regularly run portaudit -Fa. Well as some of you know, there is a vulnerability out there for kdebase-workspace-4.3.5 (I hope thats right as I am at work right now) Anyway, so I did a portupgrade on that which wanted to upgrade to 4.4.1. All is good and dandy until it decides it needs a new version of jpeg (was using jpeg-7, now it requires 8) and also an upgrade of png. As said previously that just about means I might as well install FreeBSD 8.1 Beta since just about every port and/or package needs to be updated. Eventually it got so bad that I couldnt update one without having a newer version of a port that required the old dependency. SO it was pretty much like a deadlock. After like 5 hours of being angry and trying to resolve the issue I ended up reformatting, and refusing to upgrade any ports. So now I have a vulnerability with kdebase-workspace. Having wasted an entire night and not accomplishing anything other than a headache I beg the question: How do you prevent this, and how do you know whether or not this will happen before you get in over your head? Please advise.

[vermaden]

All is good and dandy until it decides it needs a new version of jpeg (was using jpeg-7, now it requires 8) and also an upgrade of png.

Solving that case is generally pretty easy (it works at least for png/jpeg/curl/libogg ports on my box).

# cd /usr/ports/category/someport
# make deinstall package clean
# make deinstall
# pkg_add -r someport
# pkg_add -f /usr/ports/packages/All/someport-X.Y.Z.tbz

You will end up with BOTH versions of the libraries for that port (jpeg/png/libogg/...), but of course some files overlap:

[vermaden]

% pkg_info -L png\*
Information for png-1.2.40:

Files:
/usr/local/man/man3/libpng.3.gz
/usr/local/man/man3/libpngpf.3.gz
/usr/local/man/man5/png.5.gz
/usr/local/bin/libpng-config
/usr/local/include/libpng/png.h
/usr/local/include/libpng/pngconf.h
/usr/local/lib/libpng.a
/usr/local/lib/libpng.so
/usr/local/lib/libpng.so.5
/usr/local/libdata/pkgconfig/libpng12.pc

Information for png-1.4.1_1:

Files:
/usr/local/man/man3/libpng.3.gz
/usr/local/man/man3/libpngpf.3.gz
/usr/local/man/man5/png.5.gz
/usr/local/bin/libpng-config
/usr/local/include/libpng/png.h
/usr/local/include/libpng/pngconf.h
/usr/local/include/libpng/pngpriv.h
/usr/local/lib/libpng.a
/usr/local/lib/libpng.so
/usr/local/lib/libpng.so.6
/usr/local/libdata/pkgconfig/libpng14.pc

[vermaden]

These that have '2' on the beginning of the line are overwritten:
% pkg_info -L png\* | sort -n | uniq -c | sort -r
4
2 Files:
2 /usr/local/man/man5/png.5.gz
2 /usr/local/man/man3/libpngpf.3.gz
2 /usr/local/man/man3/libpng.3.gz
2 /usr/local/lib/libpng.so
2 /usr/local/lib/libpng.a
2 /usr/local/include/libpng/pngconf.h
2 /usr/local/include/libpng/png.h
2 /usr/local/bin/libpng-config
1 Information for png-1.4.1_1:
1 Information for png-1.2.40:
1 /usr/local/libdata/pkgconfig/libpng14.pc
1 /usr/local/libdata/pkgconfig/libpng12.pc
1 /usr/local/lib/libpng.so.6
1 /usr/local/lib/libpng.so.5
1 /usr/local/include/libpng/pngpriv.h

[vermaden]

By this way, You have OLD version to have OLD ports/packages installed working correctly and a NEW version and NEW sources/headers under /(...)/include to satisfy new ports.

I am using this method for some time and all things work great.

The problem comes if a dependency for a lot of ports have a vulnerability, then You will have rebuild it all to be secure.

TO ADMINS: I needed to split my post to small parts because I got these errors:

Error 503 Service Unavailable

Service Unavailable
Guru Meditation:

XID: 400838167
Varnish

[z662]

I see vermaden, so essentially the only 'good' way to resolve the issue involves the installation of packages? If thats the case, what is the benefit of installing ports in the first place? I always was under the impression ports where preferred since you can customize the application however you wanted, and the fact that it is being compiled on your machine hence being the 'optimal' installation method. (I do not really know the reasons, but I am guessing when your machine compiles it, it is better???)

[vermaden]

I see vermaden, so essentially the only 'good' way to resolve the issue involves the installation of packages?
Port after being built becomes a installed package generally, You can also create a *.tbz package in the process, it does not matter how the port/package is installed, by make install compilation process or only by adding an already compiled *.tbz packages, its the same.

If thats the case, what is the benefit of installing ports in the first place?
I do not use port if I do not must, but there are no pacakges for ALL ports (lame for example).

I always was under the impression ports where preferred since you can customize the application however you wanted
None of the methods is prefered, You use what You like most or what better works for You.

and the fact that it is being compiled on your machine hence being the 'optimal' installation method.
Its non important, You only loose time.

[jb_fvwm2]

I should update my post (#16 in this thread) to
include the /usr/local/lib/compat/[place old .so. here]
method I've since tried (see the other thread that
mentions it). That at least lends "stability" to
some of the upgrades (here anyway).

[oliverh]

>Its non important, You only loose time.

That's maybe true for you, but as general point of view it's mere nonsense. We aren't talking of Gentooish madness (I don't count numbers), but customizing. Customizing is essential for many people, I don't need lots of rubbish installed per default (see e.g. Debian). I prefer slim applications with few dependencies. Time is of course an argument, but packages aren't the answer.

Furthermore packages in FreeBSD are a pain in the backside. Over a period of time you will _always_ experience lots of problems, especially if you have to use certain ports. Apart from that, handling of packages is a major pain in the backside too. It's the sorry-state of todays FreeBSD.

Do you know Greg Lehey? Well ...

This is another nail in the coffin of FreeBSD. What new user is going to install it if he can't even work out how to start X? On the same hardware, Ubuntu installs and works out of the box. There's a certain “submit patches” mentality in the FreeBSD project, and it's lethal. Ubuntu is nice and comfortable for people who want what it provides; once you get outside that area, it becomes more difficult. Somehow that's preferable to gratuitous changes that mean that even people like myself with decades of experience can't get the bloody thing to start.

Sunday, 6 June 2010, http://www.lemis.com/grog/diary-jun2010.php#bottom

or

Building a new FreeBSD machine is becoming more and more of an issue. I had planned and started an upgrade anyway last month, but given it up because of some weird bug that seems to bite only me. And though building ports from source seems like a good idea, and in the past I've supported it, it's becoming more and more fragile in the course of time: I would bet that I wouldn't be able to install all the software I wanted to without some configuration or compilation issues. And somehow, though processors are now nearly 1000 times faster than they were when I started using FreeBSD, building software takes about the same time.

or

Somehow it's like the end of an era: using Linux instead of BSD simply because it's easier. Of course, it's not as simple as that, but it seems to be the first step. Sad, somehow. In my experience, BSD has always been cleaner and more reliable, but it's getting a bit rough at the edges.

both from Saturday, 15 May 2010, http://www.lemis.com/grog/diary-may2010.php#bottom


I startet with FreeBSD 5.0 because of Leheys articles and musings in his diary. I don't leave FreeBSD, but I have to use Linux once in a while for more ambitious work. And it's "nice" to see, that even someone, as essential to FreeBSD, like Greg Lehey made similar experiences. That said, I'm aware of the many problems especially regarding ports and the stepchild packages, but at the end of the day you need something which is able to do more than email and browser.

I'm sorry it's rather offtopic, but I'm fed up with this kind of hype aka "there is no problem, Linux has got lots of problems too".

[z662]

Well, is the advice he provided in response to my initial problem (post 46) still valid?

[joag]

I don't know whether this is going to help or not as this post is too long and I think it'll continue to grow.

I'm a big Linux fan and I've installed almost any distro you can think of just for the sake of knowing which one was the right one for me, at the end Debian won because it is easy to maintain, use and I use it as my web server; for the desktop side I decided to use Ubuntu as it is Debian based but I got tired of instability brought by to frequent changes in new releases; aptitude is the best package manager I've ever used IMHO though.

Going back to what should be in this post and replies
For installing software I prefer ports
portsnap fetch extract ==> when you first install the ports system
portsnap fetch update ==> used when you want to update the ports tree

The last option you run it every few weeks or whenever you want and forget about running it if your current packages are not up2date from your current ports tree because you can cause or not sever issues to your system depending on the update(just don't do it til all our packages are up2date).

How do yo know what to update:
portversion -l "<"

This last command give you the packages needing an update according to the current installed ports tree.

how to upgrade this packages:
portupgrade package-1 package-2 ... package-n --batch

--batch means run the update in batch mode :D which for short is say yes to everything so you don't have to provide any input it just everything by defaults.

And all this combined with some experiences dealing with broken ports which you'll get after few months working on a BSD based system; I'm relatively new dealing with this things and this is what I've been using without problems and now FreeBSD is my main desktop OS.

Hope this can help a bit to newcomers to FreeBSD.

[darehanl]

I don't know whether this is going to help or not as this post is too long and I think it'll continue to grow.

I'm a big Linux fan and I've installed almost any distro you can think of just for the sake of knowing which one was the right one for me, at the end Debian won because it is easy to maintain, use and I use it as my web server; for the desktop side I decided to use Ubuntu as it is Debian based but I got tired of instability brought by to frequent changes in new releases; aptitude is the best package manager I've ever used IMHO though.
OT: Pacman is my preferred package manager:)

How do yo know what to update:
portversion -l "<"

This last command give you the packages needing an update according to the current installed ports tree.
Another alternative would be:
pkg_version -vI | grep -v '='
The 'I' means check against /usr/ports/INDEX-*, so if you have any local changes to the ports tree take it out.


how to upgrade this packages:
portupgrade package-1 package-2 ... package-n --batch

portmaster -Db package-1 package-2...

[wblock@]

How do yo know what to update:
portversion -l "<"

This last command give you the packages needing an update according to the current installed ports tree.

I prefer
portversion -vL=
because it doesn't have any shell redirection symbols. :)

how to upgrade this packages:
portupgrade package-1 package-2 ... package-n --batch

The -r option is useful, and I use it by default:
portupgrade -r port1 port2
will upgrade port1, port2, and anything that depends on either. This helps keep things from breaking.

-C shows you all the config screens and lets you set options. For ports that you haven't done make config on before, at least. It shows all the config screens up front, before any building, so you can set them all and then leave it to build and install everything.

[cracauer@]

I have posted this in various forums, the most complete version on slashdot.

I am convinced that we need to have a stable branch of the ports tree analogous to what Debian does.

I know everybody has different pet peeves, mine is Xorg. Xorg breaks things for me at a very rapid pace starting from 7.2. It is unbearable. That is bad enough as such, and it gets worse by the fact that this affects both my Linux and my FreeBSD boxes. But the real trick is that my supposedly most stable OS - FreeBSD - upgrades me to broken new Xorg versions first, not last like it should. Other people have other problems but alas the situation on Debian is better since Xorg is in a stable third-party software branch on Debian.

Now, I won't slam FreeBSD too much since Debian/testing now also has unusable and Debian/stable too broken Xorg. I have switched the first Desktop to run Xorg (an older version) in a chroot tree and expect to run like this for years to come. So FreeBSD isn't too much worse off anymore.

I have a more complete rant here:
http://bsd.slashdot.org/comments.pl?sid=1396293&cid=29675375

[fwaggle]

I am convinced that we need to have a stable branch of the ports tree analogous to what Debian does.

To play devil's advocate, I don't really see how this would make things better. It's my understanding most of the "omfg breakage" with ports is discovered when people try to use it - a certain amount is avoided through testing, I'm sure. But I don't really see how having a slower-moving ports tree will make things any better. The problems that are plaguing the current system right now don't surface until mass amounts of people try to use them, so the issues will still make it into the "stable" ports tree anyway before being caught.

The only thing I can see that might help matters is if you had the option to keep ports in sync with the versions they were at the time your release came out (-STABLE and -CURRENT would obviously track the current ports tree), then have some retention on old tarballs and backport security issues, and doing both until the EOL for a given release. Except then you're basically asking each and every port maintainer to exponentially increase their workload - if just a small percentage of them don't want to do it, you're left with what you have now... either track the ports tree as it moves and hope stuff don't break, or sit at your -RELEASE's ports tree and hope the tarballs stay put and there's no security issues.

I think a "-RELEASE" ports tree (I think calling it "-STABLE" is a misnomer, the tree we have right now fulfills the obligations of what FreeBSD calls "-STABLE") would just increase the workload and you'd still have almost the same amount of breakage you have now. What you're basically doing is insisting on other people doing the testing and getting tripped up by the breakage - if no one wants to do that, who's going to do it?

[gilinko]

Now, I won't slam FreeBSD too much since Debian/testing now also has unusable and Debian/stable too broken Xorg. I have switched the first Desktop to run Xorg (an older version) in a chroot tree and expect to run like this for years to come. So FreeBSD isn't too much worse off anymore.

The main "we need a stable ports system" comes from using FreeBSD as a desktop and thusly depend on Xorg, which for the last two years has been in a transition state starting with Dbus, hald, xrandr etc. A _lot_ of new and unstable technologies has been incorporated and has broken a lot of things over and over and over again.

I have been using both Fedora and Ubuntu on the desktop and not one release has gone by where it hasn't been broken someway on the desktop and needed either an experimental patch not worse there is no solution. For Fedora 13 is was the total absence of drivers for ANY ATI/AMD GFX card due to a bump in the Xorg version.

So it's not the ports version of Xorg that's unstable, it's the Xorg software itself... Which is one reason I use Mac OS X for my desktop now.

We rely on the ports tree use the latest released version of a software. Not a RC, Beta or something else, and rely on the testing done by the ones that do the releases of their software(like Xorg, Mozilla, Apache etc). The main problem is not the individual softwares but the integration of them all. Especially when we are talking about the desktop (ie Xorg/Gnome/KDE etc), and frankly that's not a problem for the ports to handle but something that has to be solved upstream.

In an ideal world of course... ;)

[bsd10]

I think we should have a ports tree that only has security patches for the ports in the most recent STABLE branch extended release. A full FreeBSD distribution with binary security updates for base and packages could be helpful to a lot of users.

If anyone is interested in doing this, I would be willing to contribute a server to host the ports repository and build the packages. It would probably make sense to start with the ports in 8.1-RELEASE when it comes out.

Edit: The following was added after my post was merged here

I am convinced that we need to have a stable branch of the ports tree analogous to what Debian does.

I completely agree.

I don't really see how having a slower-moving ports tree will make things any better.

What I would propose is not a slower-moving tree, but rather a RELEASE tree. It would work just like the RELEASE branch, by providing only security updates, so breakage would be very unlikely, as no new features would be added unless absolutely necessary. This would allow users to do a full binary security update of both the base system and ports. Each RELEASE comes with a fully functional ports tree, but it is never updated. If security updates were provided for that tree, I think it could potentially be the best option for all users like me who love FreeBSD for its stability.

[chalbersma]

From my understanding ZFS could significantly help with this process (I currently don't use ZFS but I will when I get my next system :).

You could install your system, run it as normal. Then with zfs create a snapshot of the system. Now upgrade your ports. You new ports break everything? Rollback to your snapshot.

If someone with ZFS could give me a heads up on whether this will work or not that would be cool.

[cracauer@]

To play devil's advocate, I don't really see how this would make things better. It's my understanding most of the "omfg breakage" with ports is discovered when people try to use it - a certain amount is avoided through testing, I'm sure. But I don't really see how having a slower-moving ports tree will make things any better. The problems that are plaguing the current system right now don't surface until mass amounts of people try to use them, so the issues will still make it into the "stable" ports tree anyway before being caught.


We are talking different things here. You talk ports breakage as in the ports tree doesn't build, has dependency problems and the like.

I talk about a perfectly fine building ports tree pulling in software packages that are broken upstream.


The only thing I can see that might help matters is if you had the option to keep ports in sync with the versions they were at the time your release came out (-STABLE and -CURRENT would obviously track the current ports tree), then have some retention on old tarballs and backport security issues, and doing both until the EOL for a given release. Except then you're basically asking each and every port maintainer to exponentially increase their workload - if just a small percentage of them don't want to do it, you're left with what you have now... either track the ports tree as it moves and hope stuff don't break, or sit at your -RELEASE's ports tree and hope the tarballs stay put and there's no security issues.

I think a "-RELEASE" ports tree (I think calling it "-STABLE" is a misnomer, the tree we have right now fulfills the obligations of what FreeBSD calls "-STABLE") would just increase the workload and you'd still have almost the same amount of breakage you have now. What you're basically doing is insisting on other people doing the testing and getting tripped up by the breakage - if no one wants to do that, who's going to do it?

Right, that won't work.

The Debian model is the only model I can see working.

[cracauer@]

The main "we need a stable ports system" comes from using FreeBSD as a desktop and thusly depend on Xorg, which for the last two years has been in a transition state starting with Dbus, hald, xrandr etc. A _lot_ of new and unstable technologies has been incorporated and has broken a lot of things over and over and over again.


My impression is more than the Xorg people just make everything run that a user coming from Windows might expect and although they leave options in to switch to things that traditional Unix did better those options become broken. This is both a developers problem and a QA problem. Neither seem to be interested in "powerusers". It's all very interactive and all developed without much of a background in what's out there in the Unix world. I mean they just broke xterm by just not knowing that certain modifier keys with certain chars are in use by some curses apps.


We rely on the ports tree use the latest released version of a software. Not a RC, Beta or something else, and rely on the testing done by the ones that do the releases of their software(like Xorg, Mozilla, Apache etc).

In the Xorg case the problem is upstream, and it is a perfectly valid but broken release.

The main problem is not the individual softwares but the integration of them all.

Not for me. The main problem is broken upstream releases. I can deal with integration problems on my own (in fact the ports tree usually takes a week to go through cleanly on my Thinkpad).

Especially when we are talking about the desktop (ie Xorg/Gnome/KDE etc), and frankly that's not a problem for the ports to handle but something that has to be solved upstream.

In an ideal world of course... ;)

If I didn't lack any interest in X11 programming I'd split Xorg at the 7.1 or 7.2 release point right now.

But that's the problem, right? Those people who just want a working Xorg for traditional Unix functionality do not want to hack X11. So they get taken on a wild ride.

Add to that their extremely messy build system with everything splattered into different source tar files, but requiring their own same specific versions in a strict dependency tree, their "solution" to that problem that is a Python script that didn't make it through 1/10th of the tree (didn't keep error messages, sorry) and the inability to compile into anything but default paths (yes I just tried, no go, too many -Is and -Ls missing for the new prefix and a couple hardcoded /usr/.... places).

Apple of course is a source of sanity against some of that klickibunti Linuxism but they have opted not to use X11 in the first place.

[cracauer@]

From my understanding ZFS could significantly help with this process (I currently don't use ZFS but I will when I get my next system :).

You could install your system, run it as normal. Then with zfs create a snapshot of the system. Now upgrade your ports. You new ports break everything? Rollback to your snapshot.

If someone with ZFS could give me a heads up on whether this will work or not that would be cool.

I can rollback by taring up a safety copy of /usr/local and /var/db/ports just fine, I don't need ZFS.

But I need updates for software that didn't opt to break things in new releases. Security fixes just for starters. You can't just not update.

And Xorg breakage does not get repaired. You can't rollback your ports tree until it's fixes. It's broken in releases upstream. It's not a one-liner fix in your port's Makefile.

[fwaggle]

We are talking different things here. You talk ports breakage as in the ports tree doesn't build, has dependency problems and the like.

I talk about a perfectly fine building ports tree pulling in software packages that are broken upstream.

No, I think we're talking about the same thing. For example, Xorg being broken upstream - in my very, very limited understanding of the problems, I get the impression that most of the problems won't be found until the masses start using the given port in a bunch of different configurations. If there was any kind of test case, I'd be willing to bet most of the port maintainers wouldn't knowingly put a broken port into the tree in favor of an older one that works without a good reason. I'm sure the person who's email address is in the port doesn't want to be inundated with angry or exasperated emails without a good reason behind it.

So I guess what I mean is that having one "-STABLE" (and calling the current ports tree "-CURRENT") doesn't really work because where do you draw the line at something "not working"? If you have a ports system that allows you to keep the versions you have and just add security patches (a la tracking RELENG_X_Y) then the workload for ports maintainers goes up massively, further increasing the likelihood something will break.

Finally, if I'm not mistaken, ports pulls many of it's tarballs in from the third party sites - how can you guarantee those tarballs will be retained until the EOL for a given release (which is probably a pretty reasonable support time frame anyway)?

I guess what I'm trying to say is the root cause of the problem appears to be lack of manpower in the ports tree in general, and I haven't seen a single proposed solution that wouldn't exacerbate that root cause. I haven't seen anyone willing to volunteer to maintain a tree, and I think nominating every port maintainer to at least double their workload is probably not a good move. If you want to make sure your ports tree doesn't become broken, keep your ports tree static and then backport security patches yourself. :D

[chalbersma]

...
Finally, if I'm not mistaken, ports pulls many of it's tarballs in from the third party sites - how can you guarantee those tarballs will be retained until the EOL for a given release (which is probably a pretty reasonable support time frame anyway)?
...


If I'm not mistaken the checksums in the port assure this.

[DutchDaemon]

Checksums do not guarantee that the author himself will not remove tarballs for older versions of his software ...

[chalbersma]

Checksums do not guarantee that the author himself will not remove tarballs for older versions of his software ...

Got me there.

[mickey]

It is my opinion, that having a STABLE branch of the ports tree would not improve the situation, but only increase the workload and steal resources that would be of more benefit, if spent otherwise. Having a certain amount of QA in the ports tree should not be an option, but the default.

I have been using the ports tree exclusively like forever now. The thing that has bothered me the most within the last 1-2 years now, has been that updates which cause lots of dependant ports needing to be upgraded too, have not been deployed in a coordinated fashion. We have seen updates to jpeg, png, gettext, libtool and whatnot. Nearly each of these required you to rebuild something like 75% of all your installed ports, causing major headaches. Everytime you were thinking: "hey, i have my system up to date again", the next major breakage was just around the corner.

Also looking at some ports, I get the impression, that someone has to apply the same patches with every new version of the software that gets released. And I keep asking myself, why don't these patches go upstream, so that the effort of porting the next release of the software will eventually become nil. Do the upstream developers simply not care about the 'portability' of their software? I understand, that many software gets mainly developed on linux, but does that imply to be ignorant to the fact, that people using other OSes would like to use that software, too?

[jb_fvwm2]

The thing that has bothered me the most within the last 1-2 years now, has been that updates which cause lots of dependant ports needing to be upgraded too, have not been deployed in a coordinated fashion. We have seen updates to jpeg, png, gettext, libtool and whatnot. Nearly each of these required you to rebuild something like 75% of all your installed ports, causing major headaches. Everytime you were thinking: "hey, i have my system up to date again", the next major breakage was just around the corner.


edit
the below does not apply to "rebuild everything" with portmaster or portupgrade.
I have too many ports installed to ever do that...
also, I am not sure about the use of /usr/local/lib/compat/pkg as opposed to /usr/local/
lib/compat/...
(using usually tty's concurrently, sometimes with "portmaster -d port port port "
END edit


I thought so too for years. Then a few weeks ago a thread here
pointed out that breakages could be lessened by interim copying
of the .so. files to be obsoleted temporarily to /usr/local/lib/compat.
...
Just using it with gettext so far (2 .so.) but anticipate using it for
curl, png, jpeg, (etc) as well. (Having updated a subset of
the gettext-dependent ports but not others). For me, turns the problem
of port breakage wholly around for the better. Only caveat, one must
remember to eventually remove the stale .so.'s (maybe during a buildworld
cycle).
....
I was thinking, maybe a /port/ or a switch to pkg_delete could place
.so. 's in /usr/local/lib/compat and setup an "at" command to
every-so-often echo to the screen: "have you removed the 'something'
stale .so. files yet" or something. (and bail out maybe if there
are too many .so. files to put there, or if the /port/ is too
intermediate (has too many dependencies) and might not work as
smoothly. (I suspect there would be untoward effects, but have
not tried any other than /gettext/).

[mickey]

I thought so too for years. Then a few weeks ago a thread here
pointed out that breakages could be lessened by interim copying
of the .so. files to be obsoleted temporarily to /usr/local/lib/compat.

Normally portupgrade per default seems to copy outdated shared libs to /usr/local/lib/compat/pkg. So that is not the point. For me the 'breakage' is that I cannot use my system for some 1-2 days, cause it is busily recompiling half of the installed packages and me having to use my old notebook in the meantime :e

Only caveat, one must remember to eventually remove the stale .so.'s (maybe during a buildworld cycle).

Just through with that yesterday, after updating gettext and stuff. Deleted all stuff from /usr/local/lib/compat/pkg, located all binaries/libraries with unresolved dependencies, and reinstalled those. Now everything seems to be working smoothly again ... until the next major update :P
I was thinking, maybe a /port/ or a switch to pkg_delete could place .so. 's in /usr/local/lib/compat and setup an "at" command to every-so-often echo to the screen: "have you removed the 'something' stale .so. files yet" or something.

How about the Windows approach?

System indicates that the shared object foobar.so.42
is no longer used by other system components.
Do you wish to delete it permanently?

;-)

[cracauer@]

Checksums do not guarantee that the author himself will not remove tarballs for older versions of his software ...

Not an issue.

Third-party site hosted tarballs are always copied to freebsd.org sites that serves as backup for as long as the port references that tarball. Even if the original disappears the freebsd copy will be there.

[Alt]

Dunno when ports changed perl5.8 to 5.10, but now im updating all pkgs with portmaster.
Got loads of errors thats horror :P
Most of errors is that pkg_info says perl module is installed. But they are in wrong(old) directory so perl dont see them..

I wonder if there a way to just apply security patches for installed pkgs (as freebsd-update), so i can run server non-stop

[bsd10]

I wonder if there a way to just apply security patches for installed pkgs (as freebsd-update), so i can run server non-stop

There is not, but I think having this would make FreeBSD incredibly popular. I have offered to host a STABLE ports tree and package build server, but I have not gotten any response here.

[cracauer@]

Dunno when ports changed perl5.8 to 5.10, but now im updating all pkgs with portmaster.
Got loads of errors thats horror :P
Most of errors is that pkg_info says perl module is installed. But they are in wrong(old) directory so perl dont see them..

I wonder if there a way to just apply security patches for installed pkgs (as freebsd-update), so i can run server non-stop

In this particular case you failed to read the instructions in /usr/ports/UPDATING :)

20090328:
AFFECTS: users of lang/perl*
AUTHOR: skv@FreeBSD.org

lang/perl5.10 is out. If you want to switch to it from, for example
lang/perl5.8, that is:

Portupgrade users:
0) Fix pkgdb.db (for safety):
pkgdb -Ff

1) Reinstall new version of Perl (5.10):
env DISABLE_CONFLICTS=1 portupgrade -o lang/perl5.10 -f perl-5.8.\*

2) Reinstall everything that depends on Perl:
portupgrade -fr perl

Portmaster users:
env DISABLE_CONFLICTS=1 portmaster -o lang/perl5.10 lang/perl5.8
portmaster -r perl-

Note: If the "perl-" glob matches more than one port you will need to
specify the name of the Perl directory in /var/db/pkg explicitly.



That isn't to say that I don't get ports messes even when following UPDATING once in a while.

[ckester]

I wonder if there a way to just apply security patches for installed pkgs (as freebsd-update), so i can run server non-stop

If the upstream author(s) provide security patches against previous versions, sure, it might be possible.

But if the 20+ ports I maintain are typical, very few of those fixes have been backported by the upstream authors. Instead, the expectation seems to be that users will upgrade to the latest version.

Granted, most of my ports are fairly obscure. The situation might be different with heavily-used programs like Firefox, Python, GTK+, etc.

I'm not sure it's realistic to expect the maintainers to identify and backport the patches themselves. As I said previously in this thread, maintainers and committers already have enough to do. If you're seriously interested in having patches backported, you need to be ready to pitch in and contribute toward this yourself.

[Alt]

Dont know how it can be possible.. If we talking about binary patches, there must be binary version for each version for each port.. For source patches - here needed a tool that update only 1 port, saving dependency list if its possible..

Atm, i see only 1 way to update vulnerable pkg without touching others (if portaudit says its vulnerable): `pkg_add -fr`. But sometimes this can lead to heavy consequences...

[mickey]

Dunno when ports changed perl5.8 to 5.10, but now im updating all pkgs with portmaster.
Got loads of errors thats horror :P
Most of errors is that pkg_info says perl module is installed. But they are in wrong(old) directory so perl dont see them..

Looks like you didn't upgrade your perl installation properly, as is suggested in /usr/ports/UPDATING:

20090328:
AFFECTS: users of lang/perl*
AUTHOR: skv@FreeBSD.org

lang/perl5.10 is out. If you want to switch to it from, for example
lang/perl5.8, that is:

Portupgrade users:
0) Fix pkgdb.db (for safety):
pkgdb -Ff

1) Reinstall perl with new 5.10:
env DISABLE_CONFLICTS=1 portupgrade -o lang/perl5.10 -f perl-5.8.\*

2) Reinstall everything that depends on Perl:
portupgrade -fr perl

Portmaster users:
env DISABLE_CONFLICTS=1 portmaster -o lang/perl5.10 lang/perl5.8
portmaster -r perl-

Note: If the "perl-" glob matches more than one port you will need to
specify the name of the perl directory in /var/db/pkg explicitly.

[Alt]

mickey, cracauer@ said about this in post #77 =) I got the point, it was my fault

[gore]

Looks like you didn't upgrade your perl installation properly, as is suggested in /usr/ports/UPDATING:

20090328:
AFFECTS: users of lang/perl*
AUTHOR: skv@FreeBSD.org

lang/perl5.10 is out. If you want to switch to it from, for example
lang/perl5.8, that is:

Portupgrade users:
0) Fix pkgdb.db (for safety):
pkgdb -Ff

1) Reinstall perl with new 5.10:
env DISABLE_CONFLICTS=1 portupgrade -o lang/perl5.10 -f perl-5.8.\*

2) Reinstall everything that depends on Perl:
portupgrade -fr perl

Portmaster users:
env DISABLE_CONFLICTS=1 portmaster -o lang/perl5.10 lang/perl5.8
portmaster -r perl-

Note: If the "perl-" glob matches more than one port you will need to
specify the name of the perl directory in /var/db/pkg explicitly.


That actually just failed, literally just now, on my machine. I got tired of having all my ports with security holes mailed to me each day, so I decided to try an update. Ran Portsnap to get everything, and then thought about how to do the rest.

freebsd-update works like apt-get and doesn't have any issues at all. Ports not being included in security patches, which is like 95% of what most people actually use anyway, means using cvsup, or portsnap, THEN once that is done, using either portupgrade, which has broken my boxes more than I care to count, (Heh, I remember reading "just do portupgrade -af" and having an unusable machine after that) so I decided pormanager and portmaster might be worth a try.

I'm on round 3 now, because it's broke and terminated twice.

I've been using Debian and SUSE for almost 10 years, and I'm a little shocked that someone said something has broken. I've literally not seen SUSE crash, ever. And I've used it as a desktop, a server, and everything else, and rebooting is only for Kernel updates, so, I had pretty good uptime.

And of course updating non base install software doesn't take a week on slower machines. I did ONE have an Nvidia driver update screw up X for me. I told Marcus Meissner, who went into work early that day and fixed it for me. That's the one time I've actually had an issue with SUSE, and the person who wrote the Kernel patch, came in early to take care of it for me.

Right now, portmaster -a is running on my machine. I always install security patches except for ports because it takes so long, and the machine isn't exactly usable while it's happening. And being that I don't code, I don't actually want to sit here telling it how to compile something. But, I'm trying again. It's the reason my server runs Slackware; I can type one line of commands, and everything is upgraded and working. If a Kernel update was there, I reboot, and I'm done. If there wasn't one... I type the command (Like swaret --update && swaret --upgrade or slaptget, or slackpkg, whichever I want to use) and then, it runs, and finishes, and I'm done. No down time.

My Debian machine has been running for about 2 years. It has a lot of stuff running on it, and has about....12,000 things installed? And apt-get update && apt-get upgrade once a day or so, and if there are any, the get installed and I'm done.

I literally not seen much in crashes. And I push hardware pretty hard.

Third party repos exist for basically everything, but be it SUSE, Debian, or whatever, I haven't had any issues. Adding a line of text to sources.list is a lot easier than tarballs ever were.

[ikbendeman]

I'm surprised no one has pointed out that OpenBSD has this? Maybe OpenBSD would be a better system for you. I personally like FreeBSD ports better. :)

Oh and to the above poster, portupgrade never worked for me, and I almost gave up on FreeBSD because of it. I decided not to use anything, and only "make install clean". Then I found portmaster, which made my life much better. Now I have the best of both worlds (things just working like "make install clean") and easy management (with portmaster).

[oliverh]

>I've been using Debian and SUSE for almost 10 years, and I'm a little shocked that someone said something has broken. I've literally not seen SUSE crash, ever. And I've used it as a desktop, a server, and everything else, and rebooting is only for Kernel updates, so, I had pretty good uptime.

Well good luck then for the future, such experiences are rare. Usually most *BSD users are coming from the Linux-world and have a long experience with this other free operating system. Whereas most Linux-users are coming from Windows and they're usually happy if they're getting "something better" than this Windows. I don't want to start a flame-war, but if I do know some of the lows of *BSD, it's usually rather easy to avoid them and then I usually get a rock-stable system. Linux distros are a moving target, an ever-changing environment with some highs and a plethora of lows, especially if it comes to documentation and continuity.

Look at Debian for example, they have lots of developers, but compared to OpenBSD the quality is rather low, at least in recent years. And OpenBSD itself has got just about 80+ developers, they're focused on security, they don't get support from big companies, but they made rather good progress regarding the audio-system, acpi etc. pp. They stay on focus, they don't do politics. It's a pity, but Debian is a declining star of the post-millennium.

[sk8harddiefast]

Hi. In my opinion, ports are stable enough and I always make ports tree update.
We must do ports tree update because we can't stay in old versions for a long long time. Will start to be incompatible with a lot of things.
For example. If you use a very old version of Kdenlive, could be incompatible with HD movies. (Just an example to understand what I mean.)
Also latest versions have new stuff, more choices, new packages etc witch is very good.
But sometimes makes me to be a little disappointed. I will speak as a user, who is not expert on FreeBSD.
I will explain what I mean.
This time on ports we have Conky 1.8 witch is broken the last 3 - 4 months and is not working well. With simple words, if you want Conky, you must portdowngrade to 1.7 version witch is not on ports. (It should be as the latest stable version.)
Minitube exists on ports but is not streaming at all. (Well, is new project and I can accept that is under a lot of development to be workable for all Unix systems.)
I have an HD camera and I work on my new skate movie. When import them to Kdenlive, I take an error: "Clip invalid or missing".
In previous version of Kdenlive just crashed.
Speaking as user, I want to do my job but I can't.
But the truth is that rarely I see ports that cannot be build or programs that are not working well.
This also could be my fault, so I can't say that ports are not stable for this reason.
Always I see error fixes.
I know you do your best :)
Keep going :)

[Eponasoft]

I personally like staying on the cutting edge of ports... not always but usually. Recently, I've had problems getting vlc 1.1.4 to work properly after being built so I've had to downgrade to 1.0.6 via pkg_add but that's the only real issue I've had and it's probably just a configuration problem (damn that thing has a monster 'make config' dialog). Upgrading pcmanfm from the 0.5.x line to the 0.9.x line caused a couple of issue with my fbpanel config, but it also allowed me to redo my .xinitrc and run fewer programs. No problem though... I simply find another way to make things work when something goes awry.

[sk8harddiefast]

(damn that thing has a monster 'make config' dialog)
I will agree with that. Yesterday I made it work but now HD run very fast. Also, after some seconds sound stop working on video. One of the problems on vlc, is that have a billion options! Is impossible to know them all, to make a video work.

[mdg583]

There are two strategies I am trying for keeping my desktop system sane using packages:

- one is only updating the ports tree to the date of the last package build. There can be a line in a supfile like this:
*default date=2010.09.27.12.00.16
so I've made a simple script to check this page:
http://pointyhat.freebsd.org/errorlogs/i386-8-full/cvsdone
to see when the package build last finished. That way things built via ports are never ahead of things installed by packages.

- the other is using pkgupgrade.py, which can be found on the internet. It is only kind of working, but I really like that upgrade strategy: download all packages, make backups, uninstall all packages, reinstall all packages. I only do it once every few months. So far no issues, and I can't think of why there would be issues, since it involves reinstalling all software. It is pretty fast too.

[captobvious]

I am a recent slackware -> FreeBSD convert; in my short experience with the port tree, I think it is worlds better than either the Debian package system or Slackbuilds.

Why to make backups of all installed packages, when you can simply make zfs snapshot (from your post I assume, you are using it), and if anything goes wrong, simply rollback

Hopefully ZFS v28 will be an install option in 8.2/9.0. Hard disk space is cheap; since I employ FreeBSD as a desktop OS, my base system and installed ports are less than a 20th in size compared to my MP3 folder.

As killasmurf86 stated, a ZFS snapshot on a weekly schedule might be 10-20Gb for 2-3 months worth of snapshots? Make a cron job every Monday, 3am.

[oliverh]

>Hopefully ZFS v28 will be an install option in 8.2/9.0. Hard disk space is cheap

Anything is cheap, even such sayings. The main use of UNIX-like operating systems is: as server, in embedded systems or scientific workstations. The consumer-desktop is a fraction of a fraction on UNIX-like operating systems and beyond that dominated by Apple and Windows. Even Linux has no reasonable share on the desktop, apart from a plethora of sick hype. Last not least, the so-called consumer-desktop moves toward more slim hardware, like netbooks or ARM-powered all-in-one devices. The time for fat quadcore driven PCs, with lots of gigabytes of memory and XX terabytes of drive space is counted. Leave them for hardcore gamers or enthusiasts with too much money in their pockets.

Apart from that, you'll not see more than ZFS v15 in 8.2 and maybe v28 will not be ready for FBSD 9.0 (maybe 9.1). v28 is a test, it's even in the Solaris environment nothing more than a test.

That said, anything is alright as long as there is freedom of choice.