Parallel builds+downloads, 'make build-recursive'

vanessa

vanessa

There are multiple threads in the forum discussing how to speed up building ports. All target the -j option or the MAKE_JOBS_SAFE variable. But half of the time goes away downloading dist files.

So why can't these two processes run in parallel - the next distfile being downloaded while building the current one? Or am I just missing an existent nifty switch for enabling this behaviour?
 
Good tip - I didn't know that! I am still building ports manually because I (think to) have more control of the steps involved.

However I severely miss one other function besides parallel downloads: 'make build-recursive'. As ports often fail to compile halfway down the chain, I end up with some of the dependancies already installed while others are still not compiled. It would be fine to build all of the dependencies first but install them altogether in a second step and in one go.

Does anybody know of a workflow how to achieve this?
 
vanessa said:
However I severely miss one other function besides parallel downloads: 'make build-recursive'. As ports often fail to compile halfway down the chain, I end up with some of the dependancies already installed while others are still not compiled.
Click to expand...
The problem here is basically the structure of the ports; in the end they are all individual components which may have a dependency but that dependency only goes as far as to check if a specific version is installed or not.

More than often a dependency is either set as "if product x is present" or "if version y or higher of product x is present". The system doesn't really care much more beyond that.

And that's what you're experiencing here. The best way to solve some of these problems is to write a script which can handle some of the hassle and keep track of your ports and their updates. Well, it has been mentioned before but that's where ports-mgmt/portmaster comes in. It's basically a shell script which checks for all this.

I think it's a very good idea to start off manually so that you really get a good feeling for the way things work. Been there myself earlier this year. However, while you do maintain more control over your ports you'll also notice that you may end up with too much control since the ports collection hasn't automated much itself.

What you basically have are several small features which, when combined, can lead up to a very powerful setup. The only thing you'd need to do is piece the whole thing together.

Which could make a very good experience, but you'll notice that most people don't bother anymore these days and simply resort to portmaster (myself included).

Hope this can give some ideas.
 
Well, let me explain what my point is. Since Thursday (!) I am trying to build KDE on FreeBSD 10 HEAD. As it doesn't work out of the box (meaning with the predefined options for the ports involved), I am constantly either fixing something or dropping things I don't really need. The problem is, that every time a compilation fails, I end up with many packages installed. This prevents going on by simply reconfiguring and building again, because:
1) building them only doesn't work, as dependencies get automatically installed (despite telling the meta port 'make build' only)
2) I have to first deinstall the packages, before building them again with different options (or not building them at all because of dropped options). Otherwise I could only force overwriting them by setting FORCE_PKG_REGISTER. Either way it's a mess and a huge time loosing factor.

Actually this is a bug, because 'make build' should build only and not install anything, even for dependencies or meta ports. Or there should be a 'make build-recursive' like I am proposing here.
 
It starts to make some sense now. First of all though, you might already be aware of this, but please keep in mind that FreeBSD-10 is basically marked unstable. It's the developers playground so to speak, so while it may work for you now it can just as easily break tomorrow due to applied changes.

vanessa said:
The problem is, that every time a compilation fails, I end up with many packages installed. This prevents going on by simply reconfiguring and building again, because:
1) building them only doesn't work, as dependencies get automatically installed (despite telling the meta port 'make build' only)
Click to expand...
First off keep in mind that there are 2 types of dependencies for a port. You have the run dependency which is required for the port to function, but you also have the so called build dependencies. These are required to actually build or compile the port. Sometimes they overlap, but very often they don't.

So if you issue # make build then all the build dependencies will get automatically installed, simply because the port needs them at this stage. You can check these for yourself by using the make build-depends-list while make run-depends-list will show you the ports required for usage (see also ports(7)).

vanessa said:
2) I have to first deinstall the packages, before building them again with different options (or not building them at all because of dropped options). Otherwise I could only force overwriting them by setting FORCE_PKG_REGISTER. Either way it's a mess and a huge time loosing factor.
Click to expand...
I wonder if you would be able to use # make deinstall-all here. Not sure because the manual page is a bit unclear, but it seems to me as if it would uninstall the main port and all its dependencies.

So this really isn't a bug but simply another type of dependency.

But do keep in mind that using FreeBSD-10 is bound to give you problems and issues like these. The best way to overcome those is to stick with the production version which is FreeBSD-9.2 at the time of writing.

(Edit):

When going over the thread I suddenly realized that I forgot something I was planning to mention. Better late then never I suppose. Maybe you already know about this, in that case just ignore ]Unix(-like)[/noparse] commandline. So if you're working with a pristine setup (where root uses the csh shell) then you could use this when you're inside the ports directory you're working on:

Code: 
foreach a (`make build-depends-list`)
  make -C $a deinstall clean;
  end;
After typing the first line you'll get into special command prompt where you can enter the rest of the commands.

This is also doable in more common shells such as sh or bash. Then you'd use something like this: # for a in `make build-depends-list`; do make -C $a deinstall clean; done.

Hope this can help too.
 
The build step involves making sure necessary build dependencies are already installed. So it will build and install those dependencies. In other words, "build" applies to this port, not the dependencies.
 
ShelLuser said:
First of all though, you might already be aware of this, but please keep in mind that FreeBSD-10 is basically marked unstable. It's the developers playground so to speak, so while it may work for you now it can just as easily break tomorrow due to applied changes.
Click to expand...

Well, the 'playground' should/must be already quite stable - in two months it's showtime.
 
ShelLuser said:
(Edit):

When going over the thread I suddenly realized that I forgot something I was planning to mention. Better late then never I suppose. Maybe you already know about this, in that case just ignore ;)

About having to de-install those dependencies.. It can be bothersome but don't forget that it's relatively easy to automate the procedure, one of the cool things about working on a Unix(-like) commandline. So if you're working with a pristine setup (where root uses the csh shell) then you could use this when you're inside the ports directory you're working on:

Code: 
foreach a (`make build-depends-list`)
  make -C $a deinstall clean;
  end;
After typing the first line you'll get into special command prompt where you can enter the rest of the commands.

This is also doable in more common shells such as sh or bash. Then you'd use something like this: # for a in `make build-depends-list`; do make -C $a deinstall clean; done.

Hope this can help too.
Click to expand...

Thank you for the input, very kind of you :)

The problem with deinstall is that you can't deinstall a package when third ports depend upon it, unless:
  • you force this -> in this case you have to (remember to) re-install the port/s back again. If the next build fails or such ports fall out as dependencies, you end up with a missing bit.
  • or let it in and force the overwrite -> in this case it might be that there will be no overwrite (again, because of other dependencies), so you end up with a package nobody needs
In both cases you must manually fix the problem and delete old or unneeded stuff. But this is not that bad - the REAL problem is, that you have to track ALL those packages over several hours or even days in your head! So go compile KDE with some failing builds and your brain will go buum.
 
vanessa said:
Well, the 'playground' should/must be already quite stable - in two months it's showtime.
Click to expand...

Don't hold your breath in the mean time. Schedules have a tendency to slip.
 
vanessa said:
The problem with deinstall is that you can't deinstall a package when third ports depend upon it, unless:
  • you force this -> in this case you have to (remember to) re-install the port/s back again. If the next build fails or such ports fall out as dependencies, you end up with a missing bit.
  • or let it in and force the overwrite -> in this case it might be that there will be no overwrite (again, because of other dependencies), so you end up with a package nobody needs
Click to expand...
Well spotted. Yes, what I wrote up above is basically exactly what point 1 does. If you try to uninstall a package using the package manager (either using # pkg_delete or # pkg delete) you'll need to apply some virtual force because they won't allow you to remove anything which has dependencies.

But if you run a # make uninstall those dependencies will only turn into a warning ("removing anyway").

vanessa said:
But this is not that bad - the REAL problem is, that you have to track ALL those packages over several hours or even days in your head! So go compile KDE with some failing builds and your brain will go buum.
Click to expand...
Indeed.

Those one liners are only a liable option when you're going to pursue the build right then and there, otherwise you risk inconsistency. It's also why it's generally a good idea to eventually "give in" and start utilizing some programs which can do some of all this work for you. Well, either that or device a strategy of your own.

For example]portmaster[/file] has an option where it can keep track of the ports it just build in order to prevent you from having to re-build the same ports over and over again while you're actually focussing on an issue with only one of the whole lot.
 
In the mean time I found a killer solution: ZFS snapshots ;) Each failed build goes then straight back to hell �e
 
vanessa said:
In the mean time I found a killer solution: ZFS snapshots ;) Each failed build goes then straight back to hell �e
Click to expand...

There is even better solution. You can build everything in clean jailed environment using ports-mgmt/poudriere and upgrading your host system using generated packages repository. The main advantage is that you can do anything while building, you can even stop the process and continue any time - your system will stay unaffected.

Once everything built, you just hit # pkg upgrade and here you go: your installed ports will get upgraded in a few minutes using your repository.

HOWTO: http://forums.freebsd.org/showthread.php?t=38859
 
How could I have missed poudriere!? Yes, this is the real alternative to ZFS, thanks!

OK, to sum up the 2 ways of clean building from ports:
1) ports-mgmt/poudriere
2) ZFS snapshots

+1 for merging/moving poudriere into the default building process.
 
vanessa said:
OK, to sum up the 2 ways of clean building from ports:
1) ports-mgmt/poudriere
2) ZFS snapshots
Click to expand...
Actually, you still can use shapshots with the first method. I always create shapshots for /usr/local and /var/db/pkg before doing # pkg upgrade. Having backup wont hurt, especially if it's created within seconds :)
 
You must log in or register to reply here.
Back
Top