Parallel Port Building Script

I was annoyed when building ports used only one of my servers processor cores, and annoyed at having to install Perl/Ruby/Python just to make the INDEX-#.

After an afternoon of coding, I wrote a handy SH script to install ports in parallel, accounting for dependencies, and not requiring installing another language.

It's a little rough, but I got it working to install X and Gnome.

What do you think (See Attached)?
 

Attachments

  • pports.txt
    3.5 KB · Views: 361
How old is the ports tree on that machine? The ports system has supported multiple jobs for quite a while now. See
% less -p JOBS /usr/ports/Mk/bsd.port.mk
 
I'm aware of the JOBS support in Ports, but in my experience, most ports do not have that enabled, and forcing it usually doesn't work, especially when dependencies are required.

This way parallel building is supported for EVERY port.
 
Updated to version 0.2
- Fixed falsely removing dependencies from to-be-installed list
- Fixed default number of jobs to CPUs+1
- Fixed port count display
- Output pkg_add to /dev/null (Quiet)
 

Attachments

  • pports-0.2.txt
    3.7 KB · Views: 323
Updated to version 0.3
- Fixed checking for already installed packages
- Improved usage help message
- Improved output messages (removed some, corrected others)

That's the last update for a while, next will focus on speeding up the process further:
- Speed up dependency gathering phase
- Build larger packages first
- Background distfile download and extraction
 

Attachments

  • pports-0.3.txt
    4.2 KB · Views: 361
Can you make a port of your script? It's a bit ugly this way - to share scripts in a text file, I think. Anyway I'll try it on next build. Can it work with ccache/portmaster?
 
@Alt

Can you make a port your script?

The concept of installing a port to install ports never sat well with me. There are also 1000's of port management ports out there.

Can it work with ccache/portmaster

My goal was to create this using a minimum amount of additional software, so that it can be used on any system. What I will probably try to do is to integrate it into the Ports system (bsd.pports.mk or something), that way you can go into a port, and specify make parallel-package or make parallel-install clean and have it go.

That being said, this uses the standard make package clean method of installing ports, so if you install ccache, and add its configuration to the /etc/make.conf configuration file, it should still work.
 
Upgraded to version 0.4
- Continuously update the ports dependencies list, to find ports that can be built
- Greatly reduces (to almost none) the number of ports that are installed serially.
 

Attachments

  • pports-0.4.sh.txt
    4.7 KB · Views: 361
Version 0.4.1 - Bug fix - dependencies were being installed out of order ... sorry.
 

Attachments

  • pports-0.4.1.sh.txt
    4.8 KB · Views: 373
Can you clarify ..., in a case, when more then 1 origin is specified, i.e;
Code:
pports-0.4.1.sh ftp/wget lang/php5
Will it firstly build ftp/wget and simultaneously build all its dependencies, while lang/php5 is waiting OR will it be simultaneously building both origins and simultaneously all their dependencies?

Thanks!

Ok, I've been impatient, waiting longer then one minute for a reply, so I've taken a peak in your code and I say: it builds ftp/wget first and simultaneously builds all its dependencies, while lang/php5 is waiting.
 
@Seeker

Excellent question! The answer is:
...it be simultaneously building both origins and simultaneously all theirs dependencies

The code gathers the dependencies for all of the origins specified serially (one at a time), but will then build all dependencies together. So it is possible the second origin completes before the first origin.

The algorithm goes something like this:
- Run make config-conditional for all specified origins, unless -b is specified
- Gather list of dependencies for all specified origins (one at a time), store in "index"
- Gather list of dependencies for all dependencies, store in "index"
- Repeat until all possible dependencies are indexed
- Find all ports in the above "index" that are leaves (have no dependencies of their own), store in "install"
- Build all leaves listed in the "install" file, up to -j amount at a time.
- Rebuild "index" as needed, removing listed dependencies for an origin if the dependency was installed (will reveal more "leaves" that have no further dependencies)
- Repeat until all ports are installed.

Every now and then, there will be a time where the system is just building one port at a time, this is normal. It happens when a major port (such as devel/gettext or devel/libiconv) is being built, and everything requires it. I've made an effort to reduce the amount of time only one port is being built, but it will still happen.

The code's not easy to read (for that I apologize). This is more of a proof-of-concept than anything else. As I mentioned, I'll probably try to get this code added to the Ports system properly, in the /usr/ports/Mk/ files. That way a user can specify [cmd=]make -C /usr/ports/lang/php52 parallel-package[/cmd] and it's done, much faster than the normal serial building method.
 
You have a problem! Installing ports into DESTDIR doesn't work!

Code:
===>  Chrooted make in /tmp/md.sh.cy9DSHlJ succeeded
===>  Cleaning up...
Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
make: chdir ===>: No such file or directory
.make: chdir Creating: No such file or directory
make: chdir Creating: No such file or directory
.make: chdir some: No such file or directory
make: chdir some: No such file or directory
.make: chdir important: No such file or directory
make: chdir important: No such file or directory
10make: chdir subdirectories: No such file or directory
make: chdir subdirectories: No such file or directory
.make: chdir Starting: No such file or directory
make: chdir Starting: No such file or directory
.make: chdir chrooted: No such file or directory
make: chdir chrooted: No such file or directory
.make: chdir make: No such file or directory
make: chdir make: No such file or directory
.make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
.make: chdir Chrooted: No such file or directory
make: chdir Chrooted: No such file or directory
.make: chdir succeeded: No such file or directory
make: chdir succeeded: No such file or directory
20make: chdir Cleaning: No such file or directory
make: chdir Cleaning: No such file or directory
.make: chdir up...: No such file or directory
make: chdir up...: No such file or directory

Gathering Dependencies for lang/php5..make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
.make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
.make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
30make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory

Gathering Dependencies for sysutils/fusefs-ntfs..make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
40make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory

[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
grep: /tmp/pports-2119/index3: No such file or directory
grep: /tmp/pports-2119/index3: No such file or directory
rm: /tmp/pports-2119/index3: No such file or directory
Done!

You have to fix this!
;)
 
Well, Seeker, I have no idea what you did to cause that! The script requires a full base system to run, especially /usr/ports, /var/db/pkgs, etc.

If you want to build the ports in a full chroot environment, including ports tree, you can copy the script into the chroot, and run
[cmd=]chroot /mnt /root/pports.sh lang/php52[/cmd]

If you just want to install the ports to the mountpoint, I would run the script and build the ports on the main system, then use the packages created in /usr/ports/packages (created via my script.)
[cmd=]pkg_add -P /mnt /usr/ports/packages/All/php-5.2.tbz[/cmd]

Anyway, good luck!
 
cakersq said:
Well, Seeker, I have no idea what you did to cause that! ...
My DESTDIR, has a fresh FreeBSD installation.

Your script should behave in a same way as ports d, that is, if I:
Code:
cd /usr/ports/ftp/wget
make install clean
it will chroot into DESTDIR and successfully install the port, EVEN if DESTDIR doesn't have a /usr/ports tree, because it mounts into the chrooted DESTDIR, via nullfs, what it needs.

As you intend to integrate this, into the /usr/ports/Mk/ files, you have to fix this, as your script has to handle this, in the same way as the ports system does.

So good luck to you, fixing your script.
;)
 
Just for you Seeker:

Version 0.4.3 2011/06/06
- Works with DESTDIR for ports (specified in /etc/make.conf)
- Added DestDir option to script
- pkg_add honors DestDir variable
- Only checks for already installed ports once
- Fixed error handling when a specified origin doesn't exist
 

Attachments

  • pports-0.4.3.sh.txt
    5.7 KB · Views: 197
DESTDIR now works!
However, your script will hang endlessly, because of a daemonized dialogs. That is:
Code:
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'

pports.sh $ports

Dialogs will be thrown at the start, for each origin, before builds occur. However, once any dependency is started to be built, its dialog will hang endlessly, as it won't be thrown to user.

Code:
last pid: 41730;  load averages:  2.07,  2.19,  2.17                                                    up 0+03:09:45  11:10:32
83 processes:  3 running, 80 sleeping
CPU: 52.1% user,  0.0% nice, 46.4% system,  1.5% interrupt,  0.0% idle
Mem: 58M Active, 309M Inact, 114M Wired, 2508K Cache, 111M Buf, 513M Free
Swap: 2014M Total, 2014M Free

  PID USERNAME        THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
18232 root              1 107    0  3668K  1576K RUN     53:27 49.76% dialog
 5822 root              1 107    0  3668K  1576K RUN     55:53 48.49% dialog

I had to kill the dialog processes to continue builds..

Code:
18232  ??  R     54:22.88 /usr/bin/dialog --checklist Options for m4 1.4.16,1 21 70 15 LIBSIGSEGV Use libsigsegv for better

You have to fix this.

I would recomend, hammering a human with ALL dialogs, before building starts, via:
Code:
# make config-recursive
So human can take a walk, doing its "human stuff".
 
Additionally ...
I need to run this code twice:
Code:
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
# It only BUILDS, so DESTDIR ends up empty
pports.sh $ports
Code:
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
# This installs ...
pports.sh $ports

Also devfs remained mounted in chroot
 
This seems to be a problem with Ports system not honoring the BATCH flag when DESTDIR is specified. I'll have to go digging and figure out why.
 
Couldn't fix the BATCH issue when DESTDIR is specified so I wrote my own DESTDIR algorithm for this script.

Check it out, tell me what you think.
 

Attachments

  • pports-0.5.sh.txt
    6.5 KB · Views: 201
Fixed script, was not checking for ports currently being installed correctly...
 

Attachments

  • pports-0.5.1.sh.txt
    6.5 KB · Views: 178
I fixed a bug before you even found it!

Version 0.5.2
- Fixed that port dependencies are inaccurate when DESTDIR is used and make config changes options from the defaults.
 

Attachments

  • pports-0.5.2.sh.txt
    6.7 KB · Views: 355
cakersq said:
I fixed a bug before you even found it!

Version 0.5.2
- Fixed that port dependencies are inaccurate when DESTDIR is used and "make config" changes options from the defaults.

If you want I can provide Mercurial repository for your project at http://hg.bsdroot.lv also bugzilla via bugs.bsdroot.lv
Or you can use github :)
 
Code:
===>  Directory /tmp/md.sh.zBpkvMWn does not exist
===>  Please set DESTDIR to a valid jail environment.
*** Error code 1

Stop in /usr/ports/ftp/wget.
===>  Directory /tmp/md.sh.zBpkvMWn does not exist
===>  Please set DESTDIR to a valid jail environment.
*** Error code 1

Stop in /usr/ports/lang/php5.
Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
make: chdir ===>: No such file or directory
.make: chdir Directory: No such file or directory
make: chdir Directory: No such file or directory
.make: don't know how to make package-name. Stop
make: chdir /tmp/md.sh.zBpkvMWn: No such file or directory
.make: chdir does: No such file or directory
make: chdir does: No such file or directory
.make: chdir not: No such file or directory
make: chdir not: No such file or directory
.make: chdir exist: No such file or directory
make: chdir exist: No such file or directory
.make: chdir Please: No such file or directory
make: chdir Please: No such file or directory
.make: chdir set: No such file or directory
make: chdir set: No such file or directory
10make: chdir DESTDIR: No such file or directory
make: chdir DESTDIR: No such file or directory
.make: chdir valid: No such file or directory
make: chdir valid: No such file or directory
.make: chdir jail: No such file or directory
make: chdir jail: No such file or directory
.make: chdir environment.: No such file or directory
make: chdir environment.: No such file or directory
.make: chdir boot.txz: Not a directory
make: chdir boot.txz: No such file or directory
.make: chdir Error: No such file or directory
make: chdir Error: No such file or directory
.make: chdir code: No such file or directory
make: chdir code: No such file or directory
.make: chdir 1: No such file or directory
make: chdir 1: No such file or directory
.make: chdir Stop: No such file or directory
make: chdir Stop: No such file or directory
.make: chdir in: No such file or directory
make: chdir in: No such file or directory
20make: chdir /usr/ports/ftp/wget.: No such file or directory
make: chdir /usr/ports/ftp/wget.: No such file or directory

Gathering Dependencies for lang/php5..make: chdir /usr/ports/lang/php5.: No such file or directory
make: chdir /usr/ports/lang/php5.: No such file or directory

[1/2] Installing ftp/wget
[2/2] Installing lang/php5
Done!
Last two ports are installed, because there was a previous run, from yesterday. DESTDIR is valid, as was used by other parts of MY script, before and after calling your script.
;)
 
How are you setting DESTDIR?

Edit: Never mind, I figured it out. You're setting it in the environment, via export or setenv. ... Fix will be provided shortly.
 
Back
Top