Parallel Port Building Script

C, C++, Python, Perl, Shell, etc.

Parallel Port Building Script

Postby cakersq » 02 Jun 2011, 23:04

I was annoyed when building ports used only one of my servers processor cores, and annoyed at having to install Perl/Ruby/Python just to make the INDEX-#.

After an afternoon of coding, I wrote a handy SH script to install ports in parallel, accounting for dependencies, and not requiring installing another language.

It's a little rough, but I got it working to install X and Gnome.

What do you think (See Attached)?
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby wblock@ » 02 Jun 2011, 23:22

How old is the ports tree on that machine? The ports system has supported multiple jobs for quite a while now. See
[cmd="%"]less -p JOBS /usr/ports/Mk/bsd.port.mk[/cmd]
User avatar
wblock@
Old Fart
 
Posts: 10169
Joined: 07 Sep 2009, 23:23
Location: Milky Way galaxy

Postby cakersq » 02 Jun 2011, 23:36

I'm aware of the JOBS support in Ports, but in my experience, most ports do not have that enabled, and forcing it usually doesn't work, especially when dependencies are required.

This way parallel building is supported for EVERY port.
User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby wblock@ » 03 Jun 2011, 00:06

Bringing this up on the ports mailing list might be good. Maybe the advantages of both can be combined.
User avatar
wblock@
Old Fart
 
Posts: 10169
Joined: 07 Sep 2009, 23:23
Location: Milky Way galaxy

Postby cakersq » 03 Jun 2011, 01:06

Updated to version 0.2
- Fixed falsely removing dependencies from to-be-installed list
- Fixed default number of jobs to CPUs+1
- Fixed port count display
- Output pkg_add to /dev/null (Quiet)
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby cakersq » 03 Jun 2011, 01:40

Updated to version 0.3
- Fixed checking for already installed packages
- Improved usage help message
- Improved output messages (removed some, corrected others)

That's the last update for a while, next will focus on speeding up the process further:
- Speed up dependency gathering phase
- Build larger packages first
- Background distfile download and extraction
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby Alt » 03 Jun 2011, 06:54

Can you make a port of your script? It's a bit ugly this way - to share scripts in a text file, I think. Anyway I'll try it on next build. Can it work with ccache/portmaster?
User avatar
Alt
Member
 
Posts: 726
Joined: 18 Nov 2008, 12:22
Location: Mother Russia

Postby cakersq » 03 Jun 2011, 11:30

@Alt

Can you make a port your script?


The concept of installing a port to install ports never sat well with me. There are also 1000's of port management ports out there.

Can it work with ccache/portmaster


My goal was to create this using a minimum amount of additional software, so that it can be used on any system. What I will probably try to do is to integrate it into the Ports system (bsd.pports.mk or something), that way you can go into a port, and specify [FILE]make parallel-package[/FILE] or [FILE]make parallel-install clean[/FILE] and have it go.

That being said, this uses the standard [FILE]make package clean[/FILE] method of installing ports, so if you install [FILE]ccache[/FILE], and add its configuration to the [FILE]/etc/make.conf[/FILE] configuration file, it should still work.
User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby cakersq » 04 Jun 2011, 22:52

Upgraded to version 0.4
- Continuously update the ports dependencies list, to find ports that can be built
- Greatly reduces (to almost none) the number of ports that are installed serially.
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby cakersq » 04 Jun 2011, 23:44

Version 0.4.1 - Bug fix - dependencies were being installed out of order ... sorry.
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby Seeker » 05 Jun 2011, 11:50

Can you clarify ..., in a case, when more then 1 origin is specified, i.e;
Code: Select all
pports-0.4.1.sh ftp/wget lang/php5

Will it firstly build [port]ftp/wget[/port] and simultaneously build all its dependencies, while [port]lang/php5[/port] is waiting OR will it be simultaneously building both origins and simultaneously all their dependencies?

Thanks!

Ok, I've been impatient, waiting longer then one minute for a reply, so I've taken a peak in your code and I say: it builds [port]ftp/wget[/port] first and simultaneously builds all its dependencies, while [port]lang/php5[/port] is waiting.
User avatar
Seeker
Member
 
Posts: 855
Joined: 19 May 2009, 19:47
Location: Europe/Croatia/Zagreb

Postby cakersq » 05 Jun 2011, 12:31

@Seeker

Excellent question! The answer is:
...it be simultaneously building both origins and simultaneously all theirs dependencies


The code gathers the dependencies for all of the origins specified serially (one at a time), but will then build all dependencies together. So it is possible the second origin completes before the first origin.

The algorithm goes something like this:
- Run [FILE]make config-conditional[/FILE] for all specified origins, unless [FILE]-b[/FILE] is specified
- Gather list of dependencies for all specified origins (one at a time), store in "index"
- Gather list of dependencies for all dependencies, store in "index"
- Repeat until all possible dependencies are indexed
- Find all ports in the above "index" that are leaves (have no dependencies of their own), store in "install"
- Build all leaves listed in the "install" file, up to [FILE]-j[/FILE] amount at a time.
- Rebuild "index" as needed, removing listed dependencies for an origin if the dependency was installed (will reveal more "leaves" that have no further dependencies)
- Repeat until all ports are installed.

Every now and then, there will be a time where the system is just building one port at a time, this is normal. It happens when a major port (such as [port]devel/gettext[/port] or [port]devel/libiconv[/port]) is being built, and everything requires it. I've made an effort to reduce the amount of time only one port is being built, but it will still happen.

The code's not easy to read (for that I apologize). This is more of a proof-of-concept than anything else. As I mentioned, I'll probably try to get this code added to the Ports system properly, in the [file]/usr/ports/Mk/[/file] files. That way a user can specify [cmd=]make -C /usr/ports/lang/php52 parallel-package[/cmd] and it's done, much faster than the normal serial building method.
User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby Seeker » 05 Jun 2011, 15:39

You have a problem! Installing ports into [FILE]DESTDIR[/FILE] doesn't work!

Code: Select all
===>  Chrooted make in /tmp/md.sh.cy9DSHlJ succeeded
===>  Cleaning up...
Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
make: chdir ===>: No such file or directory
.make: chdir Creating: No such file or directory
make: chdir Creating: No such file or directory
.make: chdir some: No such file or directory
make: chdir some: No such file or directory
.make: chdir important: No such file or directory
make: chdir important: No such file or directory
10make: chdir subdirectories: No such file or directory
make: chdir subdirectories: No such file or directory
.make: chdir Starting: No such file or directory
make: chdir Starting: No such file or directory
.make: chdir chrooted: No such file or directory
make: chdir chrooted: No such file or directory
.make: chdir make: No such file or directory
make: chdir make: No such file or directory
.make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
.make: chdir Chrooted: No such file or directory
make: chdir Chrooted: No such file or directory
.make: chdir succeeded: No such file or directory
make: chdir succeeded: No such file or directory
20make: chdir Cleaning: No such file or directory
make: chdir Cleaning: No such file or directory
.make: chdir up...: No such file or directory
make: chdir up...: No such file or directory

Gathering Dependencies for lang/php5..make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
.make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
.make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
30make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory

Gathering Dependencies for sysutils/fusefs-ntfs..make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
40make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory

[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
grep: /tmp/pports-2119/index3: No such file or directory
grep: /tmp/pports-2119/index3: No such file or directory
rm: /tmp/pports-2119/index3: No such file or directory
Done!


You have to fix this!
;)
User avatar
Seeker
Member
 
Posts: 855
Joined: 19 May 2009, 19:47
Location: Europe/Croatia/Zagreb

Postby cakersq » 05 Jun 2011, 20:48

Well, Seeker, I have no idea what you did to cause that! The script requires a full base system to run, especially [FILE]/usr/ports[/FILE], [FILE]/var/db/pkgs[/FILE], etc.

If you want to build the ports in a full chroot environment, including ports tree, you can copy the script into the chroot, and run
[cmd=]chroot /mnt /root/pports.sh lang/php52[/cmd]

If you just want to install the ports to the mountpoint, I would run the script and build the ports on the main system, then use the packages created in [FILE]/usr/ports/packages[/FILE] (created via my script.)
[cmd=]pkg_add -P /mnt /usr/ports/packages/All/php-5.2.tbz[/cmd]

Anyway, good luck!
User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby Seeker » 06 Jun 2011, 07:38

cakersq wrote:Well, Seeker, I have no idea what you did to cause that! ...

My [FILE]DESTDIR[/FILE], has a fresh FreeBSD installation.

Your script should behave in a same way as ports d, that is, if I:
Code: Select all
cd /usr/ports/ftp/wget
make install clean

it will [FILE]chroot[/FILE] into [FILE]DESTDIR[/FILE] and successfully install the port, EVEN if [FILE]DESTDIR[/FILE] doesn't have a [FILE]/usr/ports[/FILE] tree, because it mounts into the chrooted [FILE]DESTDIR[/FILE], via [FILE]nullfs[/FILE], what it needs.

As you intend to integrate this, into the [FILE]/usr/ports/Mk/[/FILE] files, you have to fix this, as your script has to handle this, in the same way as the ports system does.

So good luck to you, fixing your script.
;)
User avatar
Seeker
Member
 
Posts: 855
Joined: 19 May 2009, 19:47
Location: Europe/Croatia/Zagreb

Postby cakersq » 06 Jun 2011, 23:59

Just for you Seeker:

Version 0.4.3 2011/06/06
- Works with [FILE]DESTDIR[/FILE] for ports (specified in [FILE]/etc/make.conf[/FILE])
- Added [FILE]DestDir[/FILE] option to script
- [FILE]pkg_add[/FILE] honors [FILE]DestDir[/FILE] variable
- Only checks for already installed ports once
- Fixed error handling when a specified origin doesn't exist
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby Seeker » 07 Jun 2011, 09:28

[FILE]DESTDIR[/FILE] now works!
However, your script will hang endlessly, because of a daemonized dialogs. That is:
Code: Select all
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'

pports.sh $ports


Dialogs will be thrown at the start, for each origin, before builds occur. However, once any dependency is started to be built, its dialog will hang endlessly, as it won't be thrown to user.

Code: Select all
last pid: 41730;  load averages:  2.07,  2.19,  2.17                                                    up 0+03:09:45  11:10:32
83 processes:  3 running, 80 sleeping
CPU: 52.1% user,  0.0% nice, 46.4% system,  1.5% interrupt,  0.0% idle
Mem: 58M Active, 309M Inact, 114M Wired, 2508K Cache, 111M Buf, 513M Free
Swap: 2014M Total, 2014M Free

  PID USERNAME        THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
18232 root              1 107    0  3668K  1576K RUN     53:27 49.76% dialog
 5822 root              1 107    0  3668K  1576K RUN     55:53 48.49% dialog


I had to kill the dialog processes to continue builds..

Code: Select all
18232  ??  R     54:22.88 /usr/bin/dialog --checklist Options for m4 1.4.16,1 21 70 15 LIBSIGSEGV Use libsigsegv for better


You have to fix this.

I would recomend, hammering a human with ALL dialogs, before building starts, via:
Code: Select all
# make config-recursive

So human can take a walk, doing its "human stuff".
User avatar
Seeker
Member
 
Posts: 855
Joined: 19 May 2009, 19:47
Location: Europe/Croatia/Zagreb

Postby Seeker » 07 Jun 2011, 12:50

Additionally ...
I need to run this code twice:
Code: Select all
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
# It only BUILDS, so DESTDIR ends up empty
pports.sh $ports

Code: Select all
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
# This installs ...
pports.sh $ports


Also devfs remained mounted in chroot
User avatar
Seeker
Member
 
Posts: 855
Joined: 19 May 2009, 19:47
Location: Europe/Croatia/Zagreb

Postby cakersq » 07 Jun 2011, 20:39

This seems to be a problem with Ports system not honoring the [FILE]BATCH[/FILE] flag when [FILE]DESTDIR[/FILE] is specified. I'll have to go digging and figure out why.
User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby cakersq » 07 Jun 2011, 22:30

Couldn't fix the [FILE]BATCH[/FILE] issue when [FILE]DESTDIR[/FILE] is specified so I wrote my own [FILE]DESTDIR[/FILE] algorithm for this script.

Check it out, tell me what you think.
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby cakersq » 08 Jun 2011, 01:14

Fixed script, was not checking for ports currently being installed correctly...
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby cakersq » 08 Jun 2011, 02:11

I fixed a bug before you even found it!

Version 0.5.2
- Fixed that port dependencies are inaccurate when [FILE]DESTDIR[/FILE] is used and [FILE]make config[/FILE] changes options from the defaults.
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Postby graudeejs » 08 Jun 2011, 10:02

cakersq wrote:I fixed a bug before you even found it!

Version 0.5.2
- Fixed that port dependencies are inaccurate when DESTDIR is used and "make config" changes options from the defaults.


If you want I can provide Mercurial repository for your project at http://hg.bsdroot.lv also bugzilla via bugs.bsdroot.lv
Or you can use github :)
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby Seeker » 08 Jun 2011, 14:55

Code: Select all
===>  Directory /tmp/md.sh.zBpkvMWn does not exist
===>  Please set DESTDIR to a valid jail environment.
*** Error code 1

Stop in /usr/ports/ftp/wget.
===>  Directory /tmp/md.sh.zBpkvMWn does not exist
===>  Please set DESTDIR to a valid jail environment.
*** Error code 1

Stop in /usr/ports/lang/php5.
Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
make: chdir ===>: No such file or directory
.make: chdir Directory: No such file or directory
make: chdir Directory: No such file or directory
.make: don't know how to make package-name. Stop
make: chdir /tmp/md.sh.zBpkvMWn: No such file or directory
.make: chdir does: No such file or directory
make: chdir does: No such file or directory
.make: chdir not: No such file or directory
make: chdir not: No such file or directory
.make: chdir exist: No such file or directory
make: chdir exist: No such file or directory
.make: chdir Please: No such file or directory
make: chdir Please: No such file or directory
.make: chdir set: No such file or directory
make: chdir set: No such file or directory
10make: chdir DESTDIR: No such file or directory
make: chdir DESTDIR: No such file or directory
.make: chdir valid: No such file or directory
make: chdir valid: No such file or directory
.make: chdir jail: No such file or directory
make: chdir jail: No such file or directory
.make: chdir environment.: No such file or directory
make: chdir environment.: No such file or directory
.make: chdir boot.txz: Not a directory
make: chdir boot.txz: No such file or directory
.make: chdir Error: No such file or directory
make: chdir Error: No such file or directory
.make: chdir code: No such file or directory
make: chdir code: No such file or directory
.make: chdir 1: No such file or directory
make: chdir 1: No such file or directory
.make: chdir Stop: No such file or directory
make: chdir Stop: No such file or directory
.make: chdir in: No such file or directory
make: chdir in: No such file or directory
20make: chdir /usr/ports/ftp/wget.: No such file or directory
make: chdir /usr/ports/ftp/wget.: No such file or directory

Gathering Dependencies for lang/php5..make: chdir /usr/ports/lang/php5.: No such file or directory
make: chdir /usr/ports/lang/php5.: No such file or directory

[1/2] Installing ftp/wget
[2/2] Installing lang/php5
Done!

Last two ports are installed, because there was a previous run, from yesterday. [FILE]DESTDIR[/FILE] is valid, as was used by other parts of MY script, before and after calling your script.
;)
User avatar
Seeker
Member
 
Posts: 855
Joined: 19 May 2009, 19:47
Location: Europe/Croatia/Zagreb

Postby cakersq » 09 Jun 2011, 00:37

How are you setting [FILE]DESTDIR[/FILE]?

Edit: Never mind, I figured it out. You're setting it in the environment, via export or setenv. ... Fix will be provided shortly.
User avatar
cakersq
Junior Member
 
Posts: 31
Joined: 05 May 2011, 23:13
Location: USA

Next

Return to Userland Programming & Scripting

Who is online

Users browsing this forum: No registered users and 0 guests