28f34
![]() |
|
|
|
|
|||||||
| Userland Programming & Scripting C, Shell, Perl, Sed & Awk |
![]() |
|
|
Thread Tools | Display Modes |
|
#1
|
||||
|
||||
|
I was annoyed when building ports used only one of my servers processor cores, and annoyed at having to install Perl/Ruby/Python just to make the INDEX-#.
After an afternoon of coding, I wrote a handy SH script to install ports in parallel, accounting for dependencies, and not requiring installing another language. It's a little rough, but I got it working to install X and Gnome. What do you think (See Attached)? |
|
#2
|
||||
|
||||
|
How old is the ports tree on that machine? The ports system has supported multiple jobs for quite a while now. See
% less -p JOBS /usr/ports/Mk/bsd.port.mk
|
|
#3
|
||||
|
||||
|
I'm aware of the JOBS support in Ports, but in my experience, most ports do not have that enabled, and forcing it usually doesn't work, especially when dependencies are required.
This way parallel building is supported for EVERY port. |
|
#4
|
||||
|
||||
|
Bringing this up on the ports mailing list might be good. Maybe the advantages of both can be combined.
|
|
#5
|
||||
|
||||
|
Updated to version 0.2
- Fixed falsely removing dependencies from to-be-installed list - Fixed default number of jobs to CPUs+1 - Fixed port count display - Output pkg_add to /dev/null (Quiet) |
|
#6
|
||||
|
||||
|
Updated to version 0.3
- Fixed checking for already installed packages - Improved usage help message - Improved output messages (removed some, corrected others) That's the last update for a while, next will focus on speeding up the process further: - Speed up dependency gathering phase - Build larger packages first - Background distfile download and extraction |
|
#7
|
||||
|
||||
|
Can you make a port of your script? It's a bit ugly this way - to share scripts in a text file, I think. Anyway I'll try it on next build. Can it work with ccache/portmaster?
Last edited by DutchDaemon; June 3rd, 2011 at 21:18. Reason: Mind your writing style: http://forums.freebsd.org/showthread.php?t=18043 |
|
#8
|
||||
|
||||
|
@Alt
Quote:
Quote:
That being said, this uses the standard make package clean method of installing ports, so if you install ccache, and add its configuration to the /etc/make.conf configuration file, it should still work. Last edited by DutchDaemon; June 3rd, 2011 at 21:19. Reason: Proper formatting: http://forums.freebsd.org/showthread.php?t=8816 |
|
#9
|
||||
|
||||
|
Upgraded to version 0.4
- Continuously update the ports dependencies list, to find ports that can be built - Greatly reduces (to almost none) the number of ports that are installed serially. |
|
#10
|
||||
|
||||
|
Version 0.4.1 - Bug fix - dependencies were being installed out of order ... sorry.
|
|
#11
|
||||
|
||||
|
Can you clarify ..., in a case, when more then 1 origin is specified, i.e;
Code:
pports-0.4.1.sh ftp/wget lang/php5 Thanks! Ok, I've been impatient, waiting longer then one minute for a reply, so I've taken a peak in your code and I say: it builds ftp/wget first and simultaneously builds all its dependencies, while lang/php5 is waiting. Last edited by DutchDaemon; June 5th, 2011 at 17:22. |
|
#12
|
||||
|
||||
|
@Seeker
Excellent question! The answer is: Quote:
The algorithm goes something like this: - Run make config-conditional for all specified origins, unless -b is specified - Gather list of dependencies for all specified origins (one at a time), store in "index" - Gather list of dependencies for all dependencies, store in "index" - Repeat until all possible dependencies are indexed - Find all ports in the above "index" that are leaves (have no dependencies of their own), store in "install" - Build all leaves listed in the "install" file, up to -j amount at a time. - Rebuild "index" as needed, removing listed dependencies for an origin if the dependency was installed (will reveal more "leaves" that have no further dependencies) - Repeat until all ports are installed. Every now and then, there will be a time where the system is just building one port at a time, this is normal. It happens when a major port (such as devel/gettext or devel/libiconv) is being built, and everything requires it. I've made an effort to reduce the amount of time only one port is being built, but it will still happen. The code's not easy to read (for that I apologize). This is more of a proof-of-concept than anything else. As I mentioned, I'll probably try to get this code added to the Ports system properly, in the /usr/ports/Mk/ files. That way a user can specify make -C /usr/ports/lang/php52 parallel-package and it's done, much faster than the normal serial building method.
Last edited by DutchDaemon; June 5th, 2011 at 17:25. Reason: Proper formatting: http://forums.freebsd.org/showthread.php?t=8816 |
|
#13
|
||||
|
||||
|
You have a problem! Installing ports into DESTDIR doesn't work!
Code:
===> Chrooted make in /tmp/md.sh.cy9DSHlJ succeeded ===> Cleaning up... Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory make: chdir ===>: No such file or directory .make: chdir Creating: No such file or directory make: chdir Creating: No such file or directory .make: chdir some: No such file or directory make: chdir some: No such file or directory .make: chdir important: No such file or directory make: chdir important: No such file or directory 10make: chdir subdirectories: No such file or directory make: chdir subdirectories: No such file or directory .make: chdir Starting: No such file or directory make: chdir Starting: No such file or directory .make: chdir chrooted: No such file or directory make: chdir chrooted: No such file or directory .make: chdir make: No such file or directory make: chdir make: No such file or directory .make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory .make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory .make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory .make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory .make: chdir Chrooted: No such file or directory make: chdir Chrooted: No such file or directory .make: chdir succeeded: No such file or directory make: chdir succeeded: No such file or directory 20make: chdir Cleaning: No such file or directory make: chdir Cleaning: No such file or directory .make: chdir up...: No such file or directory make: chdir up...: No such file or directory Gathering Dependencies for lang/php5..make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory .make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory .make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory 30make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory Gathering Dependencies for sysutils/fusefs-ntfs..make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory .make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory .make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory .make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory 40make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator [: /var/db/pkg/===>: unexpected operator grep: /tmp/pports-2119/index3: No such file or directory grep: /tmp/pports-2119/index3: No such file or directory rm: /tmp/pports-2119/index3: No such file or directory Done!
Last edited by DutchDaemon; June 5th, 2011 at 17:25. |
|
#14
|
||||
|
||||
|
Well, Seeker, I have no idea what you did to cause that! The script requires a full base system to run, especially /usr/ports, /var/db/pkgs, etc.
If you want to build the ports in a full chroot environment, including ports tree, you can copy the script into the chroot, and run chroot /mnt /root/pports.sh lang/php52If you just want to install the ports to the mountpoint, I would run the script and build the ports on the main system, then use the packages created in /usr/ports/packages (created via my script.) pkg_add -P /mnt /usr/ports/packages/All/php-5.2.tbzAnyway, good luck! Last edited by DutchDaemon; June 6th, 2011 at 00:09. Reason: Proper formatting: http://forums.freebsd.org/showthread.php?t=8816 |
|
#15
|
||||
|
||||
|
My DESTDIR, has a fresh FreeBSD installation.
Your script should behave in a same way as ports d, that is, if I: Code:
cd /usr/ports/ftp/wget make install clean As you intend to integrate this, into the /usr/ports/Mk/ files, you have to fix this, as your script has to handle this, in the same way as the ports system does. So good luck to you, fixing your script.
Last edited by DutchDaemon; June 6th, 2011 at 11:23. |
|
#16
|
||||
|
||||
|
Just for you Seeker:
Version 0.4.3 2011/06/06 - Works with DESTDIR for ports (specified in /etc/make.conf) - Added DestDir option to script - pkg_add honors DestDir variable - Only checks for already installed ports once - Fixed error handling when a specified origin doesn't exist Last edited by DutchDaemon; June 7th, 2011 at 13:54. |
|
#17
|
||||
|
||||
|
DESTDIR now works!
However, your script will hang endlessly, because of a daemonized dialogs. That is: Code:
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs' pports.sh $ports Code:
last pid: 41730; load averages: 2.07, 2.19, 2.17 up 0+03:09:45 11:10:32 83 processes: 3 running, 80 sleeping CPU: 52.1% user, 0.0% nice, 46.4% system, 1.5% interrupt, 0.0% idle Mem: 58M Active, 309M Inact, 114M Wired, 2508K Cache, 111M Buf, 513M Free Swap: 2014M Total, 2014M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 18232 root 1 107 0 3668K 1576K RUN 53:27 49.76% dialog 5822 root 1 107 0 3668K 1576K RUN 55:53 48.49% dialog Code:
18232 ?? R 54:22.88 /usr/bin/dialog --checklist Options for m4 1.4.16,1 21 70 15 LIBSIGSEGV Use libsigsegv for better I would recomend, hammering a human with ALL dialogs, before building starts, via: Code:
# make config-recursive Last edited by DutchDaemon; June 7th, 2011 at 13:56. |
|
#18
|
||||
|
||||
|
Additionally ...
I need to run this code twice: Code:
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs' # It only BUILDS, so DESTDIR ends up empty pports.sh $ports Code:
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs' # This installs ... pports.sh $ports |
|
#19
|
||||
|
||||
|
This seems to be a problem with Ports system not honoring the BATCH flag when DESTDIR is specified. I'll have to go digging and figure out why.
Last edited by DutchDaemon; June 8th, 2011 at 15:50. |
|
#20
|
||||
|
||||
|
Couldn't fix the BATCH issue when DESTDIR is specified so I wrote my own DESTDIR algorithm for this script.
Check it out, tell me what you think. Last edited by DutchDaemon; June 8th, 2011 at 15:50. |
|
#21
|
||||
|
||||
|
Fixed script, was not checking for ports currently being installed correctly...
Last edited by cakersq; June 8th, 2011 at 02:40. |
|
#22
|
||||
|
||||
|
I fixed a bug before you even found it!
Version 0.5.2 - Fixed that port dependencies are inaccurate when DESTDIR is used and make config changes options from the defaults. Last edited by DutchDaemon; June 8th, 2011 at 15:50. Reason: Proper formatting: http://forums.freebsd.org/showthread.php?t=8816 |
|
#23
|
||||
|
||||
|
Quote:
Or you can use github
|
|
#24
|
||||
|
||||
|
Code:
===> Directory /tmp/md.sh.zBpkvMWn does not exist ===> Please set DESTDIR to a valid jail environment. *** Error code 1 Stop in /usr/ports/ftp/wget. ===> Directory /tmp/md.sh.zBpkvMWn does not exist ===> Please set DESTDIR to a valid jail environment. *** Error code 1 Stop in /usr/ports/lang/php5. Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory make: chdir ===>: No such file or directory .make: chdir Directory: No such file or directory make: chdir Directory: No such file or directory .make: don't know how to make package-name. Stop make: chdir /tmp/md.sh.zBpkvMWn: No such file or directory .make: chdir does: No such file or directory make: chdir does: No such file or directory .make: chdir not: No such file or directory make: chdir not: No such file or directory .make: chdir exist: No such file or directory make: chdir exist: No such file or directory .make: chdir Please: No such file or directory make: chdir Please: No such file or directory .make: chdir set: No such file or directory make: chdir set: No such file or directory 10make: chdir DESTDIR: No such file or directory make: chdir DESTDIR: No such file or directory .make: chdir valid: No such file or directory make: chdir valid: No such file or directory .make: chdir jail: No such file or directory make: chdir jail: No such file or directory .make: chdir environment.: No such file or directory make: chdir environment.: No such file or directory .make: chdir boot.txz: Not a directory make: chdir boot.txz: No such file or directory .make: chdir Error: No such file or directory make: chdir Error: No such file or directory .make: chdir code: No such file or directory make: chdir code: No such file or directory .make: chdir 1: No such file or directory make: chdir 1: No such file or directory .make: chdir Stop: No such file or directory make: chdir Stop: No such file or directory .make: chdir in: No such file or directory make: chdir in: No such file or directory 20make: chdir /usr/ports/ftp/wget.: No such file or directory make: chdir /usr/ports/ftp/wget.: No such file or directory Gathering Dependencies for lang/php5..make: chdir /usr/ports/lang/php5.: No such file or directory make: chdir /usr/ports/lang/php5.: No such file or directory [1/2] Installing ftp/wget [2/2] Installing lang/php5 Done!
Last edited by DutchDaemon; June 8th, 2011 at 16:08. |
|
#25
|
||||
|
||||
|
How are you setting DESTDIR?
Edit: Never mind, I figured it out. You're setting it in the environment, via export or setenv. ... Fix will be provided shortly. Last edited by cakersq; June 9th, 2011 at 01:49. |
![]() |
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Parallel to Parallel cable + additional serial | mk | General | 7 | January 4th, 2011 17:05 |
| Port dependencies as packages rather than building more ports recursively | kpedersen | Installation and Maintenance of FreeBSD Ports or Packages | 9 | November 25th, 2010 22:55 |
| Parallel Port (inb, outb, /sys/io.h) | Hornpipe2 | Porting New Software | 2 | September 1st, 2009 15:13 |
| I need a second parallel port... | michaelrmgreen | System Hardware | 7 | May 12th, 2009 15:26 |
| port linux script to bsd | perhansen | Porting New Software | 0 | April 30th, 2009 17:44 |