1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Parallel Port Building Script

Discussion in 'Userland Programming and Scripting' started by cakersq, Jun 3, 2011.

  1. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    I was annoyed when building ports used only one of my servers processor cores, and annoyed at having to install Perl/Ruby/Python just to make the INDEX-#.

    After an afternoon of coding, I wrote a handy SH script to install ports in parallel, accounting for dependencies, and not requiring installing another language.

    It's a little rough, but I got it working to install X and Gnome.

    What do you think (See Attached)?
     

    Attached Files:

  2. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,723
    Thanks Received:
    2,272
    How old is the ports tree on that machine? The ports system has supported multiple jobs for quite a while now. See
    % less -p JOBS /usr/ports/Mk/bsd.port.mk
     
  3. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    I'm aware of the JOBS support in Ports, but in my experience, most ports do not have that enabled, and forcing it usually doesn't work, especially when dependencies are required.

    This way parallel building is supported for EVERY port.
     
  4. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,723
    Thanks Received:
    2,272
    Bringing this up on the ports mailing list might be good. Maybe the advantages of both can be combined.
     
  5. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Updated to version 0.2
    - Fixed falsely removing dependencies from to-be-installed list
    - Fixed default number of jobs to CPUs+1
    - Fixed port count display
    - Output pkg_add to /dev/null (Quiet)
     

    Attached Files:

  6. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Updated to version 0.3
    - Fixed checking for already installed packages
    - Improved usage help message
    - Improved output messages (removed some, corrected others)

    That's the last update for a while, next will focus on speeding up the process further:
    - Speed up dependency gathering phase
    - Build larger packages first
    - Background distfile download and extraction
     

    Attached Files:

  7. Alt

    Alt New Member

    Messages:
    726
    Thanks Received:
    79
    Can you make a port of your script? It's a bit ugly this way - to share scripts in a text file, I think. Anyway I'll try it on next build. Can it work with ccache/portmaster?
     
  8. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    @Alt

    The concept of installing a port to install ports never sat well with me. There are also 1000's of port management ports out there.

    My goal was to create this using a minimum amount of additional software, so that it can be used on any system. What I will probably try to do is to integrate it into the Ports system (bsd.pports.mk or something), that way you can go into a port, and specify make parallel-package or make parallel-install clean and have it go.

    That being said, this uses the standard make package clean method of installing ports, so if you install ccache, and add its configuration to the /etc/make.conf configuration file, it should still work.
     
  9. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Upgraded to version 0.4
    - Continuously update the ports dependencies list, to find ports that can be built
    - Greatly reduces (to almost none) the number of ports that are installed serially.
     

    Attached Files:

  10. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Version 0.4.1 - Bug fix - dependencies were being installed out of order ... sorry.
     

    Attached Files:

  11. Seeker

    Seeker Member

    Messages:
    859
    Thanks Received:
    11
    Can you clarify ..., in a case, when more then 1 origin is specified, i.e;
    Code:
    pports-0.4.1.sh ftp/wget lang/php5

    Will it firstly build ftp/wget and simultaneously build all its dependencies, while lang/php5 is waiting OR will it be simultaneously building both origins and simultaneously all their dependencies?

    Thanks!

    Ok, I've been impatient, waiting longer then one minute for a reply, so I've taken a peak in your code and I say: it builds ftp/wget first and simultaneously builds all its dependencies, while lang/php5 is waiting.
     
  12. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    @Seeker

    Excellent question! The answer is:
    The code gathers the dependencies for all of the origins specified serially (one at a time), but will then build all dependencies together. So it is possible the second origin completes before the first origin.

    The algorithm goes something like this:
    - Run make config-conditional for all specified origins, unless -b is specified
    - Gather list of dependencies for all specified origins (one at a time), store in "index"
    - Gather list of dependencies for all dependencies, store in "index"
    - Repeat until all possible dependencies are indexed
    - Find all ports in the above "index" that are leaves (have no dependencies of their own), store in "install"
    - Build all leaves listed in the "install" file, up to -j amount at a time.
    - Rebuild "index" as needed, removing listed dependencies for an origin if the dependency was installed (will reveal more "leaves" that have no further dependencies)
    - Repeat until all ports are installed.

    Every now and then, there will be a time where the system is just building one port at a time, this is normal. It happens when a major port (such as devel/gettext or devel/libiconv) is being built, and everything requires it. I've made an effort to reduce the amount of time only one port is being built, but it will still happen.

    The code's not easy to read (for that I apologize). This is more of a proof-of-concept than anything else. As I mentioned, I'll probably try to get this code added to the Ports system properly, in the /usr/ports/Mk/ files. That way a user can specify make -C /usr/ports/lang/php52 parallel-package and it's done, much faster than the normal serial building method.
     
  13. Seeker

    Seeker Member

    Messages:
    859
    Thanks Received:
    11
    You have a problem! Installing ports into DESTDIR doesn't work!

    Code:
    ===>  Chrooted make in /tmp/md.sh.cy9DSHlJ succeeded
    ===>  Cleaning up...
    Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
    make: chdir ===>: No such file or directory
    .make: chdir Creating: No such file or directory
    make: chdir Creating: No such file or directory
    .make: chdir some: No such file or directory
    make: chdir some: No such file or directory
    .make: chdir important: No such file or directory
    make: chdir important: No such file or directory
    10make: chdir subdirectories: No such file or directory
    make: chdir subdirectories: No such file or directory
    .make: chdir Starting: No such file or directory
    make: chdir Starting: No such file or directory
    .make: chdir chrooted: No such file or directory
    make: chdir chrooted: No such file or directory
    .make: chdir make: No such file or directory
    make: chdir make: No such file or directory
    .make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
    make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
    .make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
    make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
    .make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
    make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
    .make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
    make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
    .make: chdir Chrooted: No such file or directory
    make: chdir Chrooted: No such file or directory
    .make: chdir succeeded: No such file or directory
    make: chdir succeeded: No such file or directory
    20make: chdir Cleaning: No such file or directory
    make: chdir Cleaning: No such file or directory
    .make: chdir up...: No such file or directory
    make: chdir up...: No such file or directory
    
    Gathering Dependencies for lang/php5..make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
    make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
    .make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
    make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
    .make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
    make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
    30make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory
    make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory
    
    Gathering Dependencies for sysutils/fusefs-ntfs..make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
    make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
    .make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
    make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
    .make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
    make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
    .make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
    make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
    40make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory
    make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory
    
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    [: /var/db/pkg/===>: unexpected operator
    grep: /tmp/pports-2119/index3: No such file or directory
    grep: /tmp/pports-2119/index3: No such file or directory
    rm: /tmp/pports-2119/index3: No such file or directory
    Done!


    You have to fix this!
    ;)
     
  14. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Well, Seeker, I have no idea what you did to cause that! The script requires a full base system to run, especially /usr/ports, /var/db/pkgs, etc.

    If you want to build the ports in a full chroot environment, including ports tree, you can copy the script into the chroot, and run
    chroot /mnt /root/pports.sh lang/php52

    If you just want to install the ports to the mountpoint, I would run the script and build the ports on the main system, then use the packages created in /usr/ports/packages (created via my script.)
    pkg_add -P /mnt /usr/ports/packages/All/php-5.2.tbz

    Anyway, good luck!
     
  15. Seeker

    Seeker Member

    Messages:
    859
    Thanks Received:
    11
    My DESTDIR, has a fresh FreeBSD installation.

    Your script should behave in a same way as ports d, that is, if I:
    Code:
    cd /usr/ports/ftp/wget
    make install clean

    it will chroot into DESTDIR and successfully install the port, EVEN if DESTDIR doesn't have a /usr/ports tree, because it mounts into the chrooted DESTDIR, via nullfs, what it needs.

    As you intend to integrate this, into the /usr/ports/Mk/ files, you have to fix this, as your script has to handle this, in the same way as the ports system does.

    So good luck to you, fixing your script.
    ;)
     
  16. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Just for you Seeker:

    Version 0.4.3 2011/06/06
    - Works with DESTDIR for ports (specified in /etc/make.conf)
    - Added DestDir option to script
    - pkg_add honors DestDir variable
    - Only checks for already installed ports once
    - Fixed error handling when a specified origin doesn't exist
     

    Attached Files:

  17. Seeker

    Seeker Member

    Messages:
    859
    Thanks Received:
    11
    DESTDIR now works!
    However, your script will hang endlessly, because of a daemonized dialogs. That is:
    Code:
    ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
    
    pports.sh $ports


    Dialogs will be thrown at the start, for each origin, before builds occur. However, once any dependency is started to be built, its dialog will hang endlessly, as it won't be thrown to user.

    Code:
    last pid: 41730;  load averages:  2.07,  2.19,  2.17                                                    up 0+03:09:45  11:10:32
    83 processes:  3 running, 80 sleeping
    CPU: 52.1% user,  0.0% nice, 46.4% system,  1.5% interrupt,  0.0% idle
    Mem: 58M Active, 309M Inact, 114M Wired, 2508K Cache, 111M Buf, 513M Free
    Swap: 2014M Total, 2014M Free
    
      PID USERNAME        THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
    18232 root              1 107    0  3668K  1576K RUN     53:27 49.76% dialog
     5822 root              1 107    0  3668K  1576K RUN     55:53 48.49% dialog


    I had to kill the dialog processes to continue builds..

    Code:
    18232  ??  R     54:22.88 /usr/bin/dialog --checklist Options for m4 1.4.16,1 21 70 15 LIBSIGSEGV Use libsigsegv for better


    You have to fix this.

    I would recomend, hammering a human with ALL dialogs, before building starts, via:
    Code:
    # make config-recursive

    So human can take a walk, doing its "human stuff".
     
  18. Seeker

    Seeker Member

    Messages:
    859
    Thanks Received:
    11
    Additionally ...
    I need to run this code twice:
    Code:
    ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
    # It only BUILDS, so DESTDIR ends up empty
    pports.sh $ports

    Code:
    ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
    # This installs ...
    pports.sh $ports


    Also devfs remained mounted in chroot
     
  19. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    This seems to be a problem with Ports system not honoring the BATCH flag when DESTDIR is specified. I'll have to go digging and figure out why.
     
  20. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Couldn't fix the BATCH issue when DESTDIR is specified so I wrote my own DESTDIR algorithm for this script.

    Check it out, tell me what you think.
     

    Attached Files:

  21. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    Fixed script, was not checking for ports currently being installed correctly...
     

    Attached Files:

  22. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    I fixed a bug before you even found it!

    Version 0.5.2
    - Fixed that port dependencies are inaccurate when DESTDIR is used and make config changes options from the defaults.
     

    Attached Files:

  23. graudeejs

    graudeejs Well-Known Member

    Messages:
    4,594
    Thanks Received:
    633
    If you want I can provide Mercurial repository for your project at http://hg.bsdroot.lv also bugzilla via bugs.bsdroot.lv
    Or you can use github :)
     
  24. Seeker

    Seeker Member

    Messages:
    859
    Thanks Received:
    11
    Code:
    ===>  Directory /tmp/md.sh.zBpkvMWn does not exist
    ===>  Please set DESTDIR to a valid jail environment.
    *** Error code 1
    
    Stop in /usr/ports/ftp/wget.
    ===>  Directory /tmp/md.sh.zBpkvMWn does not exist
    ===>  Please set DESTDIR to a valid jail environment.
    *** Error code 1
    
    Stop in /usr/ports/lang/php5.
    Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
    make: chdir ===>: No such file or directory
    .make: chdir Directory: No such file or directory
    make: chdir Directory: No such file or directory
    .make: don't know how to make package-name. Stop
    make: chdir /tmp/md.sh.zBpkvMWn: No such file or directory
    .make: chdir does: No such file or directory
    make: chdir does: No such file or directory
    .make: chdir not: No such file or directory
    make: chdir not: No such file or directory
    .make: chdir exist: No such file or directory
    make: chdir exist: No such file or directory
    .make: chdir Please: No such file or directory
    make: chdir Please: No such file or directory
    .make: chdir set: No such file or directory
    make: chdir set: No such file or directory
    10make: chdir DESTDIR: No such file or directory
    make: chdir DESTDIR: No such file or directory
    .make: chdir valid: No such file or directory
    make: chdir valid: No such file or directory
    .make: chdir jail: No such file or directory
    make: chdir jail: No such file or directory
    .make: chdir environment.: No such file or directory
    make: chdir environment.: No such file or directory
    .make: chdir boot.txz: Not a directory
    make: chdir boot.txz: No such file or directory
    .make: chdir Error: No such file or directory
    make: chdir Error: No such file or directory
    .make: chdir code: No such file or directory
    make: chdir code: No such file or directory
    .make: chdir 1: No such file or directory
    make: chdir 1: No such file or directory
    .make: chdir Stop: No such file or directory
    make: chdir Stop: No such file or directory
    .make: chdir in: No such file or directory
    make: chdir in: No such file or directory
    20make: chdir /usr/ports/ftp/wget.: No such file or directory
    make: chdir /usr/ports/ftp/wget.: No such file or directory
    
    Gathering Dependencies for lang/php5..make: chdir /usr/ports/lang/php5.: No such file or directory
    make: chdir /usr/ports/lang/php5.: No such file or directory
    
    [1/2] Installing ftp/wget
    [2/2] Installing lang/php5
    Done!

    Last two ports are installed, because there was a previous run, from yesterday. DESTDIR is valid, as was used by other parts of MY script, before and after calling your script.
    ;)
     
  25. cakersq

    cakersq New Member

    Messages:
    31
    Thanks Received:
    1
    How are you setting DESTDIR?

    Edit: Never mind, I figured it out. You're setting it in the environment, via export or setenv. ... Fix will be provided shortly.