Tinderbox repeatedly rebuilds same ports

jef · May 25, 2012

Tiring of having to run ports-mgmt/portmaster in numerous jails to rebuild the same thing over and over again, I installed ports-mgmt/tinderbox and configured it using the README.

I created a "jail" (built from source) and a "ports tree" (complete) for it, created a "build" and then added a list of ports consistent from what I would have added by hand; the ports I explicitly wanted, letting ./tc addPort deal with the dependencies. The 62 ports I added, not unexpectedly, resulted in 353 total ports added to the build. I then added the build to the tinderd queue using ./tc addBuildPortsQueueEntry (without a -p option, as described in the README).

As I watch the build process, I see ports being built repeatedly (and successfully), even though neither the jail nor the ports tree has changed. The web UI indicates success for these ports' previous build, so tinderbox should already have a "good" build for them. It appears to me that this is when the port is a dependency of another port. Ones I have caught by eye include; automake, autoconf, db42, gettext, gtk20, gconf2, gperf, the suite of docbook variants, and a bunch of libs.

Even with distfile caching and ccache enabled, it is painfully slow (coming up on a day and I'm still not half-way through things).

Code:

tinderd_flags="-nullfs -plistcheck"

What am I missing about how the logic for determining what to build and when works for tinderd and tinderbuild?

Before I make a decision if I'm going to abandon ports-mgmt/tinderbox, I'd at least like to understand why it is doing what it does, to give it a fair shake.

(For those with less appetite for experimentation, there is some good advice for using portmaster from DougB.)

jef · May 26, 2012

Looking at the code for tinderbox and the Makefile generated suggests that it "does the right thing" -- using make and a set of rules that rebuild something if any of its dependencies change. (Excerpted and edited for clarity)

Code:

all: libxml2-2.7.8_3.tbz
libxml2-2.7.8_3: libxml2-2.7.8_3.tbz
libxml2-2.7.8_3.tbz: gettext-0.18.1.1.tbz libiconv-1.14.tbz gmake-3.82.tbz pkg-config-0.25_1.tbz
        @/usr/local/tinderbox/scripts/lib/portbuild # followed by args to build libxml2

So why all the rebuilding?

It looks like the answer is in ${pb}/scripts/lib/tc_command.sh in the tinderbuild() function, before the build is done:

Code:

    if [ ${norebuild} = 0 ]; then
        for port in ${ports}; do
            pkgname=$(${tc} getPortLastBuiltVersion -b ${build} -d ${port})
            if [ ! -z "${pkgname}" ]; then
                find -H ${pkgDir} -name ${pkgname}${PKGSUFFIX} -delete
            fi
        done
    fi

${pkgDir} is set in ${pb}/scripts/lib/tinderlib.sh to ${pb}/packages/${build} meaning this pre-deletes the target.

Adding -norebuild to the tinderd_flags (or command line) prevents this behavior.

I'm too cross-eyed after looking at all that code (which is pretty readable, I must say) to completely think through why the recursive rebuild is happening, but I'm guessing that this is the key to preventing it. (I'm guessing one build trounces on another's dependencies, which then causes a ripple effect.)

Tinderbox repeatedly rebuilds same ports

jef

jef