Building in one jail, installing in another

Shew · #1 October 7th, 2010, 06:29

Hello.

I am trying to use a chroot jail to compile ports such that I can install them in other jails. I have multiple jails and previously I had compiled programs in each jail, which meant that a huge amount of compile time had to be duplicated for each jail. I am trying to discover a better method, but it seems that everything I have tried has be very unreliable (such that I can only do it for one or two programs before failing).

I am currently trying to do something like this:
chroot /usr/shew/chroots/compile make -C /usr/ports/"$category"/"$program" package-recursive clean

Previously I had tried the following:
Compiling without using make clean and then mounting the ports folders in other jails (via nullfs) and using make install
Both package and install the ports when installing to and individual jail, then using a combination of pkg_add and make to install new ports in other jails. Installing some ports common to all of the jails, and then copying the entire jail directory or using unionfs to use that jail as a template for others.

Nothing seems to work as well as duplicating the work for all the jails. With what I am currently trying, for example, I get an error with Python:

Code:

tar: lib/python2.6/lib-dynload/_multiprocessing.so: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
pkg_create: make_dist: tar command failed with code 256

Now, the packing lists seem to be fairly static, even though one might change the package options (though, I have not in this case), so I'm not sure this is going to work once I start trying to customize my options.

Is there some recommended general method for installing ports across jails? I sure would appreciate a few tips.

Thanks.

**SirDice** · #2 October 7th, 2010, 07:16

Use a jail for building. That's what I've been doing for years. Build packages, make use of portmaster. Then mount_nullfs your ports tree and install the built packages inside each other jail.

Shew · #3 October 10th, 2010, 07:04

I just tried make package-recursive in a jail and I am getting the same errors (different ports reporting that files are missing when tarring a package). I haven't tried portmaster yet, so I guess I will try that next. There seems to be something buggy going on with package-recursive though.

anomie · #4 October 13th, 2010, 00:49

@Shew: what about building packages in a full-on FreeBSD jail - not a "chroot jail"? (Did I misunderstand you?)

Shew · #5 October 16th, 2010, 00:36

@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

It seems my problem was that I was making a mistake with the port configuration such that the port was not set to the default configuration. I have that fixed now, but it still leaves my that problem when I start to customize the build options: If the customized builds make an extra file that needs to be packaged, or don't make a file that is packaged in the default build, it will fail. I looked in the Porter's Handbook and it suggests that you can get the port to build a packing list on the fly, but it looks like this would have to be baked into the Makefile. Is there a different way I can package customized ports? Some ports seem to handle it OK and some ports don't (like Python and threading).

acheron · #6 October 16th, 2010, 18:08

You should have a look at ports-mgmt/tinderbox.

**SirDice** · #7 October 19th, 2010, 15:08

Quote:

Originally Posted by Shew

@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

To stop further confusion, chroot != jail. Do not refer to a chroot'ed environment as a jail. They are completely different.

mecano · #8 November 25th, 2011, 11:47

Quote:

Originally Posted by SirDice

Use a jail for building. That's what I've been doing for years. Build packages, make use of portmaster. Then mount_nullfs your ports tree and install the built packages inside each other jail.

Is that to say you are maintaining a "building jail" ports tree separate from main host one?
For now I'm using the main host mount_nullfs'ed ports tree in a "building jail" and some awk commands that check dependencies on packages, generate a list of packages, then in target jail, copy them, add them, replace them (just in case), quiet.. awkward.
Your solution looks far more elegant, any caveats to mention ? I thought it was a very bad idea to maintain more than one ports tree per box, not the case?

**SirDice** · #9 November 25th, 2011, 13:54

Quote:

Originally Posted by mecano

Is that to say you are maintaining a "building jail" ports tree separate from main host one?

No. It's the same tree.
# mount -t nullfs /usr/ports /jail/build/usr/ports/

Because I do have seperate filesystems for packages and distfiles I also need to do
# mount -t nullfs /usr/ports/distfiles /jail/build/usr/ports/distfiles/
# mount -t nullfs /usr/ports/packages /jail/build/usr/ports/packages/

And I'm lazy so I added those to the jail's fstab. In the host's rc.conf:

Code:

jail_enable="YES"
jail_list="build"
jail_build_rootdir="/jails/build/"
jail_build_hostname="build.dicelan.home"
jail_build_flags="-l -U root -n build"
jail_build_ip="192.168.100.200"
jail_build_interface="re0"
jail_build_mount_enable="YES"
jail_build_devfs_enable="YES"

For jail_build_mount_enable to work you need to create a seperate /etc/fstab.build for the jail:

Code:

/usr/src/             /jail/build/usr/src/             nullfs  ro,noatime  0  0
/usr/ports            /jail/build/usr/ports            nullfs  rw,noatime  0  0
/usr/ports/distfiles  /jail/build/usr/ports/distfiles  nullfs  rw,noatime  0  0
/usr/ports/packages   /jail/build/usr/ports/packages   nullfs  rw,noatime  0  0

This will make sure the filesystems are mounted/unmounted when the jail is started/stopped.
I've mounted /usr/src/ read-only as some ports need access to the sources.

When the jail is running just:
# jexec build /usr/bin/su -
The su(1) will make sure the environment is setup correctly. Once inside the jail just build packages as you would normally do.

mecano · #10 December 6th, 2011, 09:35

Dear Sir, thanks for such a detailed answer! Are you deploying ports packages by mounting nullfs as well? For example temporary mounting /var/ports/packages (or another location where you built packages are stored) in the target jail as read only and install from there?

**SirDice** · #11 December 6th, 2011, 10:32

Quote:

Originally Posted by mecano

Are you deploying ports packages by mounting nullfs as well? For example temporary mounting /var/ports/packages (or another location where you built packages are stored) in the target jail as read only and install from there?

On the same machine, yes. For other machines I just have my /usr/ports/packages/ NFS exported read-only.

fbsd1 · #12 December 7th, 2011, 02:09

You are trying to RE-Invent the wheel again.
Install the "qjail" port and use it to build all your jails. Read the documentation for simple way to populate your jails with a pre-installed group of ports.

mecano · #13 December 8th, 2011, 15:19

So does ezjail with flavours.

When installing/updating ports in target jail, is it better to
pkg_add -r then pkg_replace -Bf or to pkg_replace -Bf then pkg_add -r?

**dougb@** · #14 December 31st, 2011, 23:34

Quote:

Originally Posted by Shew

@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

It seems my problem was that I was making a mistake with the port configuration such that the port was not set to the default configuration. I have that fixed now, but it still leaves my that problem when I start to customize the build options: If the customized builds make an extra file that needs to be packaged, or don't make a file that is packaged in the default build, it will fail.

Someone else already suggested portmaster as your tool to build packages, I will ever-so-humbly state that I agree.

As long as the infrastructure in the ports tree is correct, using the -g option for portmaster will produce correct packages.

The issue of customization is more complex however. There are 3 ways to accomplish this. If all of the jails can mount /usr/ports and /var/db/ports then you can use the nullfs trick that others have mentioned so that they all see the same options.

If they are physically separate, then you have 2 choices. The simplest is to copy the package and all of its dependencies to the remote jails and install them by hand. Obviously that doesn't scale well.

The other option is to create a custom INDEX that reflects your customizations and install on the jails using portmaster --packages-only --index-only. This has advantages of not needing to mount anything in the remote jails, but requires that you set up INDEX creation/distribution, and package distribution on your package-building system. It's not particularly difficult, but it is a bit of a project. There is some information in the portmaster man page, and the pkg_add man page should also be helpful. For custom INDEX creation I highly recommend ports-mgmt/p5-FreeBSD-Portindex. It takes some time to set up, but subsequent INDEX creation is then very fast.

Hope this helps,

Doug

Norrland · #15 June 5th, 2012, 14:27

Quote:

Originally Posted by mecano

So does ezjail with flavours.

When installing/updating ports in target jail, is it better to
pkg_add -r then pkg_replace -Bf or to pkg_replace -Bf then pkg_add -r?

Been wondering the same thing.
I'm using ezjails with one "build" jail in which I build ports. portmaster -g <port> and then install on destination jail with pkg_add <package>.

It's the update part I'm worried about, can't get the workflow straight in my head.
Hope this is relevant to the thread.

**SirDice** · #16 June 5th, 2012, 14:30

Just wipe the jail and start fresh. Then you'll never have any updating issues. I takes a little more effort but it pays off in the end.

Norrland · #17 June 5th, 2012, 14:33

Hm, so you mean pkg_delete -a? Or create a whole new jail?

**SirDice** · #18 June 5th, 2012, 14:38

Quote:

Originally Posted by Norrland

Hm, so you mean pkg_delete -a? Or create a whole new jail?

Creating a whole new jail is faster

You don't want to keep any settings anyway as nothing is configured inside the build jail. You only use it to build things.

Norrland · #19 June 5th, 2012, 14:42

Hehe. Don't think that'll work for some of my jails (IRC shell for some friends

).

When creating a jail, with ezjail and flavours I've noticed that it won't install the packages I put in $FLAVOUR/pkg. Maybe I've missed something there.

bbzz · #20 June 5th, 2012, 14:48

There's no need to blow up a whole build jail every time unless there are major changes (like recent) and especially if you are building so much different things (such as KDE and GNOME) for different machines.

Simply use portmaster in build jail to rebuild what you need, make sure everything is working as far as dependencies go (pkg_libchk) and then use portmaster -PP on install machine to update packets.

One thing you want to make sure is that build jail and install machine point to same ports tree (or at least INDEX file), and options, /var/db/ports. That way there's no issues.

Norrland · #21 June 6th, 2012, 11:16

Quote:

Originally Posted by bbzz

There's no need to blow up a whole build jail every time unless there are major changes (like recent) and especially if you are building so much different things (such as KDE and GNOME) for different machines.

Simply use portmaster in build jail to rebuild what you need, make sure everything is working as far as dependencies go (pkg_libchk) and then use portmaster -PP on install machine to update packets.

One thing you want to make sure is that build jail and install machine point to same ports tree (or at least INDEX file), and options, /var/db/ports. That way there's no issues.

How do I make sure that the jails use the same ports tree? Is it described in some config file, or do I have too mount the build-jail ports into application jails /usr/ports?

**SirDice** · #22 June 6th, 2012, 11:24

Mount your own ports tree in the jail using nullfs(5):

# mount -t nullfs /usr/ports /jails/j1/usr/ports

I have, in /etc/rc.conf:

Code:

jail_build_mount_enable="YES"

And created an /etc/fstab.build:

Code:

tmpfs                           /jails/j-build-amd64/tmp                        tmpfs   rw,mode=1777    0       0
/usr/src                        /jails/j-build-amd64/usr/src                    nullfs  ro,noatime      0       0
/usr/ports                      /jails/j-build-amd64/usr/ports                  nullfs  rw,noatime      0       0
/usr/ports/distfiles            /jails/j-build-amd64/usr/ports/distfiles        nullfs  rw,noatime      0       0
/usr/ports/packages             /jails/j-build-amd64/usr/ports/packages         nullfs  rw,noatime      0       0

That makes sure the filesystems are automatically mounted/unmounted when the jail starts/stops.

This same ports tree is also NFS exported (read-only) and used on all my other machines.

Norrland · #23 June 6th, 2012, 12:08

Okay, easey peasey then.

Any special flag in portmaster to build packages for ports which a new version is available? Or should I build with build # portmaster -g type/port for the packages that need an update, and then install with j1 # pkg_add /usr/ports/packages/package.tar.gz?

bbzz · #24 June 6th, 2012, 12:52

It's better to update with portmaster since it will also update dependencies, whereas pkg_add will add missing packages and give you warning about dependencies which are not in check (e.g cairo-1.10.2 needed but cairo_1.10.1 installed, etc).

# portmaster -PP -D -B --packages-local --local-packagedir=/usr/ports/packages -y

Norrland · #25 June 26th, 2012, 07:32

I've been trying out the things mentioned in previous posts.
And the thing I found working somewhat OK is this.
host # ezjail-admin update -P Update ports tree.
build-jail # portmaster -D -a To update stuff in the jails.
app-jail # portmaster -D -B -a Update ports on specific jail.

My setup looks something like this.
host # mount | grep ports

Code:

/usr/local/jails/build/var/ports on /usr/local/jails/ns1/var/ports (nullfs, local, noatime)
/usr/local/jails/build/var/db/ports on /usr/local/jails/ns1/var/db/ports (nullfs, local, noatime)

When using this I had to have /var/ports writable on the app-jail. While this is working as far as I can see, I'm not sure if I'm creating more work for myself

.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Solved] Error starting jail on 8.1# jail: execvp: /bin/sh: Exec format error	ghostcorps	General	5	September 13th, 2010 04:34
[Solved] Installing openssh in a jail	sand_man	Installing & Upgrading	2	July 29th, 2010 08:19
[Solved] black window after building and installing kernel for update	beaute	Installing & Upgrading	2	June 4th, 2010 09:28
Installing software from host to Jail	ligregni	Installation and Maintenance of FreeBSD Ports or Packages	3	May 31st, 2010 12:00
[Solved] qt error when building jahshaka 2.0	pietro3dartist	Porting New Software	3	November 27th, 2009 20:51

The Following User Says Thank You to SirDice For This Useful Post:
mecano (November 25th, 2011)

The Following User Says Thank You to SirDice For This Useful Post:
mecano (December 6th, 2011)

The Following 3 Users Say Thank You to bbzz For This Useful Post:
jb_fvwm2 (June 7th, 2012), mecano (June 7th, 2012), Norrland (June 6th, 2012)