Building in one jail, installing in another

Installing and maintaining the FreeBSD Ports Collection or FreeBSD Packages (i.e. third party software).

Building in one jail, installing in another

Postby Shew » 07 Oct 2010, 05:29

Hello.

I am trying to use a chroot jail to compile ports such that I can install them in other jails. I have multiple jails and previously I had compiled programs in each jail, which meant that a huge amount of compile time had to be duplicated for each jail. I am trying to discover a better method, but it seems that everything I have tried has be very unreliable (such that I can only do it for one or two programs before failing).

I am currently trying to do something like this:
[CMD=" "]chroot /usr/shew/chroots/compile make -C /usr/ports/"$category"/"$program" package-recursive clean[/CMD]

Previously I had tried the following:
Compiling without using [cmd=]make clean[/cmd] and then mounting the ports folders in other jails (via nullfs) and using [cmd=]make install[/cmd]
Both package and install the ports when installing to and individual jail, then using a combination of pkg_add and make to install new ports in other jails. Installing some ports common to all of the jails, and then copying the entire jail directory or using unionfs to use that jail as a template for others.

Nothing seems to work as well as duplicating the work for all the jails. With what I am currently trying, for example, I get an error with Python:

Code: Select all
tar: lib/python2.6/lib-dynload/_multiprocessing.so: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
pkg_create: make_dist: tar command failed with code 256

Now, the packing lists seem to be fairly static, even though one might change the package options (though, I have not in this case), so I'm not sure this is going to work once I start trying to customize my options.

Is there some recommended general method for installing ports across jails? I sure would appreciate a few tips.

Thanks.
Shew
Junior Member
 
Posts: 3
Joined: 07 Oct 2010, 05:08

Postby SirDice » 07 Oct 2010, 06:16

Use a jail for building. That's what I've been doing for years. Build packages, make use of portmaster. Then mount_nullfs your ports tree and install the built packages inside each other jail.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby Shew » 10 Oct 2010, 06:04

I just tried [CMD=" "]make package-recursive[/CMD] in a jail and I am getting the same errors (different ports reporting that files are missing when tarring a package). I haven't tried portmaster yet, so I guess I will try that next. There seems to be something buggy going on with [FILE]package-recursive[/FILE] though.
Shew
Junior Member
 
Posts: 3
Joined: 07 Oct 2010, 05:08

Postby anomie » 12 Oct 2010, 23:49

@Shew: what about building packages in a full-on FreeBSD jail - not a "chroot jail"? (Did I misunderstand you?)
"Do Not Engage in Useless Activity" --Miyamoto Musashi
User avatar
anomie
Member
 
Posts: 783
Joined: 17 Nov 2008, 04:37
Location: Texas

Postby Shew » 15 Oct 2010, 23:36

@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

It seems my problem was that I was making a mistake with the port configuration such that the port was not set to the default configuration. I have that fixed now, but it still leaves my that problem when I start to customize the build options: If the customized builds make an extra file that needs to be packaged, or don't make a file that is packaged in the default build, it will fail. I looked in the Porter's Handbook and it suggests that you can get the port to build a packing list on the fly, but it looks like this would have to be baked into the Makefile. Is there a different way I can package customized ports? Some ports seem to handle it OK and some ports don't (like Python and threading).
Shew
Junior Member
 
Posts: 3
Joined: 07 Oct 2010, 05:08

Postby acheron » 16 Oct 2010, 17:08

You should have a look at [port]ports-mgmt/tinderbox[/port].
acheron
Junior Member
 
Posts: 92
Joined: 18 Aug 2010, 11:37
Location: France

Postby SirDice » 19 Oct 2010, 14:08

Shew wrote:@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

To stop further confusion, chroot != jail. Do not refer to a chroot'ed environment as a jail. They are completely different.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby mecano » 25 Nov 2011, 10:47

SirDice wrote:Use a jail for building. That's what I've been doing for years. Build packages, make use of portmaster. Then mount_nullfs your ports tree and install the built packages inside each other jail.


Is that to say you are maintaining a "building jail" ports tree separate from main host one?
For now I'm using the main host mount_nullfs'ed ports tree in a "building jail" and some awk commands that check dependencies on packages, generate a list of packages, then in target jail, copy them, add them, replace them (just in case), quiet.. awkward.
Your solution looks far more elegant, any caveats to mention ? I thought it was a very bad idea to maintain more than one ports tree per box, not the case?
mecano
Member
 
Posts: 106
Joined: 04 Feb 2009, 12:42
Location: France

Postby SirDice » 25 Nov 2011, 12:54

mecano wrote:Is that to say you are maintaining a "building jail" ports tree separate from main host one?

No. It's the same tree.
[cmd=#]mount -t nullfs /usr/ports /jail/build/usr/ports/[/cmd]

Because I do have seperate filesystems for packages and distfiles I also need to do
[cmd=#]mount -t nullfs /usr/ports/distfiles /jail/build/usr/ports/distfiles/[/cmd]
[cmd=#]mount -t nullfs /usr/ports/packages /jail/build/usr/ports/packages/[/cmd]

And I'm lazy so I added those to the jail's [file]fstab[/file]. In the host's [file]rc.conf[/file]:

Code: Select all
jail_enable="YES"
jail_list="build"
jail_build_rootdir="/jails/build/"
jail_build_hostname="build.dicelan.home"
jail_build_flags="-l -U root -n build"
jail_build_ip="192.168.100.200"
jail_build_interface="re0"
jail_build_mount_enable="YES"
jail_build_devfs_enable="YES"

For [file]jail_build_mount_enable[/file] to work you need to create a seperate [file]/etc/fstab.build[/file] for the jail:
Code: Select all
/usr/src/             /jail/build/usr/src/             nullfs  ro,noatime  0  0
/usr/ports            /jail/build/usr/ports            nullfs  rw,noatime  0  0
/usr/ports/distfiles  /jail/build/usr/ports/distfiles  nullfs  rw,noatime  0  0
/usr/ports/packages   /jail/build/usr/ports/packages   nullfs  rw,noatime  0  0

This will make sure the filesystems are mounted/unmounted when the jail is started/stopped.
I've mounted [file]/usr/src/[/file] read-only as some ports need access to the sources.

When the jail is running just:
[cmd=#]jexec build /usr/bin/su -[/cmd]
The [man=1]su[/man] will make sure the environment is setup correctly. Once inside the jail just build packages as you would normally do.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby mecano » 06 Dec 2011, 08:35

Dear Sir, thanks for such a detailed answer! Are you deploying ports packages by mounting [FILE]nullfs[/FILE] as well? For example temporary mounting [FILE]/var/ports/packages[/FILE] (or another location where you built packages are stored) in the target jail as read only and install from there?
mecano
Member
 
Posts: 106
Joined: 04 Feb 2009, 12:42
Location: France

Postby SirDice » 06 Dec 2011, 09:32

mecano wrote:Are you deploying ports packages by mounting nullfs as well? For example temporary mounting /var/ports/packages (or another location where you built packages are stored) in the target jail as read only and install from there?

On the same machine, yes. For other machines I just have my [file]/usr/ports/packages/[/file] NFS exported read-only.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby fbsd1 » 07 Dec 2011, 01:09

You are trying to RE-Invent the wheel again.
Install the "qjail" port and use it to build all your jails. Read the documentation for simple way to populate your jails with a pre-installed group of ports.
FreeBSD Install Guide www.a1poweruser.com
fbsd1
Member
 
Posts: 213
Joined: 26 Feb 2010, 09:43
Location: Angeles City, Philippines

Postby mecano » 08 Dec 2011, 14:19

So does [FILE]ezjail[/FILE] with flavours.

When installing/updating ports in target jail, is it better to
[FILE]pkg_add -r[/FILE] then [FILE]pkg_replace -Bf[/FILE] or to [FILE]pkg_replace -Bf[/FILE] then [FILE]pkg_add -r[/FILE]?
mecano
Member
 
Posts: 106
Joined: 04 Feb 2009, 12:42
Location: France

Postby dougb@ » 31 Dec 2011, 22:34

Shew wrote:@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

It seems my problem was that I was making a mistake with the port configuration such that the port was not set to the default configuration. I have that fixed now, but it still leaves my that problem when I start to customize the build options: If the customized builds make an extra file that needs to be packaged, or don't make a file that is packaged in the default build, it will fail.


Someone else already suggested [FILE]portmaster[/FILE] as your tool to build packages, I will ever-so-humbly state that I agree. :) As long as the infrastructure in the ports tree is correct, using the [FILE]-g[/FILE] option for [FILE]portmaster[/FILE] will produce correct packages.

The issue of customization is more complex however. There are 3 ways to accomplish this. If all of the jails can mount [FILE]/usr/ports[/FILE] and [FILE]/var/db/ports[/FILE] then you can use the [FILE]nullfs[/FILE] trick that others have mentioned so that they all see the same options.

If they are physically separate, then you have 2 choices. The simplest is to copy the package and all of its dependencies to the remote jails and install them by hand. Obviously that doesn't scale well.

The other option is to create a custom [FILE]INDEX[/FILE] that reflects your customizations and install on the jails using [FILE]portmaster --packages-only --index-only[/FILE]. This has advantages of not needing to mount anything in the remote jails, but requires that you set up [FILE]INDEX[/FILE] creation/distribution, and package distribution on your package-building system. It's not particularly difficult, but it is a bit of a project. There is some information in the [FILE]portmaster[/FILE] man page, and the [FILE]pkg_add[/FILE] man page should also be helpful. For custom [FILE]INDEX[/FILE] creation I highly recommend [port]ports-mgmt/p5-FreeBSD-Portindex[/port]. It takes some time to set up, but subsequent [FILE]INDEX[/FILE] creation is then very fast.

Hope this helps,

Doug
dougb@
Junior Member
 
Posts: 28
Joined: 20 Feb 2008, 22:44
Location: Los Angeles, California, USA

Postby Norrland » 05 Jun 2012, 13:27

mecano wrote:So does [FILE]ezjail[/FILE] with flavours.

When installing/updating ports in target jail, is it better to
[FILE]pkg_add -r[/FILE] then [FILE]pkg_replace -Bf[/FILE] or to [FILE]pkg_replace -Bf[/FILE] then [FILE]pkg_add -r[/FILE]?


Been wondering the same thing.
I'm using ezjails with one "build" jail in which I build ports. [FILE]portmaster -g <port>[/FILE] and then install on destination jail with [FILE]pkg_add <package>[/FILE].

It's the update part I'm worried about, can't get the workflow straight in my head.
Hope this is relevant to the thread. :)
Norrland
 

Postby SirDice » 05 Jun 2012, 13:30

Just wipe the jail and start fresh. Then you'll never have any updating issues. I takes a little more effort but it pays off in the end.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby Norrland » 05 Jun 2012, 13:33

Hm, so you mean [FILE]pkg_delete -a[/FILE]? Or create a whole new jail?
Norrland
 

Postby SirDice » 05 Jun 2012, 13:38

Norrland wrote:Hm, so you mean [FILE]pkg_delete -a[/FILE]? Or create a whole new jail?


Creating a whole new jail is faster ;) You don't want to keep any settings anyway as nothing is configured inside the build jail. You only use it to build things.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby Norrland » 05 Jun 2012, 13:42

Hehe. Don't think that'll work for some of my jails (IRC shell for some friends ;)).

When creating a jail, with [FILE]ezjail[/FILE] and flavours I've noticed that it won't install the packages I put in [FILE]$FLAVOUR/pkg[/FILE]. Maybe I've missed something there.
Norrland
 

Postby bbzz » 05 Jun 2012, 13:48

There's no need to blow up a whole build jail every time unless there are major changes (like recent) and especially if you are building so much different things (such as KDE and GNOME) for different machines.

Simply use [FILE]portmaster[/FILE] in build jail to rebuild what you need, make sure everything is working as far as dependencies go ([FILE]pkg_libchk[/FILE]) and then use [FILE]portmaster[/FILE] [FILE]-PP[/FILE] on install machine to update packets.

One thing you want to make sure is that build jail and install machine point to same ports tree (or at least INDEX file), and options, [FILE]/var/db/ports[/FILE]. That way there's no issues.
bbzz
Member
 
Posts: 858
Joined: 04 Nov 2010, 01:07
Location: random

Postby Norrland » 06 Jun 2012, 10:16

bbzz wrote:There's no need to blow up a whole build jail every time unless there are major changes (like recent) and especially if you are building so much different things (such as KDE and GNOME) for different machines.

Simply use [FILE]portmaster[/FILE] in build jail to rebuild what you need, make sure everything is working as far as dependencies go ([FILE]pkg_libchk[/FILE]) and then use [FILE]portmaster[/FILE] [FILE]-PP[/FILE] on install machine to update packets.

One thing you want to make sure is that build jail and install machine point to same ports tree (or at least INDEX file), and options, [FILE]/var/db/ports[/FILE]. That way there's no issues.


How do I make sure that the jails use the same ports tree? Is it described in some config file, or do I have too mount the build-jail ports into application jails [FILE]/usr/ports[/FILE]?
Norrland
 

Postby SirDice » 06 Jun 2012, 10:24

Mount your own ports tree in the jail using [man=5]nullfs[/man]:

[cmd=#]mount -t nullfs /usr/ports /jails/j1/usr/ports[/cmd]

I have, in [file]/etc/rc.conf[/file]:
Code: Select all
jail_build_mount_enable="YES"


And created an [file]/etc/fstab.build[/file]:
Code: Select all
tmpfs                           /jails/j-build-amd64/tmp                        tmpfs   rw,mode=1777    0       0
/usr/src                        /jails/j-build-amd64/usr/src                    nullfs  ro,noatime      0       0
/usr/ports                      /jails/j-build-amd64/usr/ports                  nullfs  rw,noatime      0       0
/usr/ports/distfiles            /jails/j-build-amd64/usr/ports/distfiles        nullfs  rw,noatime      0       0
/usr/ports/packages             /jails/j-build-amd64/usr/ports/packages         nullfs  rw,noatime      0       0


That makes sure the filesystems are automatically mounted/unmounted when the jail starts/stops.

This same ports tree is also NFS exported (read-only) and used on all my other machines.
Senior UNIX Engineer at Unix Support Nederland
Experience is something you don't get until just after you need it.
User avatar
SirDice
Old Fart
 
Posts: 16196
Joined: 17 Nov 2008, 16:50
Location: Rotterdam, Netherlands

Postby Norrland » 06 Jun 2012, 11:08

Okay, easey peasey then.

Any special flag in [FILE]portmaster[/FILE] to build packages for ports which a new version is available? Or should I build with [CMD="build #"]portmaster -g type/port[/CMD] for the packages that need an update, and then install with [CMD="j1 #"]pkg_add /usr/ports/packages/package.tar.gz[/CMD]?
Norrland
 

Postby bbzz » 06 Jun 2012, 11:52

It's better to update with [FILE]portmaster[/FILE] since it will also update dependencies, whereas [FILE]pkg_add[/FILE] will add missing packages and give you warning about dependencies which are not in check (e.g [FILE]cairo-1.10.2[/FILE] needed but [FILE]cairo_1.10.1[/FILE] installed, etc).

[CMD="#"]portmaster -PP -D -B --packages-local --local-packagedir=/usr/ports/packages -y[/CMD]
bbzz
Member
 
Posts: 858
Joined: 04 Nov 2010, 01:07
Location: random

Postby Norrland » 26 Jun 2012, 06:32

I've been trying out the things mentioned in previous posts.
And the thing I found working somewhat OK is this.
[CMD="host #"]ezjail-admin update -P[/CMD] Update ports tree.
[CMD="build-jail #"]portmaster -D -a[/CMD] To update stuff in the jails.
[CMD="app-jail #"]portmaster -D -B -a[/CMD] Update ports on specific jail.

My setup looks something like this.
[CMD="host #"]mount | grep ports[/CMD]
Code: Select all
/usr/local/jails/build/var/ports on /usr/local/jails/ns1/var/ports (nullfs, local, noatime)
/usr/local/jails/build/var/db/ports on /usr/local/jails/ns1/var/db/ports (nullfs, local, noatime)


When using this I had to have [FILE]/var/ports[/FILE] writable on the app-jail. While this is working as far as I can see, I'm not sure if I'm creating more work for myself :).
Norrland
 

Next

Return to Installation and Maintenance of FreeBSD Ports or Packages

Who is online

Users browsing this forum: free-and-bsd, grehan@, strandbal and 2 guests