Solved How to run multiple jails simultaneously without building already built packages in other jail

[nxjoseph]

Hello. Is it possible to run multiple jails simultaneously without building packages that already built in other jail? I created a jail while other one is running but once I tried to bulk with dry-run mode to see but it said that it would build a lot of packages but they are already built in other jail. Can i prevent this? Can I share same pkg repository between those jails?

 

[Erichans]

I don't know all/the best scenarios for installing/building/managing packages for inside jails, but the way you phrased:

I created a jail while other one is running but once I tried to bulk with dry-run mode to see but it said that it would build a lot of packages but they are already built in other jail.

implies to me that you expect that a build/installed package collection inside one jail should be available to other jails (not being a jail-inside-that-jail), that's not the case because in essence FreeBSD jails are a lightweight OS-level virtualisation mechanism and as such each jail forms an environment pretty much on its own.

Install package inside jail vs install package from outside may help you further on your way.

 

[nxjoseph]

so, should i install packages that are already built inside first jail into second jail with pkg? When I build a package, does it get installed to first jail by pkg?

 

[SirDice]

First question you should ask, do you really need to build from ports? If you do, because of changed options/defaults for example, then you should consider setting up your own repository and build your packages on a separate system. You can 'share' that repository to all your other hosts and/or jails.

 

[nxjoseph]

First question you should ask, do you really need to build from ports?

I don't like to see missing packages so I decided to build my own packages with custom options.

then you should consider setting up your own repository and build your packages on a separate system.

Why do I need a seperate system? Can't i have my own repository and build packages on same system that I daily drive.

You can 'share' that repository to all your other hosts and/or jails.

By jails, I meant poudriere's jails, not jails that I use for other things. What I don't want is building same packages that already built in first jail in second jail too. I might run a build job in first jail and a job in second jail too but second jail seems it's going to built packages that first jail had already built.

 

[SirDice]

Why do I need a seperate system? Can't i have my own repository and build packages on same system that I daily drive.

I use a separate system so building doesn't interfere with services. If you want to build fast you're going to choke out all your cores at 100% and generate a lot of I/O and memory presure. I regularly manage to trigger OOM that kills every running service on that machine too. Not a fun experience if you're actually using those services.

What I don't want is building same packages that already built in first jail in second jail too.

So you're creating more than one repository? Any reason for that? What makes the second jail different from the first?

And you generally want to run only one poudriere-bulk(8) run at a time. That way the bulk run can use all available resources of the machine and doesn't get bogged down by another bulk run. It's not always advantageous to run everything at once, a 'staggered' solution is often much faster in the end.

My new build server has 24 cores and I only use 6-8 builders concurrently. That leaves plenty of cores "free" for ALLOW_MAKE_JOBS to use multiple cores for each individual build. A complete 'bulk' run now finishes faster than having 24 builders using a single core each.

 

[nxjoseph]

So you're creating more than one repository? Any reason for that? What makes the second jail different from the first?

I didn't want to create more than one repository, i want to use the repository that first jail is built in second jail too. I want to have one single repository which i currently have and i don't want to build same packages in second jail.

And you generally want to run only one poudriere-bulk(8) run at a time. That way the bulk run can use all available resources of the machine and doesn't get bogged down by another bulk run. It's not always advantageous to run everything at once, a 'staggered' solution is often much faster in the end.

My new build server has 24 cores and I only use 6-8 builders concurrently. That leaves plenty of cores "free" for ALLOW_MAKE_JOBS to use multiple cores for each individual build. A complete 'bulk' run now finishes faster than having 24 builders using a single core each.

I see, thanks for a good explanation. I understand more now.

I use a separate system so building doesn't interfere with services. If you want to build fast you're going to choke out all your cores at 100% and generate a lot of I/O and memory presure. I regularly manage to trigger OOM that kills every running service on that machine too. Not a fun experience if you're actually using those services.

It could be better if i could build on a separate system but i don't have another system and my financial situation wouldn't allow it.

 

[Erichans]

If you use the quarterly branch of the ports tree (instead of main), you'll have a more consistent code base (=less frequent changes). This results in having less packages being rebuild for each successive run of poudriere-bulk (after a "git-update" from the remote FreeBSD repository of course).

Apart from this, there are lots of options to influence a poudriere run. You have a choice between two poudriere ports:

• ports-mgmt/poudriere; see poudriere.8 and poudriere bulk.8
• ports-mgmt/poudriere-devel; see poudriere.8 devel and poudriere bulk.8 devel
  A usable alternative with some extra, interesting options.

 

[nxjoseph]

If you use the quarterly branch of the ports tree (instead of main), you'll have a more consistent code base (=less frequent changes). This results in having less packages being rebuild for each successive run of poudriere-bulk (after a "git-update" from the remote FreeBSD repository of course).

I use main branch of the ports tree but I don't think of updating it much frequently.

Apart from this there are lots of options to influence a poudriere run. You have a choice between two poudriere ports:

• ports-mgmt/poudriere; see poudriere.8 and poudriere bulk.8
• ports-mgmt/poudriere-devel; see poudriere.8 devel and poudriere bulk.8 devel
  A usable alternative with some extra, interesting options.

I currently use non-devel poudriere and I'd like to know what are these extra options. Maybe, one of them would be useful to me. I'll check it, thanks.

 

[SirDice]

What's this "second" jail you keep talking about? I am thoroughly confused at what you're trying to do here. What does poudriere jails -l output?

 

[dkh]

Hello. Is it possible to run multiple jails simultaneously without building packages that already built in other jail? I created a jail while other one is running but once I tried to bulk with dry-run mode to see but it said that it would build a lot of packages but they are already built in other jail. Can i prevent this? Can I share same pkg repository between those jails?

Yes, read the bits in the handbook about thin jails.

 

[cy@]

Hello. Is it possible to run multiple jails simultaneously without building packages that already built in other jail? I created a jail while other one is running but once I tried to bulk with dry-run mode to see but it said that it would build a lot of packages but they are already built in other jail. Can i prevent this? Can I share same pkg repository between those jails?

I set up a directory tree at /jails/template/root, with 15-CURRENT, updated when the host is updated -- I use a script to build my /usr/obj on a machine and NFS share it to client machines, including the one running the jails.

On the jail machine /jails/template/root packages are installed/updated using pkg -c (using a chroot path), also installworld DESTDIR. /jails/template/root is the basis for read-only nullfs mounts. /etc in each jail is a unionfs on top of a baseline /jails/template/root/etc. Select directories within each jail has its own /var (/var/db/pkg is empty since packages are never updated from within the jail).

The basic building blocks are the nullfs and unionfs filesystems in fstab. The rest is mounted by ZFS.

The jails are vnet and non-vnet jails, depending on their purpose.

 

[nxjoseph]

What's this "second" jail you keep talking about? I am thoroughly confused at what you're trying to do here. What does poudriere jails -l output?

First one is the main jail of poudriere, i build packages that i use with this jail. If I try to build some package in second jail, packages other than the ones that i want to build are going to be built. I mean i don't want to build packages that i've already built in my first jail again.

Yes, read the bits in the handbook about thin jails.

I think I couldn't tell the problem well.

 

[SirDice]

First one is the main jail of poudriere, i build packages that i use with this jail.

Explain this further. You mentioned earlier you have no jails for "other things", what do you mean with "this jail"?

If I try to build some package in second jail

What "second" jail? A second poudriere-jail(8)?

Post the output of poudriere jail -l please.

 

[nxjoseph]

Explain this further. You mentioned earlier you have no jails for "other things", what do you mean with "this jail"?

I mean the first is the one that i use to build my packages. I mean I don't talking about a general purpose jail, im talking about jails of poudriere and they are only being used for building packages.
Following is the first jail, i've removed second one because I couldn't make what i've wanted.

# poudriere jail -l
JAILNAME VERSION     ARCH  METHOD TIMESTAMP           PATH
141x64   14.1-STABLE amd64 http   2024-06-03 23:50:48 /poudriere/jails/141x64

 

[SirDice]

Following is the first jail,

Ok, that's information I can work with. I assume the packages you build with the listed '141x64' jail in poudriere are used to update the system itself? You know you can use this repository on other systems too? You just need to install a webserver, then you can share it as a 'remote' repository for other systems on your network.

i've removed second one because I couldn't make what i've wanted.

Right. For what reason did you create that second jail? What did you try to do?

 

[nxjoseph]

Ok, that's information I can work with. I assume the packages you build with the '141x64' jail in poudriere are used to update the system itself?

Yes, I use what i build on it. If you mean upgrading my FreeBSD source (world), It's not used for that.

You know you can use this repository on other systems too? You just need to install a webserver, then you can share it as a 'remote' repository for other systems on your network.

Thanks but I don't use another system.

Right. For what reason did you create that second jail? What did you try to do?

My intention was to build another packages in the second jail while the 141x64 (first and main jail) are building something. When I tried to bulk with second jail, it started to build everything from scratch (the dependencies that already built in first jail).

 

[SirDice]

My intention was to build another packages in the second jail while the 141x64 (first and main jail) are building something.

Right. You can't start two bulk builds simultaneously using the same jail/port combination. You can interrupt the bulk build, add the other packages to the list and start the bulk build again. Or wait until that first bulk build is done.

When I tried to bulk with second jail, it started to build everything from scratch (the dependencies that already built in first jail).

Yes, because you're creating a second repository doing this. It starts off empty.

 

[Erichans]

.

 

[malavon]

I'm also not 100% sure what you're doing, but I'm guessing it's either of 2 options:
1. you're building packages one by one, something like this
poudriere bulk -j 141x64 xorg
poudriere bulk -j 141x64 firefox
This is simple, check out poudriere-bulk(), especially its "-f" option:
poudriere bulk -j 141x64 -f /usr/local/etc/poudriere.d/packagelist

2. You want two versions of some packages, with different options. Say you want to create jails or FreeBSD VM's that don't have X11 etc as dependencies. In this case you'd build 2 different repositories, but you don't need two jails for this. That's what the "-z <setname>" option is for.
poudriere bulk -j 141x64 -f /usr/local/etc/poudriere.d/packagelist-leanandmean -z server
poudriere bulk -j 141x64 -f /usr/local/etc/poudriere.d/packagelist-alotmore -z desktop

I ommitted e.g. the "-J" option since I don't know your machine and don't want to give out commands that might perform far from optimal. Nonetheless it's a really important one. I would experiment with it, a value I tend to use (on many-core machines with a plenty of RAM) is "-J <physical cores>:<physical cores/2>".

Also sometimes useful: it is possible to run 2 jails (with different -z package-sets) concurrently. I use it when my first bulk build is nearly finished and cores are no longer all in use (e.g. building packages, or non-parallelized builds).

Personal tip note, although I think I'm one of the only people doing this: I name my jails something like "FreeBSD:14:amd64" because then I can use "${ABI}" as a placeholder in my pkg() repositories. See pkg.conf() man page for other possible variables. Of course it's also possible to create a symlink, which saves you typing the "FreeBSD:" part when running poudriere.

Also make sure to read the man pages poudriere.conf() and make.conf() (make sure you read poudriere-bulk() so you know the make files it'll (try to) read (hint: it's NOT /etc/make.conf). This will help you achieve better & faster builds on your machine (especially TMPFS=all).

 

[nxjoseph]

First question you should ask, do you really need to build from ports?

I don't like to see missing packages so I decided to build my own packages with custom options.

I thought about it again and i found out that i actually don't need to build all packages from ports. I currently use 14.1-RELEASE with quarterly branch.

I use x11-wm/dwm, x11/dmenu and x11/sterm and I built them from ports with my own local customization patches and packaged it into my own little repository (/usr/ports/packages) to prevent pkg from reinstalling them with default options.