Packages from poudriere's set custom repository to jail

[mecano]

Hello,
I have one ports-mgmt/poudriere jail with custom sets and I am trying to deploy sets into jails.
So ports-mgmt/pkg is said to allow to deploy packages to jails from the outside which is a thrilling feature. Note that I am not trying this over the network yet but all on the same box.
So I set custom repos corresponding to ports-mgmt/poudriere package directories like this :
vim /usr/local/etc/pkg/repos/proxy.conf

proxy: {                                                                                                                                                                             
  url : "file:///usr/local/poudriere/data/packages/92x64-default-proxy",
  enable : yes,
  signature_type : NONE,
  mirror_type : NONE
}

Then tried to install a package to jail :
pkg -j proxy install -r proxy bash

Updating repository catalogue
No valid repository found.

but
pkg -vv

Repositories:
  proxy: { 
    url             : "file:///usr/local/poudriere/data/packages/92x64-default-proxy",
    enabled         : yes
  }

and
ls /usr/local/poudriere/data/packages/92x64-default-proxy/

All Latest digests.txz packagesite.txz

in /usr/local/etc/poudriere.d/make.conf I have WITH_PKGNG=yes
I have created an option directory for the set at /usr/local/etc/poudriere.d/proxy-options but left it empty for now.
If I try to local install with pkg -j proxy add /usr/local/poudriere/data/packages/92x64-default-proxy/All/bash*

pkg: /usr/local/poudriere/data/packages/92x64-default-proxy/All/bash-4.3.0_1.txz: No such file or directory
pkg: Was 'pkg install /usr/local/poudriere/data/packages/92x64-default-proxy/All/bash-4.3.0_1.txz' meant?

Failed to install the following 1 package(s): /usr/local/poudriere/data/packages/92x64-default-proxy/All/bash-4.3.0_1.txz

where bash-4.3.0_1.txz exists in All:
ls -la /usr/local/poudriere/data/packages/92x64-default-proxy/All/bash-4.3.0_1.txz

-rw-r--r-- 1 root wheel 1234188 Apr 1 22:58 /usr/local/poudriere/data/packages/92x64-default-proxy/All/bash-4.3.0_1.txz

Running on FreeBSD 9.2-RELEASE-p3, ports-mgmt/pkg is not installed in target jail.
/bin/sh, /var/db/pkg and /var/cache/pkg are available in jail.
Any hint?

 

[mecano]

So it looks like you need to have the repo mounted inside the jail as well as a repo configuration file, I was able to update the catalog for the jail after few steps.
Adding

/usr/local/poudriere/data/packages/92x64-default-proxy/ /usr/jails/proxy/var/ports/packages nullfs ro 0 0

to /etc/fstab.proxy
restart the jail.
mkdir -p /usr/jails/proxy/usr/local/etc/pkg/repos
setting /usr/jails/proxy/usr/local/etc/pkg/repos/proxy.conf with

proxy: {
  url : "file:///var/ports/packages",
  enable : yes,
  signature_type : NONE,
  mirror_type : NONE
}

and catalog update pkg -j proxy update works

Updating repository catalogue
digests.txz 100% 928 0.9KB/s 0.9KB/s 00:00
packagesite.txz 100% 5724 5.6KB/s 5.6KB/s 00:00
Incremental update completed, 13 packages processed:
0 packages updated, 0 removed and 13 added.

now trying pkg -j proxy install -r proxy bash* stops here

Updating repository catalogue
pkg: No packages matching 'bash*' available in the repositories

It's like playing mouse and cat

 

[SirDice]

now trying pkg -j proxy install -r proxy bash* stops here

Updating repository catalogue
pkg: No packages matching 'bash*' available in the repositories

It's like playing mouse and cat

You need to escape the asterisk or the shell will try to evaluate it before handing over the results to the command.

pkg -j proxy install -r proxy bash\*

I usually disable the standard FreeBSD repository, just edit /etc/pkg/FreeBSD.conf and set enabled to "no".

 

[kpa]

Check what pkg -vv outputs when run inside the jail, that should give you some clues of what is missing.

 

[mecano]

now trying pkg -j proxy install -r proxy bash* stops here

Updating repository catalogue
pkg: No packages matching 'bash*' available in the repositories

It's like playing mouse and cat

You need to escape the asterisk or the shell will try to evaluate it before handing over the results to the command.

pkg -j proxy install -r proxy bash\*

I usually disable the standard FreeBSD repository, just edit /etc/pkg/FreeBSD.conf and set enabled to "no".

Thanks for the escaping tip @SirDice
I was planning to use pkg -j proxy install -r proxy *.* to install all packages from the repo with one simple command. :OOO Is there any built in ports-mgmt/pkg? Something like an all switch maybe? Can't find anything on these in the man.
Thanks @kpa for joining in , I'm trying to not run or install ports-mgmt/pkg inside the jail since it has abilities to work on jail from the outside. I want to install the jail, run ports-mgmt/pkg on it and have my first jump in the jail be like sitting on my cosy good old living room (with the help of a customized sysutils/ezjail newjail, not even touching flavours).

 

[SirDice]

I was planning to use pkg -j proxy install -r proxy *.* to install all packages from the repo with one simple command. :OOO Is there any built in ports-mgmt/pkg? Something like an all switch maybe? Can't find anything on these in the man.

No, it's not there. But you don't want everything anyway, as it will also install all the build dependencies that got packaged.

 

[kpa]

Well you don't have to install ports-mgmt/pkg in the jail since it can be run inside a jail with the -j option. However, when run inside a jail it is not aware of the environment or files outside the jail and that's why there has to be a repository configuration file inside the jail as well as have the repository mounted in it. I asked you to check what pkg -vv outputs inside the jail because that command also lists the repository definitions that are in effect, if there's a mistake somewhere that's the place to go look for one.

 

[mecano]

No, it's not there. But you don't want everything anyway, as it will also install all the build dependencies that got packaged.

Yes, yes I want everything! Because I'm using custom sets in ports-mgmt/poudriere I can deploy all packages from one particular set to one particular jail, so it's easier to deploy and update and it looks so flexible and faster to set up that way. I can have general sets for let's say a web server with databases/mysql55 and another one with databases/postgresql93, if I need a particular port for one jail only, I just create a new set and be done. With the help of sysutils/ezjail for populating base configuration files it looks like a great and efficient combo.

Well you don't have to install ports-mgmt/pkg in the jail since it can be run inside a jail with the -j option. However, when run inside a jail it is not aware of the environment or files outside the jail and that's why there has to be a repository configuration file inside the jail as well as have the repository mounted in it. I asked you to check what pkg -vv outputs inside the jail because that command also lists the repository definitions that are in effect, if there's a mistake somewhere that's the place to go look for one.

Are you sure? I understand ports-mgmt/pkg can be run from the outside against a jail but how can it be run from the jail itself without being installed? Or do you mean to run it from the outside like this
pkg -j proxy -vv
because
root@proxy:/ > pkg -j proxy -vv or pkg -vv
outputs

The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: N

And I don't want to install it in it if I don't need to :beergrin

 

[SirDice]

No, it's not there. But you don't want everything anyway, as it will also install all the build dependencies that got packaged.

Yes, yes I want everything!

No, really you don't

Because I'm using custom sets in ports-mgmt/poudriere I can deploy all packages from one particular set to one particular jail, so it's easier to deploy and update and it looks so flexible and faster to set up that way. I can have general sets for let's say a web server with databases/mysql55 and another one with databases/postgresql93, if I need a particular port for one jail only, I just create a new set and be done. With the help of sysutils/ezjail for populating base configuration files it looks like a great and efficient combo.

Yes, sure I get that. But it's probably worthwhile to invest some time in a custom "meta-port". One that only has the things you need as a run dependency. Installing that package would trigger all the dependencies to be installed. Then you only get the stuff you really need to get it running, instead of everything but the kitchen-sink that was needed to build it.

 

[mecano]

Yes, sure I get that. But it's probably worthwhile to invest some time in a custom "meta-port". One that only has the things you need as a run dependency. Installing that package would trigger all the dependencies to be installed. Then you only get the stuff you really need to get it running, instead of everything but the kitchen-sink that was needed to build it.

Yup that's what I'm doing, I leave dependencies checking to ports-mgmt/poudriere.
I use ports-mgmt/poudriere sets and only put what I know I need for a particular job in each set list (meta-ports).
I only plan on building for one os version/arch so things will be easier to maintain.
What is great with ports-mgmt/poudriere is that a set=a repo in the end.
So once all (these meta-ports and dependencies) has been built for that set, I want all of it because 'a poudriere set'='a pkg repo'='one or more jails tools' and I want all (and only) the tools that are needed for a particular job to be installed in the target jail. I won't have any junk I guess or am I missing something?

 

[kpa]

I meant to run it outside the jail. This worked just fine for me on jail set up for testing:

freebsd10 ~ # pkg -j testjail -vv
...snip...

Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg.FreeBSD.org/freebsd:10:x86:32/latest",
    enabled         : yes,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }

I tested installing editors/vim using the official repository and the installation didn't want to install ports-mgmt/pkg at all:

freebsd10 ~ # pkg -j testjail install vim
...
freebsd10 ~ # pkg -j testjail info pkg   
pkg: No package(s) matching pkg
freebsd10 ~ #

 

[mecano]

instead of everything but the kitchen-sink that was needed to build it.

Do you mean things like devel/gmake and such will be installed with pkg install?
Previously I was
portmaster -g shells/bash in a building jail.
then
portmaster -PP -D -B --packages-local --local-packagedir=/pkg pkgname in the target jail.
ports-mgmt/portmaster is a very nice tool too and I didn't noticed that, and wouldn't expect that the kitchen-sink was brought over.
It is really like this with ports-mgmt/pkg?

Well you don't have to install ports-mgmt/pkg in the jail since it can be run inside a jail with the -j option.

that was the misleading part When I red it, I understood "from the inside of the jail issue the command".

I meant to run it outside the jail. This worked just fine for me on jail set up for testing

Yes it works great and you can have custom repo as well, really great!

 

[mecano]

Ok sorry I'm really tired and I'm posting too fast, need to eat too
@SirDice you mean that ports-mgmt/poudriere may create packages for things like devel/gmake when building a port that will then picked up by ports-mgmt/pkg and so I need a meta-ports list for my packages installation then? damn… I just could use the ports-mgmt/poudriere set list then? Can you feed ports-mgmt/pkg with such a thing? It's a one line per port list in the 'category/port name' format.

 

[SirDice]

@SirDice you mean that ports-mgmt/poudriere may create packages for things like devel/gmake when building a port that will then picked up by ports-mgmt/pkg and so I need a meta-ports list for my packages installation then?

Yes, that's what I meant. If you install everything from the repository you will also install devel/gmake and all the other build dependencies.

I just could use the ports-mgmt/poudriere set list then? Can you feed ports-mgmt/pkg with such a thing? It's a one line per port list in the 'category/port name' format.

Yeah, you could do that. I don't think pkg-install(8) has an option to read from a file but it should be relatively easy to script. Something like this might do the trick:

#!/bin/sh

cat my_package.lst | while read line
do
   pkg install $line
done

This may also work but not tested: pkg install `cat my_package.lst`. But this may cause problems if the list is too long.

 

[mecano]

Yeah, you could do that. I don't think pkg-install(8) has an option to read from a file but it should be relatively easy to script. Something like this might do the trick:

#!/bin/sh

cat my_package.lst | while read line
do
   pkg install $line
done

This may also work but not tested: pkg install `cat my_package.lst`. But this may cause problems if the list is too long.

Thanks for response @SirDice!
If I have a dozen of meta-ports and some with a lot of dependencies, having all these ports-mgmt/pkg instances at the (nearly) same time won't slow down things too much? Ok we are talking packages not ports so maybe not?
I was going the awk way, looks like ports-mgmt/pkg has a -g glob option (but does it mean multiple packages installation?), so I thought a one liner may work. Didn't try yet though.