general/other CLOUDINIT image, firstboot, and HTTP proxying

[bcrashkopf]

I'm setting up a short script that can create a FreeBSD guest instance for running ports-mgmt/poudriere without any user intervention. The sort of thing I can tell someone, "clone this repo, run this script" and it just sets itself up and starts doing work. Right now I have it grabbing the latest CLOUDINIT image and injecting the initial configuration via nuageinit.

For testing I'd really like to have the guest use a local Squid cache when grabbing things like packages, the ports tree, etc. What I can't figure out is how to set $HTTP_PROXY before the firstboot_pkg_upgrade script runs. Anybody know if this can be done?

I could set up Squid as a transparent poxy, but that's a different rabbit-hole...

 

[T-Daemon]

What I can't figure out is how to set $HTTP_PROXY before the firstboot_pkg_upgrade

Do you mean here sysutils/firstboot-pkg-upgrade?

If that's the case, you could modify its rc(8) script to include the PKG_ENV variable from pkg.conf(5) ( similar to env AUTOCLEAN=ON ... in script ).

OPTIONS

     PKG_ENV: Key/Value list
                  This tells pkg(8) to set key/values to be passed in the
                  environment.  This allow setting variables to configure the
                  http(s) fetcher.  Which accepts environment variables
                  compatible with fetch(3), e.g., HTTP_PROXY.  Default: not
                  set.

If your " firstboot_pkg_upgrade" is a custom script, you could use the packages script as a draft.

 

[bcrashkopf]

Do you mean here sysutils/firstboot-pkg-upgrade?

Indeed I did. It's installed in the stock CLOUDINIT-BASIC image and enabled in rc.conf.

EDIT: SirDice gave an answer that was a slightly more specific version of what T-Daemon suggested, and as such the below is only of academic interest.

If that's the case, you could modify its rc(8) script to include the PKG_ENV variable from pkg.conf(5) ( similar to env AUTOCLEAN=ON ... in script ).

You gave me an idea, so I did a bit of digging through /etc/rc and its includes, and I think I figured out a slightly better way:

1. Append export HTTP_PROXY='http://<host ip>:<port>/' to /etc/rc.conf
2. Raise SIGALRM, which causes rc to source /etc/rc.conf again
3. Edit: make sure you are accessing the package repos via plaintext HTTP

Here's the relevant portions of my user-data file:

write_files:
  - path: /etc/rc.conf
    append: 1
    content: |
        export HTTP_PROXY='http://<host ip>:<port>/'
        echo "Setting HTTP_PROXY: $HTTP_PROXY"
  - path: /etc/rc.conf.d/hostname
    append: 1
    content: |
        kill -ALRM $$
  - path: /usr/local/etc/pkg/repos/FreeBSD.conf
    content: |
        FreeBSD-ports: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly", }
        FreeBSD-ports-kmod: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/kmods_quarterly_${VERSION_MINOR}", }
        FreeBSD-base: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/base_release_${VERSION_MINOR}", enabled: yes }

This appends the kill -ALRM $$ command to /etc/rc.conf.d/hostname, so when rc runs load_rc_config("hostname") (see rc.subr(8)) it sources the file and raises SIGALRM. I used hostname because for some reason raising the signal from /etc/rc.conf.d/firstboot_pkg_upgrade is too late to have an effect.

It's a bit of a hack, but it seems promising so far. I'll update after I set up Squid.

 

[bcrashkopf]

...and it works! Thanks for your help, T-Demon!

 

[bcrashkopf]

One final update: though the proxy basically works, most of the package file requests get 302 redirected to HTTPS URLS. Getting Squid to re-write the Location header in the redirect response is non-trivial. This path appears to be a dead end. Oh well...

 

[SirDice]

Set PKG_ENV in /usr/local/etc/pkg.conf.

     PKG_ENV: Key/Value list
                  This tells pkg(8) to set key/values to be passed in the
                  environment.  This allow setting variables to configure the
                  http(s) fetcher.  Which accepts environment variables
                  compatible with fetch(3), e.g., HTTP_PROXY.  Default: not
                  set.

pkg.conf(5)

 

[bcrashkopf]

Set PKG_ENV in /usr/local/etc/pkg.conf.

     PKG_ENV: Key/Value list
                  This tells pkg(8) to set key/values to be passed in the
                  environment.  This allow setting variables to configure the
                  http(s) fetcher.  Which accepts environment variables
                  compatible with fetch(3), e.g., HTTP_PROXY.  Default: not
                  set.

pkg.conf(5)

This works. I supposed I over-focused on one particular aspect of the problem: setting a variable in the rc script environment, for which nuageinit doesn't seem to present a ready solution. I still don't have an easy solution for HTTP access to the distribution mirrors. I'm not sure if the unrequested upgrade to HTTPS is an operational oversight or mandated by the distribution team's current security policy. Either way, the question of how to set HTTP_PROXY is clearly answered. Thank you both for trying to point me in the right direction.