Best practices for Poudriere and serving a repository with HTTP or other methods

[degauss4]

Hello everyone! A few weeks ago I started learning FreeBSD, reading the handbook and Michael W. Lucas books. Everything is so wonderful about FreeBSD and I can't wait to use it more in my production environment.

I have setup a simple poudriere jail on my FreeBSD host just to have packages available in my local network and with options enabled, including insults for sudo of course, classic fortune messages and some others. If you have some other recommendations for lesser known port options I'd love to hear about them

Poudriere has built the packages and if I configure the host to use as a repository the protocol file:///<path> it works. I want to serve this repository to other hosts in the local network. I see some solutions, but I am wondering if there is like a most common way to set this up. Some options I see are:

• Setup www/nginx in the FreeBSD host itself to serve the poudriere directory which has the build status and packages. Would be the simplest, but it is a tad insecure since you are serving files on the actual host. I tried looking up if it was possible to install a webserver on the poudriere jail but I think its not possible, might be that the poudriere jail is special.
• Setup a jail and install Poudriere there. I am uncertain if that would work because that would mean the jail would also setup another jail in it. Would the second level poudriere jail be able to access system resources like ZFS datasets? If it did work however, you would install the webserver on the first level jail and serve the files.
• Setup a bhyve VM on the host, install FreeBSD and setup Poudriere on the VM. Might be the most complicated option to setup initially, but then you have a VM that is easy to manage and migrate and move around across different systems and branches, and installing a webserver is simple. One use case I see for this is to install FreeBSD-LATEST and provide the latest ports available. The virtualized FreeBSD would have to run using UFS, so some poudriere configuration lines need to change.
• What if you were to serve the poudriere packages and web files that are built on the host through NFS (or symlinking) to jail that runs a webserver?

I don't think I will need help on setting the nginx site. I see there are many configuration templates available like here

I see many solutions to this problem, but I'd like to know what is thought to be the best practice for this case.

Thanks!

 

[SirDice]

Setup a jail and install Poudriere there.

Don't need to. Set up the jail and only install nginx there. Mount the poudriere data/ directory in the jail (read-only). Set HTML_TYPE="hosted" in poudriere.conf, that causes everything you need to be inside the data/ directory. Set root in your nginx.conf to data/logs/bulk/.html. You can use the rest of the example config. I'd also add a /packages directory to share your repositories.

        location /packages {
                alias /data/packages;

                fancyindex on;
                }

(I've mounted /usr/local/poudiere/data using nullfs(5) in the jail's /data directory)

I don't think I will need help on setting the nginx site.

Examples for nginx, Apache and Caddy can be found in /usr/local/share/examples/poudriere/.

 

[degauss4]

Don't need to. Set up the jail and only install nginx there. Mount the poudriere data/ directory in the jail (read-only). Set HTML_TYPE="hosted" in poudriere.conf, that causes everything you need to be inside the data/ directory. Set root in your nginx.conf to data/logs/bulk/.html. You can use the rest of the example config. I'd also add a /packages directory to share your repositories.

        location /packages {
                alias /data/packages;

                fancyindex on;
                }

(I've mounted /usr/local/poudiere/data using nullfs(5) in the jail's /data directory)


Examples for nginx, Apache and Caddy can be found in /usr/local/share/examples/poudriere/.

Ah, that is another solution I had not thought about, and it is usually the simplest haha. Thanks!

I will investigate different types of mounts, including nullfs(5) which I have not used yet. I assume the mount needs to be done from the host, since the nginx jail cannot see anything above its root, and the Poudriere software should still be able to see the directory /usr/local/poudriere/data so it can write to it.

 

[SirDice]

I assume the mount needs to be done from the host

Correct. You can create a /etc/fstab.<jailname> and it'll get automatically mounted when the jail starts and unmounted when it stops.

 

[degauss4]

Correct. You can create a /etc/fstab.<jailname> and it'll get automatically mounted when the jail starts and unmounted when it stops.

I will also look into this, thanks

One question though, I set up the nginx jail and used nullfs to mount the poudriere data directory. I also set the nginx root to be data/logs/bulk/.html, however I see access error logs because the website is trying to find .data.json, however that file is outside the .html directory.

This is the directory listing inside the nginx jail

root@nginx:/ # ls -la /data/logs/bulk/
total 78
drwxr-xr-x    5 root wheel   6 Aug  9 09:33 .
drwxr-xr-x    3 root wheel   3 Aug  7 18:30 ..
-rw-r--r--    1 root wheel 447 Aug  9 09:33 .data.json
drwxr-xr-x    3 root wheel   7 Aug  5 21:09 .html
drwxr-xr-x    9 root wheel  12 Aug  9 09:33 amd64-141-2024Q3
drwxr-xr-x  199 root wheel 199 Aug  9 09:22 latest-per-pkg

Is this expected behaviour, or did I misconfigure something? I can probably solve it my making a symbolic link from the host, but I want to know if there is a configuration line I could check out. I've set HTML_TYPE=hosted in the main poudriere conf file (it is also the default value).

Thanks!

 

[SirDice]

This is my entire server section in nginx.conf:

    server {
        listen          80;
        server_name     ports.dicelan.home;
        root            /data/logs/bulk/.html;

        # Allow caching static resources
        location ~* ^.+\.(jpg|jpeg|gif|png|ico|svg|woff|css|js|html)$ {
                add_header Cache-Control "public";
                expires 2d;
        }

        location /data {
                alias /data/logs/bulk;

                # Allow caching dynamic files but ensure they get rechecked
                location ~* ^.+\.(log|txz|tbz|bz2|gz)$ {
                        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                }

                # Don't log json requests as they come in frequently and ensure
                # caching works as expected
                location ~* ^.+\.(json)$ {
                        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                        access_log off;
                        log_not_found off;
                }

                # Allow indexing only in log dirs
                location ~ /data/?.*/(logs|latest-per-pkg)/ {
                        autoindex on;
                }

                break;
        }
        location /packages {
                alias /data/packages;

                fancyindex on;
                }
        location /release {
                alias /storage/release;
                fancyindex on;
        }

        location /up.txt {
                return 200;
                access_log off;
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }

    }

You don't need to worry about the /release or /up.txt locations, they're for serving my release(7) images and HAProxy HTTP checks.

You have to do at least one bulk build run with HTML_TYPE=hosted , that will create the files in bulk/.html.

 

[degauss4]

Thank you! Looking at your config made me see a mistake I made in one of the aliases. I had copied the example poudriere nginx config like you suggested and didnt change the alias for /data

Now the website works!