I can't count the number of deployment scripts and infrastructure I've created over my career, solving one specific problem here, another specific problem there, etc., etc. After not too long you realize that, time and time again, you're solving pretty much the same problems over and over again, even if with slight differences here and there, but almost always with quite a bit of fragility, actually, when those solutions that you're putting together, however good, are 100% homegrown (e.g. high bus factor, among others).
github.com
Ansible manages my jail hosts, Bastille the jails themselves, perfect match. Full automation and recording in VCS of infrastructure at every level, the right tool for the job in each of them.I think you should take your deployment script(s) and infrastructure written in Bastille to Ansible then (this is the basis of my point).
Here is a link to the basic infrastructure I have for my git server. I can deploy this without any "jail manager" and is not dependable on anything other than sh(1).
jcreate/example/git at main · JohnKaul/jcreate
Script to create a FreeBSD jail.conf style jail with setup script, mounts, copy-in abilities - JohnKaul/jcreate
My actual git-server side scripts can be put inside a FreeBSD jail, *nix host, and/or on a MacBook laptop.
Not a question of inferior but different. You do not but others may want a more standard solution. Just answering your "why go to all that trouble" question!
Exactly, I never ever said anything about anyone's solutions being inferior nor superior to others. You make your own choices and decide what solutions and tools are a good fit for your problems and environment and context, plain and simple. And, to that point, I simply described my choice of solutions and tools and how I arrived at the conclusions I reached, and why for me developing and maintaining them was not only not a burden but, actually, quite an enjoyable process. No more, no less.
Of course I am interested
Please share if possible.My GitLab instance is private, so exposing it and extending an invite to give you access to the code would be a tad more work than I can do right now. So, instead, let me see how far I can get pasting code here…Of course I am interested
I am not interested in any private data - just the GitLab/Jails configs/setup - nothing more
ARG+ ROOT_KEYS
ARG+ DEPLOYMENT_ENV=production
ARG+ GITLAB_HOST
ARG+ GITLAB_PORT=443
ARG+ USE_HTTPS=true
ARG+ GITLAB_KAS_SECRET
ARG+ GITLAB_SHELL_SECRET
ARG+ GITLAB_WORKHORSE_SECRET
ARG+ DB_HOST
ARG+ DB_USERNAME
ARG+ DB_PASSWORD
ARG+ DB_DATABASE
ARG+ REDIS_HOST
ARG+ REDIS_PASSWORD
ARG+ REDIS_DB
ARG+ SMTP_HOST
ARG+ SMTP_PORT=587
ARG+ SMTP_USERNAME
ARG+ SMTP_PASSWORD
ARG+ SMTP_AUTH_METHOD
ARG+ PROXY_IP4
ARG+ BACKUPS_LIFETIME=86340
INCLUDE shared/base \
--arg PKG_REPO=latest \
--arg DEPLOYMENT_TYPE=private \
--arg ROOT_KEYS=${ROOT_KEYS} \
PKG \
www/gitlab \
devel/gitlab-shell \
www/nginx \
net/rsync \
CP files/usr /
CP files/var /
CP files/tmp /
CP ${GITLAB_KAS_SECRET} /usr/local/www/gitlab/.gitlab_kas_secret
CP ${GITLAB_SHELL_SECRET} /usr/local/www/gitlab/.gitlab_shell_secret
CP ${GITLAB_WORKHORSE_SECRET} /usr/local/www/gitlab/.gitlab_workhorse_secret
RENDER /usr/local/www/gitlab/config/gitlab.yml
RENDER /usr/local/www/gitlab/config/database.yml
RENDER /usr/local/www/gitlab/config/resque.yml
RENDER /usr/local/www/gitlab/config/initializers/smtp_settings.rb
RENDER /usr/local/www/gitlab/lib/support/nginx/gitlab
RENDER /usr/local/etc/gitlab/
RENDER /var/cron/tabs/git
CMD \
cat /tmp/sshd_additions >> /etc/ssh/sshd_config && \
rm -f /tmp/sshd_additions && \
chown -vv -R git:git \
/usr/local/git && \
chown -vv git:git \
/usr/local/www/gitlab \
/usr/local/www/gitlab/config \
/usr/local/www/gitlab/config/gitlab.yml \
/usr/local/www/gitlab/config/database.yml \
/usr/local/www/gitlab/config/secrets.yml \
/usr/local/www/gitlab/config/resque.yml \
/usr/local/www/gitlab/config/initializers/smtp_settings.rb \
/usr/local/www/gitlab/.gitlab_kas_secret \
/usr/local/www/gitlab/.gitlab_shell_secret \
/usr/local/www/gitlab/.gitlab_workhorse_secret && \
chmod -vv 0640 \
/usr/local/www/gitlab/config/gitlab.yml \
/usr/local/www/gitlab/config/database.yml \
/usr/local/www/gitlab/config/secrets.yml \
/usr/local/www/gitlab/config/resque.yml \
/usr/local/www/gitlab/config/initializers/smtp_settings.rb \
/usr/local/www/gitlab/.gitlab_kas_secret \
/usr/local/www/gitlab/.gitlab_shell_secret \
/usr/local/www/gitlab/.gitlab_workhorse_secret && \
chmod -vv 1777 \
/tmp && \
su -l git -c "yarn config set python /usr/local/bin/python3.11" \
MOUNT /mnt/services/${JAIL_NAME}/repositories /usr/local/git/repositories nullfs rw 0 0
MOUNT /mnt/services/${JAIL_NAME}/builds /usr/local/www/gitlab/builds nullfs rw 0 0
MOUNT /mnt/services/${JAIL_NAME}/shared /usr/local/www/gitlab/shared nullfs rw 0 0
MOUNT /mnt/services/${JAIL_NAME}/uploads /usr/local/www/gitlab/public/uploads nullfs rw 0 0
MOUNT /mnt/services/${JAIL_NAME}/backups /mnt/backups nullfs rw 0 0
SYSRC gitlab_env="RAILS_ENV=${DEPLOYMENT_ENV}"
SYSRC nginx_enable="YES"
CONFIG set allow.raw_sockets
RESTART
CMD \
echo "Jail ${JAIL_NAME} created successfully. Please remember to run the post-install script at /usr/local/etc/gitlab/post-install to finish configuring the GitLab service and restart the jail." \
TAG web gitlab--arg-file $file flag pointing the template application to a $file with ARG=value declarations.shared/base is a a simple shared template that installs some base packages like preferred shell, Python (for the benefit of Ansible), etc., copies shell configuration files into place, sets up the chosen pkg repo, copies SSH keys, and whatever else common setups you may want to include across a number of jails.ROOT_KEYS, GITLAB_KAS_SECRET, etc., are absolute paths, and bastille template is smart enough to copy them from the host into the target jail, which helps a great deal to keep that content outside of version control.files/tmp I keep the necessary sshd additions for GitLab's database-based SSH key lookups:# GitLab fast SSH key lookup:
Match User git
AuthorizedKeysCommand /usr/local/share/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
AuthorizedKeysCommandUser git
Match allfiles/var simply contains a crontab for the nightly backup:#minute hour mday month wday command
0 4 * * * RAILS_ENV=${DEPLOYMENT_ENV} CRON=1 STRATEGY=copy rake -f /usr/local/www/gitlab/Rakefile gitlab:backup:createfiles/usr are much more extensive, so I should probably describe those in another post.files/usr:usr/local/etc/gitlab/compile-assets:#!/bin/sh
su -l git -c "(
set -e
cd /usr/local/www/gitlab
RAILS_ENV=${DEPLOYMENT_ENV} rake gitlab:assets:clean
yarn install --production --pure-lockfile --ignore-optional
node scripts/frontend/infection_scanner/infection_scanner.mjs
RAILS_ENV=${DEPLOYMENT_ENV} USE_DB=false SKIP_STORAGE_VALIDATION=true rake gitlab:assets:compile
)"usr/local/etc/gitlab/post-install:#!/bin/sh
set -e
chown -vv git /usr/local/share/gitlab-shell
su -l git -c "(
set -e
cd /usr/local/www/gitlab
RAILS_ENV=${DEPLOYMENT_ENV} rake gitlab:setup
RAILS_ENV=production rake db:migrate
/usr/local/etc/gitlab/compile-assets
)"
chown -vv root /usr/local/share/gitlab-shell
sysrc gitlab_enable="YES"usr/local/etc/gitlab/post-upgrade:#!/bin/sh
set -e
chown -vv git /usr/local/share/gitlab-shell
su -l git -c "(
set -e
cd /usr/local/www/gitlab
rm -f Gemfile.lock
RAILS_ENV=${DEPLOYMENT_ENV} rake db:migrate
/usr/local/etc/gitlab/compile-assets
RAILS_ENV=${DEPLOYMENT_ENV} rake cache:clear
)"
chown -vv root /usr/local/share/gitlab-shellusr/local/etc/gitlab/status:#!/bin/sh
su -l git -c "(
set -e
cd /usr/local/www/gitlab
RAILS_ENV=${DEPLOYMENT_ENV} rake gitlab:check
RAILS_ENV=${DEPLOYMENT_ENV} rake gitlab:env:info
)"newsyslog configurations placed at /usr/local/etc/newsyslog.conf.d for GitLab, GitLab Shell, and Nginx that you can place into a single or separate files, as you choose:/var/log/gitlab-shell/gitlab-shell.log git:git 640 7 * $D0 N
/usr/local/www/gitlab/log/*.log git:git 640 7 * $D0 GN
/var/log/nginx/gitlab_*.log git:wheel 640 7 * $D0 G /var/run/nginx.pid SIGUSR1/usr/local/etc/nginx/nginx.conf that, essentially, defines the expected http {} block, as already documented by GitLab and Matthias, and includes within it the /usr/local/www/gitlab/lib/support/nginx/gitlab file (no SSL, this GitLab deployment sits behind an HAProxy SSL-terminated balancer.usr/local/git/.gitconfig:[core]
autocrlf = input
fsync = objects,derived-metadata,reference
[gc]
auto = 0
[receive]
advertisePushOptions = true
[repack]
writeBitmaps = truefiles/usr/local/www/gitlab directory, which is also large enough to warrant its own post.files/usr/local/www/gitlab, I'm thinking that pasting their contents actually adds little value to these posts, because they're pretty much just the stock GitLab configuration files, which usually come with .sample counterparts, plus my local abstractions and customizations for the sake of version control. But I can certainly describe them:files/usr/local/www/gitlab/config/initializers/smtp_settings.rb: configure settings such as user_name, password, etc., with corresponding ${ARG} string literals, e.g. ${SMTP_USERNAME}, and Bastille's RENDER command takes care of injecting the values you pass in at template application time.files/usr/local/www/gitlab/config/database.yml: same for settings such as the database host, username, password, etc. In my case, this is a standalone PostgreSQL deployment also managed by Bastille.files/usr/local/www/gitlab/config/gitlab.yml: here you'd abstract your GitLab's host, port, SSH host, proxy IP, etc.files/usr/local/www/gitlab/config/resque.yml: here you'd abstract all necessary settings to point GitLab to your chosen Redis service, which in my case is also a separate Bastille jail with a standalone Redis deployment.files/usr/local/www/gitlab/lib/support/nginx/gitlab: again, pretty much the GitLab-provided stock file, but in my case with the following modifications to work without issues behind my SSL-terminated proxy: real_ip_recursive on; ## If you enable 'on'
(...)
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $http_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;RENDER instructions you see in the Bastillefile take care of injecting the deployment-time values chosen for any ${ARG} abstractions that may be found in any of these files, which is super convenient to keep the templates as abstract as possible and free of hardcoded secrets.ARG+ PKG_REPO
ARG+ DEPLOYMENT_TYPE
ARG+ ROOT_KEYS
ARG+ POSTGRESQL_VERSION=17
INCLUDE shared/base \
--arg PKG_REPO=${PKG_REPO} \
--arg DEPLOYMENT_TYPE=${DEPLOYMENT_TYPE} \
--arg ROOT_KEYS=${ROOT_KEYS} \
PKG \
databases/postgresql${POSTGRESQL_VERSION}-client \
databases/postgresql${POSTGRESQL_VERSION}-contrib \
databases/postgresql${POSTGRESQL_VERSION}-docs \
databases/postgresql${POSTGRESQL_VERSION}-server \
CP files/var /
CMD \
install -v -d -o postgres -g postgres /var/db/postgres \
CONFIG set sysvshm new
RESTART
SERVICE postgresql oneinitdb
CP files/postgresql.conf /var/db/postgres/data${POSTGRESQL_VERSION}/
CP files/pg_hba.conf /var/db/postgres/data${POSTGRESQL_VERSION}/
MOUNT /mnt/services/${JAIL_NAME}/backups /mnt/backups nullfs rw 0 0
SYSRC postgresql_enable="YES"
SERVICE postgresql start
TAG database postgresqlfiles/postgresql.conf & files/pg_hba.conf files pretty much follow the same ideas as for the GitLab configuration files: take stock configuration files, add them to version control and customize whatever you want, and abstract deployment-time settings and secrets via ${ARG} abstractions that are replaced as a appropriate by the RENDER command.files/var/cron/tabs/root you have the backup command:#minute hour mday month wday command
0 0 * * * pg_dumpall -U postgres --clean --if-exists --quote-all-identifiers --load-via-partition-root --encoding=utf8 > /mnt/backups/$(date '+\%Y-\%m-\%d_\%H-\%M').sqlARG+ PKG_REPO
ARG+ DEPLOYMENT_TYPE
ARG+ ROOT_KEYS
ARG+ ACL_FILE
INCLUDE shared/base \
--arg PKG_REPO=${PKG_REPO} \
--arg DEPLOYMENT_TYPE=${DEPLOYMENT_TYPE} \
--arg ROOT_KEYS=${ROOT_KEYS} \
PKG \
databases/redis \
CP files/usr /
CP files/var /
CP ${ACL_FILE} /usr/local/etc/redis.acl
CMD \
chown -vv redis:redis /usr/local/etc/redis.acl \
&& chmod -vv 0600 /usr/local/etc/redis.acl
MOUNT /mnt/services/${JAIL_NAME}/backups /mnt/backups nullfs rw 0 0
SYSRC redis_enable="YES"
SERVICE redis start
TAG database redisACL_FILE argument is an absolute path, provided at jail deployment time, to keep your Redis ACLs out of version control.files/usr you're also going to find the main /usr/local/etc/redis.conf configuration file, which receives the exact same version control and abstraction treatment as already described for other major configuration files, newsyslog log rotation configs as desired, and the backup script placed at /usr/local/etc/backup:#!/bin/sh
set -e
! [ -r ~/.redis_auth ] || . ~/.redis_auth
db_path=$(redis-cli CONFIG GET dir | tail -1)/$(redis-cli CONFIG GET dbfilename | tail -1)
previous_save=$(redis-cli LASTSAVE)
redis-cli BGSAVE > /dev/null
while :
do
current_save=$(redis-cli LASTSAVE)
if [ $current_save -gt $previous_save ]
then
break
fi
sleep 1
done
install -m 0440 $db_path /mnt/backups/$(date -j -r $current_save '+%Y-%m-%d_%H-%M').rdb/var/cron/tabs/root:#minute hour mday month wday command
0 * * * * /usr/local/etc/backupproduction Rail environment strings! All those should have been ${DEPLOYMENT_ENV} abstractions.DEPLOYMENT_ENV=staging, but I ran into way too many errors doing that, using any value other than production, that I can only guess make sense to a GitLab developer, rather than to a GitLab user/administrator, and I ran out of time trying to figure those out. So I just called it quits and even my staging GitLab deployment had a Rails DEPLOYMENT_ENV configuration of production, and I simply pointed it to testing PostgreSQL and Redis hosts/databases via simple --ARG=value deployment-time customizations.