Scripting Desktop Installs

decuser

decuser

After installing and reinstalling my personal desktop some zillion times over the past couple of decades, I have recently gotten tired of following and updating my notes, and taking days to get stuff set up. So, as part of the process of cleaning up my notes, I started thinking like the programmer I am and decided that "all this" could be scripted... not in the traditional scripted to the n'th degree in painfully difficult to read and maintain but awesomely efficient shell scripts way, but rather in the, I'd be able to run a script and get my stuff installed and configured automagically as much as possible and with minimal intervention. So, I've launched the plan and it's working out pretty well.

Here's what I done... First I created a directory structure for my work in ~/0wa/_CD_DVD/freebsd14 (where I put install stuff):


packages/
scripts/
scripts/post-install
logs

Then I created a 00-install-stuff.sh at the top level. This script sets up the install process and runs any scripts it finds in scripts, then it runs any scripts in scripts/post-install. It logs everything (tee and sh +x stuff to logs. So far, it's pretty extensible, I've only made super minor tweaks to the top level script and when I figure out a new thing to install and how it works, I just add a script in scripts. I'm not trying real hard to manage dependencies (but I do try to write scripts that don't depend on each other) but the poor man's dependency control is to name the scripts 01-whatever, 01-whatever-else, 02-something-else (depends on 01-whatever). Any files that need to be installed are put in packages.

A sample of files in packages:

mint-backgrounds.tar.gz
mysql-connector-j-8.0.33.jar
org.apache.commons.commons-codec_1.16.0.jar
org.jkiss.dbeaver.ext.generic_2.3.209.202402111410.jar
org.jkiss.dbeaver.slf4j_2.0.93.202402111410.jar
org.osgi.service.event_1.4.1.202109301733.jar
otf.zip
slf4j.api_2.0.9.jar
spain1.ovpn

A sample of files in scripts:
01-firefox-fonts.sh
01-fonts.sh
01-javadocs.sh
01-jdftweak.sh
01-jdiskreport.sh
01-latex.sh
01-mint-backgrounds.sh
01-mysql-connector.sh
01-nordvpn.sh
01-perl.sh
01-ruby.sh

Post-install stuff is like adding users and stuff.

Here's the 01-ruby.sh script to give you a feel for the scripts:
Code: 
#!/usr/local/bin/bash -x

#********** ruby
pushd ~
rm -fr ~/.rbenv
git clone https://github.com/rbenv/rbenv.git ~/.rbenv

# skip if ark restore
cat << "EOF" >> ~/.bashrc
export PATH=$HOME/.rbenv/bin:$PATH
eval "$(rbenv init -)"
EOF

export PATH=~/.rbenv/bin:$PATH
eval "$(rbenv init -)"

mkdir -p "$(rbenv root)"/plugins
git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build
curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-doctor | bash

MAKE=make rbenv install 3.3.0
rbenv global 3.3.0
gem update --system
gem install rubocop

echo "alias sl='/usr/local/bin/sublime'" >> ~/.bashrc
popd

nevermind that sublime isn't ruby... it's cruft that I should clean up, but hey, it's a work in progress.

I also mirror the directory in test and that's where I try out new scripts.

Anyhow, I thought I'd be transparent and share to see if y'all are doing anything similar, but maybe better/easier. My objective isn't to perfect a script system, just to make it so I can simplify my build process. With the current setup, I can be back up and running from bare metal in about two hours, most of the time is waiting on the scripts. It used to be a couple of days of tweaking this setting, running that custom installer, copy this, copying that, etc.

Note: the scripts have some interactions cuz I'm not gonna spend the time to figure out how to do it without answering prompts unless the package provides --no-interaction type arguments that are easy to follow (an example is texlive - it's much faster to download the iso, save it into packages, and let the script mount it, install it, add environment variables, etc. than it is to do it via download, thankfully it has --no-interaction and reasonable arguments, otherwise, you'd have to run the tui/gui and they are decidedly not script friendly).

Cheers.
 
rbranco said:
Use Ansible. It's better
Click to expand...
Hmm.. I'm a little skeptical...

Not that it isn't better than my homebrewed solution... I'm sure it is, just that it doesn't look simple even though it says simple everywhere all over the docs. It looks great for it pros managing a handful to 65000 machines with only minor variations here and there and a slow change cycle of the catalog, but it doesn't look great for desktop use. Maybe I'm wrong, thinking this way. I see a bunch of how to do ansible videos and tutorials and they're gobbledegook - lots of here's how you id machines, connect to them, check if stuff's installed on them (checksumming and whatnot), add what's missing, etc. But super light on actual small, real life, examples. Actually, it kind of reminds me of ANT for some reason... Looks like a lot of setup to me, even for the individual tasks (none of the videos/tutorials I saw actually showed anything more complex than unpacking a package and adding users, stuff pkg already does).

Do you have some great stuff you can point to?
 
grahamperrin said:
I guess, some of what's done by outpaddling sysutils/desktop-installer might avoid reinventing a wheel or two.

github.com

GitHub - outpaddling/desktop-installer: Quickly configure a FreeBSD, NetBSD, or OpenBSD desktop system

Quickly configure a FreeBSD, NetBSD, or OpenBSD desktop system - outpaddling/desktop-installer
github.com github.com
Click to expand...

That's an interesting project. I wonder what it would take to add the ruby example to the workflow... Looking at their quickstart, it looks geared toward automating the "desktop" install, meaning getting the desktop running. I'm able to do that manually in about the same amount of time the tool does (really not too much more than sudo pkg install the right packages, sysrc the right services and loader settings, add the user to the right groups). My idea of getting the "desktop" up is having all of my packages in place and configured, where all I need to do mimic a restore from backup, but with no leftover cruft, is to restore my user data (Pictures, Music, Zotero files, fossil sandbox, git sandbox, Thuderbird emails, that sort of thing).
 
decuser said:
Hmm.. I'm a little skeptical...
Click to expand...
Understandable. ansible has a learning curve, as any reasonable configuration tool.

That being said, if you invest time into learning ansible, you can end up with this:
1) install (or reinstall) a machine, including adding user accounts and getting it onto the network
2) run one ansible-playbook command to install and configure ansible on that machine
3) run another ansible-playbook command to have the whole recorded configuration for the machine installed on it.
Steps 2 and 3 can be combined into one.

The downside is that you will have to maintain that ansible configuration for the machines you have in ansible. Everytime you make a change on the machine (that you want to keep) you will need to update the ansible configuration for it.

I believe this is common for most / all configuration tools in use.
However for me it is easier to force myself to update the ansible configuration in question than remember to keep a backup of all the important files everytime I make a change on a machine.

Full disclosure: I currently use ansible only on my servers / virtual machines, I haven't statred using in on my workstations / desktop machines yet.
 
Having had plenty of use with Ansible in the past, I tried using it to configure my Linux desktop a few years ago. The thing that made me give up and roll something pretty close to what decuser has was conditionals.
While Ansible is really nice for installing packages and making sure that a few lines are in a specific file, etc, when there isn't a module that does what you want you're stuck either writing a script to do it, or using several steps to register something then conditionally acting depending on the check. I found I was doing this enough that I could roll my own solution using shell scripts simply enough.

Plus, between using Ansible daily and for my desktop, then stopping using it daily at work, then stopping using it for my deskop, I found I wasn't changing/running it much on my desktop and so modules changed/became deprecated/etc. It was just another layer of management that I didn't need for one machine.

I still use Ansible to configure my simple home DHCP/DNS server, mostly because I use templating heavily to take a yaml file of all my home machines and create the correct configuration for them - wouldn't fancy doing that in a shell script!

tingo said:
The downside is that you will have to maintain that ansible configuration for the machines you have in ansible. Everytime you make a change on the machine (that you want to keep) you will need to update the ansible configuration for it.
...
However for me it is easier to force myself to update the ansible configuration in question than remember to keep a backup of all the important files everytime I make a change on a machine.
Click to expand...
In all the servers I configured Ansible with I had a simple policy: you manually do anything on the server and I will come to remove your fingers - or at least that was the threat ?
Even with simple shell framework for my desktop, I never install packages/edit system files/etc manually, instead it gets added to the relevant file(s) and I run the scripts again.
 
tingo said:
Full disclosure: I currently use ansible only on my servers / virtual machines, I haven't statred using in on my workstations / desktop machines yet.
Click to expand...
I also use it for my workstation and I'm comfortable reinstalling the OS with my separate /home partition knowing that Ansible will leave it configured as before.
 
decuser said:
Hmm.. I'm a little skeptical...

Not that it isn't better than my homebrewed solution... I'm sure it is, just that it doesn't look simple even though it says simple everywhere all over the docs. It looks great for it pros managing a handful to 65000 machines with only minor variations here and there and a slow change cycle of the catalog, but it doesn't look great for desktop use. Maybe I'm wrong, thinking this way. I see a bunch of how to do ansible videos and tutorials and they're gobbledegook - lots of here's how you id machines, connect to them, check if stuff's installed on them (checksumming and whatnot), add what's missing, etc. But super light on actual small, real life, examples. Actually, it kind of reminds me of ANT for some reason... Looks like a lot of setup to me, even for the individual tasks (none of the videos/tutorials I saw actually showed anything more complex than unpacking a package and adding users, stuff pkg already does).

Do you have some great stuff you can point to?
Click to expand...
With Ansible it's better to learn by doing, not by reading tutorials or videos. Start small. With 20% of Ansible knowledge you can solve 80% of your problems. It's very Pareto. No need to know it all. When I don't know something for a specific task I just look it up and find the answer.

I use this for my VM:
github.com

GitHub - ricardobranco777/ansible-freebsd: Ansible configuration for FreeBSD 14

Ansible configuration for FreeBSD 14. Contribute to ricardobranco777/ansible-freebsd development by creating an account on GitHub.
github.com github.com
 
Once you have a system installed/configured with the applications you want, save the output of this:
pkg prime-list

That easily gives you a list of the packages to install, so you can do something like this:
pkg install < pkgs.that.i.must.have

Configuration changes should be relatively easy to track, the bulk wind up in:
/etc/rc.conf
/etc/periodic.conf
 
Before Ansible I was tracking /etc and /usr/local/etc in git like this:

```
cd /etc
sudo git init
sudo git add .
sudo git commit -m init
sudo chmod 700 .git
```

Then I generated a diff that I could apply with patch. I now only use it to track any changes in the configuration with updates.
 
A year and a half (actually, 2 years) of using ansible to script installs across a variety of systems, I've gone back to my simple script approach - OMG, it's like breathing clean air again after 6 months at a coal plant. Stuff breaks, it's easy to fix, none of this isomorphic perfection, just run the @#$! script and if there's an error, check the logs - the exact problem will present, not some side effect buried in a wrapper of an abstraction of some command handler. If you're a professional maintaining dozens or more systems that have a limited number of customizations, have fun - ansible's a nice programming environment. If you have less than a dozen highly customized systems, scripting is SOOOOO much simpler. System breaks, wipe it, restore any zpools you like, run the script, a few hours later, system is pristine. Any problem encountered is of the variety, "tried to do X, but was unable due to Y". Very straightforward, easy to understand, easy to fix. Even after 18 months with using another approach (where every time something broke, it was a mini-investigation into ansible's plumbing).
 
decuser said:
A year and a half (actually, 2 years) of using ansible to script installs across a variety of systems, I've gone back to my simple script approach - OMG, it's like breathing clean air again after 6 months at a coal plant. Stuff breaks, it's easy to fix, none of this isomorphic perfection, just run the @#$! script and if there's an error, check the logs - the exact problem will present, not some side effect buried in a wrapper of an abstraction of some command handler. If you're a professional maintaining dozens or more systems that have a limited number of customizations, have fun - ansible's a nice programming environment. If you have less than a dozen highly customized systems, scripting is SOOOOO much simpler. System breaks, wipe it, restore any zpools you like, run the script, a few hours later, system is pristine. Any problem encountered is of the variety, "tried to do X, but was unable due to Y". Very straightforward, easy to understand, easy to fix. Even after 18 months with using another approach (where every time something broke, it was a mini-investigation into ansible's plumbing).
Click to expand...
I sometimes feel like this. Translating ideas to Ansible is awful, but LLM's are good at it. My only issue with Ansible is the Python dependency. Scripts are nice for simple stuff but get messy after a hundred lines or so.
 
I've never used Ansible (I'm not a professional "sysadmin"--I'm not in the IT field so forgive me if that's not the proper title) but the 'Ansible workflow' looks like a bit of a mess to me; Python dependency, YAML, etc.. I would think doing this action (deployment) would require something more auditable -i.e. less moving parts to debug and what not.

There are two sides though: "reliability" can be either on the tool (Ansible) or the script but I'm not sure which would be better. And, I'm not sure I'd like to accept the responsibility to decide which is easier to debug for most situations!

I have a tool which is very "auditable" you can use to deploy scripts/configs with if you'd be interested.
 
decuser said:
A year and a half (actually, 2 years) of using ansible to script installs across a variety of systems, I've gone back to my simple script approach - OMG, it's like breathing clean air again after 6 months at a coal plant. Stuff breaks, it's easy to fix, none of this isomorphic perfection, just run the @#$! script and if there's an error, check the logs - the exact problem will present, not some side effect buried in a wrapper of an abstraction of some command handler. If you're a professional maintaining dozens or more systems that have a limited number of customizations, have fun - ansible's a nice programming environment. If you have less than a dozen highly customized systems, scripting is SOOOOO much simpler. System breaks, wipe it, restore any zpools you like, run the script, a few hours later, system is pristine. Any problem encountered is of the variety, "tried to do X, but was unable due to Y". Very straightforward, easy to understand, easy to fix. Even after 18 months with using another approach (where every time something broke, it was a mini-investigation into ansible's plumbing).
Click to expand...
When I was using Linux Fedora, everyone in that community recommended Ansible. I didn't even bother, it looked so uninviting to any customization. I had so much customization during installs that scripting was just so much easier. I strongly believe Ansible is "Installs For Dummies" type of a tool. I'm so glad you shared you experience so we can convince everyone else to stop peddling some gross abstraction to something that can be nice and pleasant.
 
I stick on this simple idea: if you want to experiment a technology, use the well-crafted software that people advise you. If you want to daily use this technology, make your own software as simple as you can for your needs. Make evolve it depending on the move of the underlying bricks and your new needs, if any.

Oh, and well, I can't stand python.
 
Emrion said:
I stick on this simple idea: if you want to experiment a technology, use the well-crafted software that people advise you. If you want to daily use this technology, make your own software as simple as you can for your needs. Make evolve it depending on the move of the underlying bricks and your new needs, if any.

Oh, and well, I can't stand python.
Click to expand...
Good training is achieving full desktop with everything you need from the "live shell" from the FreeBSD installer. Create malloc md devices as virtual drives, mount your wanted filesystems over the image read-only dirs and just pkg install everything in RAM and start X. Apply trial and error to everything that goes wrong while trying. I ran systems for months like this just for fun. Stable is recommended because of the panic-chance of newest code updates.
 
MG said:
Good training is achieving full desktop with everything you need from the "live shell" from the FreeBSD installer. Create malloc md devices as virtual drives, mount your wanted filesystems over the image read-only dirs and just pkg install everything in RAM and start X. Apply trial and error to everything that goes wrong while trying. I ran systems for months like this just for fun. Stable is recommended because of the panic-chance of newest code updates.
Click to expand...
Shouldn't bootstrapping from an existing installation be better?
 
rbranco said:
I sometimes feel like this. Translating ideas to Ansible is awful, but LLM's are good at it. My only issue with Ansible is the Python dependency. Scripts are nice for simple stuff but get messy after a hundred lines or so.
Click to expand...
Without chat, I never would have gotten off the ground. The docs are scary awful - remind me of the modern intel cpu guides - overdetailed, no big picture, gotta read everything to find the three incantations that make sense. With chat, you just say what you wanna do, it says oh, there's a widget that handles that, edit the yaml, this way. It works, it's repeatable, and mostly understandable. When things go wrong, though, that's where the pain kicks in. Chat's good at that too, but I have a compulsion to actually understand what went wrong, how things wound up there, and make it go away forever, chat could tell me what went wrong and how to fix it, but wasn't really good at helping me to prevent the next occurrence or teaching and helping me understand the deeper underlying causes. It hallucinated more in the beginning but improved over time. I use claude pro now and I'm pretty sure it'd be much better now. Still love my scripts - most all of them are 20 lines or less.

As for python as dependency, that's annoying for sure. I heart python, but it's sprawling and requires online a lot of the time, as do rust and nodejs and ruby. Go, too... but that's more manageable - compiles to a single executable, dependencies can be vendored, easy to offline and move around withouth network dependencies - just copy the golang tarball, tar up the project with it's deps vendored, and you can move it to another machine and see it run completely offline (unless you go and linke it to fyne or something, then you'll need an os dependency like xorg-dev, but that's reasonable, in my estimation.
 
Using Bash scripts is how Omarchy does it, so if it works for you, it's ok.

I don't like YAML. Kubernetes had to come with KYAML. That's the other issue with Ansible.
 
You must log in or register to reply here.
Back
Top