Hi gang!
Editorial
There are many reasons why I'm passionate about FreeBSD as an operating system and why it's my all time favorite, two of which are fully related to this article:
Disclaimer: In most of my shared examples you'll see that I'm (ab)using the root user. The only reason I do so is because I'm using my Psi jail to provide all the examples which doesn't have any extra user accounts (mostly because it's not being used on a network). But please keep well in mind that you shouldn't use root for most of these examples; a regular account will work just as fine and it'll be a lot safer!
What is version control?
Version control is performed with a version control system ("VCS") and sometimes also referred to as software configuration management ("SCM"). It's basically a discipline most often practiced in software development where you store every change you make to your system and provide some extra comments on why you made those changes. This can help you to keep track of its history, it allows you to revert your changes back to a prior version at any time you'd like (either permanent or simply to study a prior version) and you can always fully check the specific differences between those (previous) versions.
It can also help to share your work with others and work on something together, but that part is beyond the scope of this tutorial.
Why use VCS for system configuration?
I assume that you keep backups of your system, but how high is your retention? On my main server I keep daily backups through ZFS snapshots which retention only lasts 7 days. There's also a real (off-site) backup which gets made on a weekly basis with a few incrementals sent during the week. These weekly backups are stored for 4 weeks.
So basically my retention is about a month.
Now imagine that I upgraded PHP from 5.6 to 7.0 a few weeks ago which also included a drastic overhaul of my php.ini file. The new configuration works perfectly but I made a few changes of which I no longer remember why I did that. I also seem to recall that I had better results with some previous settings and although I can look up the properties online (in the documentation) it would be much easier if I could check out my previous configuration again.
Unfortunately this would mean that I'd have to dig this up from my backup server, which can become quite a hassle, so it's probably easier to spend the next 20 minutes skimming through the documentation again.
If only there were a system which could store all these changes in an optimal (redundant) way and which would give us full access to previous versions.
Guess what? That's exactly what a version control system such as Subversion can do for you!
The advantages should be obvious:
Subversion manages data in two different ways. All data gets stored in a so called repository; this is somewhat comparable to a database (it basically is a database). Normally you don't have to do much with a repository, but it has some interesting tweaks which we'll check out later. One repository can store many projects, but because of that it will become important to make sure that you keep track of how you stored this data.
In order to access the data (so the actual files which are stored inside the repository) we need a so called working copy. This is basically the place where we work with both our data and the main repository, hence its name.
Creating the repository
The first thing we need is a repository to store our data, this is where svnliteadmin comes into play. You can basically store this anywhere you want, but I prefer using /var/db/repo:
Because this repository is going to be used to store server configuration files I strongly recommend that you make sure that no unauthorized users will be able to access this. For my own servers I prefer to limit access to the wheel group, but for my company servers we decided to utilize a dedicated group. So, the next step will be:
You have a few options here... Use chmod (and optionally chown) to make sure that wheel (or another group) has full writing permissions. Or use svnliteserve to access the repository. It can be used as a daemon which would mean that you'd have to customize /var/db/repo/conf/passwd in order to set up specific Subversion accounts.
In my example we'll stick with file system access, so to keep people out of our repository we rely on the chmod command shared earlier.
Deciding what data gets added
The most obvious locations to add to our new repository are of course /etc and /usr/local/etc. But the latter requires a bit more attention. See, the problem is that although packages never contain actual configuration files they do often provide examples:
These files could easily change whenever we upgrade the software they're related to, which would then immediately trigger our VCS system. Now, this doesn't have to be a bad thing of course: getting alerted to such changes can be a good thing. It's just that we need to decide if we want to set up such a scenario or not.
Another point of attention are binary data files. Take for example /usr/local/etc/sasldb2.db. This is a data file used by Cyrus SASL; a library used for authentication purposes. Or what to think about /etc/pwd.db?
Now, in this scenario these files wouldn't become a big problem; Subversion actually knows how to recognize and handle non-ascii files, even if you don't tell it to. But there is something to say about the risk(s) involved with storing password information in a somewhat insecure setup like our repository (it's not that insecure, but Subversion also wasn't build with high-end security in mind).
So we're going to exclude these files, and for that we need to reconfigure Subversion a bit. This is done by editing the configuration found in ~/.subversion. If this directory doesn't exist yet then you can create it by simply running
Now edit config and find & edit this section:
This will make sure that we don't have to worry about those sample and data files anymore.
Adding the data
One of the major advantages of Subversion is that it can handle large amounts of data very well. And it's not only about adding the data: working with this data is quite easy too. For example: there are many files within the /etc directory which could become a hindrance if you're only interested in the SSH configuration. Fortunately Subversion accounted for that and it allows you to access individual parts of your data if you need to.
This is why I prefer to simply add the entire /etc and /usr/local/etc structures, especially since we already made sure to exclude some unwanted overhead. However we do need to be careful and make sure that we store both directories within separate locations inside our repository.
Don't worry, it sounds more difficult than it actually is:
Note that you can omit the . if you want to (Subversion imports the current directory by default) but I prefer not to take any chances. If you want to add more information about this data than just the description then leave out the -m parameter; then your default editor will open which allows you to provide extra information ("meta data").
Don't be fooled by the file:// prefix; we're not adding a single file here but stored our data under the etc.local header. Subversion requires an URL to specify the main repository, and by using file:// we're basically telling Subversion that the repository should be accessed using our current filesystem.
After this command Subversion will show a list of files which it has added to the repository and end with a confirmation:
The revision is a unique identifier which Subversion uses to keep track of our changes. Since this is the first addition to our repository it got revision number 1, makes sense right? After you do the same for /etc you'll notice that Subversion will now tell you that you commited revision 2.
Putting our configuration under version control
Now, adding the data is only step one of the whole procedure, something you can see for yourself if you ask svnlite about the current status:
Therefor we'll need to tell Subversion that we wish to use the data which we just stored. We do that by checking out this data, like so:
You will now see that Subversion has extracted several files and directories:
Congratulations, your /usr/local/etc directory is now under version control!
So what happened here? Subversion is very careful with handling data and it normally expects to 'extract' data into a new (empty) working directory. However, our directory wasn't empty which would have triggered Subversion to ask for confirmation if we really wanted to use the existing data, that's why I used --force to enforce this.
But there's more: despite the use of the parameter it will not force Subversion to blindly overwrite any existing data, far from it:
See what I mean? Sure, the access time is just a while ago (I'm setting up these examples while I'm writing this article) but notice the modification timestamp (the sequence is: access time, modification time, creation time & birth time)? Even though we told Subversion to enforce "extraction" it didn't blindly overwrite any of our current data at all.
If you're more familiar with Subversion then you could already tell this from the previous output. That E you saw? It doesn't mean 'extracted' which is something I hinted at but it actually means Existed; Subversion indicated that the files and directories already existed, and therefor it merely added them under version control, but nothing more.
Making changes
So now the only thing you need to remember is that whenever you make a change to the system you should also tell Subversion about it. When in doubt just ask Subversion what the current status is:
Remember: in true Unix fashion 'no news' means 'good news'. So Subversion is telling us that we don't need to worry about the current state of our working copy. If you want more detailed information about our setup then you can use:
So lets reconfigure PKG. I'm going to remove all the comments and get rid of the default aliases. Then I'm going to set up a new repository directory:
So now let's see what Subversion has to say about this:
As you can see it has detected that we modified pkg.conf ('M') From here on we have several options... If you want to see exactly what has changed then you can use
And finally if you wish to keep your changes then you should commit them using:
This is one of the reasons why I think that Subversion is a very good tool to use for this, because you can look up all that extra information at a later time if you need to.
Keep in mind though that once you committed a change then you should also update the current working copy again, you can do that by using
Allowing someone else to make changes?
So let's say that our SSH server has gone bonkers and we can't fix this ourselves. Someone else has offered to help us and he can edit the config files for us. However, that would require direct access and since this person isn't a part of our usual team we can't "just" give him access.
Fortunately we have Subversion at our disposal!
Note that in this example I made sure that the keys aren't included (by not adding them in the first place) but this is an important aspect because it could be a security risk.
So now our helper can safely edit these files. We could tell him to commit the changes once he's done, but this would mean that he'd need to have access to our repository. So another option is to make him apply the changes after which we'll commit them instead.
The advantage is that our helper never gets their hands on the actual configuration. And thanks to Subversion we can check out exactly what he changed long before actual implementing it (remember
This is also one of the advantages which Subversion provides: once the changes got committed they became immediately available to our main working directory.
It's also an example of what I mentioned earlier on: how Subversion was quite capable of handling large amounts of data. As you can see we never had to check out ('extract') the entire /etc directory. And of course you can expand on this; you could even use such a setup to provide the same configuration across a series of servers.
Securing our data
Now, obviously the idea is to include /var/db/repo in your common backup scheme so that the whole repository will be safely stored away. But what about making a security copy just to be safe? Once again Subversion to the rescue:
So, the reason why I used svnliteadmin instead of tar right away is because there was no way for me to tell if the repository could be in use. This is somewhat comparable to using vi to edit /etc/passwd during which someone else also changes their password. So even if someone was working on our system configuration then svnliteadmin would have made sure that it couldn't interfere with our backup.
If you need more help...
All Subversion commands have build-in help screens which can be shown by using the help command. So if you want to know more details about the commit command then you can use:
And there you have it...
From here on you can basically continue to commit & update any changes after which you'll eventually build up quite a log. The main advantage though is that even after a year of updating you'll still have access to the way things were back then. And without all the hassle that a regular backup brings along (nor its massive size).
Also: since you already have a repository at your disposal there's no reason why you shouldn't use that for some actual development. Do you maintain your own shell scripts? Maybe it could be a good idea to add those as well!
Editorial
There are many reasons why I'm passionate about FreeBSD as an operating system and why it's my all time favorite, two of which are fully related to this article:
- FreeBSD is a complete system, meaning that you may end up surprised if you stop to think about the sheer functionality provided by just the base system.
- Curious? compiler & debugger (nowadays this seems like an afterthought, but I lived the days that you actually had to pay to get your hands on a compiler other than GCC (and I'm not only referring to Microsoft here!)), very extensive & fully customizable mailserver (Sendmail), simple file encryption, a more extensive encryption suite & extensive disk encryption facilities (/usr/bin/enigma, /usr/bin/openssl and /sbin/geli), an intrusion detection system (/usr/sbin/mtree), network monitoring / managing support (/usr/sbin/tcpdump and many other related tools) and... "out of the box" support for version control through Subversion.
- FreeBSD doesn't do popularity contests. Of course people keep a check on which tool does the best job, but when the next hip & modern tool comes out then it doesn't "just" jump on another bandwagon because something managed to gain extra popularity (it's also important to realize that we're talking about 'out of the box' functionality, something which is also very easy to customize so that the system fully suits your specific needs).
Disclaimer: In most of my shared examples you'll see that I'm (ab)using the root user. The only reason I do so is because I'm using my Psi jail to provide all the examples which doesn't have any extra user accounts (mostly because it's not being used on a network). But please keep well in mind that you shouldn't use root for most of these examples; a regular account will work just as fine and it'll be a lot safer!
What is version control?
Version control is performed with a version control system ("VCS") and sometimes also referred to as software configuration management ("SCM"). It's basically a discipline most often practiced in software development where you store every change you make to your system and provide some extra comments on why you made those changes. This can help you to keep track of its history, it allows you to revert your changes back to a prior version at any time you'd like (either permanent or simply to study a prior version) and you can always fully check the specific differences between those (previous) versions.
It can also help to share your work with others and work on something together, but that part is beyond the scope of this tutorial.
Why use VCS for system configuration?
I assume that you keep backups of your system, but how high is your retention? On my main server I keep daily backups through ZFS snapshots which retention only lasts 7 days. There's also a real (off-site) backup which gets made on a weekly basis with a few incrementals sent during the week. These weekly backups are stored for 4 weeks.
So basically my retention is about a month.
Now imagine that I upgraded PHP from 5.6 to 7.0 a few weeks ago which also included a drastic overhaul of my php.ini file. The new configuration works perfectly but I made a few changes of which I no longer remember why I did that. I also seem to recall that I had better results with some previous settings and although I can look up the properties online (in the documentation) it would be much easier if I could check out my previous configuration again.
Unfortunately this would mean that I'd have to dig this up from my backup server, which can become quite a hassle, so it's probably easier to spend the next 20 minutes skimming through the documentation again.
If only there were a system which could store all these changes in an optimal (redundant) way and which would give us full access to previous versions.
Guess what? That's exactly what a version control system such as Subversion can do for you!
The advantages should be obvious:
- Safeguarding: By letting Subversion track your configuration you can always undo any unwanted changes, or verify these changes against previous versions.
- Back up the backup
By including the Subversion data in your main backup you're basically safe keeping it; while it can be quite a hassle to dig into your backup history this doesn't hold true for a Subversion repository (= the place where Subversion stores its data). So you're basically making a backup of your full configuration backup - Useful on other servers. Though I'm not going to address this here you could provide your data to other (local) servers, which might help to set them up much quicker. For example: instead of having to configure Apache from the ground up you could also grab your existing configuration and use that as a starting point.
- Documentation. All it takes is a simple description of why you made a certain change, something which might help you later to verify why you made certain changes.
- Support is provided "out of the box". The moment you install a FreeBSD system is the moment when you can access your repository.
Subversion manages data in two different ways. All data gets stored in a so called repository; this is somewhat comparable to a database (it basically is a database). Normally you don't have to do much with a repository, but it has some interesting tweaks which we'll check out later. One repository can store many projects, but because of that it will become important to make sure that you keep track of how you stored this data.
In order to access the data (so the actual files which are stored inside the repository) we need a so called working copy. This is basically the place where we work with both our data and the main repository, hence its name.
Creating the repository
The first thing we need is a repository to store our data, this is where svnliteadmin comes into play. You can basically store this anywhere you want, but I prefer using /var/db/repo:
Code:
root@psi:~ # svnliteadmin create /var/db/repo
root@psi:~ # ls /var/db/repo/
README.txt db/ hooks/
conf/ format locks/ # chmod 770 /var/db/repo, which will prevent unauthorized access.You have a few options here... Use chmod (and optionally chown) to make sure that wheel (or another group) has full writing permissions. Or use svnliteserve to access the repository. It can be used as a daemon which would mean that you'd have to customize /var/db/repo/conf/passwd in order to set up specific Subversion accounts.
In my example we'll stick with file system access, so to keep people out of our repository we rely on the chmod command shared earlier.
Deciding what data gets added
The most obvious locations to add to our new repository are of course /etc and /usr/local/etc. But the latter requires a bit more attention. See, the problem is that although packages never contain actual configuration files they do often provide examples:
Code:
root@psi:~ # pkg which /usr/local/etc/*.sample
/usr/local/etc/clamd.conf.sample was installed by package clamav-0.100.2
/usr/local/etc/freshclam.conf.sample was installed by package clamav-0.100.2
/usr/local/etc/pkg.conf.sample was installed by package pkg-1.10.5_5Another point of attention are binary data files. Take for example /usr/local/etc/sasldb2.db. This is a data file used by Cyrus SASL; a library used for authentication purposes. Or what to think about /etc/pwd.db?
Now, in this scenario these files wouldn't become a big problem; Subversion actually knows how to recognize and handle non-ascii files, even if you don't tell it to. But there is something to say about the risk(s) involved with storing password information in a somewhat insecure setup like our repository (it's not that insecure, but Subversion also wasn't build with high-end security in mind).
So we're going to exclude these files, and for that we need to reconfigure Subversion a bit. This is done by editing the configuration found in ~/.subversion. If this directory doesn't exist yet then you can create it by simply running
svnlite help:
Code:
root@psi:~ # ls .subversion
ls: .subversion: No such file or directory
root@psi:~ # svnlite help > /dev/null
root@psi:~ # ls .subversion/
README.txt auth/ config servers
Code:
### Section for configuring miscellaneous Subversion options.
[miscellany]
### Set global-ignores to a set of whitespace-delimited globs
### which Subversion will ignore in its 'status' output, and
### while importing or adding files and directories.
### '*' matches leading dots, e.g. '*.rej' matches '.foo.rej'.
global-ignores = *.sample *.dbAdding the data
One of the major advantages of Subversion is that it can handle large amounts of data very well. And it's not only about adding the data: working with this data is quite easy too. For example: there are many files within the /etc directory which could become a hindrance if you're only interested in the SSH configuration. Fortunately Subversion accounted for that and it allows you to access individual parts of your data if you need to.
This is why I prefer to simply add the entire /etc and /usr/local/etc structures, especially since we already made sure to exclude some unwanted overhead. However we do need to be careful and make sure that we store both directories within separate locations inside our repository.
Don't worry, it sounds more difficult than it actually is:
Code:
root@psi:~ # cd /usr/local/etc
root@psi:/usr/local/etc # svnlite import . file:///var/db/repo/etc.local -m "Initial commit of /usr/local/etc"Don't be fooled by the file:// prefix; we're not adding a single file here but stored our data under the etc.local header. Subversion requires an URL to specify the main repository, and by using file:// we're basically telling Subversion that the repository should be accessed using our current filesystem.
After this command Subversion will show a list of files which it has added to the repository and end with a confirmation:
Code:
Adding rc.d
Adding rc.d/clamav-clamd
Adding rc.d/clamav-freshclam
Adding rc.d/imapd
Adding rc.d/nsd
Adding slsh.rc
Adding ssl
Adding ssl/cert.pem
Committing transaction...
Committed revision 1.Putting our configuration under version control
Now, adding the data is only step one of the whole procedure, something you can see for yourself if you ask svnlite about the current status:
Code:
root@psi:/usr/local/etc # svnlite status
svn: warning: W155007: '/usr/local/etc' is not a working copy
Code:
root@psi:/usr/local/etc # svnlite co --force file:///var/db/repo/etc.local .
Code:
E clamd.conf
E cyrus.conf
E freshclam.conf
E imapd.conf
E pkg.conf
E slsh.rc
Checked out revision 2.So what happened here? Subversion is very careful with handling data and it normally expects to 'extract' data into a new (empty) working directory. However, our directory wasn't empty which would have triggered Subversion to ask for confirmation if we really wanted to use the existing data, that's why I used --force to enforce this.
But there's more: despite the use of the parameter it will not force Subversion to blindly overwrite any existing data, far from it:
Code:
root@psi:/usr/local/etc # stat pkg.conf
3878456435 35404 -rw-r--r-- 1 root wheel 4294967295 2143 "Nov 24 06:41:08 2018" "Oct 18 01:11:17 2018" "Nov 2 15:28:03 2018" "Oct 18 01:11:17 2018" 4096 6 0x800 pkg.conf
root@psi:/usr/local/etc # date
Sat Nov 24 07:00:57 UTC 2018If you're more familiar with Subversion then you could already tell this from the previous output. That E you saw? It doesn't mean 'extracted' which is something I hinted at but it actually means Existed; Subversion indicated that the files and directories already existed, and therefor it merely added them under version control, but nothing more.
Making changes
So now the only thing you need to remember is that whenever you make a change to the system you should also tell Subversion about it. When in doubt just ask Subversion what the current status is:
Code:
root@psi:/usr/local/etc # svnlite status
root@psi:/usr/local/etc #
Code:
root@psi:/usr/local/etc # svnlite info
Path: .
Working Copy Root Path: /usr/local/etc
URL: file:///var/db/repo/etc.local
Relative URL: ^/etc.local
Repository Root: file:///var/db/repo
Repository UUID: 6f0b2ca5-adef-e811-9a4c-00123f2e3b36
Revision: 2
Node Kind: directory
Schedule: normal
Last Changed Author: root
Last Changed Rev: 1
Last Changed Date: 2018-11-24 06:41:09 +0000 (Sat, 24 Nov 2018)
Code:
root@psi:/usr/local/etc # cat pkg.conf
# System-wide configuration file for pkg(8)
# For more information on the file format and
# options please refer to the pkg.conf(5) man page
REPOS_DIR [
"/etc/pkg/",
"/usr/local/etc/pkg/repos/",
"/root/repos/"
]
Code:
root@psi:/usr/local/etc # svnlite status
M pkg.conf svnlite diff. If you want to undo these changes then you can use svnlite revert pkg.conf after which things will be changed back to the way they were, this can be an invaluable command if you made accidental changes and you don't have a ZFS snapshot at your disposal.And finally if you wish to keep your changes then you should commit them using:
svnlite commit. I suggest that you don't use the -m parameter but instead use the full editing screen. This will allow you to add any data you might need; not merely a description but also information about possible problem reports, information if someone approved the change (and who that person is, same applies to reporting, reviewing and submitting) and even specific security related information can be added.This is one of the reasons why I think that Subversion is a very good tool to use for this, because you can look up all that extra information at a later time if you need to.
Keep in mind though that once you committed a change then you should also update the current working copy again, you can do that by using
svnlite update. Need an overview of all the recent changes? Couldn't be easier: svnlite log | less.Allowing someone else to make changes?
So let's say that our SSH server has gone bonkers and we can't fix this ourselves. Someone else has offered to help us and he can edit the config files for us. However, that would require direct access and since this person isn't a part of our usual team we can't "just" give him access.
Fortunately we have Subversion at our disposal!
Code:
root@psi:~ # cd /home/helper/
root@psi:/home/helper # svnlite co file:///var/db/repo/etc/ssh
A ssh/moduli
A ssh/ssh_config
A ssh/sshd_config
Checked out revision 3.
root@psi:/home/helper # chmod 666 ssh/*configSo now our helper can safely edit these files. We could tell him to commit the changes once he's done, but this would mean that he'd need to have access to our repository. So another option is to make him apply the changes after which we'll commit them instead.
The advantage is that our helper never gets their hands on the actual configuration. And thanks to Subversion we can check out exactly what he changed long before actual implementing it (remember
svnlite diff?). And speaking of which... once we are satisfied with all the changes then we also no longer need to mess around with cp or mv. Just use Subversion to update the actual configuration:
Code:
root@psi:/etc/ssh # svnlite update
Updating '.':
U sshd_config
Updated to revision 4.It's also an example of what I mentioned earlier on: how Subversion was quite capable of handling large amounts of data. As you can see we never had to check out ('extract') the entire /etc directory. And of course you can expand on this; you could even use such a setup to provide the same configuration across a series of servers.
Securing our data
Now, obviously the idea is to include /var/db/repo in your common backup scheme so that the whole repository will be safely stored away. But what about making a security copy just to be safe? Once again Subversion to the rescue:
Code:
root@psi:~ # svnliteadmin hotcopy /var/db/repo repo.copy
* Copied revision 0.
* Copied revision 1.
* Copied revision 2.
* Copied revision 3.
* Copied revision 4.
root@psi:~ # tar cJf svnrepo.txz repo.copy/
root@psi:~ # rm -r repo.copy/If you need more help...
All Subversion commands have build-in help screens which can be shown by using the help command. So if you want to know more details about the commit command then you can use:
svnlite help commit | less, if you simply need an overview of all admin commands at your disposal: svnliteadmin help, and so on.And there you have it...
From here on you can basically continue to commit & update any changes after which you'll eventually build up quite a log. The main advantage though is that even after a year of updating you'll still have access to the way things were back then. And without all the hassle that a regular backup brings along (nor its massive size).
Also: since you already have a repository at your disposal there's no reason why you shouldn't use that for some actual development. Do you maintain your own shell scripts? Maybe it could be a good idea to add those as well!
