How do you keep track of all server changes?

fred974

fred974

Hi,

I run many webjail on top of FreeBSD...
I recently find out that the hard way that one of our website stopped working because of a php upgrade we did weeks ago.
The problem is that we had no idead what version it was untill rolled back from backup..

With that in mind, could you guys tell me what do you use to keep track of the changes you do on the server?
Software version update, configuration change etc...

Is there any best practices?

Thank you in advance
 
We mostly use Puppet and a local repository for this. This ensures everything is installed in exactly the same way using exactly the same versions. We test updates on a couple of test servers beforehand.
 
For keeping track of configuration files changes you can use Revision Control System (RCS).
It works this way: you put a file under RCS, it will keep track of your changes, and will give you the ability to see which changes where made or to restore a previous version.

To put a file under RCS you use the check in command ci(1) this way:
Code: 
# ci -u filename
You will be asked for a description of the file, then a file called filename,v will be created. This is the file where RCS will keep track of the changes you will made. Since you are giving filename in the hands of RCS for the first time, the actual copy will be the revision 1.1 of the file.
Moreover, note that I used the -u switch: this will keep a working copy of the file instead only the one with the ",v" extension.
It's particularly useful for configuration files, since they can't miss.

But if you forget to use the -u switch, you can obtain a working copy of your file checking it out with co(1):
Code: 
# co filename
Now a thing that may look strange: both the file itself, that the one created by RCS will be in read-only mode (r--r--r--)!
This is normal: since the file is under RCS control you must ask it to edit the file, in order to let it keep track of the changes:
Code: 
# co -l filename
Here I used the -l switch with co: this means that I'm locking the file in order to be sure that no other people can make changes contemporary (and probably vanishing mine ones).

When you have done with your edit, RCS will ask you a comment on the changes you made. This will remind you what you modified in that file.

Finally to release the file from the lock, check it in again using the -u switch:
Code: 
# ci -u filename

Done! You successfully put a file under RCS and edited it! :)

Now to see the list of changes you made use rlog(1):
Code: 
# rlog filename
and to see what changed in the file between to different revisions use rcsdiff(1)
Code: 
# rcsdiff -u -r1.1 -r1.2 filename
In this example I checked for differences between the first (1.1) and the second (1.2) revision of the file.

A last suggestion: if you have a lot of file under RCS, ,v files can clutter the directory. To avoid this, create a directory called RCS (yup, all uppercase), and RCS will use it automatically. :)

HTH
 
I'd suggest that either Subversion or Git are a better choice of version control system than RCS at this point. Newer (supported/maintained/improved), more powerful, and yet easier to use. Regardless, the idea of having a history of changes that were made to files is very valuable.

SirDice's comments are also important. Automating installs makes them consistent and repeatable. Also, when changes are necessary, they can be made in one place.
 
Subversion is kind of out of the picture because you can't use it without a central repository, that's not very nice for just version controlling a local directory. Git on the other hand is very simple, you just initialize a git repo in a directory that you want to control and you can start working right away, no other repository set up needed.

Code: 
mkdir directory
cd directory
git init .
vim myfile.txt
git add myfile.txt
git commit

The git init . operation is not destructive so you can use it on existing directories such as /etc.
 
fred974 said:
I recently find out that the hard way that one of our website stopped working because of a php upgrade we did weeks ago.
The problem is that we had no idead what version it was untill rolled back from backup..
Click to expand...
Are you using binary packages or the Ports collection?

I never really liked to depend on the binary packages myself because I always ran into a few issues here and there. Ranging from packages which were unavailable, packages which used different default settings than I needed so eventually I decided to fully rely on the Ports collection.

The main advantage, the way I see it, is that although it might require a little more time to use I'm always in full control over what happens. A mere # portsnap fetch update | portmaster -L | tee portmaster.log will be enough to get me a full overview of which new versions are available.

Because the servers I use all have the same configuration I simply provide my package repository to them and let them use binary package installs, but coming from my own repository.

But even if you do rely on binary packages you'd still have ways to check which versions get installed or updated before you actually apply the update.
 
Sorry Mister wblock@ I deleted my earlier post. Tried to look it up myself.
My question was: How to do revision control with the Git method.
Very informative thread.
 
Basically it's something like this:
Code: 
cd /some/directory
git init
vi somefile
git add somefile
git commit -m "Added somefile"
The git init only has to be done once, it initializes things. Basic workflow is, edit file, git add file, git commit.
 
gpatrick said:
Another thing is an application that tracks changes
Click to expand...
Hi, What application do you use?
SirDice said:
We mostly use Puppet and a local repository for this. This ensures everything is installed in exactly the same way using exactly the same versions.
Click to expand...
Not all jail are equal:( we have some website that was written over 10 years ago that still require php55 and other older package.
We are in the process of rewritting the code and we are looking at Synth. Synth look much easier that Puppet or Poudriere.
OJ said:
fred974 Do you mind editing the title? It's rather irritating. :)
Click to expand...
Sure, what would you like?

Thank you all for sharing your experience with me. At first I was thinking of installing a wiki to keep track of the changes that I make but I need to revisit I think
 
wblock@ said:
I'd suggest that either Subversion or Git are a better choice of version control system than RCS at this point.
Click to expand...

As project management systems, yes. Mercurial is my personal favorite, and is intended to be a sort of "spiritual successor" to Subversion.

However, all the super-neato wave-of-the-future revision control systems suffer from one major limitation compared to RCS: they cannot track individual files, instead tracking all contents of a directory in which a repository is initialized. Create a Git repository in /etc, for example, and every time you type git status Git will list every file in /etc.

This can get you into serious trouble if you have multiple repositories nested within a larger repository, and the fact that I've seen several dozen proposals on how best to manage lots of configurations using Git or Mercurial just makes me think it's a terrible idea.
 
You must log in or register to reply here.
Back
Top