Hi gang,
What is mtree?
mtree(8) is a utility included in the base system (/usr/sbin/mtree) and can be used to compare two directory structures thus allowing you to spot any kind of difference. By default it does this by comparing file size (in bytes) and type, last modification time, file owner, group owner, the permissions as a numeric value, any optional flags (see
But there's a whole lot more you can do here.
Why I'm writing this guide (short editorial section)
The main reason I'm writing this guide is because I think not many people use mtree to its full potential. But to make matters worse I also think the manual page doesn't do a good job. I mean... If all you do in the EXAMPLES section is to point people to some parameters without actually showing them any examples on how to use those...
It also doesn't help that the manual page tells you that all parameters are optional, yet when you then try to use mtree this happens:
Nothing.... Why doesn't it 'do' something?
So yeah, instead of just complaining I figured I'd also try to do something useful myself
How to use mtree
mtree can check a directory hierarchy (so all files and sub directories) and collect specific data on them. I already mentioned the default values which are used, but mtree allows you to collect just about any kind of information you want including hash checksums using MD5, SHA1, SHA256, SHA384, SHA512 and/or RMD160. This makes mtree also usable as an Intrusion Detection System (IDS).
The collected data can then be compared against another set of data, this is the default behavior. The comparison results can then also be used to perform certain actions such as removing files or modifying them (setting modification time, device type and/or symbolic links when they don't match).
Note that a selection of collected data by mtree is referred to as a specification.
Making our first specification
Go to your home directory (using
So what did we do here? Well, by using the -c parameter we told mtree to create a specification for the active directory (the one we're currently in) and print that to stdout. By using the pipe we then saved all this information in home.mtree. Note that the name is entirely optional, but I prefer this extension so that I know what kind of data I'm dealing with.
Also important: it is usually best to save the specification outside of your working space because its not officially part of the directory structure to begin with. If this cannot be done then you might want to exclude it, but I'll explain more about that later on.
So what is this specification file?
If we take a closer look at the specification file you'll quickly notice plenty of familiar patterns:
As you can see it first defines several default settings such as file type, user and group ID and mode. Then it lists all the individual entries and only adds specific details which don't match with the initial entry. It then repeats this process for every sub directory it finds.
How to do a comparison
So now that we got our specification lets use this to compare something against. I'm going to use a ZFS snapshot made from my home directory one day earlier:
If we then look at the results you'll notice that mtree has given us a list of all the file entries which it found in the current directory which do not match with the specification:
In case you recognize some of the file entries here: yes, these entries are part of a (private) Minecraft server.
So what mtree is telling us here is that the modification time of ./spigot11/world_nether differs from that in the specification (which basically points to /home/peter/spigot11/world_nether). Lets test this for ourselves:
I don't know about you but I'd say mtree did a fine job here.
Also noteworthy is that any missing files will also being reported:
Customizing your comparisons
mtree uses keywords to define what kind of information to collect from the file entries. You can check mtree(8) for a list of all the available keywords. As you might have guessed: you can fully customize this list to your own needs.
You have several options: you can either add specific keywords to the current list using -K, you can remove specific (currently used) keywords by using -R or you can specify your own specific keyword(s) using -k where it should be noted that the type keyword will be used by default (but it can be omitted by using the -R parameter).
Lets look at some real examples, shall we?
So lets say you're fine with the default set of information, but would like to add an MD5 checksum to your files. You could use something like this:
You're comparing a huge directory list (so speed & efficiency are important), so you decide that you'll only be checking for user and group id, permission bits, size and an SHA1 checksum. You could use something like this:
You copied a directory and you're only interested to check if all the files (and directories) have actually been copied: could there be anything missing? You can use:
You're going to archive a directory with files for use on another FreeBSD system. Just to make sure you decide to also provide checksums. You might be tempted to use the cksum(1) command, but that wouldn't make it very easy to check those values on the other server again. So instead you use mtree:
Also important to know: you only need to specify specific keywords while creating a specification, but not when you're checking against one. It's simple really: when comparing then mtree only uses the values which it finds in the specification, simply because there isn't anything else available to use
Making an exclude file
As I mentioned earlier the best place to store a specification is outside your working area. There are two easy ways. First is to direct the output somewhere outside ('above') the current directory. For example:
But what if you can't? Or what if you don't want to include a specific file and/or directory in the comparison to begin with?
For example: you're working on a Minecraft project, you're sending your world save directory but you don't want to include the structures directory as well. For non-Minecraft players: the structures directory is a place where so called structure files can be placed. These are basically exported structures (buildings) with a max. size of 30 x 30 x 30 which can also be imported again.
For this to work we'll need to set up a so called exclude file. The mtree(8) manual page refers us to fnmatch(3): "The specified file contains fnmatch(3) patterns matching files to be excluded from the specification". Unfortunately this is yet another example of poor documentation in my opinion because when we read up on the fnmatch() function then we'll soon learn: "The fnmatch() function matches patterns according to the rules used by the shell.".
So to simplify this a little bit: we can set up a list of files to exclude where wildcards recognized by your shell can also be included. When using / in the name pattern then it will be matched against pathnames, otherwise it's only matched against the basename entries. Note that matching is relative to the working directory. Meaning: if the full path to ignore would be: /home/peter/spigot11/world/structures then we can still omit nearly everything because /home/peter/spigot11/world would be my working directory. So the actual entry to ignore is basically: ./structures or better put: structures.
I'm going to name my exclude file: ignore.mtree, and I'll be using mc.mtree as the specification to create. The default set of keywords is fine with me, but I will be including md5.
So here is my exclude file:
And this is the command I'd be using:
MTree used on the FreeBSD base system itself
mtree isn't just a useful gizmo, it's actually being used by FreeBSD to protect its own integrity as well. Have you ever looked in /etc/mtree? You should
Have you ever wondered if your base system is still fully intact? Is it possible that someone has changed some of the default directory entries, maybe a permission bit? Or maybe you changed something and would now like to reset that back.
Easily done! For this example I changed my structure as follows:
And the result? Well, very obvious:
Boy, someone really messed up here Note that I used -e so that all the file entries were ignored and mtree would only focus itself on the directory hierarchy.
How to read this? Simple, the values between brackets show what the value should be followed by what it actually is.
But as mentioned earlier: mtree can do a whole lot more than just compare. I won't be showcasing everything because mtree(8) can show you, yet with this small exception. Because what if you needed to fix your FreeBSD hierarchy asap (as shown above)? Simple:
And everything was ok with the base again
And there you have it...
A closer look at mtree, in my opinion one of the more underrated (and maybe misunderstood?) default commands.
What is mtree?
mtree(8) is a utility included in the base system (/usr/sbin/mtree) and can be used to compare two directory structures thus allowing you to spot any kind of difference. By default it does this by comparing file size (in bytes) and type, last modification time, file owner, group owner, the permissions as a numeric value, any optional flags (see
ls -lo, and also chflags(1)) and finally any optional soft or hard links the file might have.But there's a whole lot more you can do here.
Why I'm writing this guide (short editorial section)
The main reason I'm writing this guide is because I think not many people use mtree to its full potential. But to make matters worse I also think the manual page doesn't do a good job. I mean... If all you do in the EXAMPLES section is to point people to some parameters without actually showing them any examples on how to use those...
It also doesn't help that the manual page tells you that all parameters are optional, yet when you then try to use mtree this happens:
Code:
macron:/home/peter/temp $ mtreeSo yeah, instead of just complaining I figured I'd also try to do something useful myself
How to use mtree
mtree can check a directory hierarchy (so all files and sub directories) and collect specific data on them. I already mentioned the default values which are used, but mtree allows you to collect just about any kind of information you want including hash checksums using MD5, SHA1, SHA256, SHA384, SHA512 and/or RMD160. This makes mtree also usable as an Intrusion Detection System (IDS).
The collected data can then be compared against another set of data, this is the default behavior. The comparison results can then also be used to perform certain actions such as removing files or modifying them (setting modification time, device type and/or symbolic links when they don't match).
Note that a selection of collected data by mtree is referred to as a specification.
Making our first specification
Go to your home directory (using
cd) then issue this command: mtree -c > home.mtree. Depending on the size of your home directory this can take a short while, so if you think this is going to be too much then just use any other location of course.So what did we do here? Well, by using the -c parameter we told mtree to create a specification for the active directory (the one we're currently in) and print that to stdout. By using the pipe we then saved all this information in home.mtree. Note that the name is entirely optional, but I prefer this extension so that I know what kind of data I'm dealing with.
Also important: it is usually best to save the specification outside of your working space because its not officially part of the directory structure to begin with. If this cannot be done then you might want to exclude it, but I'll explain more about that later on.
So what is this specification file?
If we take a closer look at the specification file you'll quickly notice plenty of familiar patterns:
Code:
# .
/set type=file uid=1001 gid=1001 mode=0644 nlink=1 flags=none
. type=dir mode=0755 nlink=22 time=1495958731.424230000
.cshrc size=967 time=1432929643.000000000
.gitconfig size=66 time=1433101592.000000000
.history mode=0600 size=31 time=1468775113.891051000
.kshrc size=124 time=1432996941.000000000
.lesshst mode=0600 size=816 time=1495958688.841798000
.login size=251 time=1432929643.000000000How to do a comparison
So now that we got our specification lets use this to compare something against. I'm going to use a ZFS snapshot made from my home directory one day earlier:
Code:
breve:/home/.zfs/snapshot/270517/peter $ mtree -f /home/peter/home.mtree > /home/peter/diff.mtree
Code:
.: modification time (Sun May 28 10:19:38 2017, Thu May 18 12:09:54 2017)
spigot11/world_nether:
modification time (Sun May 28 10:14:48 2017, Sat May 27 03:07:04 2017)
spigot11/world_nether/data/villages_nether.dat:
modification time (Sun May 28 10:14:48 2017, Sat May 27 03:07:04 2017)
spigot11/world_nether/level.dat:
size (748, 745)
modification time (Sun May 28 10:14:48 2017, Sat May 27 03:07:04 2017)
spigot11/world_nether/level.dat_old:
size (749, 745)
modification time (Sun May 28 10:09:39 2017, Sat May 27 03:01:55 2017)So what mtree is telling us here is that the modification time of ./spigot11/world_nether differs from that in the specification (which basically points to /home/peter/spigot11/world_nether). Lets test this for ourselves:
Code:
breve:/home/.zfs/snapshot/270517/peter $ stat spigot11/world_nether/
359491782 105334 drwxr-xr-x 5 peter peter 4294967295 9 "May 27 03:01:38 2017" "May 27 03:07:04 2017" "May 27 03:07:04 2017" "Nov 22 16:08:46 2016" 4096 3 0x800 spigot11/world_nether/
breve:/home/.zfs/snapshot/270517/peter $ stat /home/peter/spigot11/world_nether/
1228072919 105334 drwxr-xr-x 5 peter peter 4294967295 9 "May 28 10:19:45 2017" "May 28 10:30:14 2017" "May 28 10:30:14 2017" "Nov 22 16:08:46 2016" 4096 3 0x800 /home/peter/spigot11/world_nether/Also noteworthy is that any missing files will also being reported:
Code:
./home.mtree missing
./Maildir/new/1495849065.93496_0.breve.intranet.lan missing
./Maildir/new/1495849092.93508_0.breve.intranet.lan missing
./Maildir/new/1495851398.93861_0.breve.intranet.lan missing
./Maildir/new/1495851399.93863_0.breve.intranet.lan missing
./Maildir/new/1495857910.94508_0.breve.intranet.lan missingCustomizing your comparisons
mtree uses keywords to define what kind of information to collect from the file entries. You can check mtree(8) for a list of all the available keywords. As you might have guessed: you can fully customize this list to your own needs.
You have several options: you can either add specific keywords to the current list using -K, you can remove specific (currently used) keywords by using -R or you can specify your own specific keyword(s) using -k where it should be noted that the type keyword will be used by default (but it can be omitted by using the -R parameter).
Lets look at some real examples, shall we?
So lets say you're fine with the default set of information, but would like to add an MD5 checksum to your files. You could use something like this:
mtree -c -K md5.You're comparing a huge directory list (so speed & efficiency are important), so you decide that you'll only be checking for user and group id, permission bits, size and an SHA1 checksum. You could use something like this:
mtree -c -R all -k uid,gid,mode,size,sha1. Note that instead of all we could also have used type, but I think all is easier to remember.You copied a directory and you're only interested to check if all the files (and directories) have actually been copied: could there be anything missing? You can use:
mtree -c -R all. This might seem confusing at first but makes perfect sense: although you're not collecting any specific data from the file entries you're still logging the existence of the entries themselves.You're going to archive a directory with files for use on another FreeBSD system. Just to make sure you decide to also provide checksums. You might be tempted to use the cksum(1) command, but that wouldn't make it very easy to check those values on the other server again. So instead you use mtree:
mtree -c -R all -k cksum.Also important to know: you only need to specify specific keywords while creating a specification, but not when you're checking against one. It's simple really: when comparing then mtree only uses the values which it finds in the specification, simply because there isn't anything else available to use
Making an exclude file
As I mentioned earlier the best place to store a specification is outside your working area. There are two easy ways. First is to direct the output somewhere outside ('above') the current directory. For example:
mtree -c > ../spec.mtree. Another option is to specify which path mtree should use to check: mtree -p my_dir_to_check/ > spec.mtree.But what if you can't? Or what if you don't want to include a specific file and/or directory in the comparison to begin with?
For example: you're working on a Minecraft project, you're sending your world save directory but you don't want to include the structures directory as well. For non-Minecraft players: the structures directory is a place where so called structure files can be placed. These are basically exported structures (buildings) with a max. size of 30 x 30 x 30 which can also be imported again.
For this to work we'll need to set up a so called exclude file. The mtree(8) manual page refers us to fnmatch(3): "The specified file contains fnmatch(3) patterns matching files to be excluded from the specification". Unfortunately this is yet another example of poor documentation in my opinion because when we read up on the fnmatch() function then we'll soon learn: "The fnmatch() function matches patterns according to the rules used by the shell.".
So to simplify this a little bit: we can set up a list of files to exclude where wildcards recognized by your shell can also be included. When using / in the name pattern then it will be matched against pathnames, otherwise it's only matched against the basename entries. Note that matching is relative to the working directory. Meaning: if the full path to ignore would be: /home/peter/spigot11/world/structures then we can still omit nearly everything because /home/peter/spigot11/world would be my working directory. So the actual entry to ignore is basically: ./structures or better put: structures.
I'm going to name my exclude file: ignore.mtree, and I'll be using mc.mtree as the specification to create. The default set of keywords is fine with me, but I will be including md5.
So here is my exclude file:
Code:
# Exclude the exclude file ;)
ignore.mtree
# Exclude the specification file
mc.mtree
# Exclude the structures directory
structures mtree -c -K md5 -X ignore.mtree > mc.mtree. The result would be as expected: a full specification of my Minecraft world directory while excluding all the mtree related files as well as the structures directory.MTree used on the FreeBSD base system itself
mtree isn't just a useful gizmo, it's actually being used by FreeBSD to protect its own integrity as well. Have you ever looked in /etc/mtree? You should
Have you ever wondered if your base system is still fully intact? Is it possible that someone has changed some of the default directory entries, maybe a permission bit? Or maybe you changed something and would now like to reset that back.
Easily done! For this example I changed my structure as follows:
Code:
# chmod g+w /etc/pkg
# chmod 600 /etc/security
# chown peter /rescue && chmod 760 /rescue
Code:
root@macron:~ # cd /
root@macron:/ # mtree -e < /etc/mtree/BSD.root.dist
rescue: user (0, 1001)
permissions (0755, 0760)
etc/security:
permissions (0755, 0600)
etc/pkg:
permissions (0755, 0775)How to read this? Simple, the values between brackets show what the value should be followed by what it actually is.
But as mentioned earlier: mtree can do a whole lot more than just compare. I won't be showcasing everything because mtree(8) can show you, yet with this small exception. Because what if you needed to fix your FreeBSD hierarchy asap (as shown above)? Simple:
Code:
root@macron:/ # mtree -eu < /etc/mtree/BSD.root.dist
rescue: user (0, 1001, modified)
permissions (0755, 0760, modified)
etc/security:
permissions (0755, 0600, modified)
etc/pkg:
permissions (0755, 0775, modified)And there you have it...
A closer look at mtree, in my opinion one of the more underrated (and maybe misunderstood?) default commands.
Last edited: