Syncing one directory with another

[r619]

The goal is to minimize writes to /var on SSD.

Mounting /var read-only won't solve it, obviously - it changes too frequently, not only when we decide to update something. So the solution is to mount /var as tmpfs, and sync it with the SSD copy on init (as early as possible, to eliminate the chance of something useful being overwritten by something useless) and SSD copy with tmpfs on shutdown (as late as possible, to make less useless writes).

First task is fairly simple - we just place cp -R /somedir/var.bak/ /var somewhere in rc.d.

The second task is harder: we have to compare the last modification timestamp of each file and directory contents. A custom script which would parse several ls -c outputs and then recursively check directoriess and then copy new files doesn't seem to be right way, also I'm not yet familiar enough with sh syntax.
Are there some other solutions?

 

[SirDice]

Is there some other solutions?

How about net/rsync? You can also use it for local directories.

 

[ondra_knezour]

There is mount_unionfs() which can resolve problems with initial sync, but according to BUGS section of the manpage this filesystem seems to be broken from stone age.

NanoBSD uses some tricks to save it's configuration files to read-only partitions, check there for inspiration.

 

[throAU]

Given the improvements in SSD reliability and reduced cost, considered just mirroring /var?

I'm guessing even a pair of SSDs is a lot cheaper than your time spent to craft some custom hack that isn't in common use, and deal with the fallout when THAT breaks?

If there's one thing the industry has taught me it is this: don't try and make things more complex than they need to be (they are already complex enough!).

I'd suggest it is far more likely that your custom sync solution will be more likely to cause problems or fail in some way (un-clean shutdown for example) more spectacularly than simply wearing the cost of fault tolerance and replacement for SSD reliability.

 

[Anonymous]

... Are there some other solutions?

Perhaps sysutils/clone in sync-mode. Note, I am biased here, since I wrote this utility. # clone -s /var /somedir/var.bak

Anyway, for any kind of synchronization, you need to tell the initial copy to maintain the file/dir attributes. So for cp(1)() you want to use the -p option, in addition to -R.

man cp
...
     -p    Cause cp to preserve the following attributes of each source file
           in the copy: modification time, access time, file flags, file mode,
           user ID, and group ID, as allowed by permissions. ...

# cp -pR /somedir/var.bak/ /var

However, you could use also clone for the initial copy: # clone /somedir/var.bak /var

 

[wblock@]

I'm with @throAU on this. If the system loses power, changes to /var would be lost. I also don't feel it's worth taking special precautions to limit writes to SSDs. They are there to be used. If a system writes so much to /var that it really could wear out an SSD, using a hard drive or remote syslog server would be appropriate. But really, very few systems need that.

 

[phoenix]

Even consumer-grade SSDs are rated for multiple full-disk writes per day (usually 3-5 on the crappiest SSDs; 10-20 on the good ones). /var doesn't change that much on a daily basis, so it won't be reducing the lifespan of the SSD by more than a few days or weeks.

 

[r619]

Given the improvements in SSD reliability and reduced cost, considered just mirroring /var?

I'm guessing even a pair of SSDs is a lot cheaper than your time spent to craft some custom hack that isn't in common use, and deal with the fallout when THAT breaks?

Still, it's a nice way to make yourself familiar with the new system.

I'd suggest it is far more likely that your custom sync solution will be more likely to cause problems or fail in some way (un-clean shutdown for example) more spectacularly than simply wearing the cost of fault tolerance and replacement for SSD reliability.

Backups, in case of this. At worst, I would lose just the package database.

 

[r619]

Perhaps sysutils/clone in sync-mode. Note, I am biased here, since I wrote this utility.

A bug, presumably:

# mkdir source dest && touch ./dest/file1 ./source/file2 && chmod 777 ./dest/file1 && clone -s source dest
.
dest/file1 is of unknown type, it could not be deleted.

No items copied.

 

[Anonymous]

A bug, presumably:

...
dest/file1 is of unknown type, it could not be deleted.
...

A regression in the course of 1.0.0 to 1.0.1. Earlier today, I fixed this already upstream.

The port update is on the way.

 

[throAU]

Still, it's a nice way to make yourself familiar with the new system

Well if it's a project to learn how to do replication, then fair enough. But if it's a production system...

Backups, in case of this. At worst, I would lose just the package database.

Unless you've got your logs elsewhere via remote syslog, they are gone also. Along with your mail spool. You've also got downtime while you recover from backup.

 

[r619]

Wow, just look what I've found.

That makes the task quite easier - I may simply symlink some important dirs (portsnap, pkg database) on init to /mnt/somehdd if it exists. (Yup, I don't think I need old logs.) Everything else is already done and could be explicitly turned on by pushing "populate_var=YES" and "varmfs=YES" strings in /etc/rc.conf.

And now I also understand how rc.conf works.