Shell Comparing two directories

[balanga]

I'm looking for a script to compare two directories, A and B, and if any files are found to be identical then to delete them from A. Not sure whether I should use cmp() or diff()....

 

[VladiBG]

sysutils/fdupes

 

[ShelLuser]

Easiest (and in my opinion best) solution is to use /usr/sbin/mtree. I even wrote a small tutorial on how to use it some time ago.

 

[balanga]

sysutils/fdupes

Thanks for pointing this out. It looks like a useful utility. One problem I found was that files from different sources using different character sets have slightly different filenames which sysutils/fdupes does not identify as duplicates. Not sure how to deal with this.

 

[VladiBG]

fdupes compare the files by they md5 checksum. If the contents of the files is different then md5 checksum will be different too and the files will not be identified as duplicates.

For example you can have two mp3 files with different bitrate of the same song. Even if the song is the same the md5 checksum of the files will be different and it won't be detected as duplicates.

 

[balanga]

fdupes compare the files by they md5 checksum. If the contents of the files is different then md5 checksum will be different too and the files will not be identified as duplicates.

If the files have slightly different names because of different character sets will the md5 checksum be different?

 

[balanga]

Easiest (and in my opinion best) solution is to use /usr/sbin/mtree. I even wrote a small tutorial on how to use it some time ago.

https://forums.freebsd.org/threads/small-guide-on-using-mtree.61113/



Never used mtree() Can't seem to get my head around it.

 

[VladiBG]

The file name doesn't matter. The files are compared by they size and md5 checksum.

 

[romanaOne]

Isn't this what rsync is for? I don't remember all its bazillion command line options but one I'll never forget is --dry-run.

I'm looking for a script to compare two directories, A and B, and if any files are found to be identical then to delete them from A. Not sure whether I should use cmp() or diff()....

 

[balanga]

The file name doesn't matter. The files are compared by they size and md5 checksum.

That's what I thought but I see quite a number of files which didn't get deleted....

However when I try running cmp i get

: No such file or directory

although the files are visible when I do a directory listing... They are on an NTFS partition and maybe have invalid filenames.... Must investigate further.

 

[balanga]

sysutils/fdupes

Don't suppose there is an option for deleting empty subdirectories.....

There's quite a lot left after their contents have been deleted. Maybe rm() has some useful options but I'm hesitant about running rm -R ...

 

[obsigna]

For deleting empty directories, find(1) may be utilized:
find /path/of/the/base/directory -type d -depth -and -empty -delete

 

[balanga]

I've managed to delete a lot of file thanks to this, but some caases I get subdirectories containg a single file - Thumbs.db

Not exactly sure what this file is or how to view it.

 

[obsigna]

Thumbs.db is metadata which the Windows Explorer generates automatically. I always delete this when it comes into my way:
find /path/of/the/base/directory -name Thumbs.db -delete

The same goes for .DS_Store files wich the Mac Finder creates automatically:
find /path/of/the/base/directory -name .DS_Store -delete

Beware, if you delete these files, the appearance of directories and its contents would be reset to the default views.

 

[balanga]

A note for myself... and anyone not yet familiar with the power of find()....

A large collection of find examples

Not sure if our find is functionally equivalent to theirs ...

 

[balanga]

For example you can have two mp3 files with different bitrate of the same song. Even if the song is the same the md5 checksum of the files will be different and it won't be detected as duplicates.

I'm just finding this... I'm trying to tidy up my music collection and was surprised that sysutils/fdupes did not flag numerous mp3's in different directories which had the same size. Running cmp() showed that there was a single different character but to all intents and purposes they were duplicates. Is there any way to deal with this?

 

[VladiBG]

If your goal is to remove similar music files then search for software that can compare they acoustic fingerprint or mp3 id3 tags.

 

[balanga]

If your goal is to remove similar music files then search for software that can compare they acoustic fingerprint or mp3 id3 tags.

Is audio/chromaprint the sort of program I should be looking at?

 

[VladiBG]

I don't know such program under FreeBSD maybe audio/fdmf.

Look for dupeGuru or Similarity.