I use rsync to create backups for a directory with 5.000.000 files. Would it be faster if I split the files in 100 directories (50.000 files per directory)? I want to run only 1 rsync to sync everything and not one per directory.
You shouldn't have more than 10k files or directories in one directory.
Around 60k files it gets iffy.
Above 100k files tools start to break or show strange behavior.
… I try to find a way to split them in different directories.
for k in range(5000000):
with open('file-'+str(k) + ".txt", 'w') as f:
f.write('Lorem ipsum dolor sit amet ...')
time rsync -ahisvP user/ user2/
prefix :: 7358
id :: 8264837
prefix :: 7358
filename :: my_file.jpg
1920 :: 1920.png
960 :: 960.png
thumbnail :: thumb.jpg
media/7358/8264837/my_file.jpg
media/7358/8264837/1920.png
media/7358/8264837/960.png
media/7358/8264837/thumb.jpg
I would suggest using one of these:I use rsync to create backups for a directory with 5.000.000 files. Would it be faster if I split the files in 100 directories (50.000 files per directory)? I want to run only 1 rsync to sync everything and not one per directory.
% pkg which $( which inotifywait )
/usr/local/bin/inotifywait was installed by package inotify-tools-3.21.9.6
% inotifywait -r -m -e close_write --format '%w%f' /tmp \
| while read MODFILE
do
# rsync "${MODFILE}" "${SOMEWHERE}"
done
You know that you can do replication + snapshots and then send these snapshots further away on another computer/medium?Replication != Backup
You know that you can do replication + snapshots and then send these snapshots further away on another computer/medium?
tunefs -j disable /
under single user mode) and then it can pipe the backup to gzip and send it over SSH for remote storage if it's desired. It look like this: /sbin/dump -C16 -b64 -0Lau -h0 -f - / | gzip | ssh -p 22 user1@backup.example.com dd of=/home/user1/1402022.dump.gz
chflags nodump /usr/ports
zcat /media/root.dump.gz | restore -rvf -
zcat /media/root.dump.gz | restore -ivf -
… UFS Journaling must be disabled for live backup …