Solved CP command

[gotnull]

Hi,

I experiment things, I try various methods to create jails.
1) untar archives directly (base.tar and lib32.tar, not txz) => jail = ~650Mo
2) use ' cpdup' to copy the template => jail = ~650Mo
3) use ' cp' to copy the template => jail = ~2Go

Time is what I was looking at first(less is better), that's why I choose .tar over .txz
I put ' zfs clone' aside on purpose for this, because I want to try something else, it's experimental.

Obviously cp copy things it shouldn't but I can't put my finger on the right option to make it not doing it.
So far I tried ' cp -RP' ' cp -Rx', but it still make a bigger jail.
question 1) what would be the right cp's option for that task? or is it because of the following from the cp man page ? :

Note that cp copies hard linked files as separate files. If you need to preserve hard
links, consider using tar(1), cpio(1), or pax(1) instead.

At this point I am a bit lost ...

I noticed also using cpdup implies having to use chflags to be able to delete entirely the jail (which doesn't happen with other methods).
question 2) is it okay to use cpdup this way?

thank you.

 

[covacat]

cp will probably duplicate hardlinks
cp /rescue for testing

 

[Alain De Vos]

I use "cp -axfvR" or "clone -s".
Maybe cpdup threats hardlinks and/or softlinks different than cp

 

[covacat]

Note that cp copies hard linked files as separate files. If you
need to preserve hard links, consider using tar(1), cpio(1), or
pax(1) instead.

this is from cp(1) man page

 

[cracauer@]

rsync is another option, use rsync -a -H -S.

 

[gotnull]

I use "cp -axfvR" or "clone -s".

It tried your solution sadly it makes a big folder too.
clone is not a common tool so I don't want to use it for this task, but it's noted.

Note that cp copies hard linked files as separate files. If you
need to preserve hard links, consider using tar(1), cpio(1), or
pax(1) instead.

this is from cp(1) man page

I don't know if you are quoting me or if it's your answer.
Either way I was wondering if it could be the main reason why ' cp' fails since I am not sure to fully understand the man page's explanation, that's why I quoted it in my first message.
Sorry if I was not clear enough.

rsync is another option, use rsync -a -H -S.

Looks like rsync does the job correctly.
I'll add it to the list, good catch!



Thank you guys for your help ?
So to summarize cp can't handle this task because of hardlinks, it copies them more than once, am I right with this explanation?

PS:
The result from the tests I've done so far: cpdup is quicker than other, then comes tar and rsync.

 

[Alain De Vos]

Give clone a try. When you don't need to copy over the network rsync is overkill.

 

[SirDice]

So to summarize cp can't handle this task because of hardlinks, it copies them more than once, am I right with this explanation?

Yes, it treats them as separate files. Not sure if you're aware of how hardlinks work? Hardlinks point to the same inode on the filesystem. The files in /rescue are a good example of that.

For example:

root@molly:~ # ls -li /rescue/bzcat /rescue/cp
840150 -r-xr-xr-x  146 root  wheel  13623568 Jan 28 15:44 /rescue/bzcat
840150 -r-xr-xr-x  146 root  wheel  13623568 Jan 28 15:44 /rescue/cp

This looks like two files but is in fact just one. As you can see from the output both 'files' are pointing to the same inode (first number in the output). A simple copy would create two separate files on the destination.

root@molly:~ # cp /rescue/bzcat /rescue/cp /tmp/test/
root@molly:~ # ls -li /tmp/test/
total 53232
252666 -r-xr-xr-x  1 root  wheel  13623568 Feb 17 12:11 bzcat
252668 -r-xr-xr-x  1 root  wheel  13623568 Feb 17 12:11 cp

Instead of one file (same inodes) it's now two files (different inodes), exactly the same size and content. This would double the storage because they're now independent of each other.

 

[gotnull]

Give clone a try.

I tried clone and the result is near what cpdup can accomplish which is good, I didn't expect it to be that good.
I don't know if technically they work the same way but both use the chflags.
While it seems to be an appropriate tool cpdup will be my choice because it's used in the handbook, I suppose it has been tested more than clone in this case.
But I am glad you suggested it, at least I know a bit more about it and how fast it can be.

When you don't need to copy over the network rsync is overkill.

Well it's not a problem for me, I already have an use for it locally, for example when I want to copy big files instead of using cp, just because rsync's output is more informative and describes what is happening in real time.
alias copy='rsync -a --info=progress2'

 

[gotnull]

Not sure if you're aware of how hardlinks work?

Well I definitively needed a remainder, and you did it very well.
To be sure I will need to do some tests later today, if I do not train my brain does not register ...
Now I understand better what covacat meant in #2 when he said "cp /rescue for testing", I didn't get that part at the time, but now I do.

Thank you for the detailed explanation.

 

[SirDice]

Now I understand better what @covacat meant in #2 when he said "cp /rescue for testing", I didn't get that part at the time, but now I do.

Yes, almost all the files in /rescue are hardlinks to one and the same inode. Even though there's 146 files in that directory, they all point to one actual file on disk. If you would copy that directory you end up with 146 copies, taking up 146 times more space on disk.

 

[gotnull]

All right I think it's okay for me now, so I can change the state of this thread to solved.
Thanks to all of you guys, I appreciate your help, your suggestions and your explanations

Have a nice Sunday guys.

 

[gotnull]

Got it! I knew I already read this cp advice somewhere.
Found it in the book "FreeBSD Mastery: Jails", chapter 6 _ clones:

With standard jails, clones are pretty simple. On UFS, use tar(1) or
even cp -rp to duplicate the source directory tree in another directory,
then configure the copy as a new jail in jail.conf. If you’re using ZFS,
you can use a ZFS clone.

According to the manpage cp.1, the flag -r is now discouraged and equivalent to -RL.

cp

man.freebsd.org

-L If the -R option is specified, all symbolic links are followed.

-R If source_file designates a directory, cp copies the directory and the entire subtree
connected at that point. If the source_file ends in a /, the contents of the directory
are copied rather than the directory itself. This option also causes symbolic links to
be copied, rather than indirected through, and for cp to create special files rather than
copying them as normal files. Created directories have the same mode as the
corresponding source directory, unmodified by the process' umask.

Note that cp copies hard linked files as separate files. If you need to preserve
hardlinks, consider using tar(1), cpio(1), or pax(1) instead.

From my point of view while tar is appropriate in this case, cp isn't.
I did few tests on a RPI box that runs on UFS obviously, cloning jails with tar takes less than 30s (archives are .tar only, not .tar.gz) when cp takes more time (around 5min) because it copies hardlinks more than once.
May be at the period of time the author wrote the book, this was okay I don't know.
Anyway in 2023 it's not a big deal I presume because most of people run FBSD on ZFS where a command like zfs clone performs very well, but when it's on UFS things are not the same.

To conclude, what upsets me the most is not that there is a tiny error in a book, no it's the fact that my brain remembers it while it can't remember the rest and I didn't finish the book yet!