UFS UFS storage copying using DD command does not work

F

free2bsd

Can someone please help me if I can copy a UFS storage from one drive to another? The purpose is to back up a drive and quickly copy it if the original drive fails for some reason, or to replicate to other system without going through installation and configuration which can be very cumbersome in some scenarios.


I have a thumb drive with UFS file system storage from a PFSense installation. I made it so I can easily copy it to another drive versus re-doing installation. However, it does not work.

The thumb drive info (/dev/sda):

Size: 128GB
Partitioning: GUID Partition table
Partition 1: 524 KB
Partition 2: 8.6 GB ufs, type: FreeBSD Boot

When using the following DD command:

dd if=/dev/sda of/dev/sdb bs=1M count=9000 status=progress

The second drive (a USB at /dev/sdb) info (after copying):

Size: 64GB
Partitioning: Unknown (PMBR)
Contents: Unallocated Space

This also happens after I copy from the original thumb drive to a file, the write the file to another drive.
I got this error when trying to boot the 2nd drive up:

da0: 40.000MB/s transfers
da0: 61057MB (125045424 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
GEOM: da0: corrupt or invalid GPT detected.
GEOM: da0: GPT rejected -- may not be recoverable.
mountroot: waiting for device /dev/ufsid/675eb96de3f187dc...
Mounting from ufs:/dev/ufsid/675eb96de3f187dc failed with error 19.

Loader variables:
vfs.root.mountfrom=ufs:/dev/ufsid/675eb96de3f187dc
vfs.root.mountfrom.options=rw,noatime

Manual root filesystem specification:
<fstype>:<device> [options]
Mount <device> using filesystem <fstype>
and with the specified (optional) option list.

eg. ufs:/dev/da0s1a
zfs:zroot/ROOT/default
cd9660:/dev/cd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

? List valid disk boot devices
. Yield 1 second (for background tasks)
<empty line> Abort manual input
 
free2bsd said:
Can someone please help me if I can copy a UFS storage from one drive to another?
Click to expand...
Usually, this works fine. If you copy the whole drive, then you are also copying the partition table, so the partitions on the new one will be exactly the same as on the old one. There is a small potential problem with that: The partition table is stored both at the beginning of the drive (which is always in the same place) as well as the end! And the end of the drive is only in the same place if the two drives are exactly the same size.

If the drives are different sizes, it is safer to manually create the partition table on the target drive (making each partition the same size or bigger than the source partitions), and copy the partitions one at a time.

free2bsd said:
The purpose is to back up a drive and quickly copy it if the original drive fails for some reason, or to replicate to other system without going through installation and configuration which can be very cumbersome in some scenarios.
Click to expand...
(While I do the same thing ...) You know that this technique has limitations and problems? To begin with, you probably do the copy while the system is running, so you are copying from a mounted and live file system. Which means the copy will look like a mounted file system, and when you boot from it, you'll have to start with an fsck. And if you do the copy regularly (every day or every hour), you are needlessly wasting time and write endurance on copying things that haven't actually changed. But it is a. nice simple technique, requiring much less effort than the ones that are more efficient.

free2bsd said:
dd if=/dev/sda of/dev/sdb bs=1M count=9000 status=progress
Click to expand...
That's a Linux command: It calls USB drives /dev/sdX; on FreeBSD, they are typically /dev/daX.

free2bsd said:
GEOM: da0: corrupt or invalid GPT detected.
GEOM: da0: GPT rejected -- may not be recoverable.
Click to expand...
That may be the effect of the GPT partition table (remember, one copy at the end) being in the wrong place. It may also be anything else. Two suggestions for not having this problem: (a) Do the copy partition by partition. (b) if you're doing the copying on FreeBSD, run "gpart recover" afterwards, to repair the partition table. I'm not sure it is called "recover" or "repair", read the man page.
 
Why not use something a little more powerful, like rsync perhaps? You should be able to take a backup in less time, assuming you don't have a huge amount of churn in your filesystem(s).
 
dd'ing from a read-write mounted filesystem is actually a bit worse than just having a dirty filesystem in need of fsck.

You potentially violate write order of transactions, which can lead to serious corruption, more so than just yanking a USB disk out.
 
Sweet and simple sysutils/clone
clone /mnt /mnt1
You have to make your own partitioning & scheme but that is not a bad thing.
I find it excellent when resizing/downsizing drives. Creating drive swap space.
 
free2bsd said:
Can someone please help me if I can copy a UFS storage from one drive to another? The purpose is to back up a drive and quickly copy it if the original drive fails for some reason, or to replicate to other system without going through installation and configuration which can be very cumbersome in some scenarios.
Click to expand...
I am not using UFS on my current systems, but I would think about using dump(8) and restore(8).
 
ralphbsz said:
There is a small potential problem with that: The partition table is stored both at the beginning of the drive (which is always in the same place) as well as the end! And the end of the drive is only in the same place if the two drives are exactly the same size.
Click to expand...

This probably is it. I did not copy all the source drive, because the source drive was intentionally installed using smaller partition to make copying quicker.

ralphbsz said:
If the drives are different sizes, it is safer to manually create the partition table on the target drive (making each partition the same size or bigger than the source partitions), and copy the partitions one at a time.
Click to expand...

I will try this. Would this have any issue with the boot loader code?

ralphbsz said:
(While I do the same thing ...) You know that this technique has limitations and problems? To begin with, you probably do the copy while the system is running, so you are copying from a mounted and live file system. Which means the copy will look like a mounted file system, and when you boot from it, you'll have to start with an fsck. And if you do the copy regularly (every day or every hour), you are needlessly wasting time and write endurance on copying things that haven't actually changed. But it is a. nice simple technique, requiring much less effort than the ones that are more efficient.
Click to expand...

Yes, I am aware of that. This is for installation only, and once the firewall is setup, not much should be changed from it, and I can do the changes manually. I have a problem with PFsense installation on this appliance, that has no video port, and the serial port does not work (some bad detection by the OS). It would work afterward (once I enabled it in PFSense). So I had to install in another computer, then move the hard drive over. This is very cumbersome. So I install Pfsense in a thumb drive, and tried to use that to produce an image for real drive.
ralphbsz said:
That's a Linux command: It calls USB drives /dev/sdX; on FreeBSD, they are typically /dev/daX.
Click to expand...
Yes, I used Ubuntu to copy the two drives.

ralphbsz said:
That may be the effect of the GPT partition table (remember, one copy at the end) being in the wrong place. It may also be anything else. Two suggestions for not having this problem: (a) Do the copy partition by partition. (b) if you're doing the copying on FreeBSD, run "gpart recover" afterwards, to repair the partition table. I'm not sure it is called "recover" or "repair", read the man page.
Click to expand...
 
Jim L. said:
Why not use something a little more powerful, like rsync perhaps? You should be able to take a backup in less time, assuming you don't have a huge amount of churn in your filesystem(s).
Click to expand...

Thank you. For my use case, I only do it once (right after initial installation/configuration), not on going process.
 
cracauer@ said:
dd'ing from a read-write mounted filesystem is actually a bit worse than just having a dirty filesystem in need of fsck.

You potentially violate write order of transactions, which can lead to serious corruption, more so than just yanking a USB disk out.
Click to expand...

These are unmounted drives.
 
free2bsd said:
I will try this. Would this have any issue with the boot loader code?
Click to expand...
It may have an issue with the MBR, because that's not in a partition, but part of the partition table. But I don't even remember whether modern booting even uses the MBR any longer. Quick workaround: dd the whole drive, but only the first few MB. That gives you the MBR, and a (damaged) partition table. Then use a tool like gpart to put a correct partition table on the target drive, now with both copies. And then copy the partitions, one at a time, with dd.
 
Or better: manually partition the second drive similarly to the first and then dd each partition individually.
 
Boot fails because of these errors:
Code: 
GEOM: da0: corrupt or invalid GPT detected.
GEOM: da0: GPT rejected -- may not be recoverable.
The dd-ed partition table is invalid for the second drive because of its smaller size.
 
Andriy said:
The dd-ed partition table is invalid for the second drive because of its smaller size.
Click to expand...
That would only affect the secondary table (at the end of the disk), it will complain about it but it won't fail (because the primary is still valid).
 
Wow, thank you for all the help. With your help, and Wikipedia, I managed to read the existing USB stick data:

Primary header:

Signature:
E F I P A R T
Headersize: 92
CRC header: 90 18 83 91
Compared with:
90 18 83 91
Current loc: 1
Backup header loc: 250085375
The backup location is: 128043712000
Size part entry: 128
Number of part entries: 128
Part entry array loc: 8
CRC part entry array: b7 26 6f 28

Backup header:
Signature:
E F I P A R T
Headersize: 92
CRC header: b7 26 6f 28
Compared with:
bb ee 2f 33
Current loc: 250085375
Backup header loc: 1
Size part entry: 128
Number of part entries: 128
Part entry array loc: 250085343
CRC part entry array: 0 0 0 0


Some how the backup header appears incorrect. Both CRC values are wrong.

So I fixed it:

Signature:
E F I P A R T
Headersize: 92
CRC header: 58 7d 9a 58
Compared with:
58 7d 9a 58
Current loc: 250085375
Backup header loc: 1
Size part entry: 128
Number of part entries: 128
Part entry array loc: 250085343
CRC part entry array: b7 26 6f 28

The next step would be to read the primary partition entry and write it to the back up location. Then calculate the CRC32 and update the backup header.
Also, update the primary header with the backup header's new location. I also need to calculate/update the begin and end LBA values for the new storage of different size than the original disk.

Once that code is done, then I can write a copy code similar to that of the DD command.
 
Finally, I got it all working. I will polish the code a bit and open source it. It was written just to get it to work, without any input checking. So to make it general purpose, it would take a bit more work.

I tested with an image, and I could mount and read it.

I then used it on another USB drive. It worked, and I could boot PFSense up from it.

The disk size is: 64023257088
Signature: E F I P A R T

Headersize: 92
CRC header: 8 18 79 ad

Compared with: 8 18 79 ad

Current loc: 1
Backup header loc: 125045423
The backup location is: 64023256576
Size part entry: 128
Number of part entries: 128
Part entry array loc: 8
CRC part entry array: b7 26 6f 28

-------------------------------------------------:
Signature: E F I P A R T

Headersize: 92
CRC header: 69 dd d0 fd
Compared with: 69 dd d0 fd

Current loc: 125045423
Backup header loc: 1
Size part entry: 128
Number of part entries: 128
Part entry array loc: 125045390
CRC part entry array: b7 26 6f 28


------------
 
After some coding, and testing, I found that the backup partition entry array location was 1 less than it should be.
 
You must log in or register to reply here.
Back
Top