Other How can I backup and restore a partition with dd?

B

bsdnoob

I want to backup /dev/ada0s3f which is mounted under /usr in my i386 (p4) system and put the backup in an external 1 TB USB HDD that I mount with the following; # ntfs-3g /dev/da0s1 /usbhdd. I formatted the ext. HDD with NTFS and now I have to use fuse module that can't be loaded when the OS is in single user mode. I also tried with dump but it failed with write error or broken pipe error.
Now I'm planning to backup that partition with dd as follows;
# dd if=/dev/ada0s3f of=/usbhdd/freebsd_bkup/ada0s3f.img bs=512
Is that all right? If it's not then please tell me how to do it and restore the .img with dd.
Thank you.
 
You want to back up the raw disk (the block device) while the file system on it is mounted? In that case, the backup copy will be "dirty", and at the minimum you will have to perform fsck on it before trying to mount it. There is risk of creating a corrupted copy. Let me explain why. When the file system opens the raw disk, it leaves a marker on it saying "this disk is mounted as a file system right now, and writes to it may be occurring". If on the next mount finds this marker, it knows that the previous mount never ended due to an unmount, and will know that it wants to perform fsck. If you copy the raw data, that marker will be copied with the data, and the copy is similarly in need of fsck.

Second: The copy might be corrupted. For example, imagine that the raw disk only has 9 sectors (conveniently numbered 1...9), and the file system is performing writes on them in order 1, 9, 5, 3, 9, 1. At the same time that the file system is performing these writes, the copy process is copying them, clearly in order 1...9. Depending on the exact way the two streams of IOs overlap, the state of the copy is one that never existed in the original. For example, if both start at the same time and go at the same speed, the first write to 1 may be reflected in the copy, but the second write to 9 probably is not. Similarly, the very last write to 1 is probably not reflected in the copy, but the next-to-last write to 9 might be. Then if you try to mount the copy, fsck will probably find a terrible mess, and cleanup will either fail or lead to data loss.

Long story short: If you absolutely have to perform the copy with dd, then unmount the source first. Conversely, I would not use dd to copy live file systems, instead perform a logical copy, for example using dump/restore, or rsync.

Next comment: You are doing the copies in 512-byte blocks. That will be very very slow. I would go for 1 MiB blocks.

Lastly: I have no idea why dump failed. There were probably more error messages.
 
ralphbsz said:
You want to back up the raw disk (the block device) while the file system on it is mounted? In that case, the backup copy will be "dirty", and at the minimum you will have to perform fsck on it before trying to mount it. There is risk of creating a corrupted copy. Let me explain why. When the file system opens the raw disk, it leaves a marker on it saying "this disk is mounted as a file system right now, and writes to it may be occurring". If on the next mount finds this marker, it knows that the previous mount never ended due to an unmount, and will know that it wants to perform fsck. If you copy the raw data, that marker will be copied with the data, and the copy is similarly in need of fsck.

Second: The copy might be corrupted. For example, imagine that the raw disk only has 9 sectors (conveniently numbered 1...9), and the file system is performing writes on them in order 1, 9, 5, 3, 9, 1. At the same time that the file system is performing these writes, the copy process is copying them, clearly in order 1...9. Depending on the exact way the two streams of IOs overlap, the state of the copy is one that never existed in the original. For example, if both start at the same time and go at the same speed, the first write to 1 may be reflected in the copy, but the second write to 9 probably is not. Similarly, the very last write to 1 is probably not reflected in the copy, but the next-to-last write to 9 might be. Then if you try to mount the copy, fsck will probably find a terrible mess, and cleanup will either fail or lead to data loss.

Long story short: If you absolutely have to perform the copy with dd, then unmount the source first. Conversely, I would not use dd to copy live file systems, instead perform a logical copy, for example using dump/restore, or rsync.

Next comment: You are doing the copies in 512-byte blocks. That will be very very slow. I would go for 1 MiB blocks.

Lastly: I have no idea why dump failed. There were probably more error messages.
Click to expand...
Thank you very much for your explanation. Could you please tell me how to use rsync in this scenario. I can't dismount /dev/ada0s3f but when I do with # umount -Aa, (being in single user mode) I can't mount usb hdd with fuse as fuse is not available in single user mode so I have no other option but to use dd or tar. I haven't got any other storage except one freeagent 1 tb extrn. hdd.
 
Why is the disk formatted with NTFS in the first place? Why didn't you use UFS (or ZFS)?
 
SirDice said:
Why is the disk formatted with NTFS in the first place? Why didn't you use UFS (or ZFS)?
Click to expand...
i use it to store files from other os(s) & devs such as ext4, ufs(x86solaris10) and android phones so I made a common filesystem and as it's a 1tb, couldn't format it with msdosfs(disc is too large error when I tried from a windows machine. Now I realize the mistake I did.
You are quite right Sir.
P.S I don't know how to use zfs. I never used it.
 
bsdnoob said:
I use it to store files from other os(s) & devs such as ext4, ufs(x86solaris10) and android phones so I made a common filesystem
Click to expand...
Ok, that makes sense.

bsdnoob said:
I formatted the ext. HDD with NTFS and now I have to use fuse module that can't be loaded when the OS is in single user mode.
Click to expand...
I see no reason why this wouldn't work in single user mode though. Just make sure the fusefs(5) kernel module is loaded.
 
bsdnoob said:
I want to backup /dev/ada0s3f which is mounted under /usr in my i386 (p4) system and put the backup in an external 1 TB USB HDD that I mount with the following; # ntfs-3g /dev/da0s1 /usbhdd. I formatted the ext. HDD with NTFS and now I have to use fuse module that can't be loaded when the OS is in single user mode. I also tried with dump but it failed with write error or broken pipe error.
Now I'm planning to backup that partition with dd as follows;
# dd if=/dev/ada0s3f of=/usbhdd/freebsd_bkup/ada0s3f.img bs=512
Is that all right? If it's not then please tell me how to do it and restore the .img with dd.
Thank you.
Click to expand...


Reading the ralphbsz answer(very interesting to read),why dont you boot with the FreeBSD installer image
and make the whole operation there?
you have one functional terminal to do everything without mounting nothing
and can work in "cold" mode
 
bsdnoob said:
Could you please tell me how to use rsync in this scenario.
Click to expand...
(as root)
# mkdir /usbhdd/backup_of_usr_20200710
# rsync -av /usr /usbhdd/backup_of_usr_20200710/
The trailing slash on both directory names is very important.

But note that if writes are occurring on /usr, the state will still not be a consistent snapshot at a point in time.

Let me ask a much more fundamental question: What are you really trying to accomplish? Why do you want a backup? What are you intending to do with it? If you explain that, maybe we can give you more specific help.
 
SirDice said:
I see no reason why this wouldn't work in single user mode though. Just make sure the fusefs(5) kernel module is loaded.
Click to expand...
wolffnx said:
Reading the ralphbsz answer(very interesting to read),why dont you boot with the FreeBSD installer image
and make the whole operation there?
you have one functional terminal to do everything without mounting nothing
and can work in "cold" mode
Click to expand...
Now I'm thinking about using live cd.
 
SirDice said:
Ok, that makes sense.


I see no reason why this wouldn't work in single user mode though. Just make sure the fusefs(5) kernel module is loaded.
Click to expand...
Sorry for the delay. I also tried # mount -t ntfs-3g -o rw /dev/da0s1 /usbhdd <CR>, got "Operation is not supported on this disk" kinda error and "Not found ntfs-3g" when I used # ntfs-3g /dev/da0s1 /usbhdd/ <CR> in single user mode. I made sure fuse module was loaded and existed in single user mode but from single user mode, I couldn't find ntfs-3g.

In this case I decided to return to multi user mode and take backup the unmounted /dev/ada0s3f (/usr) using
# dump -0Lauf /usbhdd/freebsd_bkup/ada0s3f.dump /dev/ada0s3f but got the following error:

mksnap_ffs: Cannot create snapshot /usr/.snap/dump_snapshot: /usr: Snapshots are not yet supported when running with journaled soft updates: Operation not supported
dump: Cannot create /usr/.snap/dump_snapshot: No such file or directory

But when I used the dump without -L, dump worked, started taking backup with warning L is required to backup live filesystem.

Thank you sir.
 
# mkdir /usbhdd/backup_of_usr_20200710
# rsync -av /usr /usbhdd/backup_of_usr_20200710/
ralphbsz said:
The trailing slash on both directory names is very important.
Click to expand...
Thank you very much. Here I see only one trailing slash; # rsync -av /usr /usbhdd/backup_of_usr_2020071/
Where is the other one?

Now, let me explain it to you(as you've wanted to know why I wanted the filesystems backup).
I want to increase the size of /var to accommodate packages for # pkg upgrade (/var was exhausted while keeping the packages).
Few days ago I tried to update and upgrade packages but got an error/warning showed version mismatch as new kernel was 1201000 and the existing kernel is 1000086. I was asked whether I'd IGNORE_OSVERSION (y/n)? I pressed n, later I tried the following:
# freebsd-update "OSVERSION"=1201000" fetch <cr>
# frebsd-update "OSVERSION=1201000" extract <cr>
# pkg "OSVERSION=1201000" update <cr>
# pkg "OSVERSION=1201000" upgrade <cr> Here /var ran out of space.

Now I'm planning to resize all the partitions (in order to make bigger /var and /usr.)
I've got two ide type hdds. 180G and 40G.
Code: 
=>       63  312581745  ada0  MBR  (149G)
         63       1985        - free -  (993K)
       2048   50331648     1  linux-data  [active]  (24G)
   50333696    2097152     2  linux-data  (1.0G)
   52430848   62914560     3  freebsd  (30G)
  115345408  197236400        - free -  (94G)

=>      63  78165297    ada1  MBR  (37G)
        63     16002          - free -  (7.8M)
     16065  31262490  ada1s1  fat32  (15G)
  31278555  46886805          - free -  (22G)

=>       0  62914560  ada0s3  BSD  (30G)
         0   2097152       1  freebsd-ufs  (1.0G)
   2097152   4194304       2  freebsd-ufs  (2.0G)
   6291456   2097152       4  freebsd-ufs  (1.0G)
   8388608   6291456       5  freebsd-swap  (3.0G)
  14680064  48234496       6  freebsd-ufs  (23G)
I will create /usr in the free space of 22 gigs 2nd hdd (ada1) restore ada0s3f's dump and resize ada0s3[b-f].
Pardon me for the delay in responding.
 
SirDice said:
Ok, that makes sense.


I see no reason why this wouldn't work in single user mode though. Just make sure the fusefs(5) kernel module is loaded.
Click to expand...
Again I'm really sorry to have made you confused. ntfs-3g is available in single user mode too but I forgot to tell you I had used
# unmount -A when I failed to unmount /dev/ada0s3f (/usr) in a normal way. unmount -a unmounted /usr and other partitions except /
What you saw was right, Sir.
 
Dear bsdnoob,
may be you could also consider not to use so many separate partitions. Then you would not face trouble with respect to the split of the data. About backup I think there are three important things.
  1. /etc because of the configurations
  2. /usr/local/etc for the same reason
  3. Your data stored in your home directory
Then you like to store the list of packages you have installed. But the other stuff is not worth to back up if there is no additional reason. As keeping log files for legal aspects or whatever if you run some business or so.
 
chrbr said:
Dear bsdnoob,
may be you could also consider not to use so many separate partitions. Then you would not face trouble with respect to the split of the data. About backup I think there are three important things.
  1. /etc because of the configurations
  2. /usr/local/etc for the same reason
  3. Your data stored in your home directory
Then you like to store the list of packages you have installed. But the other stuff is not worth to back up if there is no additional reason. As keeping log files for legal aspects or whatever if you run some business or so.
Click to expand...
Thanks. Here are some reasons if you consider those as additional reasons. I want to resize all the partitions except ada0s3a(/) and in order to maintain existing order (as it is now) I have to resize /var, /tmp, swap and finally /user to use the rest of the free space I mentioned. I will create /usr in /dev/ada1s2, restore /usr dump there, resize all the said partitions and finally restore the corresponding dumps. Is it all right?
 
Dear bsdnoob,
this is a valid reason of course. The procedure sounds good to me. By the way, I will be out for some days and can not respond to further questions on short term, just to let you know.
 
bsdnoob said:
# rsync -av /usr /usbhdd/backup_of_usr_20200710/

Thank you very much. Here I see only one trailing slash; # rsync -av /usr /usbhdd/backup_of_usr_20200710/
Click to expand...
Sorry, typo on my part. If you want to copy everything in /usr AND UNDERNEATH IT to the directory /usbhdd/...2020071 AS SUBDIRECTORIES, then both have to have a trailing slash: rsync ... /usr/ /usbhdd...0710/

My fault.

Regarding the number of partitions: That's a difficult tradeoff. Some people advocate separate partitions, so that if a runaway process fills one up, at least some part of the system is still running, and you can debug. Great idea. Except with any of the vitally important partitions (/var/log, rest of /var, /tmp, /etc ...) filled up, the system is probably going to come down hard anyway. And it is SO easy to undersize them, unless you have experience (and spare disk space to just be generous). Personally, I still use separate partitions, but I don't know whether I would recommend it, or do it again. You might be better off going to just one system partition, and then one home partition.
 
ralphbsz said:
Sorry, typo... .
Click to expand...
It's all right. I should have guessed where the other trailing slash could be but after fumbling with backup for several hours I couldn't take any risk lest something went wrong.

Please tell me just one thing;
What is the syntax of gpart command to create second slice in my 2nd HDD (ada1) in which 1st slice(15G) is msdosfs?
I want to create the second slice for UFS just after the first slice. I assume it will be ada1s2. I've been stuck at this point.
Here is the output of gpart of my second 40G HDD.
Code: 
=>      63  78165297    ada1  MBR  (37G)
        63     16002          - free -  (7.8M)
     16065  31262490  ada1s1  fat32  (15G)
  31278555  46886805          - free -  (22G)

Thank you.
 
Read the man page for gpart, and understand it. In general terms: Gpart add makes a new partition on an existing, partitioned disk (you already have a partition table, so you don't have to start with gpart create). By default, the new partition will be right at the beginning of the free space. You probably want to add "-t freebsd-ufs" to make it a UFS partition. And finally, the argument to gpart is the disk itself, like ada1. That should do it.

You might want to align the disk; I always align things on 1MiB boundaries today, but that's really overdoing it from a performance point of view; I'm just lazy and do all disk copying (with dd and backup tools) with 1MiB IOs, so aligning things on boundaries and making sure all partitions are exact multiples just makes life easier.
 
ralphbsz said:
Read the man page for gpart, and understand it. In general terms: Gpart add makes a new partition on an existing, partitioned disk (you already have a partition table, so you don't have to start with gpart create). By default, the new partition will be right at the beginning of the free space. You probably want to add "-t freebsd-ufs" to make it a UFS partition. And finally, the argument to gpart is the disk itself, like ada1. That should do it.

You might want to align the disk; I always align things on 1MiB boundaries today, but that's really overdoing it from a performance point of view; I'm just lazy and do all disk copying (with dd and backup tools) with 1MiB IOs, so aligning things on boundaries and making sure all partitions are exact multiples just makes life easier.
Click to expand...
Yes, I got it.
Thank you very much.
 
You must log in or register to reply here.
Back
Top