Restore packages in backup folder using a text file list.

C

Cedric62

Well, after some experimentation, it appears that the below set of commands does a pretty good job of backing up all of my user settings and all of my packages into a backup folder. The below is part of a script file I have been using for the past few months with reasonable success. I didn't want to bog everyone down with the entire contents of my backup script file, so I just included the below lines.

Code: 
mkdir -p /diskbkp/backup/boot
mkdir -p /diskbkp/backup/usr/local
mkdir -p /diskbkp/backup/home
mkdir -p /diskbkp/backup/etc
mkdir -p /diskbkp/backup/var/cache/pkg

rsync -avhpoHX --progress /boot/loader.conf /diskbkp/backup/boot/loader.conf
rsync -avhpoHX --progress /usr/local/etc/ /diskbkp/backup/usr/local/etc/
rsync -avhpoHX --progress /home/ /diskbkp/backup/home/
rsync -avhpoHX --progress /etc/ /diskbkp/backup/etc/

pkg create -a -o /diskbkp/backup/var/cache/pkg

Also, I have managed to coddle together the below command which will give me three different list of packages on my system. The first command creates a list of all packages, and then the second command creates a list of packages installed during the basic installation of FreeBSD, while the third command below creates a list of third party only packages, which are not part of the initial installation process.

Code: 
pkg info > /diskbkp/backup/packagelistv.txt
grep "^FreeBSD-" /diskbkp/backup/packagelistv.txt > /diskbkp/backup/base_os_only.txt
grep -v "^FreeBSD-" /diskbkp/backup/packagelistv.txt > /diskbkp/backup/third_party_only.txt

Now, the problem that I have is that the pkg create command from my above backup script file backs up all of the packages on my system to, /diskbkp/backup/var/cache/pkg. This isn't really a major problem, but if I have to do a complete reinstall, and then I use the below command contained within my restore script file, then I notice that the below command seems to waste a lot of time trying to reinstall packages which have already been during the basic reinstallation of the operating system itself.

Code: 
pkg add /diskbkp/backup/var/cache/pkg/*

So I guess that my question then becomes, is there anyway possible to modify the package add command directly above, so that it only adds packages contained within the list, /diskbkp/backup/third_party_only.txt, and it ignores everything else? I tried replacing the star with a path pointing to the third party only text file, but that didn't seem to work. Any advice greatly appreciated.
 
Well, I think that I may have finally stumbled across a set of commands which will successfully restore my packages, without getting bogged down by trying to re add packages, which have already been during the initial installation of FreeBSD.

Below is part of the script file I use to copy certain folders and to backup my installed packages. the below script also creates a list of third party packages which is later used to help re add packages during the "restore" process.
Code: 
# +++ Copy settings and home directory to backup: +++

mkdir -p /diskbkp/backup/boot
mkdir -p /diskbkp/backup/usr/local
mkdir -p /diskbkp/backup/home
mkdir -p /diskbkp/backup/etc
mkdir -p /diskbkp/backup/var/cache/pkg

# Prepare a list of installed packages with below command:
pkg prime-list > /diskbkp/backup/packagelist.txt
# Prepare a list of installed packages and version numbers with below command:
pkg info > /diskbkp/backup/packagelistv.txt
# Prepare a list of installed packages with file extensions:
pkg query "%n-%v.pkg" > /diskbkp/backup/packagelist_ext.txt
# Prepare a list of third party packages with below command:
grep -v "^FreeBSD-" /diskbkp/backup/packagelist_ext.txt > /diskbkp/backup/third_party_only.txt
# Prepare a list of base os packages with below command:
grep "^FreeBSD-" /diskbkp/backup/packagelist_ext.txt > /diskbkp/backup/base_os_only.txt

# COPY SYSTEM DIRECTORIES WITH METADATA PRESERVATION
# -H preserves hard links, -X preserves extended attributes, --delete cleans old files
rsync -avhpoHX --progress /boot/loader.conf /diskbkp/backup/boot/loader.conf
rsync -avhpoHX --progress /usr/local/etc/ /diskbkp/backup/usr/local/etc/
rsync -avhpoHX --progress /home/ /diskbkp/backup/home/
rsync -avhpoHX --progress /etc/ /diskbkp/backup/etc/

pkg create -a -o /diskbkp/backup/var/cache/pkg

I should also add that "/diskbkp" is on a second physical hard drive, so the backup folder locacted on diskbkp will not get destroyed in case I have to do a complete reinstallation.

So basically, the method I use for restoring my installation of FreeBSD would go something like the below:

1. Turn off the computer and disconnect the second physical hard drive. (I actually leave it connected more recently, but in the beginning I disconnected it, because I wasn't sure of what I was doing.)

2. Reinstall FreeBSD onto the primary hard drive. Do not create any users, do not add a desktop, but do add rsync and create a mount point for /diskbkp.

3. Power off the computer after the installation is complete, and reconnect the second physical hard drive.

4. Restart the computer and boot into the fresh installation of FreeBSD. Mount the partition diskbkp, and navigate to the scripts folder.

5. Launch the below script to "restore" the system to the way it was before it became unstable.


Code: 
#!/bin/sh

# Install FreeBSD in the usual way then use below commands to restore settings & home directory:

# RESTORE SYSTEM DIRECTORIES WITH METADATA PRESERVATION
rsync -avhpoHX --progress /diskbkp/backup/boot/loader.conf /boot/loader.conf
rsync -avhpoHX --progress /diskbkp/backup/usr/local/etc/ /usr/local/etc/
rsync -avhpoHX --progress /diskbkp/backup/home/ /home/
rsync -avhpoHX --progress /diskbkp/backup/etc/ /etc/

# NEW: Safely loop through only third-party applications with extensions
while read -r filename; do
    echo "Restoring: $filename"
    pkg add "/diskbkp/backup/var/cache/pkg/$filename"
done < /diskbkp/backup/third_party_only.txt

echo "Script finished."

Next, the first thing I do after rebooting is I log into my regular user account, and then I run the below command in order to see if all of my packages were re added correctly.

Code: 
root@Asus:/ # pkg check -s -a
Checking all packages: 100%

So, since I don't see any errors after entering the above command, and my desktop seems to be functioning correctly, is it safe to assume that this is a reliable method to backup and restore my system? I know that there are probably better ways to do this, but I'm still getting my toes wet with FreeBSD, and I like to use this method because I think that I understand it. Also, my PC's hardware is pretty old, so not sure if I can do much else.
 
Cedric62 said:
Because I'm still learning, and I'm still figuring things out the hard way. I'll get there someday.
Click to expand...

Speaking humbly, as one who once was, and indeed still is, much like you ...

The sooner you figure things out the easy way, the farther you will go, and the more you will do.

Whenever possible, do not blaze your own path. Follow the stepping stones laid down by those who have gone before.

Stand in the midst of giants, and you will have difficulty seeing the things they see. But standing on the shoulders of those giants, you can see what they see, and beyond.
 
Infostud posted cat pkg-listfile | xargs -n1 pkg install -y
where....
-n1 applies each argument in succession
-y doesn't require your explicit yes for each package and pkg command will complete if a package is not available


B

Thread 'Copying pkgs from one system to another'

What is the simplest way of copying pkgs from one system to another?

I guess I can get a list from one system and subsequently pipe that list into pkg install on the other system using

cat pkg-listfile | xargs pkg install -y

Not sure how to get that list in the first place.... maybe pkg prime-list > pkg-listfile ....

Does that sound right?

see also

B

Thread 'How to do a full system backup of freebsd so i can boot from it if my current system fails'

Hi I'm very new to FreeBSD. Anyone know how I would go about doing a full system backup of FreeBSD that I could keep to boot from and restore my system in case my install gets corrupted?
I have been looking for what utilities to use and the steps to use them to create an exact replica of my FreeBSD system in case I need to restore to new hardware.
I have the "data" backed up already by other means, but I would like to know if there is a way to create an image of the current system that could be restore in the event of a disaster.
Disk space is not a concern, I have terabytes of disk space...
 
Jim L. said:
Speaking humbly, as one who once was, and indeed still is, much like you ...

The sooner you figure things out the easy way, the farther you will go, and the more you will do.

Whenever possible, do not blaze your own path. Follow the stepping stones laid down by those who have gone before.

Stand in the midst of giants, and you will have difficulty seeing the things they see. But standing on the shoulders of those giants, you can see what they see, and beyond.
Click to expand...
I agree, but in my case, it was like I was trying to learn how to sail a boat, but every once in a while my boat would fill up with water and become unusable, mostly due to my own dumb mistakes. So, from my point of view it seemed that if I was ever going to learn even some very basic sailing skills, then I would have to come up with a method of quickly getting my boat upright again, after each time I managed to capsize or sink it. Now, at this point, it seems that I have reliable method of returning my system to the same state it was in before I managed to make it unstable, somehow, so therefore going forward my computer should be in a stable state nearly 100% of the time, and I then I should be able to spend more time actually learning how to use this operating system, than trying to fix the little mistakes I've managed to create on my own.

Also, I realize that there are much better ways to achieve the same result, but for the time being the above does work.
 
Back up your base-package data and scripts to two separate thumb drives for safety. Then the next time you test your "wipe/re-install/restore base packages" operation, do the OS re-install onto a ZFS filesystem, even just a single-drive configuration. The FreeBSD-on-ZFS installer is robust, so just use the defaults to begin with. If you're learning FreeBSD on UFS it's sort of like learning history when you might profit more from learning current events. Nearly all the modern high-leverage use cases for FreeBSD involve ZFS filesystems (except possibly for embedded systems).

The next logical step for the sort of "foot-shooting self defense" environment you're looking to create would be using jails on ZFS. All this effort you're putting in to scale to an environment of 1 would scale to N or N^2 using ZFS and jails.

Folk will be happy to guide you to man pages and resources. Just be sure to start a new thread. :)
 
Jim L. said:
Back up your base-package data and scripts to two separate thumb drives for safety. Then the next time you test your "wipe/re-install/restore base packages" operation, do the OS re-install onto a ZFS filesystem, even just a single-drive configuration. The FreeBSD-on-ZFS installer is robust, so just use the defaults to begin with. If you're learning FreeBSD on UFS it's sort of like learning history when you might profit more from learning current events. Nearly all the modern high-leverage use cases for FreeBSD involve ZFS filesystems (except possibly for embedded systems).

The next logical step for the sort of "foot-shooting self defense" environment you're looking to create would be using jails on ZFS. All this effort you're putting in to scale to an environment of 1 would scale to N or N^2 using ZFS and jails.

Folk will be happy to guide you to man pages and resources. Just be sure to start a new thread. :)
Click to expand...
A while back I tried doing a fresh install, using ZFS, onto a spare hard (same size as my regular working primary hard drive) but after I re added my packages, the darn thing wouldn't boot properly. From what I can recall, after I selected option one from the boot menu, it would just hang and wouldn't do anything else until I hit the power button. Well, maybe when I'm feeling brave I will try again, I probably did something wrong during the partitioning process, so next time I will make sure I know what I'm doing.
 
You must log in or register to reply here.
Back
Top