ZFS Install FreeBSD on ZFS on mirrored partitions

[Giteh]

Hi,
I want to install FreeBSD on disk partitions (not entire disk) in mirror mode. So I want zroot to be mirrored on ada0p1 and ada1p1 partitions. Is it possible by the BSDinstall?

Patryk

 

[zader]

I would say it's best to build the pool using the entire disk.. but you could do it.. see this example http://www.freebsddiary.org/zfs-with-gpart.php. just create your partition as needed and build the pool on that .. in this example he left 200M for boot..

as for doing it through the installer .. just drop to prompt and issue the commands to slice out your disk and create the pool.

If your trying to so something like a dual-boot.. I believe there was a good post in the how-to section.
https://forums.freebsd.org/threads/...gside-another-os-sharing-the-same-disk.75734/

 

[ralphbsz]

I would say it's best to build the pool using the entire disk..

Even if you give all of the space on a disk to ZFS, you should still use a GPT-style partition table (created using gpart), with just one partition. Why? Because it allows you to give the partition a clear and understandable string name, which doesn't change when disks are added/removed. It also means that if the disk becomes separated from the computer (for example, you find it a few years later in a drawer), you can very quickly check what is on it, because the string name of the partition will help identify it.

 

[Emrion]

Your question isn't clear. Do you have any other OS on these disks?

The installer will create as many partitions as it needs. The mirror will concern only one partition on both disks. For example, this is what you get in a zfs mirror install with EFI and BIOS booting capabilities:

gpart show
=> 40 104857520 ada0 GPT (50G)
40 409600 1 efi (200M)
409640 1024 2 freebsd-boot (512K)
410664 984 - free - (492K)
411648 4194304 3 freebsd-swap (2.0G)
4605952 100249600 4 freebsd-zfs (48G)
104855552 2008 - free - (1.0M)

=> 40 104857520 ada1 GPT (50G)
40 409600 1 efi (200M)
409640 1024 2 freebsd-boot (512K)
410664 984 - free - (492K)
411648 4194304 3 freebsd-swap (2.0G)
4605952 100249600 4 freebsd-zfs (48G)
104855552 2008 - free - (1.0M)


 

[Giteh]

Your question isn't clear. Do you have any other OS on these disks?

I'll describe my environment. I want to separate data from OS so I want to create two zpools.
I've got 6 disks. 4 of the will be only for data and two of them I'll use for OS (two because I want to create mirror). But I don't need all 1.1TB for OS so I was wondering can I create 100GB partitions on each drive and install mirrored OS there.

So if I understood correctly there is no possibility to do this by BSDinstall and I need to manually (in command prompt) do this instructions:

[UEFI/GPT] [Dual-Boot] How to install FreeBSD (with ZFS) alongside another OS (sharing the same disk)

Important notes: 1) This tutorial assumes you have the OS you want to dual-boot with already installed on your drive, and that you already have freed up some disk space. Essentially, you will be installing FreeBSD with root-on-ZFS on the remaining free space of the disk, instead of using the...

forums.freebsd.org

but create partition on two disks and create mirrored zpool. Am I right?

 

[gpw928]

You can do it, but you have to lay out the disk partitions yourself, manually.

You also have to consider how you can gainfully use the unused space on the root disks. Suggest you sort that out first.

For some instructions, see Shell Mode Partitioning the handbook, which is, sadly, pretty thin.

The process can be tricky. I tested it on a virtual machine with two "disks", because I could not afford to break the target system.

You will have to escape to a shell before, or during, the installation to lay out the disks manually.

You can then specify those pre-constructed "disk components" to use in the FreeBSD installer.

You need to use MBR for old style mirrors underlying the ZFS root.

Your layout for the zfs root mirror will need to be constructed manually something like this:

[sherman.129] # gpart status
  Name  Status  Components
ada0p1      OK  ada0
ada0p2      OK  ada0
ada0p3      OK  ada0
ada1p1      OK  ada1
ada1p2      OK  ada1
ada1p3      OK  ada1

[sherman.130] # gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  ada0p2 (ACTIVE)
                       ada1p2 (ACTIVE)

[sherman.132] $ swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap  16777212        0 16777212     0%

[sherman.133] $ zpool status zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:42 with 0 errors on Sun Nov  1 03:11:10 2020
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada1p3  ONLINE       0     0     0
        ada0p3  ONLINE       0     0     0

errors: No known data errors

In the FreeBSD Installer, I:
* escaped to the shell to partition the disks; and
* on return to the installer chose, in the "ZFS Configuration" menu to install a "mirror" and selected ada0p3 and ada1p3 when asked to select the "disks".
It's best to have swap on a dedicated mirror partition.

 

[T-Daemon]

So if I understood correctly there is no possibility to do this by BSDinstall and I need to manually (in command prompt) do this instructions:

Not necessarily. There is a modified zfsboot installer script, if you replace the one on the install image [1] with it, bsdinstall(8) let you set the zpool partition size. I have a multi OS installation on two separate disks, used this method to install 12.2-RELEASE root-on-ZFS in a partition 20 % of the disk in size, stripe, but it works also for mirror or raidz.

⚙ D22969 bsdinstall: Allow to set ZROOT partition size

reviews.freebsd.org

[1] /usr/libexec/bsdinstall/zfsboot

 

[Giteh]

Thanks to all for answers. In the end I decided to add additional disks and install FreeBSD there. I think it is to much work to do that on separate partitions and it's not worth that.

 

[ShelLuser]

Late response, but oh well...

Thanks to all for answers. In the end I decided to add additional disks and install FreeBSD there. I think it is to much work to do that on separate partitions and it's not worth that.

That might do the trick but then again... why use ZFS if you don't want to use its capabilities?

First, your question is odd. You want to use 2 pools to separate system and user data. But you also talk about a mirror, but a mirror only ensures that your data is stored on 2 different storages, how would this help to keep your stuff separated?

Most of all: why would it matter? It seems as if you think that if you use one ZFS pool all your data can't be separated but this is not true. I always have 1 ZFS pool on my servers (except one) but I can still keep my stuff easily separated. By utilizing a home directory for example (zroot/home on my end). Or keeping data in specific places. I don't host my websites in the default designated space of /usr/local/www, that seems chaotic to me. Instead I rely on /opt/websites (which is known as zroot/websites) or ones in home directories. Either way: I can pretty much always keep my user data separated from system data, usually by dedicating a dataset for it.

But this has nothing to do with ZFS, UFS or any of them. It's only about properly setting up your system. Still, this process is a lot easier with ZFS, and you don't need different pools. Only different datasets.

Heck... in the end even that different pool will be accessible as a different dataset. So there's actually hardly any difference between another pool or another dataset. After all, backups are always done on a dataset level, not by dumping whole pools. So... you don't need another pool but another dataset.

 

[Giteh]

First, your question is odd. You want to use 2 pools to separate system and user data. But you also talk about a mirror, but a mirror only ensures that your data is stored on 2 different storages, how would this help to keep your stuff separated?

About mirrors: my understanding is that keeping stuff separated and mirrors are solving completely different issue. I want to use mirror because I want to prepare for disk failure and not for keeping data separated.

Most of all: why would it matter? It seems as if you think that if you use one ZFS pool all your data can't be separated but this is not true. I always have 1 ZFS pool on my servers (except one) but I can still keep my stuff easily separated. By utilizing a home directory for example (zroot/home on my end). Or keeping data in specific places. I don't host my websites in the default designated space of /usr/local/www, that seems chaotic to me. Instead I rely on /opt/websites (which is known as zroot/websites) or ones in home directories. Either way: I can pretty much always keep my user data separated from system data, usually by dedicating a dataset for it.

In most cases you're right but I've got quite specific environment. So I've got many servers working on production and most of them with deprecated versions of FreeBSD e.g. 11.0. What's more users are not happy when I want to do updates on this environment so trying to convince them to update from 11.0 to 12.2 is very difficult because it needs many reboot and a lot of time to fetch an update. So I came up with different idea. If I have all important data on separate pool: let's call it zdata I can call 'zpool export zdata' on the old FreeBSD take new disks install there fresh FreeBSD 12.2 attach disks with zdata and call 'zpool import zdata'.

Here I'm assuming that 'zpool export zroot' (where zroot is the pool with both system and data) won't work because zpool export needs all datasets to be unmounted and it's not permitted on system itself. Please correct me if I'm wrong.

There is no risk that I destroy some data using FreeBSD update and it takes only one reboot. What's more if someone will destroy something on system e.g. fat finger I assume that there is less probability that he'll destroy the data. Also If someday I want to migrate to ZFS on Linux I just need to export zdata and it should theoretically work.

 

[ShelLuser]

About mirrors: my understanding is that keeping stuff separated and mirrors are solving completely different issue. I want to use mirror because I want to prepare for disk failure and not for keeping data separated.

That's correct.

What's more users are not happy when I want to do updates on this environment so trying to convince them to update from 11.0 to 12.2 is very difficult because it needs many reboot and a lot of time to fetch an update.

No offense but... this doesn't make sense. First... upgrading a FreeBSD system doesn't require many reboots at all, all it takes is a decent preparation. Because I host a rather customized setup I always build my stuff from source and well... you can easily build your system, maybe set the job priority decently low in order not to have it interfere with other stuff, and you're done.

Installing the update takes some downtime, yes, but wouldn't necessarily need many reboots. At most you'd need 3: one to activate the new kernel and prepare & install the userland, the second to test the libraries and remove the old stuff (if required) and finally the last to activate your system. This could become annoying for users if your services come up and down again and such but that's easily remedied by staying in single user mode. So: plan a downtime for 10 - 15 minutes and within that timeframe you should be able to sort it all out. Only thing your users would experience is a downtime. Something which shouldn't matter if you properly plan it in so that they'll know what to expect.

Running outdated OS's is a major risk. I don't think you'll have many customers left if those systems get run over somehow, which is something you should always keep in mind.

So I came up with different idea. If I have all important data on separate pool: let's call it zdata I can call 'zpool export zdata' on the old FreeBSD take new disks install there fresh FreeBSD 12.2 attach disks with zdata and call 'zpool import zdata'.

Aaaah, I think I see what you mean... Build / upgrade a shadow system in the background, then down your current system, take out the disks, and add them to the new system. Import the data and you're back up and running.

That strategy could definitely work, however... If possible I'd try to set up some kind of network storage instead. It's been a while since I worked with one of those myself but that would probably be the most ideal setup. Disconnect from one server, connect from the upgraded server and you're done. Since most have RAID solutions you also wouldn't need to worry too much about hardware failures.

 

[Giteh]

No offense but... this doesn't make sense. First... upgrading a FreeBSD system doesn't require many reboots at all, all it takes is a decent preparation. Because I host a rather customized setup I always build my stuff from source and well... you can easily build your system, maybe set the job priority decently low in order not to have it interfere with other stuff, and you're done.

I heard that the most safe way is to update in cascade so if I want to update fbsd 11.0 to 12.2 i need to update: 11.0 -> 11.1 -> 11.2 -> 11.3 -> 11.4 -> 12.2
I don't know if it's for sure but there was my assumption.

That strategy could definitely work, however... If possible I'd try to set up some kind of network storage instead. It's been a while since I worked with one of those myself but that would probably be the most ideal setup. Disconnect from one server, connect from the upgraded server and you're done. Since most have RAID solutions you also wouldn't need to worry too much about hardware failures.

You're absolutely right! I have plans to do it exactly this way in the future but right now I just need to handle this disk migration so I needed to figure out something.

Thanks for your help and advices!

 

[SirDice]

What's more users are not happy when I want to do updates on this environment so trying to convince them to update from 11.0 to 12.2 is very difficult because it needs many reboot and a lot of time to fetch an update.

1. Updates can be downloaded in advance without interrupting anything.
2. Plan them in advance, perhaps outside of the office hours. Pick a monthly date for example, like the first monday of each month, so users can expect systems to be down.
3. If systems are so "mission critical" they can't be taken offline then there should be more than one, ideally load-balanced. Then you can just take one node offline to upgrade/update it without interrupting business.
4. What if that machine has a hardware failure? Then your users certainly won't be happy. Plan ahead.

You don't guarantee the uptime of a server, you guarantee the uptime of a service.

I heard that the most safe way is to update in cascade so if I want to update fbsd 11.0 to 12.2 i need to update: 11.0 -> 11.1 -> 11.2 -> 11.3 -> 11.4 -> 12.2

You heard wrong. You can go from 11.0 to 12.2 in one go.