ZFS [HOWTO] Convert Single disk ZFS-On-Root to Mirror

Farioko

New Member

Reaction score: 6
Messages: 1

Hi,

Here is a little tutorial on how to add a disk to your zroot zfs-on-root pool, so that it will become a mirror instead of a single disk (stripe) pool. Had some troubles myself, so I thought let's share how to do this.

Once you have connected the new harddrive, we'll have to create a partition table that is exactly the same as on the main harddrive.

First let's find the harddrive using:

camcontrol devlist

It will most likely appear as ada1.

Let's check out how the partition table looks like on the old drive:

gpart show ada0
Code:
=>       34  156301421  ada0  GPT  (75G)
         34       1024     1  freebsd-boot  (512K)
       1058    4194304     2  freebsd-swap  (2.0G)
    4195362  152106093     3  freebsd-zfs  (73G)

To clean the existing partition table on the new drive execute:

gpart destroy -F ada1

Create the partition table on the new drive:

gpart create -s GPT ada1
gpart add -t freebsd-boot -l boot2 -s 512K ada1
gpart add -t freebsd-swap -l swap2 -s 2G ada1
gpart add -t freebsd-zfs -l zfs2 ada1

Run zdb and get the GUID of the disk in the zroot pool. Copy this GUID, we need it in the next step.

Now attach the new drive to the zroot pool:

zpool attach zroot $guid /dev/gpt/zfs2

Install the bootloader onto the new disk so that when one disk crashes, you'll still be able to boot the system:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

Now you can see the pool resilvering:

zpool status zroot

OPTIONAL - MIRROR SWAP

Make sure you have enough memory to turn off the swap temporarily.
Find the name of the swap device:

swapinfo

Turn off the swap partition:

swapoff /dev/gpt/swap0

Load the geom_mirror kernel module in order to create RAID devices:

kldload geom_mirror
echo 'geom_mirror_load="YES"' >> /boot/loader.conf

Create the mirrored swap:
gmirror label -b prefer -F swap gpt/swap0 gpt/swap2

Modify /etc/fstab so when the system boots up, it mounts the right swap partition:

From:
/dev/gpt/swap0 none swap sw 0 0

To:
/dev/mirror/swap none swap sw 0 0

Activate the swap partition:
service swap start
 

Ofloo

Well-Known Member

Reaction score: 12
Messages: 312

Exactly what I was looking for.

It gets better every time I check it :p

I'd like to add a small note: in my case I was converting an existing disk to the system, before booting I'd say disable swap from fstab and add it back after booting using swapon device, the reason I'm adding this note is, because my original system became ada1 after adding a new disk and my swap mounted on the root files system of the existing disk. Good thing it didn't need the swap file at all, but anyway. But it is maybe a good mental note for people who are moving an existing system to a new disk, that this can happen.
 
Last edited by a moderator:

Ofloo

Well-Known Member

Reaction score: 12
Messages: 312

Code:
# zpool attach zroot 14970465206728248144 /dev/gpt/zfs1
cannot attach /dev/gpt/zfs1 to 14970465206728248144: can only attach to mirrors and top-level disks

Any suggestions ?

Code:
# ls /dev/gpt
gptboot0  gptboot1  swap1  zfs0  zfs1
 

Ofloo

Well-Known Member

Reaction score: 12
Messages: 312

Code:
zpool attach zroot /dev/gpt/zfs0 /dev/gpt/zfs1

gave me the following result:

Code:
# zpool status -v
  pool: zroot
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Dec 29 14:24:55 2014
  21.4G scanned out of 651G at 78.1M/s, 2h17m to go
  21.4G resilvered, 3.29% done
config:

  NAME  STATE  READ WRITE CKSUM
  zroot  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  gpt/zfs0  ONLINE  0  0  0
  gpt/zfs1  ONLINE  0  0  0  (resilvering)

errors: No known data errors
 

Ofloo

Well-Known Member

Reaction score: 12
Messages: 312

My current pool looks like this:

Code:
NAME  STATE  READ WRITE CKSUM
  zroot  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  gpt/zfs1  ONLINE  0  0  0
  gpt/zfs0  ONLINE  0  0  0
  logs
  mirror-1  ONLINE  0  0  0
  gpt/log0  ONLINE  0  0  0
  gpt/log1  ONLINE  0  0  0
  cache
  gpt/cache0  ONLINE  0  0  0
  gpt/cache1  ONLINE  0  0  0

And I was wondering if for example I could use geom_mirror to mirror the cache then add it to the pool. Since for some reason cache can't be mirrored in zfs and if so would it make the system faster or not?
 

kaN5300

New Member

Reaction score: 1
Messages: 5

gpart create -s GPT ada1
gpart add -t freebsd-boot -l boot2 -s 512K ada1
gpart add -t freebsd-swap -l swap2 -s 2G ada1
gpart add -t freebsd-zfs -l zfs2 ada1

I think this method seems more elegant:

Code:
gpart backup ada0 | gpart restore -lF ada1
Where ada0 is the source (old) disk and ada1 is a new one.

But this way you have no ability to change labels. More proper way if you want to change labels before restoring is:

1) #gpart backup ada0 > /tmp/ada0.bkp
2) #cp /tmp/ada0.bkp /tmp/ada1.restore
3) #edit /tmp/ada1.restore
4) #gpart restore -lF ada1 < /tmp/ada1.restore
 

usdmatt

Daemon

Reaction score: 571
Messages: 1,494

Nice guide, the only thing I find odd is using zdb to get the GUID of the existing disk. The normal method is just to run zpool attach pool disk1 disk2, where disk1 is the device name that appears in zpool status output.
 

sko

Well-Known Member

Reaction score: 250
Messages: 460

And I was wondering if for example I could use geom_mirror to mirror the cache then add it to the pool. Since for some reason cache can't be mirrored in zfs and if so would it make the system faster or not?

The cache/L2ARC is automatically striped across all cache-providers. Striping is the fastest configuration, not mirroring.
Mirroring would only provide some fault-tolerance for the cache while sacrificing a lot of performance. ZFS does not rely on the L2ARC and doesn't loose data if a cache provider fails - ZFS will just go to disk and retreive the data from there.
 

Raqibul Hassan

New Member


Messages: 5

Code:
# zpool attach zroot 14970465206728248144 /dev/gpt/zfs1
cannot attach /dev/gpt/zfs1 to 14970465206728248144: can only attach to mirrors and top-level disks

Any suggestions ?

Code:
# ls /dev/gpt
gptboot0  gptboot1  swap1  zfs0  zfs1


Hello Ofloo,

Probably you have put the zpool GUID instead of the disk GUID.
 

jerry507

New Member


Messages: 4

I found this guide helpful so I wanted to add two notes...
First, I also had the "can only attach to mirrors and top-level disks" error that was mentioned above. I resolved this by using the command zpool attach pool disk1 disk2 suggested by usdmatt.

Second, I wanted to elaborate on bootcode. I believe the example given:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

is for the "legacy boot" (MBR???) type.

In my case I am using UEFI so the commands are slightly different. I looked at the FreeBSD wiki entry on Root on ZFS and found the commands I needed under section 1.C.v.a "Create the bootcode partition: for UEFI Boot".

In my case my partition map looked like this:

Code:
=>        40  3907029088  ada0  GPT  (1.8T)
          40      409600     1  efiboot0  (200M)
      409640        1024     2  gptboot0  (512K)
      410664         984        - free -  (492K)
      411648     8388608     3  swap0  (4.0G)
     8800256  3898228736     4  zfs0  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada1  GPT  (1.8T)
          40      409600     1  efiboot1  (200M)
      409640        1024     2  gptboot1  (512K)
      410664         984        - free -  (492K)
      411648     8388608     3  swap1  (4.0G)
     8800256  3898228736     4  zfs1  (1.8T)
  3907028992         136        - free -  (68K)

The structure on ada0 was created by the FreeBSD installer from the guided root on ZFS option. After replicating this structure on ada1 using -b and -s I needed to install bootcode. What worked for me was first running:

gpart bootcode -p /boot/boot1.efifat -i 1 ada1

to install efi partcode into the efi partition (index 1) and then running:

gpart bootcode -p /boot/gptzfsboot -i 2 ada1

to install zfsboot partcode into the gptboot partition (index 2).

After the resilvering was completed I tested by disconnecting my first hard drive and trying to boot. It seemed to work, the system came up and showed that ada0 was faulty. After reconnecting the system was able to boot again as well.

Hopefully this helps someone else. I thought this guide compiled the info I needed in a better way than the handbook did. The handbook as well as the messages after attaching the 2nd disk both show the MBR
 
Top