Other Trying to find the best way to prepare a RAID10 on a FreeBSD server

[puppydog]

My home server currently runs CentOS 7 but I'm going to be moving to FreeBSD in the near future. The most important thing my server does is act as a fileserver, using four 4TB disks in a RAID10. This is pretty easy to do in Linux with mdadm, and after it's built you just use mdadm --detail /dev/md0 to keep an eye on its status.

Unfortunately I'm having trouble finding an equivalent function in FreeBSD. In my current setup, my server has an 80 GB SSD in ext4 that holds the OS, and the 4TB disks in RAID10 with xfs for the file server. From what I've been reading, it looks like I should be using ZFS for the raid and UFS for the OS? What would be the best way to reproduce my current setup under FreeBSD?

 

[usdmatt]

I would just run one ZFS pool using the 4 disks in RAID10 or RAIDZ2 and probably use the SSD as cache. Put the OS and data on the same pool.

Im not sure how the performance would compare to mdraid/xfs as ZFS is a pretty heavyweight file system, but I think the functionality more than makes up for it. Checksums/scrub & snapshots are awesome. Also helps that the zpool/zfs commands are pretty much the most well designed storage management commands ever invented. If you think mdadm --detail /dev/md0 is a simple command to check health, it's still a lot less straight forward than zpool status mypool.

 

[puppydog]

I would just run one ZFS pool using the 4 disks in RAID10 or RAIDZ2 and probably use the SSD as cache. Put the OS and data on the same pool.

Looking around for the best way to handle RAID10 as a ZFS pool, it doesn't look like there's a single easy option for it. From what I've found, I would accomplish this by running # zpool create poolname mirror disk0 disk1 followed by # zpool add poolname mirror disk2 disk3. Is this correct? And in doing this, what is the advantage to using the SSD as a cache? Or would it only be useful if I used RAIDZ2?

 

[SirDice]

From what I've found, I would accomplish this by running # zpool create poolname mirror disk0 disk1 followed by # zpool add poolname mirror disk2 disk3. Is this correct?

Yes, that's spot on.

And in doing this, what is the advantage to using the SSD as a cache? Or would it only be useful if I used RAIDZ2?

It can be used for pretty much every configuration but it really depends on your usage patterns. My home server has 4 * 3TB in a RAIDZ, I once added an SSD as L2ARC but after a couple of weeks the cache was barely 1% in use. So it was pretty useless and it didn't improve anything for me.

 

[usdmatt]

A zpool is made up of one or more vdevs. There are multiple types of vdev - single disk, mirror, raidz, etc. If more than one vdev is added to a pool, the data is striped across them.
The command to create a pool looks like this:

# zpool create poolname vdev1 [vdev2] [vdev3] ...

Each vdev specifies the type (although you don't include that for a single disk), followed by the disks you want in that vdev.

So for example the following commands would just stripe across 3 disks:

# zpool create poolname disk1 disk2 disk3

This would create two mirrors, effectively what you're after:

# zpool create poolname mirror disk1 disk2 mirror disk3 disk4

Of course there's nothing wrong with the way you've shown; You're creating a pool with a single mirror vdev, then using zpool add, which is the command to add a new vdev to an existing pool.

 

[puppydog]

This would create two mirrors, effectively what you're after:

# zpool create poolname mirror disk1 disk2 mirror disk3 disk4

I'm backing up my current raid, it'll take a while but I expect to start the process today. Just had a couple more questions. First, I'm trying to see how the mirroring is done here. As I'm reading it, disk1 and disk2 are a mirrored pair (RAID1), as are disk3 and disk4, and the pool combines them similar to a RAID0? So if each disk is 4TB, I have a total of 8TB of usable disk space here, and can potentially lose two disks without data loss, as long as the two lost disks are not both from the same mirrored pair. e.g., I could lose disk1 and disk4 and still have 8TB of usable disk space. Did I lay things out correctly there?

And second, on an mdadm-managed Linux raid, the raid device is treated as a logical device, in my case /dev/md0, which behaves like any other disk. Is the same true of a zpool device? Will I be able to keep it mounted it at the particular mountpoint I need it to be on just like I would with my current /dev/md0?

I appreciate your help so far.

 

[usdmatt]

On the first point yes you're correct. You can lose a disk in either mirror.
Alternatively you could put all 4 disks into one RAID-Z2 vdev, which is similar to RAID6 and would allow you to lose any two disks, although performance would be slightly worse than 2 mirrors.

The second point is a bit more complex. A ZFS pool is basically just a place to store blocks of data. A ZFS pool can contain any number of file systems that are mounted wherever you want. When the pool is imported, all those file systems will be mounted where you configured them to.

You can probably think of it a bit like LVM if you're from Linux. The zpool is sort of like a LVM group. However, rather than having to split it up and format individual volumes, you can simply create as many file systems as you want and they just share space on the pool. (I may be using the wrong LVM terminology here)

By default when you create a zpool, a single ZFS file system is created with the same name as the pool, and mounted on /poolname

# zpool create mypool disk1
# mount
mypool on /mypool

You can then create additional file systems

# zfs create mypool/data
# mount
mypool on /mypool
mypool/data on /mypool/data

You do not use the 'normal' file system mounting tools with ZFS. If you want to change mount points, you do it through ZFS:

# zfs set mountpoint=/data mypool/data
# mount
mypool on /mypool
mypool/data on /data

If you export the pool those file systems will be unmounted. If you import the pool again they will be mounted automatically (Although of course you can't export a pool if it's got root on it)

 

[puppydog]

Thanks, makes sense. I've actually decided to go with RAIDZ2. Currently I keep my RAID10 as /data, and because of some configs I'll be importing I would prefer that it stay on /data. So if I understand correctly, what I'll need to do is # zpool create data raidz2 disk0 disk1 disk2 disk3, and after it finishes building I'll automagically have a ZFS-formatted RAIDZ2 with 8TB of usable space sitting on /data ready for me to use, without having to add any fstab entries or whatnot?

If this is the case, I think I'm all set. I assume the pool exporting and such is ZFS-specific things? I'll get around to learning the little things that make zfs special, but as long as it will hold my data and work like I'm expecting, that's all I need for now.

 

[usdmatt]

So if I understand correctly, what I'll need to do is # zpool create data raidz2 disk0 disk1 disk2 disk3, and after it finishes building I'll automagically have a ZFS-formatted RAIDZ2 with 8TB of usable space sitting on /data ready for me to use, without having to add any fstab entries or whatnot?

Yes. It'll take about 2 seconds.

One other thing is that it's generally encouraged to create separate file systems for different uses (where it makes sense). Unlike traditional file systems, where cutting your array up into dozens of smaller file systems probably wouldn't be the greatest idea. This is the benefit of all ZFS file systems sharing pool space, allowing you to add file systems whenever you want. For example, you may end up with ZFS file systems like this:

NAME                      MOUNTPOINT
data                      /data
data/projects             /data/projects
data/projects/project1    /data/projects/project1
data/documents            /data/documents
data/mail                 /data/mail
data/home                 /home (<- manually changed with 'zfs set' so normal /home dir is on the pool)
data/media                /data/media
data/media/video          /data/media/video
..etc..

There's a few nice things about keeping things logically separated into independent ZFS file systems:

1. You can instantly see how much space each one is using in zfs list output.
2. Things like compress/quota/etc can be set independently for each file system.
3. You can snapshot each file system separately if you want - so you might not bother keeping snapshots of your media, but may want daily or hourly snapshots of your projects so you can pull out a file from an hour ago or yesterday if you mess something up.
4. Leading on from the last point (as ZFS send/recv is based on snapshots), you can send copies of file systems to another pool separately. You may want to back up some file systems more often than others, or there may be some file systems you're not bothered about copying to another location.

Also, zpool status is your friend. It shows the configuration of the pool and current state, including any disk errors. Run regular scrubs which will verify all data is fully intact and you can watch the progress in the status output.

# zpool scrub data
# zpool status data

Here's the status from one of my machines:

# zpool status
  pool: backup
 state: ONLINE
  scan: scrub repaired 0 in 3h21m with 0 errors on Fri Nov 13 14:35:42 2015
config:

        NAME             STATE     READ WRITE CKSUM
        backup           ONLINE       0     0     0
          mirror-0       ONLINE       0     0     0
            gpt/backup0  ONLINE       0     0     0
            gpt/backup1  ONLINE       0     0     0

errors: No known data errors

In this case I've partitioned and labelled the disks using GPT which can be useful, especially if you physically label the disk the same. That way if backup0 fails, I know to pull the disk with backup0 written on it.