D
Drew Gurkowski
Guest
ZFS has proven itself to be a reliable and trustworthy filesystem over many years of use. Its RAID-Z level, an advanced form of RAID-5, had a major flaw in its initial implementation: users couldn’t add disks to a RAID-Z array. There was no easy way to expand it after initial creation, which meant creating a new array and copying data. Now FreeBSD has implemented the solution with the addition of RAID-Z expansion.
Many users with small pools previously had no reasonable path to expansion, forcing them to replace all drives at once or migrate data manually. With RAID-Z expansion, they can grow their storage capacity incrementally, reducing costs and complexity.
Storage needs often grow over time in environments using FreeBSD-based NAS systems. Before, expanding a RAID-Z vdev required backing up data, recreating the pool with additional drives, and restoring everything, which was time-consuming and carried inherent risk. With RAID-Z expansion, administrators can add new drives seamlessly, increasing storage capacity without disrupting operations or risking data loss.
RAID-Z vdev are commonly used for high-availability systems in enterprise environments. Despite how critical it is to manage growing datasets while maintaining uptime, expanding storage requires complex migrations or system downtime. With RAID-Z expansion, system administrators can scale storage dynamically without interrupting operations, ensuring continuous availability and data integrity.
The Road to RAID-Z Expansion
Expanding storage capacity has long been a challenge for RAID-Z users. Traditionally, increasing the size of a RAID-Z pool required adding an entirely new RAID-Z vdev, often doubling the number of disks—an impractical solution for smaller storage pools with limited expansion options.
To address this, the FreeBSD Foundation funded the development of RAID-Z expansion, making it both practical and easy to implement. Led by Matt Ahrens, a ZFS co-creator, the feature underwent years of rigorous testing and refinement. Although the pandemic caused some delays, the project was feature complete in 2022. Additional integration steps followed, and the feature is now generally available in the OpenZFS.
Thank You for Your Support
After years of development, industry collaboration, infrastructure testing, and nearly $100,000 investment, we are so excited to see RAID-Z expansion in the recent release of OpenZFS 2.3. We’re also grateful to iXsystems for their efforts in finalizing and integrating this feature into OpenZFS.
This marks a significant milestone in the evolution of the ZFS filesystem and reinforces its position as a cutting-edge open source filesystem for modern storage use cases.
This development work happened because of your donations to the FreeBSD Foundation. We couldn’t have made this financial commitment without your help. Thank you to all our supporters, large and small.
A Technical Look at How to Expand RAID-Z
RAID-Z expansion enables users to increase storage capacity by adding disks one at a time to an existing RAID-Z group, eliminating the need to double the number of drives.
The process works by redistributing existing data across the new disk configuration, creating a contiguous block of free space at the end of the logical RAID-Z group. Because the expansion occurs online, the pool remains accessible throughout. If interrupted—due to a system reboot, for instance—the process resumes from the last known state, ensuring fault tolerance. Importantly, RAID-Z expansion maintains data redundancy, preserving the reliability of stored data.
This feature supports all RAID-Z levels—RAID-Z1, RAID-Z2, and RAID-Z3—without altering their failure tolerance. A RAID-Z1 pool remains a RAID Z1 pool even after multiple expansions.
The benefits of RAID-Z expansion include:
- Incremental Storage Growth: Users can expand capacity by adding as little as a single disk.
- Better Resource Utilization: Previously, expanding RAID-Z required adding an entirely new vdev, often leading to inefficient use of older pools. Now, storage scales dynamically.
- Minimal Downtime: Expansion occurs while the system remains operational.
This demonstration of RAID-Z expansion uses a recent FreeBSD 15-CURRENT snapshot: bsd_FreeBSD-15.0-CURRENT-amd64-zfs-20250116-054c5ddf587a-274800.raw from 2025/01/16, which includes all the latest OpenZFS 2.3 goodies. For reference, here is the uname(1) output of the system used in this demonstration.
| # uname -prismK FreeBSD 15.0-CURRENT amd64 amd64 GENERIC 1500030 |
The demonstration will use plain files on disk – represented in the /dev tree by md(4) FreeBSD driver.
First we will create and prepare five files – the first four will be used for the initial ZFS RAID-Z pool and the 5th will be used as an expansion.
| # truncate -s 10G FILE0 # truncate -s 10G FILE1 # truncate -s 10G FILE2 # truncate -s 10G FILE3 # truncate -s 10G FILE4 # mdconfig -a -t vnode -f FILE0 md0 # mdconfig -a -t vnode -f FILE1 md1 # mdconfig -a -t vnode -f FILE2 md2 # mdconfig -a -t vnode -f FILE3 md3 # mdconfig -a -t vnode -f FILE4 md4 |
We will now create our RAID-Z ZFS pool from 4 disks.
| # zpool create RAID5 raidz md0 md1 md2 md3 # zpool status RAID5 pool: RAID5 state: ONLINE config: NAME STATE READ WRITE CKSUM RAID5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 errors: No known data errors |
Our ZFS pool is healthy and empty. We will now populate it with some /usr/lib data (4 times) to make sure we work on the populated ZFS pool.
| # du -sm FILE* 1 FILE0 1 FILE1 1 FILE2 1 FILE3 1 FILE4 # cp -a /usr/lib /RAID5/less # cp -a /usr/lib /RAID5/more # cp -a /usr/lib /RAID5/alt # cp -a /usr/lib /RAID5/lib # du -sm FILE* 1648 FILE0 1648 FILE1 1648 FILE2 1648 FILE3 1 FILE4 # ls -l /RAID5 total 75 drwxr-xr-x 11 root wheel 582 Jan 16 09:02 alt drwxr-xr-x 11 root wheel 582 Jan 16 09:02 less drwxr-xr-x 11 root wheel 582 Jan 16 09:02 lib drwxr-xr-x 11 root wheel 582 Jan 16 09:02 more # du -smc /RAID5/* 1223 /RAID5/alt 1223 /RAID5/less 1223 /RAID5/lib 1223 /RAID5/more 4889 total # zpool list RAID5 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT RAID5 39.5G 6.38G 33.1G – – 0% 16% 1.00x ONLINE – # zfs list -r RAID5 NAME USED AVAIL REFER MOUNTPOINT RAID5 4.77G 23.9G 4.77G /RAID5 |
Now the most important part – we will expand the ZFS pool with the additional disk.
| # zpool attach RAID5 raidz1-0 md4 |
Now we will watch the ZFS pool expansion ‘live’ with zpool(8) and iostat(8) tools.
| # zpool status RAID5 pool: RAID5 state: ONLINE expand: expansion of raidz1-0 in progress since Thu Jan 23 23:28:25 2025 762M / 6.38G copied at 27.2M/s, 11.67% done, 00:03:31 to go config: NAME STATE READ WRITE CKSUM RAID5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 md4 ONLINE 0 0 0 errors: No known data errors |
The zpool(8) command with iostat argument.
| # zpool iostat RAID5 5 capacity operations bandwidth pool alloc free read write read write ———- —– —– —– —– —– —– RAID5 6.38G 33.1G 0 0 0 0 RAID5 6.38G 33.1G 0 0 0 0 RAID5 6.38G 33.1G 0 0 0 0 RAID5 6.38G 33.1G 16 464 17.8M 8.54M RAID5 6.38G 33.1G 15 56 34.7M 43.9M RAID5 6.38G 33.1G 29 523 37.0M 29.9M RAID5 6.38G 33.1G 20 76 37.3M 36.1M (…) RAID5 6.38G 33.1G 6 28 32.4M 32.4M RAID5 6.38G 33.1G 11 20 21.3M 26.5M RAID5 6.38G 33.1G 815 239 3.91M 13.3M RAID5 6.38G 43.1G 150 118 89.4M 995K RAID5 6.38G 43.1G 185 43 121M 30.6K RAID5 6.38G 43.1G 125 30 113M 34.6K RAID5 6.38G 43.1G 168 0 127M 0 RAID5 6.38G 43.1G 139 26 129M 30.9K RAID5 6.38G 43.1G 185 25 113M 36.0K RAID5 6.38G 43.1G 216 30 133M 34.6K RAID5 6.38G 43.1G 350 31 153M 36.4K RAID5 6.38G 43.1G 239 24 137M 35.7K RAID5 6.38G 43.1G 134 24 125M 34.8K RAID5 6.38G 43.1G 64 67 48.7M 78.6K RAID5 6.38G 43.1G 0 0 0 0 RAID5 6.38G 43.1G 0 0 0 0 RAID5 6.38G 43.1G 0 0 0 0 |
Now the iostat(8) command.
| # iostat -x md0 md1 md2 md3 md4 5 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 0 0 0.1 0.0 0 0 0 0 0 0 md1 0 0 0.1 0.0 0 0 0 0 0 0 md2 0 0 0.1 0.0 0 0 0 0 0 0 md3 0 0 0.1 0.0 0 0 0 0 0 0 md4 0 0 0.1 0.0 0 0 0 0 0 0 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 6 119 6564.4 2419.9 83 4 0 8 2 51 md1 4 126 6563.5 4201.4 63 5 0 7 2 51 md2 6 121 6565.1 3423.5 65 4 0 7 2 49 md3 4 129 6564.0 4389.5 69 5 0 7 2 51 md4 14 25 398.6 5302.8 0 18 0 12 2 30 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 10 144 12728.0 7091.7 83 14 0 19 2 152 md1 8 146 12729.9 7536.4 72 15 0 18 2 151 md2 10 147 12730.0 7647.3 67 15 0 18 2 153 md3 12 147 12720.8 7639.5 59 15 0 18 2 152 md4 0 119 0.0 11174.0 0 19 0 19 0 121 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 4 36 19997.6 15397.3 500 408 0 416 3 208 md1 2 38 19072.7 15196.6 231 427 0 419 2 210 md2 2 37 19072.5 10519.1 218 445 0 435 3 208 md3 3 36 19072.9 10516.0 149 450 0 429 2 205 md4 0 8 0.0 19954.1 0 339 0 339 0 151 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 0 1 1350.1 5573.1 390 660 0 615 2 36 md1 2 1 4621.3 5652.5 95 624 0 284 1 38 md2 1 0 4209.0 1711.0 646 595 0 625 2 38 md3 1 0 4209.0 1710.6 949 1070 0 979 2 38 md4 0 1 0.0 6732.6 0 633 0 633 0 39 (…) extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 26 5 25217.1 7.0 103 4 0 87 2 98 md1 26 5 25420.7 7.1 96 4 0 81 3 94 md2 25 5 23877.7 7.1 106 4 0 89 3 98 md3 28 5 24668.5 7.0 92 4 0 78 3 97 md4 25 5 24230.7 7.3 104 4 0 88 3 96 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b md0 0 0 0.0 0.0 0 0 0 0 0 0 md1 0 0 0.0 0.0 0 0 0 0 0 0 md2 0 0 0.0 0.0 0 0 0 0 0 0 md3 0 0 0.0 0.0 0 0 0 0 0 0 md4 0 0 0.0 0.0 0 0 0 0 0 0 |
After the expansion is complete you will be able to see similar output as the one below.
| # zpool status RAID5 pool: RAID5 state: ONLINE scan: scrub in progress since Thu Jan 23 23:32:56 2025 3.47G / 6.38G scanned at 1.16G/s, 0B / 6.38G issued 0B repaired, 0.00% done, no estimated completion time expand: expanded raidz1-0 copied 6.38G in 00:04:31, on Thu Jan 23 23:32:56 2025 config: NAME STATE READ WRITE CKSUM RAID5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 md4 ONLINE 0 0 0 errors: No known data errors |
After the RAID-Z expansion, the ZFS scrub takes place immediately to additionally verify the integrity of all the data.
The post OpenZFS RAID-Z Expansion: A New Era in Storage Flexibility first appeared on FreeBSD Foundation.
Continue reading...