Solved ZFS data pool - 4K partitions or raw device?

graemeg

graemeg

Hi,

I have been reading post after post for the last 2 days. I think a lot of that information might be out of date (posts from 2009, 2011 etc). I'm running FreeBSD 10.1 and want to setup a new data-only pool using four WD Red Pro 3TB drives in RAIDZ2.

For my previous RAIDZ1 I used raw devices in the pool - no partition table, but that was done on FreeBSD 9.x and now has the 512K warning for 4K drives. I want to migrate that data over to the new pool.

I've created the new RAIDZ2 pool with ashift=12 and verified that the pool is ashift 12. All the other links (some a couple years old - so info might be out of date) and posts I've read talk about partition tables where partitions must be aligned on 4K boundaries for best performance. But does that apply to raw devices in a pool too? So far the performance is already MUCH better than my old pool (60MB/s vs 125MB/s for copying large data files), but not sure if that is due to WD Green drives vs WD Red Pro drives.

So what is the recommendation (and why)? Can I just use raw devices (I like this idea) or must I still create a 4k aligned partition and gnop trick, with FreeBSD 10.1?

Edited:
It seems the 4th reply in this message thread [https://forums.freebsd.org/threads/wd-red-4k-alignment.48507/] answers my question (if that info is correct). If you use raw devices (no partitions on Advanced Format drives), the ZFS pool will automatically be 4k sector aligned because it will start at sector 0.
 
As far i understand this alingnment trouble, it is good to use a raw device.

Good aligned partition:
|<physical disk start
|---|---|---|---|---|---|---|---|<physical 4K sectors
|---------------|<start of a partition, matches one 4k sector
|---------------|---|---|---|---|<logical 4k sectors perfect match physical
|---------------|||||||||||||||||<logical 512b sectors resulting in bad performance

Bad aligned partition:
|<physical disk start
|---|---|---|---|---|---|---|---|<physical 4K sectors
|-----------------|<start of a partition, does not match 4k sector
|-----------------|---|---|---|---|<logical 4k sectors does not match physical, results in very bad performance
|-----------------|||||||||||||||||<logical 512b sectors resulting in bad performance, but equal to good aligned partitions

Raw device:
|<physical disk start
|---|---|---|---|---|---|---|---|<physical 4K sectors
|<start of the "raw" partition, matches one 4k sector
|---|---|---|---|<logical 4k sectors perfect match physical
 
Using raw devices for ZFS is only relevant to Solaris and its derivatives. It has to do with Solaris not enabling write cache otherwise, thus hurting performance heavily.

You could use either raw devices or gpt partitions in FreeBSD. But make sure you understand how to make gpt partitions with correct 4K alignment before proceeding. Also, you should also consider use glable or gpart -l to label your disks.

Finally, in 10.1 and above, you could set vfs.zfs.min_auto_ashift=12 instead of using the gnop trick.
 
t1066 said:
You could use either raw devices or gpt partitions in FreeBSD. But make sure you understand how to make gpt partitions with correct 4K alignment before proceeding. Also, you should also consider use glable or gpart -l to label your disks.
Click to expand...

Thanks for all the info from both replies.

I went ahead with raw devices - so no need to fuss about with GPT partitions. I did read about assigning labels to my disks, but didn't bother. I did some tests by copying a couple a GB of data to the new data pool. Then shut down the server, swapped around the SATA cables to see if it interferes with the pool when the server boots up again. Everything worked perfectly. It seems that even though I used unlabelled raw devices it still identifies each disk in the pool with a GUID id. I then did another test - I moved the new pool to different SATA controllers and temporarily unplug the original data pool. Again the server booted just fine and mounted the new data pool without problem. So again, I didn't see the need for labelling the disks - ZFS seems to find the correct drives no matter what. Awesome!

As for the vfs.zfs.min_auto_ashift=12 setting... Because the disks are new and true 4K drives, FreeBSD correctly auto set the ashift=12 when I created the pool. But for safety sake I went ahead and set the minimum auto ashift value of the system.

I'm now in the process of moving 3TB of data to the new pool and the performance is so much better. :)
 
Using labels simplifies the procedure of quickly identifying a disk that might need to be replaced. Especially if you don't have physical access and you want someone else to remotely do that for you. I usually label all my disks using their serial number. Thus, my instructions to the DC guys are: "please remove the disk with serial number XXXXX..."
 
gkontos said:
Using labels simplifies the procedure of quickly identifying a disk that might need to be replaced. Especially if you don't have physical access and you want someone else to remotely do that for you. I usually label all my disks using their serial number. Thus, my instructions to the DC guys are: "please remove the disk with serial number XXXXX..."
Click to expand...

I've never understood how labelling a drive using the serial number is usable. How do you determine which disk has that serial number without first removing the drive to look at the serial number, leading to the removal of the incorrect drive while searching for the correct one?

Or, do you mean that you also label the outside of the drive cage with a sticker that has the serial number on it?
 
You must log in or register to reply here.
Back
Top