bhyve How to set up storage for a bhyve hypervisor? ZFS or UFS for host? Guest?

[DaLynX]

Hello,

I am considering migrating a few VMs to a new hosts and would love to make it a freebsd one.

I am not sure however of the best way to set up storage. From the Handbook I read that using ZFS on the host and ZFS volumes for the guests would be optimal. But a blog post from vermaden tells me simple files offer better performance.

ZFS for the host seems to be the easiest way to set up RAID. But to be honest I have trouble understanding what actually happens if I used ZFS on both sides. Which machine does what? Does the guest benefit from the RAID offered by ZFS on the host over its drives?

Or should I just use UFS for guests to make them lighter, if they already benefit from the host's ZFS layer underneath anyway?

Could you please help me understand my options here and their consequences?

 

[rootbert]

ZFS is a huge advantage regarding safety and consistency of your data. Furthermore, ZFS snapshots are awesome when you have to upgrade a VM - using them is easy and straightforward. I suggest using ZFS on the host, UFS in the guest. The host system then takes care of the ZFS cache. If you have ZFS in each VM it will probably eat quite a bit more RAM cause every VM then does caching for itself. While you can also do ZFS inside the VMs, be aware that this dramatically increases your write load (~4x) and thus decreases responsiveness/increases latency - read up on copy on write filesystems if you want to know what's behind this.

 

[bakul]

You have most of these choices: (host: zfs-volume, zfs-file, zfs-filesystem, ufs, RAID, rawdisk) x (guest: ufs, zfs, nfs) each with its own pros and cons. zfs-file is where you use a single file on the host zfs. zfs-filesystem is really a single file on a zfs filesystem (so that you can snapshot it as zfs doesn't allow snapshot of a single file). Obviously nfs on guest can't be used with RAID or rawdisk. (zfs-filesystem,zfs) is the most flexible & featureful but not the most performant. If you are willing to snapshot on the host, use (zfs-filesystem, ufs). zfs-volume may be faster than what I call zfs-filesystem but AFAIK snapshots on zfs volumes are more restricted. With zfs-filesystem, you can even mount snapshots as read-only filesystems from the same or other guests (theoretically -- I haven't actually tried this). Almost as good as plan9's cached WORM filesystem but faster & clunkier!

 

[cmoerz]

We recently updated the handbook on zfs use with bhyve in the virtualization chapter. If you're using zfs for the host as well as for the guest, you should limit caching on the host to metadata only:

zfs set primarycache=metadata <name>

In theory, zvols should be faster, though YMMV. I too am using file based backing storage on zfs and have the impression it is delivering more stable I/O performance. I have not measured it, so it may just be subjective opinion...

 

[DaLynX]

Thanks a lot for your answers!

We recently updated the handbook on zfs use with bhyve in the virtualization chapter. If you're using zfs for the host as well as for the guest, you should limit caching on the host to metadata only:

zfs set primarycache=metadata <name>

In theory, zvols should be faster, though YMMV. I too am using file based backing storage on zfs and have the impression it is delivering more stable I/O performance. I have not measured it, so it may just be subjective opinion...

Yes I read that. Thank you. But after reading about zvols and snapshots (meaning snapshots are limited by the size of the zvol, if I understand correctly), I think doing what bakul recommends fits me better:

You have most of these choices: (host: zfs-volume, zfs-file, zfs-filesystem, ufs, RAID, rawdisk) x (guest: ufs, zfs, nfs) each with its own pros and cons. zfs-file is where you use a single file on the host zfs. zfs-filesystem is really a single file on a zfs filesystem (so that you can snapshot it as zfs doesn't allow snapshot of a single file). Obviously nfs on guest can't be used with RAID or rawdisk. (zfs-filesystem,zfs) is the most flexible & featureful but not the most performant. If you are willing to snapshot on the host, use (zfs-filesystem, ufs). zfs-volume may be faster than what I call zfs-filesystem but AFAIK snapshots on zfs volumes are more restricted. With zfs-filesystem, you can even mount snapshots as read-only filesystems from the same or other guests (theoretically -- I haven't actually tried this). Almost as good as plan9's cached WORM filesystem but faster & clunkier!

So what you mean by zfs-filesystem is:

1. create a zfs filesystem dataset for each VM
2. following cmoerz's, and the manual's advice, I set the primarycache to metadata for that dataset to avoid performance drag
3. inside them create the disk file (.img)
4. use zfs or not in my guest VMs depending on the need for zfs features (e.g. snapshots) or not

Is that right?

Is that #2 step still needed and appropriate if I don't do zfs inside the guest? (e.g. ufs or linux guest)

 

[bakul]

Is that right?

RIght! Though note that I am unclear on the effect of #2 (caching only the metadata) as I haven't used it nor thought much about it. [Edit: as always, you should do your own benchmarking. Never blindly believe random people on the Internet like me!]

 

[vermaden]

I am not sure however of the best way to set up storage. From the Handbook I read that using ZFS on the host and ZFS volumes for the guests would be optimal. But a blog post from vermaden tells me simple files offer better performance.

Do you plan to start with this machine in Formula 1 races? If not - then it mostly does not matter - use something that fits your workflow and what seems more natural for you. When I quoted these benchmarks - the 'flat' files were faster - but that may have been fixed and its the same now (or even faster). Same as for UFS inside VMs - for example I use ZFS on the host and ZFS inside machines as that gives me more flexibility - including ZFS Boot Environments inside VMs that I can send between these VMs if needed.