ZFS Running swap on ZFS

protocelt

Daemon

Reaction score: 414
Messages: 1,253

Hi all,

After participating in Thread 50987 I have a question my "google-foo" can't seem to find answer to and didn't want to run the mentioned thread off topic.

I've been under the impression for a couple years now that running swap on ZFS was a bad idea due to possible memory exhaustion and instead should use a swap partition. I've not seen anywhere that this has changed and was wondering if anyone had any concrete knowledge they could share in regards to this. This doesn't affect me now as I've maxed out the memory in my system and run/test CURRENT so want a swap partition for dumps, however it would be nice to know for sure for future systems I may put together.
 

getopt

Aspiring Daemon

Reaction score: 616
Messages: 866

Thanks for this posting, we definitively should know more about swap on Zfs.
zfs(8) does not know the word swap, but there is an option that could be used to create it:
Code:
 # zfs create -V seize \
                    [b]-o org.freebsd:swap=on[/b] \
                    -o checksum=off \
                    -o compression=off \
                    -o dedup=off \
                    -o sync=disabled \
                    -o primarycache=none \
                    poolname/swap
What does it mean that even the manual does not know swap? Is it a hidden feature or is the manual not complete?

I do use swap on Zfs for quite some time now, but I do not need it for core dumps as they are disabled here. Hint from junovitch:
if you need to do core dumps in the event of a kernel panic the kernel cannot write to a real file system as it could cause corruption. You would need a swap partition to capture the core dump.
The first time I noticed that my swap on Zfs got used was due to a misconfiguration of Poudriere (the tread protocelt refers to) which freezes the system.
It is still not known what actually caused the system going out of memory. Was it the program running or the swap on Zfs? I tend to the first, but it has to be tested which I do not know how to do.

PS:
I found this on the topic:
http://www.schmidp.com/2014/01/05/freebsd-10-does-swap-work-on-a-zvol/
 

kpa

Beastie's Twin

Reaction score: 1,843
Messages: 6,318

The zfs(8) manual page doesn't know about swap because swap on ZFS is just a clever application of a more general concept, ZVOLs that behave like block devices. The org.freebsd:swap property is the "glue" that allows the system to use the swap automatically, the /etc/rc.d/zvol script handles the initialization on boot up.
 

getopt

Aspiring Daemon

Reaction score: 616
Messages: 866

The zfs(8) manual page doesn't know about swap because swap on ZFS is just a clever application
A "clever application" should have a clever documentation, which also should give some information whether this kind of swap is suitable for dumps or not.
 

kpa

Beastie's Twin

Reaction score: 1,843
Messages: 6,318

A "clever application" should have a clever documentation, which also should give some information whether this kind of swap is suitable for dumps or not.

Proper place to document it would be in my opinion swapon(8) since it's the utility that handles turning swap on. Its manual page even makes mention of md(4) devices so ZVOLs are not that far off.

Maybe a new manual page in category 7 that would explain the ins an outs of swap on anything, files, ZVOLs, younameit. swap(7).
 
OP
protocelt

protocelt

Daemon

Reaction score: 414
Messages: 1,253

Thanks for this posting, we definitively should know more about swap on Zfs.
zfs(8) does not know the word swap, but there is an option that could be used to create it:
Code:
 # zfs create -V seize \
                    [b]-o org.freebsd:swap=on[/b] \
                    -o checksum=off \
                    -o compression=off \
                    -o dedup=off \
                    -o sync=disabled \
                    -o primarycache=none \
                    poolname/swap
What does it mean that even the manual does not know swap? Is it a hidden feature or is the manual not complete?

I do use swap on Zfs for quite some time now, but I do not need it for core dumps as they are disabled here. Hint from junovitch:

The first time I noticed that my swap on Zfs got used was due to a misconfiguration of Poudriere (the tread protocelt refers to) which freezes the system.
It is still not known what actually caused the system going out of memory. Was it the program running or the swap on Zfs? I tend to the first, but it has to be tested which I do not know how to do.

PS:
I found this on the topic:
http://www.schmidp.com/2014/01/05/freebsd-10-does-swap-work-on-a-zvol/
In fact I had originally added that link in my original reply to your thread but after thinking about it, retracted it due to it being the only recent link I could find. My guess seems to be so far that the potential issues with swap on ZFS has not changed and depending on your setup and use case should still seriously consider using a swap partition instead.
 
OP
protocelt

protocelt

Daemon

Reaction score: 414
Messages: 1,253

Proper place to document it would be in my opinion swapon(8) since it's the utility that handles turning swap on. Its manual page even makes mention of md(4) devices so ZVOLs are not that far off.

Maybe a new manual page in category 7 that would explain the ins an outs of swap on anything, files, ZVOLs, younameit. swap(7).
The man page is the first place I looked for an answer. This would certainly be helpful.
 

gkontos

Daemon

Reaction score: 487
Messages: 2,160

I used to run my swap on ZFS (zvol) on most of my zroot installations. IMHO performance is better if you don't. But I never experienced any problems whatsoever.
 

belon_cfy

Well-Known Member

Reaction score: 7
Messages: 260

Running swap on zvol is a bad idea, it will eventually crash the server when trashing happens.

You can simulate the trashing by running the following Perl script, it will eventually fill up all your available memory.
Code:
#!/usr/bin/perl
my $count=0;
my @data;
my $temp_data;
for(my $i=0;$i<10000000;$i++) {
        $temp_data.="1234567890abcdefghijklmnopqrstuvwxyz";
}


while(1) {
        $data[$count++]=$temp_data;
}
Tested on FreeBSD 10.1 stable
With zvol swap - my 8GB ram FreeBSD server will stall within 1 minutes.
without swap or dedicated swap disk/partition - server will automatically kill that Perl process.
 
OP
protocelt

protocelt

Daemon

Reaction score: 414
Messages: 1,253

I would be happy to file a PR on this myself, however I don't feel qualified enough to be honest. I can only look at this from a simple user's perspective. Is this a technical limitation of FreeBSD's implementation of ZFS or bug that there is intention of fixing or can be fixed at some point? Is this a technical limitation of ZFS itself that is accepted and thus a possible documentation issue? I don't know.
 

scottro

Daemon

Reaction score: 874
Messages: 2,022

To jump in 7 plus months later, it looks like it can be used with some sysctl parameters, judging by comment 16 of the PR. Documentation still seems lacking. Is there anywhere where the option ... -o org.freebsd:swap=on is explained? I'm looking though the zfs(8) and swapon(8) man pages and don't see it listed.
(I haven't given it any serious testing, just added a couple of gigs of swap on a somewhat underpowered machine. but at least the machine hasn't become unresponsive.)
 
Last edited by a moderator:

vermaden

Son of Beastie

Reaction score: 1,727
Messages: 3,035

Is there anywhere where the option ... -o org.freebsd:swap=on is explained?

This seems to be the only place:
Code:
% grep org.freebsd /etc/rc.d/zvol
  # Enable swap on ZVOLs with property org.freebsd:swap=on.
  zfs list -H -o org.freebsd:swap,name -t volume | \
  # Disable swap on ZVOLs with property org.freebsd:swap=on.
  zfs list -H -o org.freebsd:swap,name -t volume | \
 
Last edited by a moderator:

scottro

Daemon

Reaction score: 874
Messages: 2,022

Thanks. Not that that clarifies it for me, but at least I know I wasn't missing something.
 

vermaden

Son of Beastie

Reaction score: 1,727
Messages: 3,035

scottro

This /etc/rc.d/zvol script just searches for org.freebsd property in the ZFS datasets and if it finds one it just does swapon on that ZFS dataset.
 

usdmatt

Daemon

Reaction score: 601
Messages: 1,543

Technically:
Dataset is the generic term for a ZFS file system, volume, snapshot or clone.
A ZFS file system is just one type of ZFS dataset, so he is correct (although putting zvol would of probably been more appropriate as it's the only type of dataset that should be used for swap). There's confusion because people often use "zfs dataset" or "zvol / zfs volume" when they really mean a "zfs file system" or "zvol". Both are datasets.

Edit: If anyone if wondering what this is going on about, it's in reference to a since deleted post.
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,289
Messages: 4,099

To jump in 7 plus months later, it looks like it can be used with some sysctl parameters, judging by comment 16 of the PR. Documentation still seems lacking. Is there anywhere where the option -o org.freebsd:swap=on is explained? I'm looking though the zfs and swapon man pages and don't see it listed.
(I haven't given it any serious testing, just added a couple of gigs of swap on a somewhat underpowered machine. but at least the machine hasn't become unresponsive.)

It's a user-defined property set on the ZFS dataset. User-defined properties are mentioned in the man pages. The "best-practise" for naming user-defined properties is to use reverse-domain notation, where you use your web domain in reverse, then a colon, then the name of the property. So, for properties defined by the FreeBSD project, they would start with org.freebsd.

User-defined properties are not used by ZFS directly. They are just meta-data attached to datasets that users can then query to do different things with.

For example, I've been toying with using user-defined properties to set the expiration date for snapshots at snapshot creation time, and then just running a cronjob that checks if today's date is greater than the expiration date to determine whether to delete the snapshot or not.
 
Top