Why I think that ZFS is not per definition more secure (or reliable) than UFS
You are mixing a whole lot of things together here. The term "security" is usually used in computers to mean that the system is resilient against unauthorized use or abuse. The discussion of ZFS versus UFS has very little to do with that. File systems can contribute to the security of a system, for example with encryption (at rest or in flight), and with access control (ACLs, capabilities, more interestingly complex authentication systems like what Coda and AFS had). But none of that has anything to do with checksums.
Checksums have to do with "reliability". In the context of file systems, that means roughly that the system is capable of returning the same data that you wrote (not wrong data), at any time: right now, tomorrow, and in a few years. An obscure part important part of reliability is correctness: If you write a certain set of data, the file system should either return exactly the data you wrote (preferably), or give you an error message. Giving you wrong data is RIGHT OUT (I'm shouting because it is a movie quote, Monte Python).
There is another related aspect of file systems, availability: A storage system should be able to read and write at all times, not go down frequently. Checksums have very little to do with that. But reliability and availability are connected, and are often confused: A system that is temporarily down, but will come back in a little while, may have an availability problem, but it still has good reliability. At least that's the way the storage community uses the language.
Yes, ZFS maintains checksums of the data it writes and checks those on the fly. Sort off... See, this doesn't happen all the time, it usually is done during write operations and not so much read operations. And isn't 100% reliable either.
Nonsense. In ZFS, checksums are calculated and written on every write, and checked on every read. At least they're supposed to. There might be obscure bugs where the wrong checksum is calculated at write time, but those would long have been found (because all subsequent reads would find the wrong checksum, so such a bug wouldn't last long). There might be obscure bugs where the checksum is not actually verified during read, or where a checksum error is detected during read but then not reported to the user as an IO error. I find that pretty unlikely.
I've experienced too many ZFS pools which didn't mention anything wrong during normal operation, and then suddenly when you scrubbed them did you get comments about failures which got repaired.
So? That's completely to be expected. To begin with, checksum validation mostly helps when reading, not when writing. In normal operation, probably files were written (with checksums). The files may have become corrupted on disk (usually, we ascribe that to the disk, but it can also be caused by memory, IO interfaces such as SATA or SAS, or RAID controllers). The time those problems will be found is at the next read. But if you look at normal file system usage, many files are written and then not read, perhaps for long times. Problems with those files will be found on the next scrub. And this is why scrubbing is so important (as the paper published by some NetApp folks demonstrated): it finds disk errors early, hopefully at a time when RAID can still fix it, before errors have gotten so extensive that they overwhelm the error correction capability of RAID.
Yet that same feature is also present in filesystems such as UFS. That too can be told to perform a filesystem check, even every sector of the filesystem, in order to ensure that nothing is wrong.
Wrong. UFS does not have the capability to check the content of every sector of a file system. It simply does not store information such as checksums that allow it to verify the content of data blocks. It does have some checksum capabilities built into metadata, but to my knowledge it doesn't even consistently checksum all metadata.
Sure, maintaining these checksums on the fly provides more chances of spotting possible errors. But so would a daily fsck.
Sorry, but a fsck does not read the file content. It reads the metadata, and verifies that it is logically correct (for example, if an inode points at an indirect block, it verifies that the indirect block is validly formatted, and points at data itself). Note that fsck runs way too fast (in a few minutes to an hour on normal disks) to actually verify all the data; that operation is a scrub, and typically takes many hours up to a day.
When you let a ZFS filesystem fill up to a point where no data can be added then you're in high risk of file system corruption because ZFS doesn't reserve space for its metadata. Now all of a sudden those checksums are going to work against you because even if you try to free up diskspace by removing data it would still need free diskspace to account for the metadata required for those write actions, which it doesn't have.
You are confusing things. Corruption is when a file system returns data that is different from what was written. I've never heard that ZFS will corrupt data when it is full. It is quite possible that ZFS has bugs that when it gets too full, it is incapable of deleting files. If it has such a bug, let's hope that it gets fixed.
UFS also knows about this issue because it immediately reserves system space (8% by default) which is explicitly reserved for metadata usage. ZFS on the other hand doesn't account for any of this and therefor it's still a requirement: make sure that your filesystems don't fill up completely!
Nonsense. To begin with, the 8% that UFS (and many other file systems reserve) can be written, just not by normal users. Root can do that easily. And I have written file systems full to the last byte many times in my life, sometimes deliberately in testing, sometimes by mistake. UFS is not immune to file system full conditions. And ZFS can do exactly the same thing: If you want, you can configure quotas on ZFS so normal (non-root) users can only fill 92% of the file system.
And by the way, ZFS does reserve a very small part of the file system (I think by default 2%) for internal operations when the normal part of the file system is full. Since ZFS is internally built on appending, logging, and copy-on-write, it does need some reserved space to work itself out of knots.
Sure: you can work your way around this, but I still think it should be factored in when comparing the reliability of filesystems. I think every Unix sysadmin has encountered a fully filled up /var once in his lifetime (either on their own server(s) or by proxy: seeing it happen somewhere else).
I would think that any file system that has a test suite goes through testing of file system full. I've never seen a file system that handles this really gracefully; all the ones I've experienced get awfully slow. But on the other hand, I haven't seen any that corrupt data or become outright unusable when they are completely full. However, you have to be really careful to do the testing correctly: most operating systems don't survive if the root file system is full, since many of their vital processes don't handle write errors correctly. So if you want to completely fill a file system, better make it not root.
When you create a UFS filesystem it will automatically create plenty of superblock copies which you can use to recover your filesystem should something go wrong somewhere. ZFS doesn't really support any of this because most of these actions happen on the fly. Automatically. Which can work both in favor as well as against you.
If the boot sector of a ZFS partition gets damaged then you're at a severe risk, because that could easily corrupt the whole pool. And a corrupt ZFS pool means losing access to all of the underlying filesystems. It also doesn't help that ZFS will happily propagate such a destructive operation onto all underlying vdevs (mirror / raidz), leaving you with a fully trashed system. It doesn't have the intelligence to recognize unwanted ("damaging") changes which prohibits it from propagating this data onto other vdevs.
I vaguely remember seeing that ZFS writes multiple copies of the most important data structures, but honestly I don't know how many and where. The idea that all of a ZFS file system tree depends on a single copy of a fundamental descriptor is too insane for any modern file system implementor to get away with.
Which is another issue... dumpfs, tunefs and fsck_ffs are all programs which you can use to safeguard your filesystem. Sure: you have to know what you're doing, but they allow for a lot of control and flexibility.
I know that a ZFS debugger exists, which allows inspecting data structures (don't know whether it allows modifying them). That's more or less equivalent to dumpfs and fsck when run in interactive mode. ZFS does not have a traditional fsck, and that's for very logical reasons. There are other high-quality file systems that don't have fsck, because (a) they don't need them, since unexpected shutdowns don't leave things dangling, and (b) handling of internal corruption is better done in a debugger than in a semi-automatic program.
That 8% system reservation on UFS I spoke of earlier? I can easily change that if I want to.
You can change the equivalent quota setting on ZFS too. Which has nothing to do with reliability (or security).
Your whole discussion of readonly access and block-level debugging is pretty unrealistic. In modern complex file systems, direct modification of file system data structures is very complex, and really can't be done by hand-patching on disk by end users. Yes, it can be done (with great care and difficulty) by experts, and I've even had to do it a customer installations (in my case, paying customers), but this is terrain that should only be trodden by people who have the source code of the file system in another window, and know what they are doing. To begin with: having checksums on everything on disk means that repairs require finding where the checksums are stored, and also updating them (and then the checksums that protect those places, and so on).
Which brings me back to the topic of checksums. As I said above: disks are very big today, and still have a non-zero rate of undetected and uncorrected errors (those are typically spec'ed at 10^-15 per bit by disk manufacturers, but there are other sources of errors than just disks). That means that the IO stack will occasionally return false data. This is enough of a problem that in my opinion, any production file system that stores data that is not itself protected needs to implement checksums, both on data and metadata. Otherwise, you will get wrong file content back too often (roughly once every 100TB that are read). Today, there are a few open-source (free) file systems that implement full checksum protection, and they are Btrfs and ZFS (perhaps HammerFS also does, I'm not sure, but to my knowledge it is not production-ready, being part of an experimental OS). Btrfs is not in a good situation as far as bugs and support is concerned, so much so that RedHat has officially given up on it. Friends of mine in the Linux storage developer community have called Btrfs "a machine to create data loss", it is so buggy. That leaves only ZFS in the free arena. The other important argument is this: In my opinion, file systems have to be tightly integrated with the RAID stack. For example, when ZFS (or any FS) finds a checksum error on one mirror copy, it needs to read the other mirror copy (assuming RAID-1), and that's something that's hard or impossible to do if the file system is not knowledgeable about the internals of the redundancy layer. Similarly, scrubbing a file system needs to be done at the level of individual disks, not at the level of virtual RAID'ed disks, otherwise some copies on disks will go unchecked. And finally (perhaps most importantly from a reliability point of view), when a disk failure occurs, the rebuild a.k.a. resilvering of the redundant copies needs to be done using occupancy information from the file system to finish faster, because that lower MTTR directly improves the data loss rate of the overall system. ZFS can do all that, and to my knowledge no other free or open source system can (there are non-free commercial systems that do, but this forum is not the place to discuss those).
I'm not meaning to say that ZFS is perfect. I'm sure it has bugs. I may have experienced some of those (although I've been lucky enough to never have driven it to extremes). It's user interfaces for management operations are different from traditional Unix-style commands, and that's for a good reason, since it deals in different entities than the traditional block device - mount paradigm. It's performance on my home machine has at times made me wonder how it is implemented to be so slow, but with my limited workload it has not been a problem. I know that there are some high-performance systems that use ZFS (Lustre comes to mind), so perhaps the problem is with my hardware or the way I have set it up.