ECC or non-ECC

ECC or non-ECC


  • Total voters
    56
I think at that time all RAM modules (SIMM) had ECC.
Not all of them. Some 30 pin SIMMs had parity, which is not the same as ECC. ECC on SIMMs was only on some of the 72 pin SIMMs, at a much later date.
 
is kind of funny, b/c ZFS the "program" really doesn't care what kind of memory is in the machine .. ECC ram adds an "extra/optional" layer of corruption protection between when the processor writes information to ram, and when that information is dumped from ram.. the only real (non-technical) difference is that you get 1 extra check with ECC ram before the data is passed off to ZFS to be written..

does it help? sure it's nice to have. ... is it required nope.. ZFS is self contained and doesn't care what the os or ram reports .. It does all of that internally. is it worth 30 messages of debit? lol nope ... I would guess most people that use ECC ram only do so because they are using a Xeon.. I forget what episode of BSD Now , but AJ and BH discussed it in depth ... you could check the show notes it you really want to deep dive into it..
 
Well, there is some logic in it: ZFS keeps a huge amount of filesystem data in memory for long times, and AFAIK it is not checksummed while being there. So if a bitflip occurs, it may hit that data, and when it is later written to disk, the flaw will not be detected and will write all mirrored copies the same. Then, if it happens that this bitflip indeed concerns the toplevel master node (or whatever the thing is called) of a pool, then data might be written out damaged in a way that the pool structure becomes unintellegible. Then it would be necessary to analyze that pool and fix it manually (there are companies offering such service, if you won't try by yourself).
This is different to other filesystems. There fsck could repair such errors. And the bitflip would rather hit cached application data, so then the application data might be silently wrong, which might be just as bad.

What about the probability? Assuming that a cosmic ray triggering a bitflip happens about once a year (at sea level), it would hit the master node (lets assume 100kB, out of 16GB) every 100'000 years.

Nevertheless, I do use ECC where I can, and I recommend it for 24/7 operation.
 
Well, there is some logic in it: ZFS keeps a huge amount of filesystem data in memory for long times,
So do other file systems. Normally, file systems fill all memory with disk cache. ZFS's caching is not particularly aggressive, nor does it keep unwritten data (write cache) in menory particularly long.

and AFAIK it is not checksummed while being there.
I don't know. I should read the slides from the talk that Kirk McKusick gave a few days ago about how ZFS is implemented on FreeBSD. I think that all data and metadata in the memory cache is checksumed.

Then, if it happens that this bitflip indeed concerns the toplevel master node (or whatever the thing is called) of a pool, then data might be written out damaged in a way that the pool structure becomes unintellegible.
Yes. But one of the design principles of ZFS is that it never overwrites anything on disk. So if a metadata block is written damaged, the old copy of it is still on disk too, which makes the problem restorable and fixable for a long time (until the old data needs to be overwritten due to space concerns). I don't know how hard it is to fix such a problem though.

This is different to other filesystems. There fsck could repair such errors.
Fsck can repair some errors, but not all. A completely shredded metadata block (like inode or directory content) if often not repairable.
And the bitflip would rather hit cached application data, so then the application data might be silently wrong, which might be just as bad.
Traditional file systems also have metadata (superblocks, inodes, directories, indirect blocks), and they also cache those in memory.

Nevertheless, I do use ECC where I can, and I recommend it for 24/7 operation.
I agree, using ECC is good, but not any all cost. You are still better off with ZFS on non-ECC than with a other file systems on non-ECC.
 
Anything the FreeNAS forum has to say on ECC is complete bullshit. (It's the birthplace of non-ECC-mem-will-break-your-entire-ZFS-pool FUD.)
As others have commented here that's entirely subjective if you have an option of to ECC or not ECC. However, I'd rather pay the extra for ECC memory if it's within my budget for my servers running ZFS than not for an additional level of data integrity. Plus some servers only support ECC memory such as my Lenovo TS430. If ECC wasn't that big of a deal then why do many servers require it at a hardware level?

ZFS only cares about available memory. ECC memory requirement is a system board issue. That's a moot point. The issue is the probability if memory corruption before ZFS writes if your system has the option of to ECC or not ECC. I'm not one to gamble with my data.
 
I don't care what you personally use. Just please stop scaring hobbyists into choosing ext4-based Linux NAS solutions. This is completely counterproductive.
 
I don't care what you personally use. Just please stop scaring hobbyists into choosing ext4-based Linux NAS solutions. This is completely counterproductive.
This is a FreeBSD forum; not Lummux. Since you're here on your high penguin, you should be ban hammered.
 
Just please stop scaring hobbyists into choosing ext4-based Linux NAS solutions. This is completely counterproductive.
I don't even know WTF you're talking about because I don't use Lummux. Speak BSD or fasthalt.
Oh my… You are welcome to report me.
Already done. I also commented in the feedback forum because I've seen people who don't follow the rules posting here that should be gone. If they need an enforcer for it, I'm volunteering.
 
To me it seems simple if you have an option to go with ECC choose ECC, ECC is always better.
If you have to make a tradeoff between ECC and an extra disk or better disk, the extra disk MIGHT be a better investment. It depends on how much disk redundancy you already have. But I agree, the extra cost of ECC DIMMs (if you already have an ECC-capable motherboard) is not very high, so in general it is a good idea.

If you are running ZFS, the best investment is to buy a sufficiently large number of disks to use a sufficient RAID code (the more fault tolerant, like Z2 or Z3, the better). The second best investment is to buy high-quality disks (enterprise grade). The third best investment is to buy or rent hardware for remote backups, to protect against destruction of the primary server (either by hardware or operator error). Perhaps the priority of "second" and "third" could be switched. And ECC is next.
 
You'll ban 3/4 of the forum members :D It's not a coincidence that the moderators are the most chill people here.
Someone called? Splendid!
I do believe that someone has been misreading things. Someone does not know how to handle things. Someone only doubles down. All this does not need to be the same someone.
Now you all pop a cold one, put up your feet and get mellow. Or else...

(Yes, the mods here are chill. Like a sleeping bear in his cave. You don't want to poke it too hard.)
 
If you have to make a tradeoff between ECC and an extra disk or better disk, the extra disk MIGHT be a better investment. It depends on how much disk redundancy you already have. But I agree, the extra cost of ECC DIMMs (if you already have an ECC-capable motherboard) is not very high, so in general it is a good idea.

If you are running ZFS, the best investment is to buy a sufficiently large number of disks to use a sufficient RAID code (the more fault tolerant, like Z2 or Z3, the better). The second best investment is to buy high-quality disks (enterprise grade). The third best investment is to buy or rent hardware for remote backups, to protect against destruction of the primary server (either by hardware or operator error). Perhaps the priority of "second" and "third" could be switched. And ECC is next.

Maybe i'm to simple minded but it's not because you get better memory that you should spend less on disks. Also you can gradually add more disks. Disks get cheaper a lot faster then ram, so buying over a bigger time frame will allow you to invest more and get better disks over time.

What you need to think about is good motherboard that allows you to expand, good cpu, good memory, good disk controller/backplane disks come later.
 
spec wise here is a couple of examples..

if your set up for racks, aka dont mind noise .. this is a killer bang for the buckl for a 12bay (works 100% with freebsd ) actually have one keeping my feet warm.. (if they havent changed it .. mine also came with a 4port 10gb intel card, not sure if that was a mistake tho)

or
if you want to build your own.. its more money but this isnt to bad granted linus is not the best resource for zfs/storage systems.. the hardware selection is ok.
View: https://www.youtube.com/watch?v=FAy9N1vX76o
 
I'd think this just comes down to priority. If your data is important, you want error correction and redundancy (immediately available or not). Some tasks have a higher importance on uptime than they do on data integrity.
If data is important, RAID does not come before backups. RAID is good at maintaining uptime and joining multiple disks into one storage area for the additional space too. RAID does not defend against accidental deletions/overwrites, viruses, software/OS/hardware issues; backups may if an originally good copy makes it out to the backup without the corruption/damage/mistake. Data integity then has to hold up in those backups until later needed. Checksums, or better yet digital signatures, are useful to make sure while multiple backups help in case one does have 'any' problem. Many communication busses have their own integrity checks (AHCI, TCP/IP, etc.) that the data will be passed through but only watch it during transit.
Bad RAM 'could' lead to corruption of ZFS structures or the data within the files as the garbage may go in checksummed. It could also impact checksumming itself. As a bigger picture, it may also corrupt the data a program was going to pass to disk well before ZFS ever saw it and ZFS cannot protect against 'garbage in'. The likely hood of each would be best guessed as quantity of RAM used for each of those tasks divided by the chance a quantity of RAM will be impacted with corruption; less RAM used makes it less likely the problem impacts your workload in any way.
Without ECC, filesystem checksums, etc. you won't know you have a problem unless it causes a program to crash/misbehave from corruption, causes a checksum failure to detect it, or causes structural/visible corruption in the data when you look at it later. A jpeg or mp3 file when reviewed later may be more obvious but a wav or bitmap may easily have corruption that is a small enough change after the value(s) drift that it goes unnoticed even after review. ECC RAM + communication buss checksums + ZFS still doesn't cover the rest of the memory in the computer such as the cache in your CPU. You are also left to wonder if your disk controller has nonbus data integrity checks too. In the end, none of that gets you past a buggy program though.
If so important, you can always perform a task more than once and check the outcome is what is expected by comparing those outcomes. Doing that across multiple pieces of hardware helps answer the question of if there was a hardware issue; as long as each piece doesn't have the same 'bug' engineered into them. The project seti@home takes results of 3 users performing a computation and if all don't match has the option to get more calculation attempts to be performed too.
I've had bad RAM in my desktop cause different kinds of corruption including causing a ZFS filesystem to have issues; the RAM was VERY unstable and months of play were still necessary to reach a point where ZFS filesystem issues could be detected despite random bits of data erroring out multiple times over two hours according to memory testers. Most times it was shown in program crashes or OS crashes. One hour of 'uptime' would require multiple tries to reach. I've also had disks overheat and cause small bits of corruption occasionally on them which the disk internally identified as unreadable sectors (rewrites caused no sector reallocations as the sectors would be seen as working on write). I had another disk in the past that as it failed would lead to corrupt data being passed with no warning but as a scratchpad for a video project I was actively working on it was easy to see data glitches that did occur.
I personally do not have ECC in my desktop as much as I'd like to; higher price hardware for lower performance on a device that doesn't make me money hasn't seemed wise at the time of shopping. I have been using ZFS for years. Its nice to think problems don't happen silently for no added $ cost but I have taken a performance hit to use ZFS; my FreeBSD box handles low memory VERY poorly with slowing down like swap is in use to use the filesystem even when swap is disabled and other nondisk slowdowns with long pauses and system freezes when I would expect the system to say, 'RAM too low; closing a program' but it just wont. ZFS + scattered data (fragmentation?) has shown large performance issues too. ZFS by design currently has no way to deal with it poorly arranging data by reoptimizing it to 'not suck' on its own and since they don't want to rewrite data to fix it and I don't want to backup/restore disks regularly to work past that bug, I usually copy a folder with data I expect to be impacted and delete the old+rename the new to the old to get a LARGE performance bump back. I have had to go back to a backup due to corruption causing a reboot during scrub and a running scrub restarts on reboot. No reliable way to fix or live with ZFS data issues (I consider the previous issue I had to go beyond, 'checksum failed; replacing data' type of a fix) means backups should be in order or my data will be unusable someday.
So is ECC a good idea...Yes. Mandatory...Not for my use. Will it eliminate all data corruption problems when combined with ZFS...No, but more could be prevented (and much more likely is that other runtime issues can also be prevented). Backups have saved my data from damage that ZFS, ECC, RAID with mirroring/parity, etc. would have just lost. Program bugs also have caused me far more grief than RAM or hard drives short of them reaching a state considered 'failed' (including intermittent issues, I consider that a fail if non-rare).
 
This is indeed incredible - it shows the mindset of "desktop" users (and why it is so difficult to deal with them).
(Thanks shkhln for pointing at this.)

I would expect the system to say, 'RAM too low; closing a program' but it just wont.

There are shell-scripts. Shell scripts can be used as a programming language, with branches, loops and decisions, where each command is actually a program to be run. When we start to unexpectedly "close" arbitrary programs, it is the same as ripping instructions out of the script, arbitrarily, at runtime.
You can have the same effect by installing defective memory or a broken harddisk. Or by shooting yourself in the foot for fun.

If the system ever gets into an OOM situation, you must reboot anyway, because all the scripted tools are no longer in a defined state afterwards. That is why one uses ample swapspace, much more than might ever be needed: to avoid the reboot.

For the OS there is no difference between a "program" and a process. Anything that runs is a process.
And the system has no means to "close a program". The only thing it can do is kill a process, violently und unexpected. And that is the worst thing that can happen. It is better to create a system panic and reboot and do the things from start again.

It is similar with cars. If you buy a car, you expect to just sit into and drive away. But if you happen to drive something that is good for professional work, say a truck or a caterpillar, then you are expected to do some adjustments to the workload before running it. Same here: if you want to run a professional OS, that can do real work, you are expected to adjust it to the workload (or pay a maintenance shop to do that for you, like you do with your car).
 
For what it's worth, I'm perfectly fine with Firefox being killed on OOM. It's practically impossible to estimate RAM requirements for that thing anyway and the alternative is a complete lockup.

And the system has no means to "close a program". The only thing it can do is kill a process, violently und unexpected. And that is the worst thing that can happen. It is better to create a system panic and reboot and do the things from start again.

Meh. This is no different from a crash on segfault. Programs definitely should be written to handle restarts.
 
For what it's worth, I'm perfectly fine with Firefox being killed on OOM. It's practically impossible to estimate RAM requirements for that thing anyway and the alternative is a complete lockup.
Meh. This is no different from a crash on segfault. Programs definitely should be written to handle restarts.

That's why I think this is basically a server-desktop conflict. Firefox is only run on desktops. OTOH, I just cannot write shellscripts so that they could handle the crash of any called function. I have, for instance, some shellscripts that handle the backup of database redo-logs (for point-in-time recovery). These are different from normal backup as one must keep them in sequence and never loose a single one of them. So I do this twice to two different backup-volumes, and there is a lot of interlocking, as the machine may be busy and the backups may delay for an undefinite time, and still another script must precisely know which logs are successfully backed away and can be deleted. That fabric will re-initialize on a complete crash, but I am not sure what it does when an arbitrary component gives a wrong status.
For the same reason, if a single worker from the postgres fails, it will propagate a full database crash. So probably, if you have to guarantee transactional integrity, there is no other way.

And no, I do not have segfaults on a server. I do see segfaults on the desktop, e.g. from cups (and that gives me a clear opinion about that ultra-crap), but on a 24/7 server any segfault should be analyzed for it's root-cause.

And, as I just notice: even my firefox seems to behave properly, these days. Can't remember it having failed for many weeks. Lets have a look:
Code:
$ /usr/local/sbin/pkg info | grep firefox
firefox-esr-68.4.2,1
$ ps axlw | grep firefox
1100  1245  1204   0  21  0 3531752 832108 select   Ss    -    407:23.97 firefox
1100  1247  1245   0  20  0 3273584 699068 select   S     -    462:20.97 /usr/local/lib/firefox/firefox -contentproc -childID 1 -is
1100  1248  1245   0  82  0 3457580 837980 -        R     -   1030:30.98 /usr/local/lib/firefox/firefox -contentproc -childID 2 -is
1100  1249  1245   0  20  0 2462732  66572 select   S     -      1:50.30 /usr/local/lib/firefox/firefox -contentproc -childID 3 -is
$ top -b
last pid: 38885;  load averages:  0.99,  1.11,  1.07  up 6+19:28:48    20:48:01
116 processes:  2 running, 114 sleeping

Mem: 788M Active, 1284M Inact, 381M Laundry, 5066M Wired, 775M Buf, 326M Free
ARC: 2389M Total, 1727M MFU, 262M MRU, 1666K Anon, 63M Header, 336M Other
     1568M Compressed, 3436M Uncompressed, 2.19:1 Ratio
Swap: 10G Total, 1733M Used, 8507M Free, 16% Inuse

Doesn't look bad. And *surprize* not even any zfs tuning is currently in place. Only my usual recommendation: reduce the number of content worker processes (within firefox) to something that makes sense, e.g. one per 4G ram. By default it is the number of CPUs, and that is usually too much.
 
For me, it is ECC where ever possible. Even my new laptop will come with ECC memory, I would never consider building/buying/using a server without it. Also a good argument for Ryzen CPUs, which mostly support ECC (I believe only Ryzen 3 does not).
Anyone who knows about subnotebooks below 1.5 kg supporting ECC, please drop me a note.
EDIT Just do not use that crap firefox. It's evil. Period ;)
 
Back
Top