ZFS Complete tool for automated backups to Amazon AWS, Google Cloud, or Microsoft Azure

ralphbsz said:
While in a mathematical sense you are correct, in a practical sense you are not. Other storage service providers have outage statistics that are MUCH MUCH better than Cloudflare. But even if that doesn't worry users, the real worry is that it indicates that their engineering has a cavalier attitude towards reliability.
Click to expand...
As You may see, in this (and other threads on this forum) i vote for CF as for DNS/proxy service, not as for reliable storage service.
From my point of view for small-middle companies are only one type of storage: own storage system with 2+ controllers connected by FC cables and FC switches 2+ 10-15k rom HDD shelfs. From GOOD manufacturer like IBM, EMC, Toshiba,... hardware and ZFS filesystem.

ralphbsz said:
Yes, but Cloudflare's track record of leaking information demonstrates, yet again, that they are careless and/or incompetent. Significantly worse than other providers.

But the real problem with Cloudflare is not that they are bad at what they should be doing (providing a reliable service, without data leakage). The real problem is that Cloudflare explicitly and deliberately serves customers that it knows are doing things that are either outright illegal (such as killing people, they were the provider to ISIS/ISIL), or ethically very bad but not yet illegal (such as providing the backend for 8chan, which various hate groups connected to mass shootings have used as a platform to organize themselves). While nobody has been able to prove that Cloudflare is itself a criminal enterprise, it willingly provides service to criminals and terrorists.

Interestingly, it has that in common with the cryptocurrency industry: While it might have been originally well-intended, today it is mostly a tool of scammers and organized crime; plus fools who think they can make a fortune from it (and usually end up poorer).
Click to expand...
In common sense You are right. But in real world MUCH MORE DANGEROUS RIGHT NOW are the situation when russia using chemicals weapon in Ukraine, targeting the nuclear power plants (and occupy one - in Zaporizzya), blocking a Black Sea (which lead to mass dies in poor countries in East and Africa), and make a public messages that “ready to shoot US by strategic nuclear weapon”...
And officially russia make friendship with Taliban and ISIS/ISL.
And all of this right now, from 24 Feb of 2022.

Or may be You forgot russian attacks on fuel pipes structures of US recent year?

In which world real or digital a You live now???

P.S. I purpose return to topic. :)
 
Sergei_Shablovsky said:
From my point of view for small-middle companies are only one type of storage: own storage system with 2+ controllers connected by FC cables and FC switches 2+ 10-15k rom HDD shelfs. From GOOD manufacturer like IBM, EMC, Toshiba,... hardware and ZFS filesystem.
Click to expand...
That can indeed be a good solution. Whether it is good or bad depends mostly on the quality of the personnel that implements it.
But today I think it is no longer a cost-effective solution. Add up all the cost of the system (two servers, disk enclosures, SAS or FC hardware, software licenses and/or support contracts, system administration personnel costs), and it is likely more expensive than outsourcing your storage to a cloud provider. But this tradeoff needs to be evaluated specifically for each situation. Part of that tradeoff is what vendor to use.

But in real world MUCH MORE DANGEROUS RIGHT NOW are the situation when russia using chemicals weapon in Ukraine, targeting the nuclear power plants (and occupy one - in Zaporizzya), blocking a Black Sea (which lead to mass dies in poor countries in East and Africa), and make a public messages that “ready to shoot US by strategic nuclear weapon”...
And officially russia make friendship with Taliban and ISIS/ISL.
Click to expand...

Absolutely true. And I'm following the news in detail, and some of my friends are rather directly involved in the conflict. But that doesn't change the fact that there are good cloud storage providers, and bad ones, with Cloudflare pretty solidly on the bad side. For a variety of reasons, some technical, some ethical.
 
ralphbsz said:
That can indeed be a good solution. Whether it is good or bad depends mostly on the quality of the personnel that implements it.
Click to expand...
Totally agree with the fact that there are extremely less numbers of quality personnel: most of youth nowadays not pay much attention to hardware & os, protocols basics and much more addicted to “work just right out of the box” solutions even TCO for that solutions are higher for company in which they working on.
ralphbsz said:
But today I think it is no longer a cost-effective solution. Add up all the cost of the system (two servers, disk enclosures, SAS or FC hardware, software licenses and/or support contracts, system administration personnel costs), and it is likely more expensive than outsourcing your storage to a cloud provider. But this tradeoff needs to be evaluated specifically for each situation. Part of that tradeoff is what vendor to use.
Click to expand...
Hm... Lets calc:
EMC/NetApp 24+ 3,5” HDD shelf (best HGST 3Tb legendary SAS drives cost $130-180/each new) cost on aftermarket ~ $200-250
2 x Xeon 5670 + 64Gb DDR3 chipkill memory controller IBM-based server cost on aftermarket ~ $200-250
great Qualcomm 6Gb/s FC controller cost on aftermarket ~ $150-200
7-8,000 kVA Emerson/ Liebert rack online interactive UPS cost ~ $500-600 (with a new batteries)
smart SDU/PDU (Athen/Servertech) + environment control (like APC EMS) cost on aftermarket ~$300-350 total
Floor stand cooling system for room 25m2 (1 main device + 1 for failure hotswap) - new cost ~$800 / both
FC cables + Eth cables ~ $50-100

FreeNAS - free license
FreeBSD + a lot of pkg - free
ok, setup fee would be $30-60/h.

Utilities bill - according Your country rates.

So on that price a You have a robust system, based on hi-quality components, expandable, with VERY low maintenance cost.
And price for all of this are equal 2-3-4 month of service of great cloud provider. (And EACH additional step like adding HDD, reconfigurations cost a You more and more...)

Where am I wrong in calculating?
ralphbsz said:
Absolutely true. And I'm following the news in detail, and some of my friends are rather directly involved in the conflict. But that doesn't change the fact that there are good cloud storage providers, and bad ones, with Cloudflare pretty solidly on the bad side. For a variety of reasons, some technical, some ethical.
Click to expand...
As a You may see, cyber-terror from russia, North China and Iran last months are only increasing...
 
hardworkingnewbie said:
Guess what's going on again right now? Another massive Cloudflare outage!
https://twitter.com/i/web/status/1539151292685332481
View: https://twitter.com/patelkjoel/status/1539151292685332481

techcrunch.com

Cloudflare fixes outage that knocked popular services offline

Cloudflare said on Tuesday it resolved a "wide-spread" outage earlier in the day that affected a large number of services including FTX, Discord, DoorDash and NordVPN.
techcrunch.com techcrunch.com
Click to expand...
Here in Ukraine, on ISP level we see some small delays, but not so much.

Mostly we using CF as DNS service (reverse proxy), and not see huge outage/delays.

May be for cryptocurrency geeks this outage issue was more valuable... ;) Am I wrong ?
 
hardworkingnewbie said:
Regarding your tool selection problem: you just need to use ZFS snapshots and integrate this with restic if you are not using a ZFS enabled backup provider.

Like this:

1. create throwaway ZFS snapshot for backup
2. use restic to backup that ZFS snapshot offsite
3. delete your ZFS snapshot
Click to expand...
Seems to be a nice workaroud. Did you test actually it?
I actually want to backup snapshots of a restic repo.
 
hardworkingnewbie said:
Well since you've asked, I'll add a few more points to that. Cloudflare started back then as honeypot project, which later added DDoS protection to their portfolio. Their "free" DDoS protection is now "protecting" many web sites. So they also ventured into the field of DNS servers, and are one of the main propagators of that abomination called DNS over HTTPS.

They are operating the free DNS server 1.1.1.1 as well as the DoH DNS server, to which Mozilla Firefox out of the box connects to, amongst many other things.

Just a few highlights out of their "career": in 2014 they opened up a challenge website when Heartbleed was all the rage, claming people could abuse Heartbleed but not retrieve their SSL certificates. Of course somebody was succesful.

The no other company on the world is causing more issues for the Tor network than Cloudflare.

Tavis Ormandy (Google Project Zero) found in 2017 the Cloudbleed bug: reverse proxies were dumping uninitialized memory.

Of course 1.1.1.1 is there to grab all our DNS query data, just like 8.8.8.8 is there for Google.

DNS over HTTPS became the standard in Mozilla spring 2020, and of course uses Cloudflare. Bert Hubert from PowerDNS about that move. https://blog.powerdns.com/2018/09/04/on-firefox-moving-dns-to-a-third-party/

They had also big DNS outages, like in 2019: https://ianix.com/pub/dnssec-outages/20190321-www.cloudflare.com/, often talking half of the internet with them.

Complete breakdown in 2019, which affected lots of web sites: https://metro.co.uk/2019/07/02/cloudflare-outage-means-websites-including-detector-10103471/

And they want you to believe that public keys are not enough for SSH security, so you should integrate them in your security architecture: https://blog.cloudflare.com/public-keys-are-not-enough-for-ssh-security/ - what could possibly go wrong?

In 2020 they've created cloud based web browsers and wanted to offer this service to people. https://www.techradar.com/in/news/cloudflare-wants-to-run-your-web-browser-in-the-cloud

And they are unable to handle DNS root zones correctly. https://lists.dns-oarc.net/pipermail/dns-operations/2020-January/019684.html

Cloudflare was rate limiting npm - by mistake. https://github.com/npm/cli/issues/836#issuecomment-587019096

And of course if people are too lazy to create SSL certificates, instead let Cloudflare handle that - OMG.

Cloudflare considered harmful. And there's oh so much more about it...
Click to expand...
Dear HardworkingNewbie!

Recently I re-read all links that You post in this tread, and dozen of related: to be sure that my arguments are still valid.

So, I need to add a little bit more for my previous answer: later or sooner, the "PROfessional internet inside ORDINARYinternet" become reality, whatever agree we with this or not.
Because a lot of factors: in next decade only really transglobal companies (or governments of rich western countries) be able to investment into infrastructure building (because of a decade of regional wars and economic recession due this), connectivity protocols stack dramatically and fastly changing, and network devices become more intellectual and powerful, post-industrial world need more people who consuming rather producing, social networks and metaverse become new reality for next generations of people, ...
In fact world would be divide on two categories: PRO - hi speed, guarantee, high availability, etc and ORDINARY - who lives not so critical bounded with net.

And CloudFlare, Google, Akamai, Amazon & Facebook just try to catch as much bigger piece of this "big cake of future money".

And of course all of them would make mistakes, take wrong decision,... Like government regulators in each country also would make unprofessional, wrong & just stupid decision...

So at the bottom line: of course CloudFlare are not Ideal, but much better in all senses than Google, cheaper (TCO) than Amazon, and great solution for those, who have no big budgets on Akamai (for example) solution for enterprise clients.

Am I wrong?
 
Sergei_Shablovsky said:
What is Your opinion about
backup/zapzend
https://github.com/oetiker/znapzend
Click to expand...

Sergei_Shablovsky said:
backup/sanoid
Click to expand...
Sergei_Shablovsky said:
backup/zfs_autobackup
github.com

GitHub - psy0rz/zfs_autobackup: ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable.

ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable. - psy0rz/zfs_autobackup
github.com github.com
Click to expand...


Please give me Your opinion about pro/cons about this two solutions.

Note:
1. We need strongly secured connection on a outside remote machine/cloud instance (in this case may be data compression also would be ok, because unstable routing with speed are vary, but on inside local bare metal machine data compression would not need at all,- the speed are more important. So, for outside secure+data compress, for inside less/no secure and no data compress);
2. We using ZFS only, on each machine.
2. The speed of whole backup procedure on a remote machine/cloud instance - are IMPORTANT;
3. less pressure on CPU/RAM/disc - are IMPORTANT.

P.S.
Let's say on a outside remote machine/cloud instance

Thank You all for detailed suggestions!
 
You won't go wrong with Sanoid/Syncoid. Take a look at its Github repo, particularly number of folks, contributors, users, issues and so on. These metrics can be helpful in deciding what solution to explore. Apply it to other interests too.
 
Sergei_Shablovsky said:
Please give me Your opinion about pro/cons about this two solutions.

Note:
1. We need strongly secured connection on a outside remote machine/cloud instance (in this case may be data compression also would be ok, because unstable routing with speed are vary, but on inside local bare metal machine data compression would not need at all,- the speed are more important. So, for outside secure+data compress, for inside less/no secure and no data compress);
2. We using ZFS only, on each machine.
2. The speed of whole backup procedure on a remote machine/cloud instance - are IMPORTANT;
3. less pressure on CPU/RAM/disc - are IMPORTANT.

P.S.
Let's say on a outside remote machine/cloud instance

Thank You all for detailed suggestions!
Click to expand...
Any comment?
 
You must log in or register to reply here.
Back
Top