This is the main backups server for all non-school sites, including web servers, groupware servers, database servers, SMTP servers, file/print servers, etc. There's approx 50-odd servers here, with only about 20 being direct-installed on physical hardware. The rest are VMs (using LVM for disk storage, not filesystem image files).
Code:
[fcash@alphadrive ~]$ zdb -DD storage
DDT-sha256-zap-duplicate: 39820749 entries, size 1166 on disk, 188 in core
DDT-sha256-zap-unique: 84371551 entries, size 1194 on disk, 193 in core
DDT histogram (aggregated over all DDTs):
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 80.5M 8.18T 5.28T 5.61T 80.5M 8.18T 5.28T 5.61T
2 22.8M 2.52T 1.80T 1.88T 50.0M 5.51T 3.95T 4.12T
4 9.10M 836G 604G 639G 46.1M 4.10T 2.97T 3.14T
8 2.52M 230G 144G 156G 27.7M 2.47T 1.52T 1.65T
16 1.67M 180G 91.9G 101G 37.6M 4.00T 2.03T 2.22T
32 981K 109G 66.2G 70.5G 41.7M 4.63T 2.84T 3.02T
64 329K 37.0G 24.4G 25.7G 28.6M 3.22T 2.07T 2.19T
128 468K 57.3G 35.3G 37.0G 85.1M 10.4T 6.38T 6.70T
256 158K 18.2G 10.6G 11.2G 43.4M 4.92T 2.92T 3.10T
512 17.3K 1.46G 884M 973M 11.0M 873G 518G 576G
1K 2.38K 165M 66.1M 82.4M 3.13M 208G 84.7G 106G
2K 737 22.9M 11.3M 16.1M 2.17M 65.5G 31.4G 45.8G
4K 431 16.4M 7.78M 10.8M 2.10M 79.1G 39.1G 53.9G
8K 310 1.55M 700K 2.93M 3.66M 19.2G 9.42G 36.4G
16K 28 261K 31K 240K 570K 5.71G 612M 4.76G
32K 7 260K 11.5K 71.9K 278K 8.90G 419M 2.72G
64K 4 257K 12.5K 48.0K 329K 21.7G 1.09G 3.91G
128K 3 1.50K 1.50K 24.0K 510K 255M 255M 3.98G
Total 118M 12.1T 8.04T 8.50T 464M 48.7T 30.6T 32.6T
dedup = 3.83, compress = 1.59, copies = 1.06, dedup * compress / copies = 5.73
Code:
[fcash@alphadrive ~]$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
storage 34.9T 4.78T 288K none
IOW, ~150 TB of data stored in just 35 TB of disk.
And this is the main backups server for schools. There's 1 Linux server per school that is the "jack-of-all-trades" server handling proxy, cache, web, mail, packet filtering (in the elems only), file/print, and diskless booting duties. Secondary schools also have separate webmail and firewall boxes.
There's approx 100-odd physical servers being backed up every night, in under 5 hours. Gotta love rsync.
Code:
[fcash@betadrive ~]$ zdb -DD storage
DDT-sha256-zap-duplicate: 18470415 entries, size 784 on disk, 177 in core
DDT-sha256-zap-unique: 60064456 entries, size 751 on disk, 169 in core
DDT histogram (aggregated over all DDTs):
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 57.3M 5.11T 3.54T 3.67T 57.3M 5.11T 3.54T 3.67T
2 11.2M 1.14T 935G 955G 24.8M 2.53T 2.04T 2.08T
4 3.72M 403G 261G 270G 18.1M 1.89T 1.24T 1.28T
8 1.11M 90.9G 54.8G 58.5G 11.6M 960G 583G 622G
16 447K 42.7G 20.2G 21.6G 8.92M 853G 405G 434G
32 809K 27.3G 18.2G 22.3G 38.5M 1.33T 922G 1.09T
64 293K 23.4G 11.3G 12.7G 26.7M 2.14T 1.03T 1.15T
128 50.8K 1.80G 782M 1.04G 8.06M 266G 119G 164G
256 10.4K 229M 132M 192M 3.55M 84.4G 49.0G 69.2G
512 5.05K 196M 126M 152M 3.48M 137G 90.1G 108G
1K 1.40K 42.6M 24.4M 32.2M 1.87M 59.3G 33.8G 44.2G
2K 1.03K 32.6M 13.1M 19.0M 2.77M 85.2G 34.6G 50.4G
4K 407 14.8M 6.50M 8.85M 2.26M 77.7G 33.6G 47.1G
8K 183 4.83M 1.68M 2.74M 2.08M 54.4G 18.2G 30.5G
16K 143 2.63M 930K 1.77M 3.27M 56.6G 19.3G 39.6G
32K 263 1.76M 716K 2.36M 9.15M 77.7G 33.0G 92.1G
64K 11 145K 12K 85.2K 867K 9.90G 898M 6.50G
128K 8 4.50K 4K 56.8K 1.41M 799M 723M 10.0G
256K 6 11.5K 6.50K 42.6K 2.01M 3.18G 1.90G 14.3G
Total 74.9M 6.82T 4.81T 4.98T 227M 15.7T 10.1T 11.0T
dedup = 2.20, compress = 1.55, copies = 1.08, dedup * compress / copies = 3.15
Code:
[fcash@betadrive ~]$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
storage 11.6T 10.2T 256K none
So, approximately 35 TB of data stored in 11 TB of disk.
The off-site backups server currently has a dead OS drive, so I can't connect to it to grab stats for that one. But it receives ZFS sends from the above servers across a gigabit link, and iftop shows it hitting peaks of 915 Mbps, with the 40s average hovering above 600 Mbps.
All of the above have LZJB compression and dedupe enabled.
Forgot to mention that all three pools are configured the same:
- 4 raidz2 vdevs with 6 drives each
- 32 GB cache vdev
- 2 GB log vdev (although this is currently only for testing)
- 16 GB UFS for OS
- 8 GB of swap (the last 4 items are all on the same SSD)
- 20, 24, 32 GB of RAM (alpha/beta/omegadrive respectively)