Solved Fastest way to transfer files between servers.

A

Alexandr Kirilov

Hello all!

Need to transfer huge amount of files between servers. The average file size is about 15Mb - 1100Mb. This is video fragments from surveillance cameras. The income video traffic 2-4Gbit. Income traffic recording into fragments and after it fragments spreading to servers.
Have some questions:
  • What is the fastest way to do it? FTP? rsync/rcp/scp? Something else?
  • Is there something to be tuned for this amount of traffic?
 
How many video streams/cameras you have? This is huge amount of traffic. I'm using H265 encoding in the camera itself which is around 5Mbit/s @30fps for 4K camera w/o sound.
 
That's about 500 Mbyte/second. Given that an individual disk drive can not sustain reading or writing much above 100 Mbyte/second, this means a parallel IO disk array. Given that spinning disks give the most bandwidth per dollar, I find it unlikely that SSDs are being used for storage, but perhaps they are.

With a good server, those kinds of speeds are easily reachable; our group was doing 18 GByte/second using a pair of servers (active/active failover) about 10 years ago, but we used several hundred disk drives per server pair (and a heck of a lot of Infiniband and SAS hardware to get the data plumbed in and out).

From a server hardware viewpoint, 1/2 Gbyte/second doesn't stress the system much. The issue is things like file system, locking, how to handle deletion, how to get parallel and cache-coherent access to work efficiently.

At that kind of speed, I would look at the system architecture first; the question of what file transfer protocol to use (rsync versus FTP) is secondary. Matter-of-fact, I would move towards a cluster file system model, where data is never moved, once it has been stored, since moving data is inefficient.

Your question "is there something to be tuned" is somewhat humorous. Yes, to reach really high speeds, you will have a team of experts spend a year or two tuning the system, using labs with dozens of test systems.
 
ralphbsz said:
That's about 500 Mbyte/second. Given that an individual disk drive can not sustain reading or writing much above 100 Mbyte/second, this means a parallel IO disk array. Given that spinning disks give the most bandwidth per dollar, I find it unlikely that SSDs are being used for storage, but perhaps they are.

With a good server, those kinds of speeds are easily reachable; our group was doing 18 GByte/second using a pair of servers (active/active failover) about 10 years ago, but we used several hundred disk drives per server pair (and a heck of a lot of Infiniband and SAS hardware to get the data plumbed in and out).

From a server hardware viewpoint, 1/2 Gbyte/second doesn't stress the system much. The issue is things like file system, locking, how to handle deletion, how to get parallel and cache-coherent access to work efficiently.

At that kind of speed, I would look at the system architecture first; the question of what file transfer protocol to use (rsync versus FTP) is secondary. Matter-of-fact, I would move towards a cluster file system model, where data is never moved, once it has been stored, since moving data is inefficient.

Your question "is there something to be tuned" is somewhat humorous. Yes, to reach really high speeds, you will have a team of experts spend a year or two tuning the system, using labs with dozens of test systems.
Click to expand...
The question was about "Fastest way to transfer files between servers". It wasn't about how to store it on server within several hundred disk drivers.
 
1762844446539.png
 
I guess the data in the source server is already compressed, if it's something like mp4. If you can do any further compression before the transfer, that might be worth exploring. For example can you reduce the video resolution during capture/encoding, or do you need the full resolution? Anything you can do to cut down the volume of data to be transferred must help, provided you have a sufficiently meaty cpu (and possibly hardware compression coprocessor) at the video capture side. What is the absolute minimum video resolution you need? You can of course get video compression accelerator cards like this, although I don't know if freebsd supports them, you would need to explore what freebsd supports.

www.l-trondirect.com

DVP-7030HE | 4-Channel Full HD PCIe Video Compression Card with SDK by Advantech - L-TronDirect

DVP-7030HE is a PCIe-bus, software compression video capture card with 4 HDMI video and 4 audio inputs. DVP-7030HE supports H.264/MPEG4 compression format at up to Full HD 1080p for each channel. With an easy-to-use software development kit (SDK) and flexibility to stack multiple cards...
www.l-trondirect.com www.l-trondirect.com

Once you've captured and compressed it... using ssh for the transfer usually has some overhead because the data has to be encrypted/decrypted. Using rsync or ftp would avoid that, rsync is probably the preferred option nowadays. If the data is already highly compressed then ssh -C may not buy you much additional compression.

To increase the link bandwidth... investigate using the lagg(4) link aggregation driver to obtain a faster link between the source and destination machines. You can aggregate links to get a faster channel. It goes without saying that you want the fastest NICs and switch (if there is a switch and not a direct link) you can get. It all depends where the bottlenecks are, you need to find that by experiment.

And I guess optimising zfs and raid configuration on the target machine, but ralphbsz has already discussed that. That's probably the bottleneck once you've got the network bandwidth up and the data size minimised. There's some discussion here that might be useful

Best way to increase zfs read+write speeds

Hi, Previously i had a 1.5tb server made up of all SSDs (2x512gb,2x256gb) and it was pretty fast, i was only ever limited by the network speed. Now i have a 20tb server made up of 4x5tb drives (1 redundant), with a 512gb ssd as l2arc and one of the 256gb drives as boot drive, and 189gb DDR4 RAM...
forums.servethehome.com forums.servethehome.com

Perhaps it's worth experimenting with building a 2-node freebsd HAST cluster as the target, which should increase the speed at which you can dump data onto the target.
 
There is one other question, which is WHY you need to transfer this huge volume of data in the first place. Do you really need to build an archive of the video data? Or is it possible to process the data at source in realtime and only archive the finished data products, and/or only archive the video segments that are interesting. Just a thought. :)
 
blackbird9 said:
I guess the data in the source server is already compressed, if it's something like mp4. If you can do any further compression before the transfer, that might be worth exploring. For example can you reduce the video resolution during capture/encoding, or do you need the full resolution? Anything you can do to cut down the volume of data to be transferred must help, provided you have a sufficiently meaty cpu (and possibly hardware compression coprocessor) at the video capture side. What is the absolute minimum video resolution you need? You can of course get video compression accelerator cards like this, although I don't know if freebsd supports them.

www.l-trondirect.com

DVP-7030HE | 4-Channel Full HD PCIe Video Compression Card with SDK by Advantech - L-TronDirect

DVP-7030HE is a PCIe-bus, software compression video capture card with 4 HDMI video and 4 audio inputs. DVP-7030HE supports H.264/MPEG4 compression format at up to Full HD 1080p for each channel. With an easy-to-use software development kit (SDK) and flexibility to stack multiple cards...
www.l-trondirect.com www.l-trondirect.com

Once you've captured and compressed it... using ssh for the transfer usually has some overhead because the data has to be encrypted/decrypted. Using rsync or ftp would avoid that, rsync is probably the preferred option nowadays. If the data is already highly compressed then ssh -C may not buy you much additional compression.

To increase the link bandwidth... investigate using the lagg(4) link aggregation driver to obtain a faster link between the source and destination machines. You can aggregate links to get a faster channel. It goes without saying that you want the fastest NICs and switch (if there is a switch and not a direct link) you can get. It all depends where the bottlenecks are, you need to find that by experiment.

And I guess optimising zfs and raid configuration on the target machine, but ralphbsz has already discussed that. That's probably the bottleneck once you've got the network bandwidth up and the data size minimised. There's some discussion here that might be useful

Best way to increase zfs read+write speeds

Hi, Previously i had a 1.5tb server made up of all SSDs (2x512gb,2x256gb) and it was pretty fast, i was only ever limited by the network speed. Now i have a 20tb server made up of 4x5tb drives (1 redundant), with a 512gb ssd as l2arc and one of the 256gb drives as boot drive, and 189gb DDR4 RAM...
forums.servethehome.com forums.servethehome.com

Perhaps it's worth experimenting with building a 2-node freebsd HAST cluster as the target, which should increase the speed at which you can dump data onto the target.

Click to expand...
Thx for reply.

1) Everything for compression or any kind of minimizing the size of files - done. Nothing more could be done. It is what it is. We have on first stage HLS translation (everything in memory on RAM disk), next stage is compressing fragments in mp4 (also in memory and only after send it to storage server, there are only one action of writing to disk on storage server). And fragments stored in cloud within HTTPS access. For now question is about how to speed up the transfer between servers.

2) Thx for lagg link aggregation driver notice. Will look into it. And for HAST, never been doing something like this

3) SSD isn't the solution because all of them have 'write limit'. We've been calculating that every ssd for 3-5 month only. After that time SSD degradation starting. For this amount of data it will be cheaper if it's made of gold. Classical HDD is what we using. We have really a lot of cameras and we must be storing data for 10-20 days, sometimes with 3D part processing.
 
blackbird9 said:
There is one other question, which is WHY you need to transfer this huge volume of data in the first place. Do you really need to build an archive of the video data? Or is it possible to process the data at source in near realtime and only archive the finished data products, and/or only archive the video segments that are interesting. Just a thought. :)
Click to expand...
To handle realtime HLS, processing video analyse and storing this huge amount of data is impossible on one server, especially when you handle about 1000 cameras per one server within ability to load archive from cloud. There must be solution for scaling storage.
 
Alexandr Kirilov said:
Thx for reply.

1) Everything for compression or any kind of minimizing the size of files - done. Nothing more could be done. It is what it is. We have on first stage HLS translation (everything in memory on RAM disk), next stage is compressing fragments in mp4 (also in memory and only after send it to storage server, there are only one action of writing to disk on storage server). And fragments stored in cloud within HTTPS access. For now question is about how to speed up the transfer between servers.

2) Thx for lagg link aggregation driver notice. Will look into it. And for HAST, never been doing something like this

3) SSD isn't the solution because all of them have 'write limit'. We've been calculating that every ssd for 3-5 month only. After that time SSD degradation starting. For this amount of data it will be cheaper if it's made of gold. Classical HDD is what we using. We have really a lot of cameras and we must be storing data for 10-20 days, sometimes with 3D part processing.
Click to expand...
It sounds pretty sophisticated already. Interesting technical problem. :)
 
So it isn't just network you need advice for, you don't have a disk solution yet either?

FWIW you would have a hard time wearing out proper datacenter SSDs.
 
cracauer@ said:
So it isn't just network you need advice for, you don't have a disk solution yet either?

FWIW you would have a hard time wearing out proper datacenter SSDs.
Click to expand...
Question is about network only.
We using only classical HDDs, everything is working perfectly on previous version of video server disks.
 
Alexandr Kirilov said:
  • What is the fastest way to do it? FTP? rsync/rcp/scp? Something else?
  • Is there something to be tuned for this amount of traffic?
Click to expand...
Frankly, your question is a bit vague asked, so I try to need to understand some things to give an usable answer:
You do have two servers, let's call them A and B.
A is producing X B/s data. A's data shall be transferred to B.

The bandwidth of the net you're using is N B/s, and apparently X > N, otherwise you wouldn't have asked.
If not, measure your data throughput N first, and get exact numbers on X. If those don't fit it's useless to talk about the protocol at all. The net's bandwidth is independently from the protocol you're using, ftp, scp, rsync... of course some are faster than others, while none increases the bandwidth.

So, by principle you have two options, only:
1. increase N
2. reduce X

For 1 - see how to get more bandwidth, you have two options: Ask or find a provider for a faster line. If there is no line available fast enough, your only options left then are: Build a line yourself. Which, of course, can be a bit problematic, if you have more to span than some few hundred meters on your own property, and money matters.
Or you can do what covacat said: Do copies of the data to storages (tapes, HDDs, SSDs,...) and ship those with some parcel service, or drive them yourself over to B. You can get amazing bandwidths this way: Just imagine you put thirty 10TB drives into a parcel, and calculate the B/s you get when this package needs 48h for delivery.

Or 2, you need reduce the amount of data that needs to be transferred. Remove all data, which doesn't need to be transferred, or at least at the same time, and select only data really needed.
As you say the data is mostly videofiles. So there is no really more you can compress anymore.
So check if there is a file format your raw videos could be saved with better compression. And see if you can reduce the video files raw data: lower resolution, FPS, colors,...

For 2 you said, everything could be done was done.
So 1 is left: get a line fast enough, or ship storages.
 
the responses seem to focus on the software end, which only buys you so much. Put multiple NICs in source and destination servers to increase the available pipe size, keeping mind the overall speed/width of the computer bus. then, focus on transfer protocols that use UDP as opposed to TCP. Since the connection will be reliable you don't need to ACK individual packets.
 
kent_dorfman766 said:
the responses seem to focus on the software end, which only buys you so much. Put multiple NICs in source and destination servers to increase the available pipe size, keeping mind the overall speed/width of the computer bus. then, focus on transfer protocols that use UDP as opposed to TCP. Since the connection will be reliable you don't need to ACK individual packets.
Click to expand...
or hook up a hot-pluggable SAS drive system to copy the data disk-to-disk
 
cracauer@ said:
So it isn't just network you need advice for, you don't have a disk solution yet either?

FWIW you would have a hard time wearing out proper datacenter SSDs.
Click to expand...
Spinning disks remain the best solution if one is optimizing for $/bandwidth (read or write), even though it isn't as perfect as we once thought (the ~500 TB/year throughput limit). Agree with you that good SSDs have fine write endurance, although it is possible to wear then out within years if they are used as fast log devices. The real answer depends on many details, in particular the required data lifetime.

Maturin said:
You do have two servers, let's call them A and B.
A is producing X B/s data. A's data shall be transferred to B.

The bandwidth of the net you're using is N B/s, and apparently X > N, otherwise you wouldn't have asked.
Click to expand...
The OP is creating 500 Mbyte/second. 10gig Ethernet boards, cables and switches are a commodity.

Exactly along the lines of what you're saying: Before we can recommend a solution, we need to know "speeds and feed". Right now we have a few more numbers from the OP. How long does the data have to be retained? How many times it is re-processed and rewritten? How much of it is ever read (the write-once, read never pattern is surprisingly common)? How intense are the reads?

Using the OP's numbers (4 Gigabit/second x 10-20 day lifetime), they need a storage capacity of about 400-900 TByte. That's not a particularly large storage system, measured in dozens of disks, not hundreds. Disk bandwidth will not be a problem from the ingress viewpoint. The crux may be in the read traffic.

As you said, the protocol question is secondary, and a minor detail.
 
You must log in or register to reply here.
Back
Top