Solved Fastest way to transfer files between servers.

Alexandr Kirilov said:
  • What is the fastest way to do it? FTP? rsync/rcp/scp? Something else?
  • Is there something to be tuned for this amount of traffic?
Click to expand...
Frankly, your question is a bit vague asked, so I try to need to understand some things to give an usable answer:
You do have two servers, let's call them A and B.
A is producing X B/s data. A's data shall be transferred to B.

The bandwidth of the net you're using is N B/s, and apparently X > N, otherwise you wouldn't have asked.
If not, measure your data throughput N first, and get exact numbers on X. If those don't fit it's useless to talk about the protocol at all. The net's bandwidth is independently from the protocol you're using, ftp, scp, rsync... of course some are faster than others, while none increases the bandwidth.

So, by principle you have two options, only:
1. increase N
2. reduce X

For 1 - see how to get more bandwidth, you have two options: Ask or find a provider for a faster line. If there is no line available fast enough, your only options left then are: Build a line yourself. Which, of course, can be a bit problematic, if you have more to span than some few hundred meters on your own property, and money matters.
Or you can do what covacat said: Do copies of the data to storages (tapes, HDDs, SSDs,...) and ship those with some parcel service, or drive them yourself over to B. You can get amazing bandwidths this way: Just imagine you put thirty 10TB drives into a parcel, and calculate the B/s you get when this package needs 48h for delivery.

Or 2, you need reduce the amount of data that needs to be transferred. Remove all data, which doesn't need to be transferred, or at least at the same time, and select only data really needed.
As you say the data is mostly videofiles. So there is no really more you can compress anymore.
So check if there is a file format your raw videos could be saved with better compression. And see if you can reduce the video files raw data: lower resolution, FPS, colors,...

For 2 you said, everything could be done was done.
So 1 is left: get a line fast enough, or ship storages.
 
the responses seem to focus on the software end, which only buys you so much. Put multiple NICs in source and destination servers to increase the available pipe size, keeping mind the overall speed/width of the computer bus. then, focus on transfer protocols that use UDP as opposed to TCP. Since the connection will be reliable you don't need to ACK individual packets.
 
kent_dorfman766 said:
the responses seem to focus on the software end, which only buys you so much. Put multiple NICs in source and destination servers to increase the available pipe size, keeping mind the overall speed/width of the computer bus. then, focus on transfer protocols that use UDP as opposed to TCP. Since the connection will be reliable you don't need to ACK individual packets.
Click to expand...
or hook up a hot-pluggable SAS drive system to copy the data disk-to-disk
 
POSIX.1 said:
Folks, do not be such a fool supporting war actors that are attacking the free and open world FreeBSD is part of.
Click to expand...
Of course I agree with that. When I read the OP's question nothing suggested military use to me though. I guess it does mention 'surveillance', but that could mean any number of things. I would have thought if it was really for military use the question would have been a bit more camouflaged to put people off the scent. And we're only guessing nationality from Alexandr's name... after all who knows who 'POSIX.1' is or what country they are from? It's very hard to tell on an open forum like this. I don't pretend to know what the answer is.
 
eternal_noob said:
I just don't want to support Russia to kill children.
Click to expand...
Me neither. I don't want anybody killing any one at all - not only children.
But besides I simply see no proof that this would really came from russia, I don't see why this was critical:
Any one can figure it out herself. All you need is the fundamental basic knowledge of bandwidths, the capability to compute, and compare some numbers, and the active use of oneself's brains.
 
To be honest I don't find it credible that professional software developers, working for big defence contractors developing military systems, would ask basic questions of this type on a public forum like this. Much more likely the question comes from someone in an academic/opensource project or perhaps in a small startup company who is trying to design something like this for the first time. Of course I could be wrong, but weapons systems development doesn't ring true to my mind.
 
blackbird9 said:
Much more likely someone in an academic project or perhaps in a small startup company who is trying to design something like this for the first time.
Click to expand...
Yep, software DVR can generate a lot of data even in small organization.

Obviosly, mil and govt have their own high level specialists, who dont need forums like this.

P.S. As international project FreeBSD must be out of politics as much as possible, or politized fanatics and propagandists will destroy it...
 
blackbird9 said:
To be honest I don't find it credible that professional software developers, working for big defence contractors developing military systems, would ask basic questions of this type on a public forum like this. Much more likely the question comes from someone in an academic/opensource project or perhaps in a small startup company who is trying to design something like this for the first time. Of course I could be wrong, but weapons systems development doesn't ring true to my mind.
Click to expand...
You don't know/care much about recent development of drone warfare, do you? "Human Safari" in Kherson?
 
dclau said:
You don't know/care much about recent development of drone warfare, do you? "Human Safari" in Kherson?
Click to expand...
I don't KNOW anything about drone warfare. Of course I deplore the war in ukraine as well as all the other wars around the world. But I only "know" what I see in the TV news each evening. All war is terrible, only a fool wants war.
 
POSIX.1 said:
As OP prefers not to answer, see https://www.arboreus.systems/
Click to expand...
I had a quick look around that website and at their github repository. They have a bunch of useful looking sample code in various different languages, and a library of articles. I noticed this one talking about why they chose freebsd for their projects: https://github.com/ArboreusSystems/...aster/freebsd/why_freebsd/eng.why_freebsd.pdf

I can't see anything that looks remotely related to military work. There is some discussion of blockchain and healthcare systems.
It looks like a typical very small company or consultancy. It all looks pretty innocent to me.
 
cracauer@ said:
So it isn't just network you need advice for, you don't have a disk solution yet either?

FWIW you would have a hard time wearing out proper datacenter SSDs.
Click to expand...
Spinning disks remain the best solution if one is optimizing for $/bandwidth (read or write), even though it isn't as perfect as we once thought (the ~500 TB/year throughput limit). Agree with you that good SSDs have fine write endurance, although it is possible to wear then out within years if they are used as fast log devices. The real answer depends on many details, in particular the required data lifetime.

Maturin said:
You do have two servers, let's call them A and B.
A is producing X B/s data. A's data shall be transferred to B.

The bandwidth of the net you're using is N B/s, and apparently X > N, otherwise you wouldn't have asked.
Click to expand...
The OP is creating 500 Mbyte/second. 10gig Ethernet boards, cables and switches are a commodity.

Exactly along the lines of what you're saying: Before we can recommend a solution, we need to know "speeds and feed". Right now we have a few more numbers from the OP. How long does the data have to be retained? How many times it is re-processed and rewritten? How much of it is ever read (the write-once, read never pattern is surprisingly common)? How intense are the reads?

Using the OP's numbers (4 Gigabit/second x 10-20 day lifetime), they need a storage capacity of about 400-900 TByte. That's not a particularly large storage system, measured in dozens of disks, not hundreds. Disk bandwidth will not be a problem from the ingress viewpoint. The crux may be in the read traffic.

As you said, the protocol question is secondary, and a minor detail.
 
Yes, because the smart people already left the country or are dead. There are only idiots left.

(To be clear i meant Russian government. I know that population != government and in fact i like the Russians like everyone else)
 
Please technical not fairytales here. So sad to attack a question asker with a technical question.

Remember the BSD license. We don't care what you do with it. We don't concern ourself with end use.
Silly, stupid or evil.
I don't care if you are building centrifuges to blow up the world.
Please continue on. This is an operating system not a social platform.

David Byrne said it best.
This Ain't No Party... This Ain't No Disco... This Ain't No Foolin' Around);
 
cracauer@ said:
Sure they would.
Click to expand...
Well, that's not my experience of working in large companies (or even small companies). I can imagine the reaction I would get if at the monday morning scrum, when the team lead asked me how the data transfer work is going and I replied "don't worry I'm going to ask on the freebsd forum and I'll have all the answers later". I'd be laughed out of the room. 😄

I mean, being serious, I guess in wartime if people are desparate it might be a bit more likely that that kind of thing might happen. But I think we have to have an "innocent until proven guilty" policy, or I would if I was administering this site. Otherwise half the questions asked potentially have military applications, and how are we, the forum users who might answer them, to distinguish the innocent questions from the ones involved in warfare? Anyway that's really a question for the forum moderators, but I haven't noticed them blocking people posting from russia or ukraine (or sudan, congo, etc..).
 
blackbird9 said:
...I mean, being serious, i guess in wartime if people are desparate it might be a bit more likely that that kind of thing might happen.
Click to expand...
I'm thinking it's some kind of illegal to use VLC on an OS for drone FPV (Fedora export control policy discussion)

What's the difference between the implied serious use of that, vs flying around a bedroom? It's an unmanned aerial vehicle involving a "produced" set-up, "controlled out of the direct natural vision of the operator" :p
 
I stand by my opinion. Whether somebody asks questions in public depends on their personality and the momentary situation, not where they work.

Google is held together by stackoverflow, too.
 
Well I guess the stackoverflow reference is a valid point :)

But we're still stuck wondering whether a question posted might be involved in "drone warfare". I don't really see how it's possible to tell, unless they specifically talk about subjects that are obviously military in nature.
 
You must log in or register to reply here.
Back
Top