Distributed processing?

PacketMan · Feb 3, 2016

I am interesting in distributed processing, and the ability to do it in FreeBSD. Note from some YouTube videos I understand you can 'mount the OS' (for lack of a better phrase) over multiple machines. I am not interested in that.

What I am interested in is some sort of port that basically allows many fully independent FreeBSD machines to cluster together the processing capabilities of user chosen programs/processes. So for example I choose to allow compiling to be spread over multiple machines, or maybe for example my processing efforts to find the next prime number, or some hard core number crunching for example. The user application port is installed on one machine, but it gets access to the resources of many machines over a LAN, and ideally over a routed WAN via a 'distributed clustering' port on all the machines.

Does such a port(s) exist? Any brilliant minds wanna write that for FreeBSD??!! (Wish I was a coder!)

BSD-Kitsune · Feb 3, 2016

Not to my knowledge does it exist in software. However, you can do this in hardware with NUMALink enabled computer systems. Examples would be HP Integrity, SGI Altix and Origin systems. But none of these run FreeBSD with the exception of the x86 versions of these machines.

ronaldlees · Feb 3, 2016

PacketMan said:
... The user application port is installed on one machine, but it gets access to the resources of many machines over a LAN, and ideally over a routed WAN via a 'distributed clustering' port on all the machines ...

Sounds like what Plan 9 does. For just any random program to share low level resources, the facility probably has to be baked into an operating system like it is with Plan 9. It's not an easy system to get used to (like FreeBSD is

)

wblock@ · Feb 3, 2016

devel/ccache used to have some capacity to spread compile jobs out over several machines. I never tried it, though.

PacketMan · Feb 3, 2016

Thanks wblock@, I have an interest in that for other reason, hmmmm might tinker with it.

da1 · Feb 3, 2016

Not sure what's the status of this https://www.google.de/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=freebsd beowulf cluster but it sounds like it can perform the task you need.

tingo · Feb 5, 2016

For that last part: note that most common clusters (of which Beowulf is one) only run cluster-aware programs written to the cluster "API" so to speak. The cluster doesn't run any random user-selected program unless that program happens to be written to the cluster "API" in use.
Or I can describe it this way: there doesn't current exist any cluster technology now that can run COTS programs on the cluster (AFAIK).

kpa · Feb 5, 2016

tingo said:
For that last part: note that most common clusters (of which Beowulf is one) only run cluster-aware programs written to the cluster "API" so to speak. The cluster doesn't run any random user-selected program unless that program happens to be written to the cluster "API" in use.
Or I can describe it this way: there doesn't current exist any cluster technology now that can run COTS programs on the cluster (AFAIK).

Another way of saying that is that if you take a random program X the chances that it could be automatically turned into something that makes use of the parallel execution capabilities is virtually zero. Posing a question "is this program X such that it could be turned into parallelized equivalent by automated means" is guaranteed to be unsolvable. This is a consequence the a famous "halting problem" that basically states that it's impossible to write programs that could deduce some non-trivial property of another program in such a way that it could do that universally for any program imaginable.