For an embarrassingly parallel task like this why does there need to be any blazingly fast communication between the nodes at all?
My guess is that they've taken HashCat and not made any changes to it at all and just ran it on this virtual cluster so that it assumes that it's talking directly to each GPU/node.
For brute forcing things there's no need for fast communication between the nodes, you just split the work up into units of n seconds each and then have them poll for work from a central host. Sure it's a bit more work/complex as you're effectively running individual copies of HashCat on each node but that's just a bit of scripting and saves you the cost of Infiniband!
The Virtual Open Cluster and Infiniband solution would be much better for tasks that require lots of intercommunication between the nodes (e.g. block Lanczos of large sparse matrices) but brute forcing passwords/hashes isn't a great example.
I guess the parallelism is why botnets are so good at it. I wonder if anyone's tried getting JavaScript web workers (or possibly, somehow WebGL) to poll a central server to do cracking. A site with just a few thousand visitors a day should be able to crack passwords faster than this machine.
I have attempted this, and I believe others on the wider Internet have as well. Essentially it ended up being slower than if the central servers CPU was used to attack the hashes instead. With only JavaScript available to do the hashing, the maximum per client was disappointingly slow, around 2000 hash/second each if I remember correctly.
It might be better now with faster JavaScript engines, but I wouldn't bet on it.
My guess is that they've taken HashCat and not made any changes to it at all and just ran it on this virtual cluster so that it assumes that it's talking directly to each GPU/node.
For brute forcing things there's no need for fast communication between the nodes, you just split the work up into units of n seconds each and then have them poll for work from a central host. Sure it's a bit more work/complex as you're effectively running individual copies of HashCat on each node but that's just a bit of scripting and saves you the cost of Infiniband!
The Virtual Open Cluster and Infiniband solution would be much better for tasks that require lots of intercommunication between the nodes (e.g. block Lanczos of large sparse matrices) but brute forcing passwords/hashes isn't a great example.