My view is that it depends entirely on the workload, and the systems with which your compute needs to interact.  A few things I’ve experienced before.

 

1. Modern ethernet networks have pretty good latency these days, and so MPI codes can run over them.   Whether IB is worth the money is a cost/benefit calculation for the codes you want to run.  The ethernet network we put in at Sanger in 2016 or so we measured as having similar latency, in practice, as FDR infiniband, if I remember correctly.  So it wasn’t as good as state-of-the-art IB at the time, but not bad.  Certainly good enough for our purposes, and we gained a lot of flexibility through software-defined networking, important if you have workloads which require better security boundaries than just a big shared network.
2. If your workload is predominantly single node, embarrassingly parallel, you might do better to go with ethernet and invest the saved money in more compute nodes.
3. If you only have ethernet, your cluster will be simpler, and require less specialised expertise to run
4. If your parallel filesystem is Lustre, IB seems to be the more well-worn path than ethernet.  We encountered a few Lustre bugs early on because of that.
5. On the other hand, if you need to talk to Weka, ethernet is the well-worn path.  Weka’s IB implementation requires the dedication of some cores on every client node, so you lose some compute capacity, which you don’t need to do if you’re using ethernet.

 

So, as any lawyer would say “it depends”.  Most of my career has been in genomics, where IB definitely wasn’t necessary.  Now that I’m in pharma, there’s more MPI code, so there’s more of a case for it.

 

Ultimately, I think you need to run the real benchmarks with real code, and as Jason says, work out whether the additional complexity and cost of the IB network is worth it for your particular workload.  I don’t think the mantra “It’s HPC so it has to be Infiniband” is a given.

 

Tim