[slurm-users] Question about IB and Ethernet networks

Hi Fellow Slurm Users,

This question is not slurm-specific, but it might develop into that.

My question relates to understanding how *typical* HPCs are designed in terms of networking. To start, is it typical for there to be a high speed Ethernet *and* Infiniband networks (meaning separate switches, NICs)? I know you can easily setup IP over IB, but is IB usually fully reserved for MPI messages? I’m tempted to spec all new HPCs with only a high speed (200Gbps) IB network, and use IPoIB for all slurm comms with compute nodes. I plan on using BeeGFS for the file system with RDMA.

Just looking for some feedback, please. Is this OK? Is there a better way? If yes, please share why it’s better.

Thanks,

Daniel Healy