[slurm-users] Question about networks and connectivity

[mercan]

Hi;

Your mpi and NAMD use your second network because of your applications 
did not compiled for infiniband. There are many compiled NAMD versions. 
the verb and ibverb versions are for using infiniband. Also, when you 
compiling the mpi source, you should check configure script detect the 
infiniband network to use infiniband. And even while compiling the slurm 
too.

Regards;

  Ahmet M.


On 5.12.2019 15:07, sysadmin.caos wrote:
> Hello,
>
> Really, I don't know if my question is for this mailing list... but I 
> will explain my problem and, then, you could answer me whatever you 
> think ;)
>
> I manage a SLURM clusters composed by 3 networks:
>
>   * a gigabit network used for NFS shares (192.168.11.X). In this
>     network, my nodes are "node01, node02..." in /etc/hosts.
>   * a gigabit network used by SLURM (all my nodes are added to SLURM
>     cluster using this network and the hostname assigned via /etc/host
>     to this second network). (192.168.12.X). In this network, my nodes
>     are "clus01, clus02..." in /etc/hosts.
>   * a Infiniband network (192.168.13.X). In this network, my nodes are
>     "infi01, infi02..." in /etc/hosts.
>
> When I submit a MPI job, SLURM scheduler offers me "n" nodes called, 
> for example, clus01 and clus02 and, there, my application runs 
> perfectly using second network for SLURM connectivity and first 
> network for NFS (and NIS) shares. By default, as SLURM connectivity is 
> on second network, my nodelist contains nodes called "clus0x".
>
> However, now, I'm getting a "new" problem. I want to use third network 
> (Infiniband), but as SLURM offers me "clus0x" (second network), my MPI 
> application runs OK but using second network. This problem also 
> occurs, for example, using NAMD (Charmrun) application.
>
> So, my questions are:
>
>  1. is this SLURM configuration correct for using both networks?
>      1. If answer is "no", how do I configure SLURM for my purpose?
>      2. But if answer is "yes", how can I ensure connections in my
>         SLURM job are going in Infiniband?
>
> Thanks a lot!!
>