[slurm-users] ssh-keys on compute nodes?

[Ole Holm Nielsen]

Hi Durai,

I can only try to explain how I understand this: The "slurm" user runs 
only the slurmctld and slurmdbd central server daemons.  On the compute 
nodes, the slurmd daemon runs as the root user so that it can start user 
tasks on behalf of normal users.

The "slurm" user should *not* have password-less SSH!

Normal users also do not need SSH if their MPI tasks are started with 
Slurm's "srun".  Users only need password-less SSH if they have some 
strange MPI software, in which case you need to set up SSH authorized_keys 
files for such users.

/Ole

On 6/9/20 11:21 AM, Durai Arasan wrote:
> Hi,
> 
> Can you please help me understand how the passwordless ssh works on SLURM?
> 
> I was under the assumption that jobs/tasks are ultimately submitted by the 
> "slurm" linux user and not by the linux user who wants to run jobs. Is 
> this not correct? So is it not sufficient for only the "slurm" linux user 
> to have passwordless ssh access to all nodes? Why do we have to give 
> passwordless ssh access to every user of the cluster?
> 
> Thanks,
> Durai
> Zentrum für Datenverarbeitung
> Tübingen
> 
> On Mon, Jun 8, 2020 at 6:43 PM Ole Holm Nielsen 
> <Ole.H.Nielsen at fysik.dtu.dk <mailto:Ole.H.Nielsen at fysik.dtu.dk>> wrote:
> 
>     On 08-06-2020 18:07, Jeffrey T Frey wrote:
>      > There's a Slurm PAM module you can use to gate ssh access --
>     basically it checks to see if the user has a job running on the node
>     and moves any ssh sessions to the first cgroup associated with that
>     user on that node.  If you don't use cgroup resource limiting I think
>     it just gates access w/o any such cgroup assignments.
> 
>     The pam_slurm_adopt[1] module is used by lots of Slurm sites for
>     restricting access by SSH.  See the discussion in
>     https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#pam-module-restrictions
> 
>     /Ole
> 
>     [1] https://slurm.schedmd.com/pam_slurm_adopt.html
> 
> 
>      >> On Jun 8, 2020, at 12:01 , Durai Arasan <arasan.durai at gmail.com
>     <mailto:arasan.durai at gmail.com>> wrote:
>      >>
>      >> Hi Jeffrey,
>      >>
>      >> Thanks for the clarification.
>      >>
>      >> But this is concerning, as the users will be able to ssh into any
>     node. How do you prevent that?
>      >>
>      >> Best,
>      >> Durai
>      >>
>      >> On Mon, Jun 8, 2020 at 5:55 PM Jeffrey T Frey <frey at udel.edu
>     <mailto:frey at udel.edu>> wrote:
>      >> User home directories are on a shared (NFS) filesystem that's
>     mounted on every node.  Thus, they have the same id_rsa key and
>     authorized_keys file present on all nodes.
>      >>
>      >>
>      >>
>      >>
-- 
Ole Holm Nielsen
PhD, Senior HPC Officer
Department of Physics, Technical University of Denmark,
Fysikvej Building 309, DK-2800 Kongens Lyngby, Denmark
E-mail: Ole.H.Nielsen at fysik.dtu.dk
Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/
Mobile: (+45) 5180 1620
>      >>> On Jun 8, 2020, at 11:42 , Durai Arasan <arasan.durai at gmail.com
>     <mailto:arasan.durai at gmail.com>> wrote:
>      >>>
>      >>> Ok, that was useful information.
>      >>>
>      >>> So when you provision user accounts, you add the public key to
>     .ssh/authorized_keys of *all* nodes on the cluster? Not just the login
>     nodes.. ?
>      >>> When we provision user accounts on our Slurm cluster we still add
>     .ssh, .ssh/id_rsa (needed for older X11 tunneling via libssh2), and
>     add the public key to .ssh/authorized_keys.
>      >>>
>      >>> Thanks,
>      >>> Durai
>