Hi Fritz,
Does pam_slurm_adopt.so exist in the right location on the node? Normally on EL hosts it would be /usr/lib64/security/pam_slurm_adopt.so
# ls /usr/lib64/security/pam_slurm_adopt.so -la -rwxr-xr-x 1 root root 291936 Mar 4 12:44 /usr/lib64/security/pam_slurm_adopt.so
If the file doesn't exist, pam would abnormally exit and not allow anyone to log in.
Sean ________________________________ From: Ratnasamy, Fritz via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, 17 June 2025 14:55 To: Kevin Buckley kevin.buckley.pawsey.org.au@gmail.com Cc: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [EXT] [slurm-users] Re: slurm_pam_adopt module not working
External email: Please exercise caution
________________________________ Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that locked all users to the cluster. That same file works on a different cluster where the files are pushed via puppet but with ansible it looks like it is locking all users to the cluster. See below config file sshd:
auth required pam_sepermit.so auth substack password-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so ##SLURM account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny account sufficient pam_access.so ##END SLURM password include password-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open env_params session required pam_namespace.so session optional pam_keyinit.so force revoke session include password-auth session include postlogin # Used with polkit to reauthorize users in remote sessions -session optional pam_reauthorize.so prepare
Fritz Ratnasamy Data Scientist Information Technology
On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users <slurm-users@lists.schedmd.commailto:slurm-users@lists.schedmd.com> wrote: On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote:
We wanted to block users from ssh to a node where there are no jobs running however it looks like users are able to do so. We have installed the slurm_pam_adopt_module and set up the slurm.conf accordingly (the same way we did on our first cluster where the pam module denies ssh access correctly).
We saw a similar issue whereby the way that we had PAM setup, meant that, and here I quote from SchedMD's Daniel Armengod:
----8<--------8<--------8<--------8<--------8<--------8<--------8<---- This is almost certainly caused by the fact that SSH's `keyboard-interactive` (not to be confused with `password`) AuthMethod forks a short-lived child process that is involved in the authentication logic. Slurm's pam_slurm_adopt module latches on to that process (which is the wrong one, of course) and things break in interesting ways from there.
SSH authmethods `publickey` and `password` do not exhibit this behaviour as SSH does not fork a child process to offload the authentication challenge-response dialogue to.
...
The key bit here is that in your last test you're forcing `PreferredAuthentications=password`, which isn't actually the `keyboard-interactive` AuthMethod that got picked before. They work differently under the hood, even if as far as the user is concerned, both methods just ask for a password.
...
In summary: try disabling the `keyboard-interactive` authentication method in your compute nodes. pam_slurm_adopt should work correctly now. ----8<--------8<--------8<--------8<--------8<--------8<--------8<----
Maybe that's also your issue.
Daniel did say that SchedMD were going to update their documentation to make that distinction, and it's effect, more explciit, so I would expect it to be in the mainstream docs by now.
HTH
-- slurm-users mailing list -- slurm-users@lists.schedmd.commailto:slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.commailto:slurm-users-leave@lists.schedmd.com CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.