Fritz Ratnasamy
Data Scientist
Information Technology
Hi Fritz,
Does pam_slurm_adopt.so exist in the right location on the node? Normally on EL hosts it would be /usr/lib64/security/pam_slurm_adopt.so
# ls /usr/lib64/security/pam_slurm_adopt.so -la-rwxr-xr-x 1 root root 291936 Mar 4 12:44 /usr/lib64/security/pam_slurm_adopt.so
If the file doesn't exist, pam would abnormally exit and not allow anyone to log in.
Sean
From: Ratnasamy, Fritz via slurm-users <slurm-users@lists.schedmd.com>
Sent: Tuesday, 17 June 2025 14:55
To: Kevin Buckley <kevin.buckley.pawsey.org.au@gmail.com>
Cc: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: [EXT] [slurm-users] Re: slurm_pam_adopt module not workingThanks, for some reason I edited the /etc/pam.d/sshd via ansible but that locked all users to the cluster. That same file works on a different cluster where the files are pushed via puppet but with ansible it looks like it is locking all users to the cluster. See below config file sshd:
External email: Please exercise caution
auth required pam_sepermit.so
auth substack password-auth
auth include postlogin
# Used with polkit to reauthorize users in remote sessions
-auth optional pam_reauthorize.so prepare
account required pam_nologin.so
##SLURM
account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny
account sufficient pam_access.so
##END SLURM
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session required pam_namespace.so
session optional pam_keyinit.so force revoke
session include password-auth
session include postlogin
# Used with polkit to reauthorize users in remote sessions
-session optional pam_reauthorize.so prepare
Fritz Ratnasamy
Data Scientist
Information Technology
On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users <slurm-users@lists.schedmd.com> wrote:
On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote:
>
> We wanted to block users from ssh to a node where there are no jobs
> running however it looks like users are able to do so. We have installed
> the slurm_pam_adopt_module and set up the slurm.conf accordingly (the same
> way we did on our first cluster where the pam module denies ssh access
> correctly).
We saw a similar issue whereby the way that we had PAM setup, meant
that, and here I quote from SchedMD's Daniel Armengod:
----8<--------8<--------8<--------8<--------8<--------8<--------8<----
This is almost certainly caused by the fact that SSH's `keyboard-interactive`
(not to be confused with `password`) AuthMethod forks a short-lived child
process that is involved in the authentication logic. Slurm's pam_slurm_adopt
module latches on to that process (which is the wrong one, of course) and
things break in interesting ways from there.
SSH authmethods `publickey` and `password` do not exhibit this behaviour as SSH
does not fork a child process to offload the authentication challenge-response
dialogue to.
...
The key bit here is that in your last test you're forcing
`PreferredAuthentications=password`, which isn't actually the
`keyboard-interactive` AuthMethod that got picked before.
They work differently under the hood, even if as far as the
user is concerned, both methods just ask for a password.
...
In summary: try disabling the `keyboard-interactive` authentication method in
your compute nodes. pam_slurm_adopt should work correctly now.
----8<--------8<--------8<--------8<--------8<--------8<--------8<----
Maybe that's also your issue.
Daniel did say that SchedMD were going to update their documentation
to make that distinction, and it's effect, more explciit, so I would
expect it to be in the mainstream docs by now.
HTH
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.
CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.