Yes the file exists in /usr/lib64/security/. Best,
*Fritz Ratnasamy*Data Scientist Information Technology
On Tue, Jun 17, 2025 at 12:17 AM Sean Crosby scrosby@unimelb.edu.au wrote:
Hi Fritz,
Does pam_slurm_adopt.so exist in the right location on the node? Normally on EL hosts it would be /usr/lib64/security/pam_slurm_adopt.so
# ls /usr/lib64/security/pam_slurm_adopt.so -la -rwxr-xr-x 1 root root 291936 Mar 4 12:44 /usr/lib64/security/pam_slurm_adopt.so
If the file doesn't exist, pam would abnormally exit and not allow anyone to log in.
Sean
*From:* Ratnasamy, Fritz via slurm-users slurm-users@lists.schedmd.com *Sent:* Tuesday, 17 June 2025 14:55 *To:* Kevin Buckley kevin.buckley.pawsey.org.au@gmail.com *Cc:* slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com *Subject:* [EXT] [slurm-users] Re: slurm_pam_adopt module not working
- External email: Please exercise caution *
Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that locked all users to the cluster. That same file works on a different cluster where the files are pushed via puppet but with ansible it looks like it is locking all users to the cluster. See below config file sshd:
auth required pam_sepermit.so auth substack password-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so ##SLURM account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny account sufficient pam_access.so ##END SLURM password include password-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open env_params session required pam_namespace.so session optional pam_keyinit.so force revoke session include password-auth session include postlogin # Used with polkit to reauthorize users in remote sessions -session optional pam_reauthorize.so prepare
*Fritz Ratnasamy *Data Scientist Information Technology
On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users < slurm-users@lists.schedmd.com> wrote:
On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote:
We wanted to block users from ssh to a node where there are no jobs running however it looks like users are able to do so. We have installed the slurm_pam_adopt_module and set up the slurm.conf accordingly (the
same
way we did on our first cluster where the pam module denies ssh access correctly).
We saw a similar issue whereby the way that we had PAM setup, meant that, and here I quote from SchedMD's Daniel Armengod:
----8<--------8<--------8<--------8<--------8<--------8<--------8<---- This is almost certainly caused by the fact that SSH's `keyboard-interactive` (not to be confused with `password`) AuthMethod forks a short-lived child process that is involved in the authentication logic. Slurm's pam_slurm_adopt module latches on to that process (which is the wrong one, of course) and things break in interesting ways from there.
SSH authmethods `publickey` and `password` do not exhibit this behaviour as SSH does not fork a child process to offload the authentication challenge-response dialogue to.
...
The key bit here is that in your last test you're forcing `PreferredAuthentications=password`, which isn't actually the `keyboard-interactive` AuthMethod that got picked before. They work differently under the hood, even if as far as the user is concerned, both methods just ask for a password.
...
In summary: try disabling the `keyboard-interactive` authentication method in your compute nodes. pam_slurm_adopt should work correctly now. ----8<--------8<--------8<--------8<--------8<--------8<--------8<----
Maybe that's also your issue.
Daniel did say that SchedMD were going to update their documentation to make that distinction, and it's effect, more explciit, so I would expect it to be in the mainstream docs by now.
HTH
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.
CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.
You say that you modified the file in a different way. It may be worth checking file permissions as for some security functions files can be ignored if they don't have the required permissions.
That said, that would show in the journal/ logs.
William
On Tue, 17 Jun 2025, 06:24 Ratnasamy, Fritz via slurm-users, < slurm-users@lists.schedmd.com> wrote:
Yes the file exists in /usr/lib64/security/. Best,
*Fritz Ratnasamy*Data Scientist Information Technology
On Tue, Jun 17, 2025 at 12:17 AM Sean Crosby scrosby@unimelb.edu.au wrote:
Hi Fritz,
Does pam_slurm_adopt.so exist in the right location on the node? Normally on EL hosts it would be /usr/lib64/security/pam_slurm_adopt.so
# ls /usr/lib64/security/pam_slurm_adopt.so -la -rwxr-xr-x 1 root root 291936 Mar 4 12:44 /usr/lib64/security/pam_slurm_adopt.so
If the file doesn't exist, pam would abnormally exit and not allow anyone to log in.
Sean
*From:* Ratnasamy, Fritz via slurm-users slurm-users@lists.schedmd.com *Sent:* Tuesday, 17 June 2025 14:55 *To:* Kevin Buckley kevin.buckley.pawsey.org.au@gmail.com *Cc:* slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com *Subject:* [EXT] [slurm-users] Re: slurm_pam_adopt module not working
- External email: Please exercise caution *
Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that locked all users to the cluster. That same file works on a different cluster where the files are pushed via puppet but with ansible it looks like it is locking all users to the cluster. See below config file sshd:
auth required pam_sepermit.so auth substack password-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so ##SLURM account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny account sufficient pam_access.so ##END SLURM password include password-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open env_params session required pam_namespace.so session optional pam_keyinit.so force revoke session include password-auth session include postlogin # Used with polkit to reauthorize users in remote sessions -session optional pam_reauthorize.so prepare
*Fritz Ratnasamy *Data Scientist Information Technology
On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users < slurm-users@lists.schedmd.com> wrote:
On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote:
We wanted to block users from ssh to a node where there are no jobs running however it looks like users are able to do so. We have installed the slurm_pam_adopt_module and set up the slurm.conf accordingly (the
same
way we did on our first cluster where the pam module denies ssh access correctly).
We saw a similar issue whereby the way that we had PAM setup, meant that, and here I quote from SchedMD's Daniel Armengod:
----8<--------8<--------8<--------8<--------8<--------8<--------8<---- This is almost certainly caused by the fact that SSH's `keyboard-interactive` (not to be confused with `password`) AuthMethod forks a short-lived child process that is involved in the authentication logic. Slurm's pam_slurm_adopt module latches on to that process (which is the wrong one, of course) and things break in interesting ways from there.
SSH authmethods `publickey` and `password` do not exhibit this behaviour as SSH does not fork a child process to offload the authentication challenge-response dialogue to.
...
The key bit here is that in your last test you're forcing `PreferredAuthentications=password`, which isn't actually the `keyboard-interactive` AuthMethod that got picked before. They work differently under the hood, even if as far as the user is concerned, both methods just ask for a password.
...
In summary: try disabling the `keyboard-interactive` authentication method in your compute nodes. pam_slurm_adopt should work correctly now. ----8<--------8<--------8<--------8<--------8<--------8<--------8<----
Maybe that's also your issue.
Daniel did say that SchedMD were going to update their documentation to make that distinction, and it's effect, more explciit, so I would expect it to be in the mainstream docs by now.
HTH
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.
CAUTION: This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Ratnasamy, Fritz -
William Brown