In the instructions for pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config, there are instructions such as:
Add the following line to the appropriate file in /etc/pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag):
This module is configurable. Add these options to the end of the
pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd or system-auth):
Assuming an OS like CentOS does this mean it should be put in both?
slurm.conf on the node has: UsePAM yes
slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so
so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so
We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets:
Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth]
Am I missing something?
Hi Robert,
The explicit configuration details of pam_slurm_adopt are described in this Wiki page: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#pam-module-re...
Important: DO NOT configure UsePAM=1 in slurm.conf! The advice for UsePAM=1 this may be found on the internet, but that is a bad advice :-(
You mention CentOS, but that OS has been dead for a long time...
IHTH, Ole
On 18-04-2025 17:26, Robert Kudyba via slurm-users wrote:
In the instructions for pam_slurm_adopt <https://slurm.schedmd.com/ pam_slurm_adopt.html#ssh_config>, there are instructions such as:
Add the following line to the appropriate file in /etc/pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag): This module is configurable. Add these options to the end of the pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd or system-auth):Assuming an OS like CentOS does this mean it should be put in both?
slurm.conf on the node has: UsePAM yes
slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so
so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so
We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets:
Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth]
Am I missing something?
Hi
Did you disable the pam_systemd.so also from the module files included by the sshd pam file ? I am asking because I had this problem when I configured the pam_slurm_adopt
Cheers, Massimo
On Fri, Apr 18, 2025 at 5:28 PM Robert Kudyba via slurm-users < slurm-users@lists.schedmd.com> wrote:
In the instructions for pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config, there are instructions such as:
Add the following line to the appropriate file in /etc/pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag):
This module is configurable. Add these options to the end of the
pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd or system-auth):
Assuming an OS like CentOS does this mean it should be put in both?
slurm.conf on the node has: UsePAM yes
slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so
so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so
We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets:
Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth]
Am I missing something?
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thanks Ole and Massimo, I definitely do not have UsePAM=1 in slurm.conf. I commented out pam_systemd here: grep pam_systemd * fingerprint-auth:-session optional pam_systemd.so fingerprint-auth-ac:-session optional pam_systemd.so password-auth:#-session optional pam_systemd.so password-auth-ac:#-session optional pam_systemd.so runuser-l:#-session optional pam_systemd.so smartcard-auth:-session optional pam_systemd.so smartcard-auth-ac:-session optional pam_systemd.so system-auth:#-session optional pam_systemd.so system-auth-ac:#-session optional pam_systemd.so
I did enable debug here; sshd:-account sufficient pam_slurm_adopt.so action_unknown=newest log_level=debug5 Latest logs:
Apr 18 13:06:08 node11 sshd[32043]: Authorized to user, krb5 principal user@ADCU.OURUNI.EDU (ssh_gssapi_krb5_cmdok) Apr 18 13:06:08 node11 sshd[32043]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Ignoring obsolete CgroupReleaseAgentDir option. Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Reading slurm.conf file: /etc/slurm/slurm.conf Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 4294967295 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 0 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Munge authentication plugin loaded Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Success. Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Connection by user user: user has only one job 4736742 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: _adopt_process: trying to get 4736742.4294967295 to adopt 32043 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Leaving stepd_add_extern_pid Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Leaving stepd_get_x11_display Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Process 32043 adopted into job 4736742 Apr 18 13:06:08 node11 sshd[32043]: fatal: Access denied for user user by PAM account configuration [preauth]
There are a few Slurm bugs mentioning: Connection by user user: user has only one job
But the "only" makes it sound like that's a bad thing?
On Fri, Apr 18, 2025 at 1:07 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Hi
Did you disable the pam_systemd.so also from the module files included by the sshd pam file ? I am asking because I had this problem when I configured the pam_slurm_adopt
Cheers, Massimo
On Fri, Apr 18, 2025 at 5:28 PM Robert Kudyba via slurm-users < slurm-users@lists.schedmd.com> wrote:
In the instructions for pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config, there are instructions such as:
Add the following line to the appropriate file in /etc/pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag):
This module is configurable. Add these options to the end of the
pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd or system-auth):
Assuming an OS like CentOS does this mean it should be put in both?
slurm.conf on the node has: UsePAM yes
slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so
so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so
We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets:
Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth]
Am I missing something?
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi Robert,
The pam_slurm_adopt has worked well and without bugs for many Slurm versions in the past. You should make sure to follow carefully the instructions in the mentioned wiki page, however.
What is your Slurm version and OS version?
Did you try a different username than "user"?
/Ole
On 18-04-2025 19:09, Robert Kudyba via slurm-users wrote:
Thanks Ole and Massimo, I definitely do not have UsePAM=1 in slurm.conf. I commented outpam_systemdhere: grep pam_systemd * fingerprint-auth:-session optional pam_systemd.so fingerprint-auth-ac:-session optional pam_systemd.so password-auth:#-session optional pam_systemd.so password-auth-ac:#-session optional pam_systemd.so runuser-l:#-session optional pam_systemd.so smartcard-auth:-session optional pam_systemd.so smartcard-auth-ac:-session optional pam_systemd.so system-auth:#-session optional pam_systemd.so system-auth-ac:#-session optional pam_systemd.so
I did enable debug here; sshd:-account sufficient pam_slurm_adopt.so action_unknown=newest log_level=debug5 Latest logs:
Apr 18 13:06:08 node11 sshd[32043]: Authorized to user, krb5 principal user@ADCU.OURUNI.EDU mailto:user@ADCU.OURUNI.EDU (ssh_gssapi_krb5_cmdok) Apr 18 13:06:08 node11 sshd[32043]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Ignoring obsolete CgroupReleaseAgentDir option. Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Reading slurm.conf file: /etc/slurm/slurm.conf Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 4294967295 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 0 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Munge authentication plugin loaded Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Success. Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Connection by user user: user has only one job 4736742 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: _adopt_process: trying to get 4736742.4294967295 to adopt 32043 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Leaving stepd_add_extern_pid Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Leaving stepd_get_x11_display Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Process 32043 adopted into job 4736742 Apr 18 13:06:08 node11 sshd[32043]: fatal: Access denied for user user by PAM account configuration [preauth]
There are a few Slurm bugs mentioning: Connection by user user: user has only one job
But the "only" makes it sound like that's a bad thing?
On Fri, Apr 18, 2025 at 1:07 PM Massimo Sgaravatto <massimo.sgaravatto@gmail.com mailto:massimo.sgaravatto@gmail.com> wrote:
Hi Did you disable the pam_systemd.so also from the module files included by the sshd pam file ? I am asking because I had this problem when I configured the pam_slurm_adopt Cheers, Massimo On Fri, Apr 18, 2025 at 5:28 PM Robert Kudyba via slurm-users <slurm-users@lists.schedmd.com <mailto:slurm- users@lists.schedmd.com>> wrote: In the instructions for pam_slurm_adopt <https:// slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: Add the following line to the appropriate file in /etc/ pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag): This module is configurable. Add these options to the end of the pam_slurm_adopt line in the appropriate file in /etc/ pam.d/ (e.g., sshd or system-auth): Assuming an OS like CentOS does this mean it should be put in both? slurm.conf on the node has: UsePAM yes slurm.conf has PrologFlags=contain and ProctrackType=proctrack/ cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets: Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth] Am I missing something?
SchedMD confirms it's pam_sss denying the user:
Apr 18 12:31:26 node11 sshd[57358]: pam_sss(sshd:account): Access denied
for user user: 6 (Permission denied)
Which is included through:
-- /etc/pam.d/sshd -- account include system-auth -- /etc/pam.d/system-auth account [default=bad success=ok user_unknown=ignore] pam_sss.so
So how can we configure this to work around sssd?
On Sat, Apr 19, 2025 at 4:47 AM Ole Holm Nielsen via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi Robert,
The pam_slurm_adopt has worked well and without bugs for many Slurm versions in the past. You should make sure to follow carefully the instructions in the mentioned wiki page, however.
What is your Slurm version and OS version?
Did you try a different username than "user"?
/Ole
On 18-04-2025 19:09, Robert Kudyba via slurm-users wrote:
Thanks Ole and Massimo, I definitely do not have UsePAM=1 in slurm.conf. I commented outpam_systemdhere: grep pam_systemd * fingerprint-auth:-session optional pam_systemd.so fingerprint-auth-ac:-session optional pam_systemd.so password-auth:#-session optional pam_systemd.so password-auth-ac:#-session optional pam_systemd.so runuser-l:#-session optional pam_systemd.so smartcard-auth:-session optional pam_systemd.so smartcard-auth-ac:-session optional pam_systemd.so system-auth:#-session optional pam_systemd.so system-auth-ac:#-session optional pam_systemd.so
I did enable debug here; sshd:-account sufficient pam_slurm_adopt.so action_unknown=newest log_level=debug5 Latest logs:
Apr 18 13:06:08 node11 sshd[32043]: Authorized to user, krb5 principal user@ADCU.OURUNI.EDU mailto:user@ADCU.OURUNI.EDU
(ssh_gssapi_krb5_cmdok)
Apr 18 13:06:08 node11 sshd[32043]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Ignoring obsolete CgroupReleaseAgentDir option. Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Reading slurm.conf file: /etc/slurm/slurm.conf Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 4294967295 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 0 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Munge authentication plugin loaded Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Success. Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Connection by user user: user has only one job 4736742 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: _adopt_process: trying to get 4736742.4294967295 to adopt 32043 Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Leaving stepd_add_extern_pid Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug: Leaving stepd_get_x11_display Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Process 32043 adopted into job 4736742 Apr 18 13:06:08 node11 sshd[32043]: fatal: Access denied for user user by PAM account configuration [preauth]
There are a few Slurm bugs mentioning: Connection by user user: user has only one job
But the "only" makes it sound like that's a bad thing?
On Fri, Apr 18, 2025 at 1:07 PM Massimo Sgaravatto <massimo.sgaravatto@gmail.com mailto:massimo.sgaravatto@gmail.com>
wrote:
Hi Did you disable the pam_systemd.so also from the module files included by the sshd pam file ? I am asking because I had this problem when I configured the pam_slurm_adopt Cheers, Massimo On Fri, Apr 18, 2025 at 5:28 PM Robert Kudyba via slurm-users <slurm-users@lists.schedmd.com <mailto:slurm- users@lists.schedmd.com>> wrote: In the instructions for pam_slurm_adopt <https:// slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: Add the following line to the appropriate file in /etc/ pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag): This module is configurable. Add these options to the end of the pam_slurm_adopt line in the appropriate file in /etc/ pam.d/ (e.g., sshd or system-auth): Assuming an OS like CentOS does this mean it should be put inboth?
slurm.conf on the node has: UsePAM yes slurm.conf has PrologFlags=contain and ProctrackType=proctrack/ cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets: Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth] Am I missing something?-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Massimo Sgaravatto -
Ole Holm Nielsen -
Robert Kudyba