Thanks Ole and Massimo, I definitely do not have UsePAM=1 in slurm.conf. I commented out pam_systemd here:
grep pam_systemd *
fingerprint-auth:-session     optional      pam_systemd.so
fingerprint-auth-ac:-session     optional      pam_systemd.so
password-auth:#-session     optional      pam_systemd.so
password-auth-ac:#-session     optional      pam_systemd.so
runuser-l:#-session optional pam_systemd.so
smartcard-auth:-session     optional      pam_systemd.so
smartcard-auth-ac:-session     optional      pam_systemd.so
system-auth:#-session   optional pam_systemd.so
system-auth-ac:#-session   optional pam_systemd.so

I did enable debug here;
sshd:-account   sufficient      pam_slurm_adopt.so action_unknown=newest  log_level=debug5
Latest logs:

Apr 18 13:06:08 node11 sshd[32043]: Authorized to user, krb5 principal user@ADCU.OURUNI.EDU (ssh_gssapi_krb5_cmdok)
Apr 18 13:06:08 node11 sshd[32043]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied)
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  Reading cgroup.conf file /etc/slurm/cgroup.conf
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  Ignoring obsolete CgroupReleaseAgentDir option.
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  Reading slurm.conf file: /etc/slurm/slurm.conf
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 4294967295
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug4: found jobid = 4736742, stepid = 0
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  Munge authentication plugin loaded
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug3: Success.
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Connection by user user: user has only one job 4736742
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  _adopt_process: trying to get 4736742.4294967295 to adopt 32043
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  Leaving stepd_add_extern_pid
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: debug:  Leaving stepd_get_x11_display
Apr 18 13:06:08 node11 pam_slurm_adopt[32043]: Process 32043 adopted into job 4736742
Apr 18 13:06:08 node11 sshd[32043]: fatal: Access denied for user user by PAM account configuration [preauth]


There are a few Slurm bugs mentioning:
Connection by user user: user has only one job

But the "only" makes it sound like that's a bad thing?

On Fri, Apr 18, 2025 at 1:07 PM Massimo Sgaravatto <massimo.sgaravatto@gmail.com> wrote:
Hi

Did you disable the pam_systemd.so also from the module files included by the sshd pam file ?
I am asking because I had this problem when I configured the pam_slurm_adopt 

Cheers, Massimo


On Fri, Apr 18, 2025 at 5:28 PM Robert Kudyba via slurm-users <slurm-users@lists.schedmd.com> wrote:
In the instructions for pam_slurm_adopt, there are instructions such as:
Add the following line to the appropriate file in /etc/pam.d, such as system-auth or sshd (you may use either the "required" or "sufficient" PAM control flag):

This module is configurable. Add these options to the end of the pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd or system-auth):
Assuming an OS like CentOS does this mean it should be put in both?

slurm.conf on the node has:
UsePAM yes

slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup
I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack.
#%PAM-1.0
auth       required pam_sepermit.so
auth       substack system-auth
auth       include postlogin
# Used with polkit to reauthorize users in remote sessions
-auth      optional pam_reauthorize.so prepare
account    required pam_nologin.so
account    include system-auth
-account   required      pam_slurm_adopt.so


so pam_sss.so is at the bottom of  /etc/pam.d/sshd 
session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
#-session     optional      pam_systemd.so
session     optional      pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_sss.so


We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets:

Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user@ouruni.EDU (ssh_gssapi_krb5_cmdok)
Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied)
Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth]


Am I missing something? 

--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com