[slurm-users] SLURM disregards LDAP configured via SSSD

[Leopold Talirz]

Hi,

I have an issue with SLURM (20.11.9) in conjunction with LDAP user accounts.

Both the scheduler node, where slurmctld is running, and the worker nodes
that are spun up by slurm are running the SSSD, which fetches user accounts
from an external LDAP server.

This works fine: I can log into the scheduler _and_ the worker nodes using
SSH as an LDAP user without problems.
This does not work: If, instead of SSH, I connect to a worker node via a
slurm job, i.e. using `srun` (or `sbatch`), I get

whoami: cannot find name for user ID 1290486416

It seems that, for some reason, SLURM does not rely on the same
authentication mechanism (configured via /etc/pam.d/*) as SSH.

Any ideas what may be causing this or which logs I should be looking at to
understand what is going on here?

Potentially relevant further information:
- The scheduler is running CentOS 7.9 (meaning /etc/pam.d is configured via
the older authconfig), while the worker nodes are running AlmaLinux 8.7
(meaning /etc/pam.d is configured via the newer authselect). As described
above, both work fine when connecting via SSH, but I don't know whether
slurm imposes additional requirements between the scheduler VM and the
workers.
- After I log in via SSH to one of the worker nodes for the first time,
`srun` then also starts working (it recognizes the user account, apparently
it is now seeing it in some cache). However, there are still differences
between the user state when logging via SSH and via srun - for example,
when using `srun` the user account does not have access to /dev/nvidia*
devices, i.e. nvidia-smi shows "no devices found", while logging in via SSH
shows the devices correctly.

Best wishes,
Leopold
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20231010/4bb146a9/attachment-0001.htm>