[slurm-users] Problem with sbatch

[Jeffrey Frey]

> So, if I understand this correctly, for some reason, `srun` does not need root privileges on the computation node side, but `sbatch` does when scheduling. I was afraid doing so would mean users could do things such as apt install and such, but it does not seem the case.

The critical part here is slurmstepd running as one user (root, or slurm in your case) attempting to seteuid/setegid/setreuid/setregid the process.  If not running as root, you'll see in the slurmstepd source code that the seteuid/setegid/setreuid/setregid calls are skipped -- thus, all job tasks run as SlurmdUser.  On a multiuser system, SlurmdUser=root is necessary and safe.



> I am not going to be managing the actual cluster, only exploring possibilities. At this point I am mostly convinced slurmdUser=sudo is safe, so that is one less potential problem.

SlurmdUser must be set to a user name present on the machine.  Setting it to "sudo" merely has it run as a user named "sudo."


> @Patrick: I do not know how to do that. I only know that I can make slurm sudoer and NOPASSWD, but slurm would still call to `chown` (not `sudo chown`). An alternative would be replacing `chown` with a small script that calls `sudo chown`, but that is likely to break a lot of stuff. I assume slurmd will also need other root-only commands to work.

The chown in question is a chown() system call in the slurmd/slurmstepd source code that's compiled into slurmd/slurmstepd.  Replacing /usr/bin/chown with a custom command that calls sudo would not help at all (and would probably create a security issue).


> @Michael Indeed, the documentation/tutorials often mention that SlurmdUser should be root, but it is not clearly explained why anywhere (e.g. https://slurm.schedmd.com/quickstart_admin.html section Daemons). It seems that `srun whoami` returns the current user (and not root), so even when slurmdUser is root, users do not have privileges, so in principle there is no problem at all.

Again, the critical part here is slurmstepd's having the ability to change the running uid/gid of the processes it starts.  With SlurmdUser=root, it can do this.  With any other user name, it cannot, and every job step runs as SlurmdUser.



> @Jeffrey It is expected to be multi-user. As for your third option, I think you refer to something similar to what I wrote for Patrick. 

I was talking about POSIX capabilities, and the possibility that all the capabilities exist to grant to slurmstepd the ability to chown() like root can, etc.  That still doesn't help with the seteuid/setegid/setreuid/setregid calls because slurmstepd doesn't even make those calls when it's not running as root.




::::::::::::::::::::::::::::::::::::::::::::::::::::::
Jeffrey T. Frey, Ph.D.
Systems Programmer V / HPC Management
Network & Systems Services / College of Engineering
University of Delaware, Newark DE  19716
Office: (302) 831-6034  Mobile: (302) 419-4976
::::::::::::::::::::::::::::::::::::::::::::::::::::::