[slurm-users] SlurmdSpoolDir

[Kamil Wilczek]

Dear Slurm Users,

recently, I have started a new instance of my cluster with Slurm 22.05.2
(built from source). Evertyhing seems to be configured properly and
working fine except "sbatch". The error is quite self-explanatory and
I thought it would be quite easy to fix directory permissions.

slurmstepd: error: execve(): 
/opt/slurm_state_dir/slurmd_spool/job00136/slurm_script: Permission denied

I read here 
(https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdSpoolDir_1) that 
the directory should
be writable by root. I did that, but it did not help. I tried
several other combinations of permissions, no improvement.

Currently:

# ls -l /opt/
drwxrwx---  3 slurm  root   26 Aug 11 19:49 slurm_state_dir

# tree -pug /opt/slurm_state_dir
/opt/slurm_state_dir
└── [drwxrwx--- root     slurm   ]  slurmd_spool
     ├── [-rw------- root     root    ]  cred_state
     ├── [-rw------- root     root    ]  cred_state.old
     └── [-rw-r--r-- root     root    ]  hwloc_topo_whole.xml

Additionaly, when I change the mode of the slurmd_spool directory,
for example to 770, restarting the slurmd service changes them
back to 755 irrespectively of user/group.

Could somoeone tell what the correct settings should be?
I did not have such problems in using 19.05.

Kind Regards
-- 
Kamil Wilczek