[slurm-users] salloc problem

[Gizo Nanava]

Sorry for this very late response.

The directory where job containers are to be created is of course already there - it is the local filesystem.
We also start slurmd as a very last process once a node is ready to accept jobs.
That seems to be either a feature of salloc or a bug in Slurm, presumable caused by some race conditions - 
in very rare cases, salloc works without this issue.
I see that doc on the Slurm power saving mentions about salloc, but not for the case of interactive use of it.

Thank you & best regards
Gizo

> On 27/10/22 4:18 am, Gizo Nanava wrote:
> 
> > we run into another issue when using salloc interactively on a cluster where Slurm
> > power saving is enabled. The problem seems to be caused by the job_container plugin
> > and occurs when the job starts on a node which boots from a power down state.
> > If I resubmit a job immediately after the failure to the same node, it always works.
> > I can't find any other way to reproduce the issue other than booting a reserved node from a power down state.
> 
> Looking at this:
> 
> > slurmstepd: error: container_p_join: open failed for /scratch/job_containers/791670/.ns: No such file or directory
> 
> I'm wondering is a separate filesystem and, if so, could /scratch be 
> only getting mounted _after_ slurmd has started on the node?
> 
> If that's the case then it would explain the error and why it works 
> immediately after.
> 
> On our systems we always try and ensure that slurmd is the very last 
> thing to start on a node, and it only starts if everything has succeeded 
> up to that point.
> 
> All the best,
> Chris
> -- 
> Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA
> 
>