[slurm-users] how can users start their worker daemons using srun?

[Chris Samuel]

On Saturday, 1 September 2018 2:33:39 AM AEST Priedhorsky, Reid wrote:

> That is, it exceeds both the CPU count (1) and memory (1KiB) that I told
> Slurm it would use. This is what I want. Is allowing such exceedance a
> common configuration? I don’t want to rely on quirks of our site.

I think you can configure Slurm to do that, but in my experience sites are 
always doing their best to constrain jobs to what they ask for and so we use 
cgroups for this (tasks can only access the cores, memory and GPUs they 
request and the kernel will prevent them accessing anything else).

For your situation using using CR_Core as your SelectTypeParameters basically 
tells Slurm to ignore memory for scheduling.

> The drawback here is that for real daemons, I’ll need “sleep infinity”, so
> I’ll need to manually kill the srun. So, this is still a workaround. The
> ideal behavior would be to have Slurm not clean up processes when the job
> step completes, but instead at the end of the job.

You've got a race condition here though then - the job doesn't complete until 
all the steps are done, and if you've got a step with processes that never end 
then the job will keep running until it hits its time limit (unless, as you 
say, you manually kill that step yourself).

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC