[slurm-users] Why does Slurm kill one particular user's jobs after a few seconds?

Ole Holm Nielsen Ole.H.Nielsen at fysik.dtu.dk
Thu Apr 15 07:07:24 UTC 2021

Hi Thomas,

I wonder if your problem is related to that reported in this list thread?

You could try to restart the slurmctld service, and also make sure your 
configuration (slurm.conf etc.) has been pushed correctly to the slurmd nodes.


On 4/14/21 9:53 AM, Thomas Arildsen wrote:
> Oh and I forgot to mention that we are using Slurm version 20.11.3.
> Best,
> Thomas
> ons, 14 04 2021 kl. 09:23 +0200, skrev Thomas Arildsen:
>> I administer a Slurm cluster with many users and the operation of the
>> cluster currently appears "totally normal" for all users; except for
>> one. This one user gets all attempts to run commands through Slurm
>> killed after 20-25 seconds (I think the cause is another job - not so
>> much the time, see further down).
>> The following minimal example reproduces the error:
>>      $ sudo -u <the_user> srun --pty sleep 25
>>      srun: job 110962 queued and waiting for resources
>>      srun: job 110962 has been allocated resources
>>      srun: Force Terminated job 110962
>>      srun: Job step aborted: Waiting up to 32 seconds for job step to
>> finish.
>>      slurmstepd: error: *** STEP 110962.0 ON <node> CANCELLED AT 2021-
>> 04-09T16:33:35 ***
>>      srun: error: <node>: task 0: Terminated
>> When this happens, I find this line in the slurmctld log:
>>      _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=110962 uid
>> <the_users_uid>
>> It only happens for '<the_user>' and not for any other user that I know
>> of. This very similar but shorter-running example works fine:
>>      $ sudo -u <the_user> srun --pty sleep 20
>>      srun: job 110963 queued and waiting for resources
>>      srun: job 110963 has been allocated resources
>> Note that when I run srun --pty sleep 20 as myself, srun does not
>> output the two srun: job... lines. This seems to me to be an additional
>> indication that srun is subject to some different settings for
>> '<the_user>'.
>> All settings that I have been able to inspect appear identical for
>> '<the_user>' as for other users. I have checked, and 'MaxWall' is not
>> set for this user and not for any other user, either. Other users
>> belonging to the same Slurm account do not experience this problem.
>> When this unfortunate user's jobs get allocated, I see messages like
>> this in
>> '/var/log/slurm/slurmctld.log':
>>      sched: _slurm_rpc_allocate_resources JobId=111855 NodeList=<node>
>> and shortly after, I see this message:
>>      select/cons_tres: common_job_test: no job_resources info for
>> JobId=110722_* rc=0
>> Job 110722_* is a pending array job by another user that is pending due
>> to 'QOSMaxGRESPerUser'. One pending part of this array job (110722_57)
>> eventually ends up taking over job 111855's CPU cores when 111855 gets
>> killed. This leads me to believe that 110722_57 causes 111855 to be
>> killed. However, 110722_57 remains pending afterwards.
>> Some of the things I fail to understand here are:
>>    - Why does a pending job kill another job, yet remains pending
>> afterwards?
>>    - Why does the pending job even have privileges to kill another job
>> in the first place?
>>    - Why does this only affect '<the_user>'s jobs but not those of other
>> users?
>> None of this is intended to happen. I am guessing it must be caused by
>> some settings specific to '<the_user>', but I cannot figure out what
>> they are and they are not supposed to be like this. If these are
>> settings we admins somehow caused, it was unintended.
>> NB: some details have been anonymized as <something> above.
>> I hope someone has a clue what is going on here. Thanks in advance,
>> Thomas Arildsen

More information about the slurm-users mailing list