[slurm-users] Strange error, submission denied

[Marcus Wagner]

Hi Andreas,

I get the same result if I set --ntasks-per-node=48 and --ntasks=48, or 
96, or whatever.

What we wanted to achieve is, that exactly ntasks-per-node tasks get 
scheduled onto one host.


Best
Marcus

On 2/14/19 7:09 AM, Henkel, Andreas wrote:
> Hi Marcus,
>
> What just came to my mind: if you don’t set —ntasks isn’t the default just 1? All examples I know using ntasks-per-node also set ntasks with ntasks >= ntasks-per-node.
>
> Best,
> Andreas
>
>> Am 14.02.2019 um 06:33 schrieb Marcus Wagner <wagner at itc.rwth-aachen.de>:
>>
>> Hi all,
>>
>> I have narrowed this down a little bit.
>>
>> the really astonishing thing is, that if I use
>>
>> --ntasks=48
>>
>> I can submit the job, it will be scheduled onto one host:
>>
>>     NumNodes=1 NumCPUs=48 NumTasks=48 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>>     TRES=cpu=48,mem=182400M,node=1,billing=48
>>
>> but as soon as I change --ntasks to --ntasks-per-node (which should be the same, as --ntasks=48 schedules onto one host), I get the error:
>>
>> sbatch: error: CPU count per node can not be satisfied
>> sbatch: error: Batch job submission failed: Requested node configuration is not available
>>
>>
>> Is there no one else, who observes this behaviour?
>> Any explanations?
>>
>>
>> Best
>> Marcus
>>
>>
>>> On 2/13/19 1:48 PM, Marcus Wagner wrote:
>>> Hi all,
>>>
>>> I have a strange behaviour here.
>>> We are using slurm 18.08.5-2 on CentOS 7.6.
>>>
>>> Let me first describe our computenodes:
>>> NodeName=ncm[0001-1032]  CPUs=48  Sockets=4 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=185000 Feature=skx8160,hostok,hpcwork                        Weight=10541 State=UNKNOWN
>>>
>>> we have the following config set:
>>>
>>> $>scontrol show config | grep -i select
>>> SelectType              = select/cons_res
>>> SelectTypeParameters    = CR_CORE_MEMORY,CR_ONE_TASK_PER_CORE
>>>
>>>
>>> So, I have 48 cores on one node. According to the manpage of sbatch, I should be able to do the following:
>>>
>>> #SBATCH --ntasks=48
>>> #SBATCH --ntasks-per-node=48
>>>
>>> But I get the following error:
>>> sbatch: error: Batch job submission failed: Requested node configuration is not available
>>>
>>>
>>> Has anyone an explanation for this?
>>>
>>>
>>> Best
>>> Marcus
>>>
>> -- 
>> Marcus Wagner, Dipl.-Inf.
>>
>> IT Center
>> Abteilung: Systeme und Betrieb
>> RWTH Aachen University
>> Seffenter Weg 23
>> 52074 Aachen
>> Tel: +49 241 80-24383
>> Fax: +49 241 80-624383
>> wagner at itc.rwth-aachen.de
>> www.itc.rwth-aachen.de
>>
>>

-- 
Marcus Wagner, Dipl.-Inf.

IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wagner at itc.rwth-aachen.de
www.itc.rwth-aachen.de