[slurm-users] [EXT]Re: only 1 job running

[Chandler]

Thanks for the explanation Brian.  Seems turning on IOMMU helped, as well as added sharing to slurm.conf:

SelectType=select/cons_res
SelectTypeParameters=CR_CPU

Now all the CPUs are being used on all the compute nodes so things are working as expected.

Thanks to everyone else on the list who helped also, Andy, Ole, Chris, appreciate it!  Looking forward to helping out where I can as well.

Brian Andrus wrote on 1/28/21 15:50:
> External Email
> 
> Yep, Looks like you are on the right track.
> 
> If the CPU count does not make sense to slurm, it will drain the node and jobs will not be able to start on them.
> 
> There does seem more to it though. Detailed info about a job and node would help.
> 
> The 'priority' pending jobs, you can ignore. Those aren't starting because another job is supposed to go first. That is the one with 'Resources' as the reason.
> 
> Resources means the scheduler has allocated the resources on the node such that there aren't any left to be used.
> My bet here is that you aren't specifying memory. If you don't specify it, slurm assumes all memory on the node for the job. So, even if you are only using 1 cpu, all the memory is allocated, leaving none for any other job to run on the unallocated cpus.
> 
> Brian Andrus
> 
> On 1/28/2021 2:15 PM, Chandler wrote:
>>
>> Brian Andrus wrote on 1/28/21 13:59:
>>> What are the specific requests for resources from a job?
>>> Nodes, Cores, Memory, threads, etc?
>>
>> Well the jobs are only asking for 16 CPUs each.  The 255 threads is weird though, seems to be related to this,
>> https://askubuntu.com/questions/1182818/dual-amd-epyc-7742-cpus-show-only-255-threads
>>
>> The vendor recommended to turn on IOMMU in the BIOS so I will try that and see if it helps....
>>
>