[slurm-users] Submitting to multiple paritions problem with gres specified

[Bas van der Vlies]

Hi,

On this cluster I have version 20.02.6 installed. We have different 
partitions for cpu type and gpu types. we want to make it easy for the 
user who not care where there job runs and for the experienced user they 
can specify the gres type: cpu_type or gpu

I have defined 2 cpu partitions:
  * cpu_e5_2650_v1
  * cpu_e5_2650_v2

and 2 gres cpu_type:
  * e5_2650_v1
  * e5_2650_v2


When no partitions are specified it will submit to both partitions:
  * srun --exclusive  --gres=cpu_type:e5_2650_v1  --pty /bin/bash  --> 
r16n18 wich has defined this gres and is in partition cpu_e5_2650_v1

Now I submit at the same time another job:
  * srun --exclusive  --gres=cpu_type:e5_2650_v1  --pty /bin/bash

This fails with: `srun: error: Unable to allocate resources: Requested 
node configuration is not available`

I would expect it gets queued in the partition `cpu_e5_2650_v1`.


When I specify the partition on the command line:
  * srun  --exclusive -p cpu_e5_2650_v1_shared 
--gres=cpu_type:e5_2650_v1 --pty /bin/bash

srun: job 1856 queued and waiting for resources


So the question is can slurm handle submitting to multiple partitions 
when we specify gres attributes?

Regards


-- 
Bas van der Vlies
| HPCV Supercomputing | Internal Services  | SURF | 
https://userinfo.surfsara.nl |
| Science Park 140 | 1098 XG Amsterdam | Phone: +31208001300 |
|  bas.vandervlies at surf.nl