[slurm-users] Requesting total GPUs or memory, not per node.

[Rob Middleton]

Hello,

I'm relatively new to administering slurm, so my apologies if I've missed
something obvious.

We have nodes of 4 GPU and nodes of 8 GPU. I would like users to be able to
request a total number of GPUs they require. The MPI software is not fussed
how many nodes it spans.

I had hoped requests such as these would work:
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --nodes=1-2

However as both "gres" (or an alternate workaround "mem") are per-node
resources rather than per-job this doesn't work -- a pair of 4-GPU boxes
can never be chosen.

So -- is there a way to do this right, or to fake it? Such jobs should run
on whatever appropriate hardware configuration is first available. The
submitted job script will then slightly reconfigure our software
configuration depending on the hardware type it lands on, before launching
via srun.


As an alternative -- I note the "heterogeneous jobs" feature. This allows
jobs which require resources of "hardware config A" AND "hardware config
B". Is there anyway to request one hardware configuration OR another?


I can almost fake it for a single use-case with "constraints", however this
syntax doesn't seem understood by the parser code:
--constraints=[grp1|grp2|grp3|grp4]&[gpuA*1&gpuB*1]
--nodes=1-2
--exclusive

With example node configuration:
NodeName=small1 Gres=gpu:4 Feature=gpuA,grp1
NodeName=small2 Gres=gpu:4 Feature=gpuB,grp1
NodeName=small3 Gres=gpu:4 Feature=gpuB,grp2
NodeName=small4 Gres=gpu:4 Feature=gpuB,grp2
NodeName=big1 Gres=gpu:8 Feature=gpuA,gpuB,grp3
NodeName=big2 Gres=gpu:8 Feature=gpuA,gpuB,grp4


All ideas are appreciated.

Thanks,
Rob Middleton.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20180222/5fb714ec/attachment-0001.html>