[slurm-users] Enforce gpu usage limits (with GRES?)

[Analabha Roy]

Hi,

Thanks, your advice worked. I used sacctmgr to create a QOS called 'nogpu'
and set MaxTRES=gres/gpu=0, then attached it to the cpu partition in
slurm.conf as

PartitionName=CPU Nodes=ALL Default=Yes QOS=nogpu MaxTime=INFINITE  State=UP

And it works! Trying to run gpu jobs in the cpu partition now fails. Qos'es
are nice!

Only thing is that the nogpu qos has a priority of 0. Should it be higher?

https://pastebin.com/VVsQAz6P

AR

On Fri, 3 Feb 2023 at 13:37, Markus Kötter <koetter at cispa.de> wrote:

> Hi,
>
>
> limits ain't easy.
>
> >
> https://support.ceci-hpc.be/doc/_contents/SubmittingJobs/SlurmLimits.html#precedence
>
>
> I think there is multiple options, starting with not having GPU
> resources in the CPU partition.
>
> Or creating qos the partition and have
> MaxTRES=gres/gpu:A100=0,gres/gpu:K80=0,gres/gpu=0
> attaching it to the CPU partition.
>
> And the configuration will require some values as well,
>
> # slurm.conf
> AccountingStorageEnforce=associations,limits,qos,safe
> AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:K80
>
> # cgroups.conf
> ConstrainDevices=yes
>
> most likely some others I miss.
>
>
> MfG
> --
> Markus Kötter, +49 681 870832434
> 30159 Hannover, Lange Laube 6
> Helmholtz Center for Information Security
>


-- 
Analabha Roy
Assistant Professor
Department of Physics
<http://www.buruniv.ac.in/academics/department/physics>
The University of Burdwan <http://www.buruniv.ac.in/>
Golapbag Campus, Barddhaman 713104
West Bengal, India
Emails: daneel at utexas.edu, aroy at phys.buruniv.ac.in, hariseldon99 at gmail.com
Webpage: http://www.ph.utexas.edu/~daneel/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230204/bdcb113c/attachment.htm>