Minimum cpu cores per node partition level configuration
Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam
Hi Jeherul Islam, Such a policy may be implemented using a job_submit plugin which you have to write yourself. You may perhaps find this Wiki page useful: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#job-submit-pl... On 3/28/25 05:36, Jeherul Islam via slurm-users wrote:
I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run.
Any suggestions will be highly appreciated.
IHTH, Ole
You can set a partition QoS which specifies a minimum. We have such a qos on our large-gpu partition; we don’t want people scheduling small stuff to it, so we have this qos: $ sacctmgr show qos large-gpu --json | jq '.QOS[] | { name: .name, min_limits: .limits.min }' { "name": "large-gpu", "min_limits": { "priority_threshold": { "set": false, "infinite": true, "number": 0 }, "tres": { "per": { "job": [ { "type": "cpu", "name": "", "id": 1, "count": 32 }, { "type": "mem", "name": "", "id": 2, "count": 262144 }, { "type": "gres", "name": "gpu", "id": 1002, "count": 3 } ] } } } } i.e. the user has to request 32 cores and at least 3 GPUs. If I try to allocate less I get an error: $ salloc -p large-gpu --gres=gpu:1 -c 32 --mem 256G salloc: error: QOSMinGRES salloc: error: Job submit/allocate failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) -- Tim Cutts Senior Director, R&D IT - Data, Analytics & AI, Scientific Computing Platform AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | From: Jeherul Islam via slurm-users <slurm-users@lists.schedmd.com> Date: Friday, 28 March 2025 at 4:39 am To: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: [slurm-users] Minimum cpu cores per node partition level configuration Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>
Hi Tim, "Cutts, Tim via slurm-users" <slurm-users-rGrgPyRx505G7+FkpxDULAC/G2K4zDHf@public.gmane.org> writes:
You can set a partition QoS which specifies a minimum. We have such a qos on our large-gpu partition; we don’t want people scheduling small stuff to it, so we have this qos:
How does this affect total throughput? Presumably, 'small' GPU jobs might potentially have to wait for resources in other partitions, even if resources are free in 'large-gpu'. Do you other policies which ameliorate this? Cheers, Loris [snip (135 lines)] -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin
As you say, it does reduce overall throughput, because those large-gpu nodes are dedicated. We don’t have many nodes in that partition, the rest are in our standard queues. Ultimately, this is a tradeoff. We have chosen to reduce total throughput slightly in order to make sure that large jobs actually get scheduled in a timely fashion. This is the boulders-and-sand scheduling problem; I don’t think there’s any single perfect configuration, it just depends on what’s most important to your business; throughput, or the turnaround time of particular jobs. Tim -- Tim Cutts Senior Director, R&D IT - Data, Analytics & AI, Scientific Computing Platform AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | From: Loris Bennett via slurm-users <slurm-users@lists.schedmd.com> Date: Friday, 4 April 2025 at 6:37 am To: Slurm Users Mailing List <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: Minimum cpu cores per node partition level configuration Hi Tim, "Cutts, Tim via slurm-users" <slurm-users-rGrgPyRx505G7+FkpxDULAC/G2K4zDHf@public.gmane.org> writes:
You can set a partition QoS which specifies a minimum. We have such a qos on our large-gpu partition; we don’t want people scheduling small stuff to it, so we have this qos:
How does this affect total throughput? Presumably, 'small' GPU jobs might potentially have to wait for resources in other partitions, even if resources are free in 'large-gpu'. Do you other policies which ameliorate this? Cheers, Loris [snip (135 lines)] -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>
participants (4)
-
Cutts, Tim -
Jeherul Islam -
Loris Bennett -
Ole Holm Nielsen