dynamic node slurm node list order
Hello, we would like to understand how the internal SLURM node list order works. More detailed info: Our setup: We have a Slurm cluster with N dynamic nodes (heterogeneous node types) in a partition and we observe that if all resources are free, jobs always get submitted to one specific node first. This node is also the node that turns up at the top of the list when doing an $ scontrol show nodes . As we do not have a defined node list in slurm.conf as it is done for non-dynamic nodes, we were wondering how the order is set up. Interestingly, when deleting (and unregistering) this node from the cluster and re-registering it again, the node is still at the same position when doing scontrol show nodes. Is there some internal node list order caching or similar? How is the node list order defined ? Thanks , Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann@psi.ch
You can get a node list by scontrol show node Not sure if that order is a definitive one But to resolve this issue you can try this all node probably have the same node weight Slurm pick first available node of the minimum node weight You can manipulate the node weight change which node is picked Either a cron job to randomise it or a prologslurmctld script to increment it.
On 10/22/25 14:41, Patrick Pun via slurm-users wrote:
You can get a node list by scontrol show node Not sure if that order is a definitive one
But to resolve this issue you can try this all node probably have the same node weight Slurm pick first available node of the minimum node weight
There's a description of this in https://slurm.schedmd.com/slurm.conf.html#OPT_Weight
All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements.
/Ole
Dear Patrick and Ole, Thanks a lot for your fast answers! As I mentioned in my email, I used scontrol show node(s) in order to display the current order - I was more interested in understanding how the order is actually defined/set up by SLURM , as it is not defined in slurm.conf for dynamic nodes. We assumed that it might be related to the time the node registers, but as I unregistered and re-registered and the node stayed in the same "order" I was wondering if there is some internal caching... Also -Yes, you are right, we have the same Weights for all nodes - but we would like to avoid setting weights for our setup, only if not possible otherwise...- but thanks again for the hint! So I would still like to clarify: * How SLURM determines the internal node order when nodes register dynamically given all weights/configs are the same ? (kinda : who/what defines the output of scrontrol show nodes) * Whether this order persists across node or controller restarts (i.e., if SLURM caches previous node information)? Best, Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann@psi.ch ________________________________ From: Ole Holm Nielsen via slurm-users <slurm-users@lists.schedmd.com> Sent: 22 October 2025 15:02 To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: dynamic node slurm node list order On 10/22/25 14:41, Patrick Pun via slurm-users wrote:
You can get a node list by scontrol show node Not sure if that order is a definitive one
But to resolve this issue you can try this all node probably have the same node weight Slurm pick first available node of the minimum node weight
There's a description of this in https://slurm.schedmd.com/slurm.conf.html#OPT_Weight
All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements.
/Ole -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
I suspect it is the order of registration. So, even if you unregister a node, it still exists internally. Even if you delete the node, it would still be in the db with a 'deleted' flag. I haven't dug into the code to validate, but that is what I have gone with and it hasn't been off for me. Brian Andrus On 10/22/2025 6:13 AM, Assmann, Greta Marie via slurm-users wrote:
Dear Patrick and Ole,
Thanks a lot for your fast answers!
As I mentioned in my email, I used *scontrol show node(s) * in order to display the current order - I was more interested in understanding *how *the order is actually defined/set up by SLURM , as it is not defined in slurm.conf for dynamic nodes. We assumed that it might be related to the time the node registers, but as I unregistered and re-registered and the node stayed in the same "order" I was wondering if there is some internal caching...
Also -Yes, you are right, we have the same Weights for all nodes - but we would like to avoid setting weights for our setup, only if not possible otherwise...- but thanks again for the hint!
So I would still like to clarify:
* How SLURM determines the internal node order when nodes register dynamically given all weights/configs are the same ? (kinda : who/what defines the output of scrontrol show nodes) * Whether this order persists across node or controller restarts (i.e., if SLURM caches previous node information)?
Best, Greta
Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann@psi.ch
------------------------------------------------------------------------ *From:* Ole Holm Nielsen via slurm-users <slurm-users@lists.schedmd.com> *Sent:* 22 October 2025 15:02 *To:* slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> *Subject:* [slurm-users] Re: dynamic node slurm node list order On 10/22/25 14:41, Patrick Pun via slurm-users wrote:
You can get a node list by scontrol show node Not sure if that order is a definitive one
But to resolve this issue you can try this all node probably have the same node weight Slurm pick first available node of the minimum node weight
There's a description of this in https://slurm.schedmd.com/slurm.conf.html#OPT_Weight
All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements.
/Ole
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thanks, Brian, That matches our experience.. Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann@psi.ch ________________________________ From: Brian Andrus via slurm-users <slurm-users@lists.schedmd.com> Sent: 22 October 2025 22:39 To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: dynamic node slurm node list order I suspect it is the order of registration. So, even if you unregister a node, it still exists internally. Even if you delete the node, it would still be in the db with a 'deleted' flag. I haven't dug into the code to validate, but that is what I have gone with and it hasn't been off for me. Brian Andrus On 10/22/2025 6:13 AM, Assmann, Greta Marie via slurm-users wrote: Dear Patrick and Ole, Thanks a lot for your fast answers! As I mentioned in my email, I used scontrol show node(s) in order to display the current order - I was more interested in understanding how the order is actually defined/set up by SLURM , as it is not defined in slurm.conf for dynamic nodes. We assumed that it might be related to the time the node registers, but as I unregistered and re-registered and the node stayed in the same "order" I was wondering if there is some internal caching... Also -Yes, you are right, we have the same Weights for all nodes - but we would like to avoid setting weights for our setup, only if not possible otherwise...- but thanks again for the hint! So I would still like to clarify: * How SLURM determines the internal node order when nodes register dynamically given all weights/configs are the same ? (kinda : who/what defines the output of scrontrol show nodes) * Whether this order persists across node or controller restarts (i.e., if SLURM caches previous node information)? Best, Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann@psi.ch<mailto:greta.assmann@psi.ch> ________________________________ From: Ole Holm Nielsen via slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Sent: 22 October 2025 15:02 To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: dynamic node slurm node list order On 10/22/25 14:41, Patrick Pun via slurm-users wrote:
You can get a node list by scontrol show node Not sure if that order is a definitive one
But to resolve this issue you can try this all node probably have the same node weight Slurm pick first available node of the minimum node weight
There's a description of this in https://slurm.schedmd.com/slurm.conf.html#OPT_Weight
All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements.
/Ole -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>
participants (4)
-
Assmann, Greta Marie -
Brian Andrus -
Ole Holm Nielsen -
Patrick Pun