Hello, Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs? We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determinethe order of scheduling for pending jobs. One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time. ( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling orderof jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. ) A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy. Help appreciated. Thanks! Bhaskar.
If you are looking to use the C API for this then showq may be a good guide: https://github.com/fasrc/slurm_showq%C2%A0 The -o option orders the pending queue in priority order.
If you are looking at native slurm commands, sprio can print out the current priority breakdown of any job and filter by partition, then you can reorder based on that. squeue also can print out current priority. You might also look at the --priority option: https://slurm.schedmd.com/squeue.html#OPT_priority
-Paul Edmon-
On 10/29/24 9:33 AM, Bhaskar Chakraborty via slurm-users wrote:
Hello,
Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs?
We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determine the order of scheduling for pending jobs.
One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time.
( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling order of jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. )
A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy.
Help appreciated.
Thanks!
Bhaskar.
I use the following horrible little shell function to give me a reasonable list of currently pending jobs in a partition, and the order they’re likely to run in (as well as individual users’ priorities):
_users_waiting() { squeue "$@" -t PD -h -O UserName | sort -u | tr -d ' ' | tr '\012' ',' | sed 's/,$//' }
priority_in_queue() { local users=$(_users_waiting "$@") echo "User Fairshare" echo "==============" sshare -U -u "$users" echo echo "Pending Job Priorities" echo "======================" sprio "$@" -S "-y" -u "$users" -o "%.8i %9r %.8u %.8o %.8Y %.6A %.10F %.10J %.9P %.7n %.6Q %.30T" }
-- Tim Cutts Scientific Computing Platform Lead AstraZeneca
Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Cataloguehttps://azcollaboration.sharepoint.com/sites/CMU993 |
From: Paul Edmon via slurm-users slurm-users@lists.schedmd.com Date: Tuesday, 29 October 2024 at 2:09 PM To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: Slurm Job Sched Priority
If you are looking to use the C API for this then showq may be a good guide: https://github.com/fasrc/slurm_showqhttps://github.com/fasrc/slurm_showq The -o option orders the pending queue in priority order.
If you are looking at native slurm commands, sprio can print out the current priority breakdown of any job and filter by partition, then you can reorder based on that. squeue also can print out current priority. You might also look at the --priority option: https://slurm.schedmd.com/squeue.html#OPT_priorityhttps://slurm.schedmd.com/squeue.html#OPT_priority
-Paul Edmon- On 10/29/24 9:33 AM, Bhaskar Chakraborty via slurm-users wrote: Hello,
Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs?
We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determine the order of scheduling for pending jobs.
One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time.
( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling order of jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. )
A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy.
Help appreciated.
Thanks!
Bhaskar.
________________________________
AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA.
This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.comhttps://www.astrazeneca.com
Hi Paul,Thanks for the tip. Looking at the code it seems to compare each job’s priorityvariable through its job_ptr. I did some experiments where I log the priority variable in job_ptrfor successive jobs submitted from same user. However, I find them equal and so they don’t reflect the first-come-first-serve order of the jobs. Is there any other way through C APIs to gather the dynamic priority? If we can at least find the start and end of a scheduling iteration then it can help usas we can evaluate the relative order of scheduling ourselves, but this is more of a 2nd best solution or a workaround. -Bhaskar.
Sent from Yahoo Mail for iPad
On Tuesday, October 29, 2024, 7:43 PM, Paul Edmon via slurm-users slurm-users@lists.schedmd.com wrote:
If you are looking to use the C API for this then showq may be a good guide: https://github.com/fasrc/slurm_showq%C2%A0 The -o option orders the pending queue in priority order.
If you are looking at native slurm commands, sprio can print out the current priority breakdown of any job and filter by partition, then you can reorder based on that. squeue also can print out current priority. You might also look at the --priority option: https://slurm.schedmd.com/squeue.html#OPT_priority
-Paul Edmon-
On 10/29/24 9:33 AM, Bhaskar Chakraborty via slurm-users wrote:
Hello, Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs? We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determine the order of scheduling for pending jobs. One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time. ( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling order of jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. ) A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy. Help appreciated. Thanks! Bhaskar.
Sadly I don't have any deeper insight into the C API for that information.
-Paul Edmon-
On 11/3/2024 2:14 AM, Bhaskar Chakraborty wrote:
Hi Paul, Thanks for the tip. Looking at the code it seems to compare each job’s priority variable through its job_ptr.
I did some experiments where I log the priority variable in job_ptr for successive jobs submitted from same user.
However, I find them equal and so they don’t reflect the first-come-first-serve order of the jobs.
Is there any other way through C APIs to gather the dynamic priority?
If we can at least find the start and end of a scheduling iteration then it can help us as we can evaluate the relative order of scheduling ourselves, but this is more of a 2nd best solution or a workaround.
-Bhaskar.
Sent from Yahoo Mail for iPad https://mail.onelink.me/107872968?pid=nativeplacement&c=Global_Acquisition_YMktg_315_Internal_EmailSignature&af_sub1=Acquisition&af_sub2=Global_YMktg&af_sub3=&af_sub4=100000604&af_sub5=EmailSignature__Static_
On Tuesday, October 29, 2024, 7:43 PM, Paul Edmon via slurm-users slurm-users@lists.schedmd.com wrote:
If you are looking to use the C API for this then showq may be a good guide: https://github.com/fasrc/slurm_showq <https://github.com/fasrc/slurm_showq> The -o option orders the pending queue in priority order. If you are looking at native slurm commands, sprio can print out the current priority breakdown of any job and filter by partition, then you can reorder based on that. squeue also can print out current priority. You might also look at the --priority option: https://slurm.schedmd.com/squeue.html#OPT_priority <https://slurm.schedmd.com/squeue.html#OPT_priority> -Paul Edmon- On 10/29/24 9:33 AM, Bhaskar Chakraborty via slurm-users wrote: Hello, Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs? We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determine the order of scheduling for pending jobs. One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time. ( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling order of jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. ) A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy. Help appreciated. Thanks! Bhaskar. -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Bhaskar Chakraborty -
Cutts, Tim -
Paul Edmon