[slurm-users] How to look for free nodes of a certain constraint efficiently

Carsten Beyer beyer at dkrz.de
Thu Oct 14 14:03:57 UTC 2021

Hi Matt,

you may have a look to sinfo/squeue command with the --format / -o 
output options, e.g.:

[root at ma1 slurm]# sinfo -t idle -o "%P %.5a %.10l %.6D %.6t %N %b"
compute    up    8:00:00     44   idle 
prepost    up   12:00:00      0    n/a  (null)
shared    up 7-00:00:00      0    n/a  (null)
gpu    up   12:00:00      5   idle mg[106-109,111] k80,256G
gpu    up   12:00:00      2   idle mg[201,203] m40,512G
gpu    up   12:00:00      1   idle mg207 v100,512G
miklip    up 2-00:00:00      2   idle m[21431,21588] 256G
compute2    up    8:00:00     30   idle 
compute2    up    8:00:00     20   idle m[21167-21168,21368-21385] 128G


Carsten Beyer
Abteilung Systeme

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45a * D-20146 Hamburg * Germany

Phone:  +49 40 460094-221
Fax:    +49 40 460094-270
Email:beyer at dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784

Am 14.10.2021 um 14:44 schrieb Thompson, Matt (GSFC-610.1)[SCIENCE 
> All,
> I work on a cluster that uses SLURM which has various types of nodes 
> that are are controlled via --constraint flags in sbatch.
> Now, I started thinking "How can I figure out how many jobs are 
> running/pending/etc on a certain type of node?". I first thought 
> obviously "squeue --constraint=foo", but...nope. No --constraint flag 
> with squeue. Okay. Constraints are just Features by another name, 
> but...you can't seem to just squeue a feature either.
> I asked a SLURM guru here and they suggested using --nodelist/-w a la:
>   squeue -a -w nodea[001-100],nodeb[001-100],... -t r
> where you pass in all the nodes of a certain type. And, yep, that 
> works! But that also means I have to know what nodes are what type. I 
> could obviously do a one-time parsing of "scontrol show nodes" and see 
> what each chunk is and be done with it...but dangit I'm lazy and SLURM 
> has so many programs and options there might just be something and I 
> haven't read the right manpage! :)
> So I was wondering if anyone out there knows of a 
> cool/elegant/efficient way of doing this?
> Thanks,
> Matt
> PS: I still might write a bash script where I've listed what the node 
> names are of constraint and realize I might have to update it once 
> every year or two. Now time to look at what parser SLURM uses for 
> nodelist. Can you use regexes and use *, etc? Or just use 
> nodea[001-100]? Time to find out!
> -- 
> Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
> NASA GSFC,    Global Modeling and Assimilation Office
> Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
> Phone: 301-614-6712                 Fax: 301-614-6246
> _http://science.gsfc.nasa.gov/sed/bio/matthew.thompson_
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20211014/305a2742/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5316 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20211014/305a2742/attachment-0001.bin>

More information about the slurm-users mailing list