[slurm-users] How to look for free nodes of a certain constraint efficiently

[Carsten Beyer]

Hi Matt,

you may have a look to sinfo/squeue command with the --format / -o 
output options, e.g.:

[root at ma1 slurm]# sinfo -t idle -o "%P %.5a %.10l %.6D %.6t %N %b"
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST ACTIVE_FEATURES
compute    up    8:00:00     44   idle 
m[10474-10475,10594-10596,10598,10600-10608,10924-10925,10927,10934-10935,10990-11013] 
64G
prepost    up   12:00:00      0    n/a  (null)
shared    up 7-00:00:00      0    n/a  (null)
gpu    up   12:00:00      5   idle mg[106-109,111] k80,256G
gpu    up   12:00:00      2   idle mg[201,203] m40,512G
gpu    up   12:00:00      1   idle mg207 v100,512G
miklip    up 2-00:00:00      2   idle m[21431,21588] 256G
compute2    up    8:00:00     30   idle 
m[20058,20110-20112,20233,20373-20375,20513,20521,20706,20709,20828-20831,20936-20941,20943-20944,21047,21053-21055,21733-21734] 
64G
compute2    up    8:00:00     20   idle m[21167-21168,21368-21385] 128G

Cheers,
Carsten

-- 
Carsten Beyer
Abteilung Systeme

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45a * D-20146 Hamburg * Germany

Phone:  +49 40 460094-221
Fax:    +49 40 460094-270
Email:beyer at dkrz.de
URL:http://www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784


Am 14.10.2021 um 14:44 schrieb Thompson, Matt (GSFC-610.1)[SCIENCE 
SYSTEMS AND APPLICATIONS INC]:
>
> All,
>
> I work on a cluster that uses SLURM which has various types of nodes 
> that are are controlled via --constraint flags in sbatch.
>
> Now, I started thinking "How can I figure out how many jobs are 
> running/pending/etc on a certain type of node?". I first thought 
> obviously "squeue --constraint=foo", but...nope. No --constraint flag 
> with squeue. Okay. Constraints are just Features by another name, 
> but...you can't seem to just squeue a feature either.
>
> I asked a SLURM guru here and they suggested using --nodelist/-w a la:
>
>   squeue -a -w nodea[001-100],nodeb[001-100],... -t r
>
> where you pass in all the nodes of a certain type. And, yep, that 
> works! But that also means I have to know what nodes are what type. I 
> could obviously do a one-time parsing of "scontrol show nodes" and see 
> what each chunk is and be done with it...but dangit I'm lazy and SLURM 
> has so many programs and options there might just be something and I 
> haven't read the right manpage! :)
>
> So I was wondering if anyone out there knows of a 
> cool/elegant/efficient way of doing this?
>
> Thanks,
>
> Matt
>
> PS: I still might write a bash script where I've listed what the node 
> names are of constraint and realize I might have to update it once 
> every year or two. Now time to look at what parser SLURM uses for 
> nodelist. Can you use regexes and use *, etc? Or just use 
> nodea[001-100]? Time to find out!
>
> -- 
>
> Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
>
> NASA GSFC,    Global Modeling and Assimilation Office
>
> Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
>
> Phone: 301-614-6712                 Fax: 301-614-6246
>
> _http://science.gsfc.nasa.gov/sed/bio/matthew.thompson_
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20211014/305a2742/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5316 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20211014/305a2742/attachment-0001.bin>