[slurm-users] free resources

[Ole Holm Nielsen]

On 26-08-2021 09:34, Pankaj Dorlikar wrote:
> However, sinfo -t idle /sinfo -t mix is only giving the node names which are
> idle/mix. To get exact number of cpus / gpus, what command need to be used.

If sinfo doesn't show what you want, perhaps you can try installing my 
"pestat" command from 
https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat

The pestat can display very many job and node properties, see "pestat 
-h".  To display GPU resources available and used, try this one:

$ pestat -t mix -G
Select only nodes with state=mix
GRES (Generic Resource) is printed after each jobid
Hostname       Partition     Node Num_CPU  CPUload  Memsize  Freemem 
GRES/node     Joblist
                             State Use/Tot  (15min)     (MB)     (MB) 
              JobID(JobArrayID) User GRES/job ...
     s002          sm3090      mix  64  80    8.15*   191000   156548 
gpu:RTX3090:10 3941466 s184362 gpu:RTX3090:1 3941465 s184362 
gpu:RTX3090:1 3941387 s184362 gpu:RTX3090:1 3940065 s184362 
gpu:RTX3090:1 3939748 s184362 gpu:RTX3090:1 3939743 s184362 
gpu:RTX3090:1 3941386 s184362 gpu:RTX3090:1 3941385 s184362 gpu:RTX3090:1


> showbf -S
> backfill window (user: 'root' group: 'root' partition: ALL) Thu Aug 26
> 12:53:23
> 
> 410 procs available for       3:57:21
> 218 procs available for       9:38:20
> 194 procs available for    1:10:51:51
> 192 procs available with no timelimit

Again, the Maui command "showbf -S" is undocumented!  Do you have 
documentation for it?

/Ole

> 
> -----Original Message-----
> From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Ole
> Holm Nielsen
> Sent: 26 August 2021 12:41
> To: slurm-users at lists.schedmd.com
> Subject: Re: [slurm-users] free resources
> 
> On 26-08-2021 08:01, Pankaj Dorlikar wrote:
>> We are using slurm-20.11..7 on ubuntu system having GPUs. What is the
>> equivalent of "showbf -S" in maui or any command in slurm for checking
>> the free resources ?
> 
> Maybe "sinfo -t idle"?
> 
> The showbf manual doesn't document any -S flag:
> https://docs.adaptivecomputing.com/maui/commands/showbf.php