[slurm-users] using resources effectively?

[Weijun Gao]

Thanks you Michael!

I've tried the following example:

     NodeName=gpunode01 Gres=gpu:1 Sockets=2 CoresPerSocket=28 
ThreadsPerCore=2 State=UNKNOWN RealMemory=380000
     PartitionName=gpu MaxCPUsPerNode=56 MaxMemPerNode=190000 
Nodes=gpunode01 Default=NO MaxTime=1-0 State=UP
     PartitionName=cpu MaxCPUsPerNode=56 MaxMemPerNode=190000 
Nodes=gpunode01 Default=YES MaxTime=1-0 State=UP

1) So when the system is idling, the following "gpu" job will start 
immediately ("gpu" partition, 1 GPU, 20 CPUs):

     srun -p gpu --gpus=1 -c 20 --pty bash -i

2) If I run the same command again, it will be queued ... this is normal 
("gpu" partition, 1 GPU, 20 CPUs):

     srun -p gpu --gpus=1 -c 20 --pty bash -i

3) Then the following "cpu" job will be queued too ("cpu" partition, 20 
x CPUs):

     srun -p cpu --gpus=0 -c 20 --pty bash -i

Is there a way to let the "cpu" job run instead of waiting?

Any suggestions?

Thanks again,

Weijun

On 12/16/2020 2:54 PM, Renfro, Michael wrote:
> *EXTERNAL EMAIL:*
>
> We have overlapping partitions for GPU work and some kinds non-GPU 
> work (both large memory and regular memory jobs).
>
> For 28-core nodes with 2 GPUs, we have:
>
> PartitionName=gpu MaxCPUsPerNode=16 … Nodes=gpunode[001-004]
>
> PartitionName=any-interactive MaxCPUsPerNode=12 … 
> Nodes=node[001-040],gpunode[001-004]
>
> PartitionName=bigmem MaxCPUsPerNode=12 … Nodes=gpunode[001-003]
>
> PartitionName=hugemem MaxCPUsPerNode=12 … Nodes=gpunode004
>
> Worst case, non-GPU jobs could reserve up to 24 of the 28 cores on a 
> GPU node, but only for a limited time (our any-interactive partition 
> has a 2 hour time limit). In practice, it's let us use a lot of 
> otherwise idle CPU capacity in the GPU nodes for short test runs.
>
> *From: *slurm-users <slurm-users-bounces at lists.schedmd.com>
> *Date: *Wednesday, December 16, 2020 at 1:04 PM
> *To: *Slurm User Community List <slurm-users at lists.schedmd.com>
> *Subject: *[slurm-users] using resources effectively?
>
> External Email Warning
>
> This email originated from outside the university. Please use caution 
> when opening attachments, clicking links, or responding to requests.
>
> ________________________________
>
> Hi,
>
> Say if I have a Slurm node with 1 x GPU and 112 x CPU cores, and:
>
>      1) there is a job running on the node using the GPU and 20 x CPU 
> cores
>
>      2) there is a job waiting in the queue asking for 1 x GPU and 20 x
> CPU cores
>
> Is it possible to a) let a new job asking for 0 x GPU and 20 x CPU cores
> (safe for the queued GPU job) start immediately; and b) let a new job
> asking for 0 x GPU and 100 x CPU cores (not safe for the queued GPU job)
> wait in the queue? Or c) is it doable to put the node into two Slurm
> partitions, 56 CPU cores to a "cpu" partition, and 56 CPU cores to a
> "gpu" partition, for example?
>
> Thank you in advance for any suggestions / tips.
>
> Best,
>
> Weijun
>
> ===========
> Weijun Gao
> Computational Research Support Specialist
> Department of Psychology, University of Toronto Scarborough
> 1265 Military Trail, Room SW416
> Toronto, ON M1C 1M2
> E-mail: weijun.gao at utoronto.ca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201216/ef574a15/attachment.htm>