[slurm-users] using resources effectively?

[Renfro, Michael]

We have overlapping partitions for GPU work and some kinds non-GPU work (both large memory and regular memory jobs).

For 28-core nodes with 2 GPUs, we have:

PartitionName=gpu MaxCPUsPerNode=16 … Nodes=gpunode[001-004]
PartitionName=any-interactive MaxCPUsPerNode=12 … Nodes=node[001-040],gpunode[001-004]
PartitionName=bigmem MaxCPUsPerNode=12 … Nodes=gpunode[001-003]
PartitionName=hugemem MaxCPUsPerNode=12 … Nodes=gpunode004

Worst case, non-GPU jobs could reserve up to 24 of the 28 cores on a GPU node, but only for a limited time (our any-interactive partition has a 2 hour time limit). In practice, it's let us use a lot of otherwise idle CPU capacity in the GPU nodes for short test runs.

From: slurm-users <slurm-users-bounces at lists.schedmd.com>
Date: Wednesday, December 16, 2020 at 1:04 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: [slurm-users] using resources effectively?
External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

________________________________

Hi,

Say if I have a Slurm node with 1 x GPU and 112 x CPU cores, and:

     1) there is a job running on the node using the GPU and 20 x CPU cores

     2) there is a job waiting in the queue asking for 1 x GPU and 20 x
CPU cores

Is it possible to a) let a new job asking for 0 x GPU and 20 x CPU cores
(safe for the queued GPU job) start immediately; and b) let a new job
asking for 0 x GPU and 100 x CPU cores (not safe for the queued GPU job)
wait in the queue? Or c) is it doable to put the node into two Slurm
partitions, 56 CPU cores to a "cpu" partition, and 56 CPU cores to a
"gpu" partition, for example?

Thank you in advance for any suggestions / tips.

Best,

Weijun

===========
Weijun Gao
Computational Research Support Specialist
Department of Psychology, University of Toronto Scarborough
1265 Military Trail, Room SW416
Toronto, ON M1C 1M2
E-mail: weijun.gao at utoronto.ca

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201216/4f5a3833/attachment-0001.htm>