[slurm-users] Suspend QOS help

[Walls, Mitchell]

Time slicing based suspend works for the QOSes submitted to the same partition so I at least think the configuration is close. It just doesn't seem to be working for separate partitions for the suspend and general QOS. I'd prefer not to timeslice in the separate partition configuration but it seems both jobs run without timeslicing or suspending when on separate partitions.

45112      test stress.s  user2  R       0:02      1 node04
45110      test stress-s  user1  S       1:00      1 node04

# A time later
45112      test stress.s  user2  R       1:17      1 node04
45110      test stress-s  user1  S       2:00      1 node04

#partition
PartitionName=test    Default=NO  Nodes=node04   OverSubscribe=FORCE:1 MaxTime=30-00:00:00             AllowQos=general,suspend

________________________________________
From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Walls, Mitchell <miwalls at siue.edu>
Sent: Friday, February 18, 2022 9:55 AM
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] Suspend QOS help

Whoops Nodes name was wrong correction here for partitions.
# test partitions
PartitionName=test    Default=NO  Nodes=node04   OverSubscribe=FORCE:1 MaxTime=30-00:00:00 Qos=general AllowQos=general
PartitionName=suspend Default=NO  Nodes=node04   OverSubscribe=FORCE:1 MaxTime=30-00:00:00 Qos=suspend AllowQos=suspend

________________________________________
From: Walls, Mitchell <miwalls at siue.edu>
Sent: Friday, February 18, 2022 9:54 AM
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] Suspend QOS help

Both jobs would be using the whole node same as below but with two nodes. I've reduced the problem space to two isolated partitions on just node04.
NodeName=node04 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=257476 Features=cpu

# qoses have stayed the same.
      Name   Priority    Preempt PreemptMode
---------- ---------- ---------- -----------
   general       1000     suspend     cluster
   suspend       100                        cluster

# test partitions
PartitionName=test    Default=NO  Nodes=cc-cpu-04   OverSubscribe=FORCE:1 MaxTime=30-00:00:00 Qos=general AllowQos=general
PartitionName=suspend Default=NO  Nodes=cc-cpu-04   OverSubscribe=FORCE:1 MaxTime=30-00:00:00 Qos=suspend AllowQos=suspend

stress-suspend.sh
#!/bin/bash
#SBATCH -p suspend
#SBATCH -C cpu
#SBATCH -q suspend
#SBATCH -c 32
#SBATCH --ntasks-per-node=1
#SBATCH -N 1
stress -c 32 -t $1

#stress.sh
#!/bin/bash
#SBATCH -p test
#SBATCH -C cpu
#SBATCH -q general
#SBATCH -c 32
#SBATCH --ntasks-per-node=1
#SBATCH -N 1
stress -c 32 -t $1


________________________________________
From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Brian Andrus <toomuchit at gmail.com>
Sent: Friday, February 18, 2022 9:36 AM
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] Suspend QOS help

First look and I would guess that there are enough resources to satisfy
the requests of both jobs, so no need to suspend.

Having the node info and the job info to compare would be the next step.

Brian Andrus


On 2/18/2022 7:20 AM, Walls, Mitchell wrote:
> Hello,
>
> Hoping someone can shed some light on what is causing jobs to run on same nodes simultaneously rather than being actually suspended for the lower priority job? I can provide more info if someone can think of something to help!
>
> # Relevant config.
> PreemptType=preempt/qos
> PreemptMode=SUSPEND,GANG
>
> PartitionName=general Default=YES Nodes=general     OverSubscribe=FORCE:1 MaxTime=30-00:00:00   Qos=general  AllowQos=general
> PartitionName=suspend Default=NO  Nodes=general     OverSubscribe=FORCE:1 MaxTime=30-00:00:00 Qos=suspend AllowQos=suspend
>
> # Qoses
>        Name   Priority    Preempt PreemptMode
> ---------- ---------- ---------- -----------
>     general       1000     suspend     cluster
>     suspend       100                        cluster
>
> # squeue (another note is I see that both processes are actually running at same time and not being timesliced in htop)
> $ squeue
>               JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
>               45085   general  stress.s   user2   R       7:33     2 node[04-05]
>               45084   suspend stress-s  user1   R       7:40     2 node[04-05]
>
> Thanks!