[slurm-users] Fwd: Using PreemptExemptTime

[John DeSantis]

Phil,

>> Does anyone have a working example using PreemptExemptTime?
>> 
>> My goal is to make a higher priority job wait 24 hours before actually
>> preempting a lower priority job. Another way, any job is entitled to 24
>> hours run time before being preempted. The preempted job should be
>> suspended, ideally. If requeue is necessary that is ok.

We do and it is working as expected.  Please see relevant snippets below.

> [~] $ scontrol show config|grep Preempt
> PreemptMode             = CANCEL
> PreemptType             = preempt/qos
> PreemptExemptTime       = 00:00:00

>                 Name Priority PreemptExe 
> -------------------- -------- ---------- 
> interactive          1000     01:00:00   
> preempt              500      01:00:00   
> preempt_short        500      00:30:00   
> rchii                1000     01:00:00   


>> Details from my test cluster below my signature. Any ideas on what I
>> should check or missing? Maybe I misunderstood something.

I don't think you've missed anything.  The only bit of information I can add is that we previously were using GraceTime (which requires PreemptMode=CANCEL).  Unfortunately, depending on the application, it wouldn't always be clear that a job was preempted in the application's output, or within the slurmctld logs.  When we switched to PreemptExemptTime, all application output and SLURM logs stated preempted as the reason.

I know you want to suspend preempted jobs, but what happens if you cancel them instead?

HTH,

John DeSantis


On 2/2/22 14:12, Phil Kauffman wrote:
> Does anyone have a working example using PreemptExemptTime?
> 
> My goal is to make a higher priority job wait 24 hours before actually
> preempting a lower priority job. Another way, any job is entitled to 24
> hours run time before being preempted. The preempted job should be
> suspended, ideally. If requeue is necessary that is ok.
> 
> It's been asked before here:
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fg%2Fslurm-users%2Fc%2FmK4_M4hpXL8%2Fm%2FsRhT53VYBQAJ&data=04%7C01%7Cdesantis%40usf.edu%7C6228bd43128249e5c34308d9e6801523%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C637794920273664110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=kjiBW8EMaKmxZPjLJJBqHvHRtTC2JluqiC%2FibtWhY3w%3D&reserved=0
> 
> I've run through many iterations attempting to set `PreemptExemptTime`
> in slurm.conf and in QOS.
> 
> Setting `PreemptType=preempt/partition_prio`:
> - The preempted job gets suspended but `PreemptExemptTime` is ignored.
> 
> Setting `PreemptType=preempt/qos`
> - Configuring inside the QOS as well as globally in slurm.conf
> - `PreemptExemptTime` is respected but both jobs continue to run at the
> same time using 200% of the resources, which is not wanted.
> 
> 
> Details from my test cluster below my signature. Any ideas on what I
> should check or missing? Maybe I misunderstood something.
> 
> Cheers,
> 
> Phil
> 
> 
> 
> In my tests I'm using 3 mins as the PreemptExemptTime.
> 
> # Nodes
> NodeName=slurm[2-5] CPUs=1 Sockets=1 CoresPerSocket=1 ThreadsPerCore=2
> RealMemory=1800 MemSpecLimit=200 State=UNKNOWN
> 
> 
> 
> ### experiment using PreemptType=preempt/qos
> PartitionName=DEFAULT OverSubscribe=FORCE:1 Nodes=slurm[2-4]
> PartitionName=active Default=YES QOS=normal
> PartitionName=hipri  Default=NO QOS=expedite
> 
> PreemptType=preempt/qos
> PreemptMode=SUSPEND,GANG
> PreemptExemptTime=00:03:00
> SchedulerParameters=preempt_strict_order
> PriorityType=priority/multifactor
> SelectType=select/cons_tres
> SelectTypeParameters=CR_Core_Memory
> # QOS
> [root at slurm2 slurm-llnl]# sacctmgr show qos -p --noheader
> normal|1|00:00:00||00:03:00|cluster|||1.000000||||||||||||||||||
> expedite|2|00:00:00|normal|00:03:00|cluster|||1.000000||||||||||||||||||
> 
> 
> 
> 
> 
> ### Experiment using PreemptType=preempt/partition_prio
> PartitionName=low    Default=NO OverSubscribe=NO      PriorityTier=10
> PreemptMode=requeue
> PartitionName=med    Default=NO OverSubscribe=FORCE:1 PriorityTier=20
> PreemptMode=suspend
> PartitionName=hi     Default=NO OverSubscribe=FORCE:1 PriorityTier=30
> PreemptMode=off
> 
> PreemptType=preempt/partition_prio
> PreemptMode=SUSPEND,GANG
> PreemptExemptTime=00:03:00
> SchedulerParameters=preempt_strict_order
> PriorityType=priority/multifactor
> SelectType=select/cons_tres
> SelectTypeParameters=CR_Core_Memory
> 
> 
> [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.