We have implemented that, but we don't do the allocation limits, so the use case is slightly different. It works well, as long as the users "remember" to use it correctly (which I guess you may force/remember them with the allocation). In a nutshell, we have done
sacctmgr add qos requeue set Priority=10 PreemptMode=REQUEUE
sacctmgr add qos cancel set Priority=10 PreemptMode=CANCEL
sacctmgr add qos high set Priority=1000 Preempt=cancel,requeue PreemptMode=cluster
sacctmgr add qos max5nodes set priority=50 MaxNodesPerUser=5 PreemptMode=cluster flags=DenyOnLimit
sacctmgr modify qos normal set priority=100
sacctmgr modify account default set qos=normal,requeue,cancel,high,max5nodes
Then make sure slurm.conf contains
PreemptType=preempt/qos
PreemptMode=REQUEUE
AccountingStorageEnforce=associations,limits,qos
The funniest pitfall we encountered is that slurm v23 in our test cluster (which we did not manage to upgrade yet, long story) accepted spaces in slurm.conf but v25 doesn't so we had a space before
preempt/qos and that worked fine in the test environment but failed in production for "unknown" reasons until we figured out.
