[slurm-users] Transparently assign different walltime limit to a group of nodes ?

[Cyrus Proctor]

Jens, you are most welcome. I'll be curious to hear a followup after you 
have evaluated it. I myself have not used it in production but I thought 
it looked really cool.

It is interesting to me for increasing overall resource utilization for 
partitions with transient "real time" demands. I should be able to take, 
say, 25% of this real time partition, overlap the nodes with another 
general production partition, set the real time partition priority 
slightly higher, and set a floating reservation to keep job turnaround 
high on this set of overlapping nodes in case the real time partition 
becomes busy. I think this would be a nice solution that does not 
involve job preemption.

Cheers,
Cyrus

On 08/13/2018 11:20 AM, Jens Dreger wrote:
> Hi Cyrus!
>
> On Mon, Aug 13, 2018 at 08:44:15AM -0500, Cyrus Proctor wrote:
>> Hi Jens,
>>
>> Check out https://slurm.schedmd.com/reservations.html specifically the "
>> Reservations Floating Through Time" section. In your case, set a walltime of 14
>> days for your partition that contains n[01-10]. Then, create a floating
>> reservation on node n[06-10] for n + 1 day where "n" is always evaluated as
>> now.
> This is just perfect! Thank you!
>
> Jens.
>
>> If you wish to allow the user more control, then specify a "Feature" in
>> slurm.conf for you nodes. Something like:
>> NodeName=n[01-05] Sockets=1 CoresPerSocket=48 ThreadsPerCore=2 State=UNKNOWN
>> Feature=long
>> NodeName=n[06-10] Sockets=1 CoresPerSocket=48 ThreadsPerCore=2 State=UNKNOWN
>> Feature=short
>>
>> The feature is an arbitrary string that the admin sets. Then a user could
>> specify in their submission as something like:
>> sbatch --constraint="long|short" batch.slurm
>>
>> Best,
>> Cyrus
>>
>> On 08/13/2018 08:28 AM, Loris Bennett wrote:
>>
>>      Hi Jens,
>>
>>      Jens Dreger <jens.dreger at physik.fu-berlin.de> writes:
>>
>>
>>          Hi everyone!
>>
>>          Is it possible to transparently assign different walltime limits
>>          to nodes without forcing users to specify partitions when submitting
>>          jobs?
>>
>>          Example: let's say I have 10 nodes. Nodes n01-n05 should be available
>>          for jobs with a walltime up to 14 days, while n06-n10 should only
>>          be used for jobs with a walltime limit less then 1 day. Then as long
>>          as nodes n06-n10 have free resources, jobs with walltime <1day should
>>          be scheduled to these nodes. If n06-n10 are full, jobs with walltime
>>          <1day should start on n01-n05. Users should not have to specify
>>          partitions.
>>
>>          Would this even be possible to do with just one partition much
>>          like nodes with different memory size using weights to fill nodes
>>          with less memoery first?
>>
>>          Background of this question is that it would be helpfull to be able
>>          to lower the walltime for a rack of nodes, e.g. when adding this rack
>>          to an existing cluster in order to be able to easily shut down just
>>          this rack after one day in case of instabilities. Much like adding
>>          N nodes to a cluster without changing anything else and have only
>>          jobs with walltime <1day on thiese nodes in the beginning.
>>
>>      If you just want to reduce the allowed wall-time for a given rack, can't
>>      you just use a maintenance reservation for the appropriate set of nodes?
>>
>>
>>      Loris
>>
>>
>>