[slurm-users] How to get an estimate of job completion for planned maintenance?

[Ryan Novosielski]

Another useful format string – and again, this is if you mess up and don’t do a reservation early enough (or your environment has no concept of a time limit) – is this one:

squeue -o %u,%i,%L

Will show you username, job id, and remaining time – which is sometimes easier to deal with than end date/time.

--
#BlackLivesMatter
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Nov 7, 2021, at 7:45 AM, Carsten Beyer <beyer at dkrz.de> wrote:
> 
> Hi Ahmad,
> 
> you could use squeue -h -t r --format="%i %e" | sort -k2 to get a list of all running jobs sorted by their endtime.
> 
> We use normaly a maintenance reservation with starttime of the mainenance (or with some leading time before) to get the system free of jobs. That make things easier, because if you drain your cluster no new jobs could start. With the reservation jobs with a shorter wallclock time could be backfilled till the reservation/maintenance starts. You can put the reservation anytime in the system but at least or before "<starttime maintenance> minus <longest MaxTime of partition>", e.g.
> 
> scontrol create reservation=<name> starttime=<starttime> duration=<duration>  user=root flags=maint nodes=ALL
> 
> Hope, that helps a little bit,
> 
> Carsten
> 
> -- 
> Carsten Beyer
> Abteilung Systeme
> 
> Deutsches Klimarechenzentrum GmbH (DKRZ)
> Bundesstraße 45a * D-20146 Hamburg * Germany
> 
> Phone:  +49 40 460094-221
> Fax:    +49 40 460094-270
> Email:  beyer at dkrz.de
> URL:    http://www.dkrz.de
> 
> Geschäftsführer: Prof. Dr. Thomas Ludwig
> Sitz der Gesellschaft: Hamburg
> Amtsgericht Hamburg HRB 39784
> 
> 
> Am 05.11.2021 um 23:16 schrieb Ahmad Khalifa:
>> If I plan maintenance on a certain day, how long before that day should I set the queue to drain mode?! Is there a way to estimate the completion date / time of current running jobs?!
>> 
>> Regards.
>