[slurm-users] How to get an estimate of job completion for planned maintenance?

Ryan Novosielski novosirj at rutgers.edu
Tue Dec 14 21:33:35 UTC 2021

Another useful format string – and again, this is if you mess up and don’t do a reservation early enough (or your environment has no concept of a time limit) – is this one:

squeue -o %u,%i,%L

Will show you username, job id, and remaining time – which is sometimes easier to deal with than end date/time.

|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark

> On Nov 7, 2021, at 7:45 AM, Carsten Beyer <beyer at dkrz.de> wrote:
> Hi Ahmad,
> you could use squeue -h -t r --format="%i %e" | sort -k2 to get a list of all running jobs sorted by their endtime.
> We use normaly a maintenance reservation with starttime of the mainenance (or with some leading time before) to get the system free of jobs. That make things easier, because if you drain your cluster no new jobs could start. With the reservation jobs with a shorter wallclock time could be backfilled till the reservation/maintenance starts. You can put the reservation anytime in the system but at least or before "<starttime maintenance> minus <longest MaxTime of partition>", e.g.
> scontrol create reservation=<name> starttime=<starttime> duration=<duration>  user=root flags=maint nodes=ALL
> Hope, that helps a little bit,
> Carsten
> -- 
> Carsten Beyer
> Abteilung Systeme
> Deutsches Klimarechenzentrum GmbH (DKRZ)
> Bundesstraße 45a * D-20146 Hamburg * Germany
> Phone:  +49 40 460094-221
> Fax:    +49 40 460094-270
> Email:  beyer at dkrz.de
> URL:    http://www.dkrz.de
> Geschäftsführer: Prof. Dr. Thomas Ludwig
> Sitz der Gesellschaft: Hamburg
> Amtsgericht Hamburg HRB 39784
> Am 05.11.2021 um 23:16 schrieb Ahmad Khalifa:
>> If I plan maintenance on a certain day, how long before that day should I set the queue to drain mode?! Is there a way to estimate the completion date / time of current running jobs?!
>> Regards.

More information about the slurm-users mailing list