[slurm-users] Handling idle sessions

[Lachlan Musicman]

On 27 May 2018 at 18:23, Nadav Toledo <nadavtoledo at cs.technion.ac.il> wrote:

> Hello forum,
>
> I am trying to deal with idle session for some time, and haven't found a
> solution i am happy with.
> The scenario is as follow: users using srun for jupyter-lab(which is fine
> and even encouraged by me) on image processing cluster with gpus.
>
> problem is, I am trying to have some kind of solution to email/cancel
> their job if their session is idle for X amount of hours.
>
> the w command or xprintidle cannot be used , since they both work with ssh
> but not with slurm(checked that)
>
> Writing a script is not as easy as one might think, If i run a script in
> admin user scope, i need later on to figure out which idle gpu belong to
> which slurm job.
> running a script in the user scope is probably better idea, but in which
> way? crontab is running even user is not logged, how can i force users to
> run something only when the job start?
>
> perhaps some combination of sreport and tres?
>

Hmm.  We address this with accounting. A tight walltime ( 40 minutes) means
that most jobs run without worrying about walltime. But some will need to
set it. The accounting system keeps people honest by making "hogging" of
resources bad for a users job priority - in so much as their next job will
be deprioritsed.

Letting people know that their next job will not be de-prioritised if they
waste the resources, we find our users behave responsibly.

L.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20180527/b2f952c3/attachment.html>