[slurm-users] How to set priorities of actual obs

[Dmitri Chebotarov]

Ketiw, 

Can this can be done in /etc/security/limits.conf file in your case?

F.e. '@users hard priority 10' will set default nice level for all users to 10 (and priority to 3. Default nice is 0 and priority is 20).

But keep in mind, if your Slurm is configured to enforce CPU core (cgroups?) , then user's job only gets requested number of CPUs and priority/nice will only apply to those CPUs. Slurm won't allow user to grab unused CPU cores, even if the core are idle. 

I'm not sure why you want to change default nice/priority in your case. 
I do it to make sure 'root' and other admin/system accounts don't fight for CPU cycle with users' jobs.

Thank you.

-----Original Message-----
From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Renfro, Michael
Sent: Friday, September 14, 2018 10:33 AM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] How to set priorities of actual obs

A 'nice -n 19' process will still consume 100% of the CPU if nothing else is going on.

‘top’ output from a dual-core system with 3 ‘dd’ processes -- 2 with default nice value of 0, and 1 with a nice value of 19:

=====

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
15743 renfro    20   0  107992    612    512 R  99.3  0.0   0:54.60 dd
15705 renfro    20   0  107992    608    512 R  98.3  0.0   1:19.81 dd
15671 renfro    39  19  107992    612    512 R   1.7  0.0   1:44.70 dd

=====

After killing one of the default priority processes, the nice value 19 process can still take up a full core:

=====

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
15671 renfro    39  19  107992    612    512 R  99.3  0.0   1:54.43 dd
15743 renfro    20   0  107992    612    512 R  99.3  0.0   1:12.58 dd

=====

So your slurm-generated processes should be running at full speed if nothing else is going on.

> On Sep 14, 2018, at 9:09 AM, kesim <ketiwsim at gmail.com> wrote:
> 
> Dear Dimitri,
> 
> Thank you for the answer. I was considering this but I would have to relay on users to submit their jobs that way. I just found a simpler solution but also somewhat limited in scope:
> If
> PropagatePrioProcess is set to 0  which is a default then the tasks will inherit slurmd priority. 
> You can set it by:  slurmd -n 19
> 
> However it would be nicer if the partitions itself can be separately managed - for short jobs with higher priority etc.
> 
> Best regards,
> 
> Ketiw
> 
> On Fri, Sep 14, 2018 at 3:37 PM Dmitri Chebotarov <dchebota at gmu.edu> wrote:
> Hi Ketiw,
> 
> Wouldn't 'nice' work in this case?
> 
> $ man nice
> ...
> NAME
>        nice - run a program with modified scheduling priority
> 
> SYNOPSIS
>        nice [OPTION] [COMMAND [ARG]...] ...
> 
> In your submit script you would run the program as
> 
> nice -n 19 <program-name>
> 
> 
> 
> -----Original Message-----
> From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of 
> Loris Bennett
> Sent: Friday, September 14, 2018 9:15 AM
> To: slurm-users at lists.schedmd.com
> Subject: Re: [slurm-users] How to set priorities of actual obs
> 
> 
> kesim <ketiwsim at gmail.com> writes:
> 
> >> On Fri, Sep 14, 2018 at 2:41 PM Loris Bennett <loris.bennett at fu-berlin.de> wrote:
> >>
> >>  Hi Ketiw,
> >>
> >>  kesim <ketiwsim at gmail.com> writes:
> >>
> >>  > Dear all,
> >>  >
> >>  > I would like to submit a job in such a way that the actual 
> >> program is  > run with the lowest priority (nice=19 on linux). At 
> >> the moment every  > task has priority 0. Is it possible to do that and how?
> >>  >
> >>  > Best regards,
> >>  >
> >>  > Ketiw
> >>
> >>  I'm not aware that this is possible. What would be the use-case 
> >> for this? Normally you would want a job to make full use of any 
> >> cores it had reserved.
> 
> > I have slurm running on several desktops and we have only a few 
> > application for its use. I would like the slurm submitted tasks to 
> > have lowest priority so they are not hampering users who normally 
> > work on their desktops.
> 
> Please write to the list, rather than to me directly.
> 
> I think you can probably achieve what you want with cgroups, but I have no experience of doing this myself.
> 
> Regards
> 
> Loris
> 
> --
> Dr. Loris Bennett (Mr.)
> ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de
>