[slurm-users] exclusive or not exclusive, that is the question
Christopher Benjamin Coffey
Chris.Coffey at nau.edu
Wed Aug 21 20:15:18 UTC 2019
Marcus, maybe you can try playing with --mem instead? We recommend our users to use --mem instead of --mem-per-cpu/task as it It makes it easier for users to request the right amount of memory for the job. --mem is the amount of memory for the whole job. This way, there is no multiplying of memory * cpu involved.
Strange that the cgroup has more memory than possible though.
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
On 8/20/19, 11:27 PM, "slurm-users on behalf of Marcus Wagner" <slurm-users-bounces at lists.schedmd.com on behalf of wagner at itc.rwth-aachen.de> wrote:
One thing, I forgot.
On 8/20/19 4:58 PM, Christopher Benjamin Coffey wrote:
> Hi Marcus,
>
> What is the reason to add "--mem-per-cpu" when the job already has exclusive access to the node?
The user (normally) does not set --exclusive directly. We have several
accounts, whose jobs by default should run exclusively, so we set that
in the job_submit plugin.
> Your job has access to all of the memory, and all of the cores on the system already. Also note, for non-mpi code like single core job, or shared memory threaded job, you want to ask for number of cpus with --cpus-per-task, or -c. Unless you are running mpi code, where you will want to use -n, and --ntasks instead to launch n copies of the code on n cores. In this case, because you asked for -n2, and also specified a mem-per-cpu request, the scheduler is doling out the memory as requested (2 x tasks), likely due to having SelectTypeParameters=CR_Core_Memory in slurm.conf.
I must say, we would be much happier with a --mem-per-task option
instead. I still do not know, why one should ask for mem-per-cpu
(logically), since in a shared memory job, you start one process, the
threads share the memory.
With an hybrid MPI-code (mpi code with openmp parallelization on the
tasks), it makes even less sense. If I know, how much memory my tasks
needs, e.g. 10 GB, I still have to divide that through the number of
threads (-c) to get the right memory request. For me as an
administrator, an openmp job is a special hybrid job with only one
requested task. So it is the same for a shared memory job. I always have
to divide the really needed memory through the number of threads (or
cpus-per-task).
Is there anyone, who can enlighten me?
Why does one have to ask for memory per smallest scheduleable (is that
word right?) unit? Isn't it better to ask for memory per task/process?
Best
Marcus
>
> Best,
> Chris
>
> —
> Christopher Coffey
> High-Performance Computing
> Northern Arizona University
> 928-523-1167
>
>
> On 8/20/19, 1:37 AM, "slurm-users on behalf of Marcus Wagner" <slurm-users-bounces at lists.schedmd.com on behalf of wagner at itc.rwth-aachen.de> wrote:
>
> Just made another test.
>
>
> Thanks god, the exclusivity is not "destroyed" completely, only on job
> can run on the node, when the job is exclusive. Nonetheless, this is
> somewhat unintuitive.
> I wonder, if that also has an influence on the cgroups and the process
> affinity/binding.
>
> I will do some more tests.
>
>
> Best
> Marcus
>
> On 8/20/19 9:47 AM, Marcus Wagner wrote:
> > Hi Folks,
> >
> >
> > I think, I've stumbled over a BUG in Slurm regarding the
> > exclusiveness. Might also, I've misinterpreted something. I would be
> > happy, if someone could explain that to me in the latter case.
> >
> > To the background. I have set PriorityFlags=MAX_TRES
> > The TRESBillingWeights are "CPU=1.0,Mem=0.1875G" for a partition with
> > 48 core nodes and RealMemory 187200.
> >
> > ---
> >
> > I have two jobs:
> >
> > job 1:
> > #SBATCH --exclusive
> > #SBATCH --ntasks=2
> > #SBATCH --nodes=1
> >
> > scontrol show <jobid> =>
> > NumNodes=1 NumCPUs=48 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> > TRES=cpu=48,mem=187200M,node=1,billing=48
> >
> > exactly, what I expected, I got 48 CPUs and therefore the billing is 48.
> >
> > ---
> >
> > job 2 (just added mem-per-cpu):
> > #SBATCH --exclusive
> > #SBATCH --ntasks=2
> > #SBATCH --nodes=1
> > #SBATCH --mem-per-cpu=5000
> >
> > scontrol show <jobid> =>
> > NumNodes=1-1 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> > TRES=cpu=2,mem=10000M,node=1,billing=2
> >
> > Why "destroys" '--mem-per-cpu' exclusivity?
> >
> >
> >
> > Best
> > Marcus
> >
>
> --
> Marcus Wagner, Dipl.-Inf.
>
> IT Center
> Abteilung: Systeme und Betrieb
> RWTH Aachen University
> Seffenter Weg 23
> 52074 Aachen
> Tel: +49 241 80-24383
> Fax: +49 241 80-624383
> wagner at itc.rwth-aachen.de
> https://nam05.safelinks.protection.outlook.com/?url=www.itc.rwth-aachen.de&data=02%7C01%7Cchris.coffey%40nau.edu%7C7c17306818aa4c9c640708d72600a33a%7C27d49e9f89e14aa099a3d35b57b2ba03%7C0%7C0%7C637019656498257836&sdata=7yFRaJHUtfoKISb75FLCesa7zz2e9ZyN6Or0iu89ggM%3D&reserved=0
>
>
>
>
--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wagner at itc.rwth-aachen.de
https://nam05.safelinks.protection.outlook.com/?url=www.itc.rwth-aachen.de&data=02%7C01%7Cchris.coffey%40nau.edu%7C7c17306818aa4c9c640708d72600a33a%7C27d49e9f89e14aa099a3d35b57b2ba03%7C0%7C0%7C637019656498257836&sdata=7yFRaJHUtfoKISb75FLCesa7zz2e9ZyN6Or0iu89ggM%3D&reserved=0
More information about the slurm-users
mailing list