[slurm-users] exclusive or not exclusive, that is the question

[Christopher Benjamin Coffey]

Hi Marcus,

What is the reason to add "--mem-per-cpu" when the job already has exclusive access to the node? Your job has access to all of the memory, and all of the cores on the system already. Also note, for non-mpi code like single core job, or shared memory threaded job, you want to ask for number of cpus with --cpus-per-task, or -c. Unless you are running mpi code, where you will want to use -n, and --ntasks instead to launch n copies of the code on n cores. In this case, because you asked for -n2, and also specified a mem-per-cpu request, the scheduler is doling out the memory as requested (2 x tasks), likely due to having SelectTypeParameters=CR_Core_Memory in slurm.conf.

Best,
Chris

—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
 

On 8/20/19, 1:37 AM, "slurm-users on behalf of Marcus Wagner" <slurm-users-bounces at lists.schedmd.com on behalf of wagner at itc.rwth-aachen.de> wrote:

    Just made another test.
    
    
    Thanks god, the exclusivity is not "destroyed" completely, only on job 
    can run on the node, when the job is exclusive. Nonetheless, this is 
    somewhat unintuitive.
    I wonder, if that also has an influence on the cgroups and the process 
    affinity/binding.
    
    I will do some more tests.
    
    
    Best
    Marcus
    
    On 8/20/19 9:47 AM, Marcus Wagner wrote:
    > Hi Folks,
    >
    >
    > I think, I've stumbled over a BUG in Slurm regarding the 
    > exclusiveness. Might also, I've misinterpreted something. I would be 
    > happy, if someone could explain that to me in the latter case.
    >
    > To the background. I have set PriorityFlags=MAX_TRES
    > The TRESBillingWeights are "CPU=1.0,Mem=0.1875G" for a partition with 
    > 48 core nodes and RealMemory 187200.
    >
    > ---
    >
    > I have two jobs:
    >
    > job 1:
    > #SBATCH --exclusive
    > #SBATCH --ntasks=2
    > #SBATCH --nodes=1
    >
    > scontrol show <jobid> =>
    >    NumNodes=1 NumCPUs=48 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
    >    TRES=cpu=48,mem=187200M,node=1,billing=48
    >
    > exactly, what I expected, I got 48 CPUs and therefore the billing is 48.
    >
    > ---
    >
    > job 2 (just added mem-per-cpu):
    > #SBATCH --exclusive
    > #SBATCH --ntasks=2
    > #SBATCH --nodes=1
    > #SBATCH --mem-per-cpu=5000
    >
    > scontrol show <jobid> =>
    >    NumNodes=1-1 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
    >    TRES=cpu=2,mem=10000M,node=1,billing=2
    >
    > Why "destroys" '--mem-per-cpu' exclusivity?
    >
    >
    >
    > Best
    > Marcus
    >
    
    -- 
    Marcus Wagner, Dipl.-Inf.
    
    IT Center
    Abteilung: Systeme und Betrieb
    RWTH Aachen University
    Seffenter Weg 23
    52074 Aachen
    Tel: +49 241 80-24383
    Fax: +49 241 80-624383
    wagner at itc.rwth-aachen.de
    https://nam05.safelinks.protection.outlook.com/?url=www.itc.rwth-aachen.de&data=02%7C01%7Cchris.coffey%40nau.edu%7C4a5803448abd497d7cde08d7254995f2%7C27d49e9f89e14aa099a3d35b57b2ba03%7C0%7C0%7C637018870287848104&sdata=HNuqCBYwrJjBcLGFGYuVKxWe9pqCxt028rrRrJ%2FTYp0%3D&reserved=0