[slurm-users] New slurm configuration - multiple jobs per host

[Jake Jellinek]

Thanks Lyn – that was exactly the problem.

Jake

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Lyn Gerner
Sent: 03 June 2022 01:51
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] New slurm configuration - multiple jobs per host

Jake, my hunch is that your jobs are getting hung up on mem allocation, such that Slurm is assigning all of memory to each job as it runs; you can verify w/scontrol show job. If that's what's happening, try setting a DefMemPerCPU value for your partition(s).

Best of luck,
Lyn

On Thu, May 26, 2022 at 1:39 PM Jake Jellinek <jakejellinek at outlook.com<mailto:jakejellinek at outlook.com>> wrote:
Hi Ole

I only added the oversubscribe option because without it, it didn’t work - so in fact, it appears not to have made any difference

I though the RealMemory option just said not to offer any jobs to the node that didn’t have AT LEAST that amount of RAM
My large node has more than 64GB RAM (and more will be allocated later) but I have yet to get to a memory issue…still working on cores


jake at compute001:~$ slurmd -C
NodeName=compute001 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=64359
UpTime=0-06:58:54


Thanks
Jake

> On 26 May 2022, at 21:11, Ole Holm Nielsen <Ole.H.Nielsen at fysik.dtu.dk<mailto:Ole.H.Nielsen at fysik.dtu.dk>> wrote:
>
> Hi Jake,
>
> Firstly, which Slurm version and which OS do you use?
>
> Next, try simplifying by removing the oversubscribe configuration.  Read the slurm.conf manual page about oversubscribe, it looks a bit tricky.
>
> The RealMemory=1000 is extremely low and might prevent jobs from starting!  Run "slurmd -C" on the nodes to read appropriate node parameters for slurm.conf.
>
> I hope this helps.
>
> /Ole
>
>
>> On 26-05-2022 21:12, Jake Jellinek wrote:
>> Hi
>> I am just building my first Slurm setup and have got everything running – well, almost.
>> I have a two node configuration. All of my setup exists on a single HyperV server and I have divided up the resources to create my VMs
>> One node I will use for heavy duty work; this is called compute001
>> One node I will use for normal work; this is called compute002
>> My compute node specification in slurm.conf is
>> NodeName=DEFAULT CPUs=1 RealMemory=1000 State=UNKNOWN
>> NodeName=compute001 CPUs=32
>> NodeName=compute002 CPUs=2
>> The partition specification is
>> PartitionName=DEFAULT State=UP
>> PartitionName=interactive Nodes=compute002 MaxTime=INFINITE OverSubscribe=FORCE
>> PartitionName=simulation Nodes=compute001 MaxTime=30 OverSubscribe=FORCE
>> I have added the OverSubscribe=FORCE option as I want more than one job to be able to land on my interactive/simulation queues.
>> All of the nodes and cluster master start up fine and they all talk to each other but no matter what I do, I cannot get my cluster to accept more than one job per node.
>> Can you help me determine where I am going wrong?
>> Thanks a lot
>> Jake
>> The entire slurm.conf is pasted below
>> # slurm.conf file generated by configurator.html.
>> ClusterName=pm-slurm
>> SlurmctldHost=slurm-master
>> MpiDefault=none
>> ProctrackType=proctrack/cgroup
>> ReturnToService=2
>> SlurmctldPidFile=/var/run/slurmctld.pid
>> SlurmctldPort=6817
>> SlurmdPidFile=/var/run/slurmd.pid
>> SlurmdPort=6818
>> SlurmdSpoolDir=/var/spool/slurmd
>> SlurmUser=slurm
>> StateSaveLocation=/home/slurm/var/spool/slurmctld
>> SwitchType=switch/none
>> TaskPlugin=task/cgroup
>> #
>> # TIMERS
>> InactiveLimit=0
>> KillWait=30
>> MinJobAge=300
>> SlurmctldTimeout=120
>> SlurmdTimeout=300
>> Waittime=0
>> #
>> # SCHEDULING
>> SchedulerType=sched/backfill
>> SelectType=select/cons_tres
>> SelectTypeParameters=CR_Core_Memory
>> #
>> # LOGGING AND ACCOUNTING
>> JobAcctGatherFrequency=30
>> JobAcctGatherType=jobacct_gather/cgroup
>> SlurmctldDebug=info
>> SlurmctldLogFile=/var/log/slurmctld.log
>> SlurmdDebug=info
>> SlurmdLogFile=/var/log/slurmd.log
>> # COMPUTE NODES
>> NodeName=DEFAULT CPUs=1 RealMemory=1000 State=UNKNOWN
>> NodeName=compute001 CPUs=32
>> NodeName=compute002 CPUs=2
>> PartitionName=DEFAULT State=UP
>> PartitionName=interactive Nodes=compute002 MaxTime=INFINITE OverSubscribe=FORCE
>> PartitionName=simulation Nodes=compute001 MaxTime=30 OverSubscribe=FORCE
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220603/f261e069/attachment-0001.htm>