<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks Lyn – that was exactly the problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Jake<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Lyn Gerner<br>
<b>Sent:</b> 03 June 2022 01:51<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] New slurm configuration - multiple jobs per host<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Jake, my hunch is that your jobs are getting hung up on mem allocation, such that Slurm is assigning all of memory to each job as it runs; you can verify w/scontrol show job. If that's what's happening, try setting a DefMemPerCPU value
for your partition(s).<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best of luck,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Lyn<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, May 26, 2022 at 1:39 PM Jake Jellinek <<a href="mailto:jakejellinek@outlook.com">jakejellinek@outlook.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi Ole<br>
<br>
I only added the oversubscribe option because without it, it didn’t work - so in fact, it appears not to have made any difference<br>
<br>
I though the RealMemory option just said not to offer any jobs to the node that didn’t have AT LEAST that amount of RAM<br>
My large node has more than 64GB RAM (and more will be allocated later) but I have yet to get to a memory issue…still working on cores<br>
<br>
<br>
jake@compute001:~$ slurmd -C<br>
NodeName=compute001 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=64359<br>
UpTime=0-06:58:54<br>
<br>
<br>
Thanks<br>
Jake<br>
<br>
> On 26 May 2022, at 21:11, Ole Holm Nielsen <<a href="mailto:Ole.H.Nielsen@fysik.dtu.dk" target="_blank">Ole.H.Nielsen@fysik.dtu.dk</a>> wrote:<br>
> <br>
> Hi Jake,<br>
> <br>
> Firstly, which Slurm version and which OS do you use?<br>
> <br>
> Next, try simplifying by removing the oversubscribe configuration. Read the slurm.conf manual page about oversubscribe, it looks a bit tricky.<br>
> <br>
> The RealMemory=1000 is extremely low and might prevent jobs from starting! Run "slurmd -C" on the nodes to read appropriate node parameters for slurm.conf.<br>
> <br>
> I hope this helps.<br>
> <br>
> /Ole<br>
> <br>
> <br>
>> On 26-05-2022 21:12, Jake Jellinek wrote:<br>
>> Hi<br>
>> I am just building my first Slurm setup and have got everything running – well, almost.<br>
>> I have a two node configuration. All of my setup exists on a single HyperV server and I have divided up the resources to create my VMs<br>
>> One node I will use for heavy duty work; this is called compute001<br>
>> One node I will use for normal work; this is called compute002<br>
>> My compute node specification in slurm.conf is<br>
>> NodeName=DEFAULT CPUs=1 RealMemory=1000 State=UNKNOWN<br>
>> NodeName=compute001 CPUs=32<br>
>> NodeName=compute002 CPUs=2<br>
>> The partition specification is<br>
>> PartitionName=DEFAULT State=UP<br>
>> PartitionName=interactive Nodes=compute002 MaxTime=INFINITE OverSubscribe=FORCE<br>
>> PartitionName=simulation Nodes=compute001 MaxTime=30 OverSubscribe=FORCE<br>
>> I have added the OverSubscribe=FORCE option as I want more than one job to be able to land on my interactive/simulation queues.<br>
>> All of the nodes and cluster master start up fine and they all talk to each other but no matter what I do, I cannot get my cluster to accept more than one job per node.<br>
>> Can you help me determine where I am going wrong?<br>
>> Thanks a lot<br>
>> Jake<br>
>> The entire slurm.conf is pasted below<br>
>> # slurm.conf file generated by configurator.html.<br>
>> ClusterName=pm-slurm<br>
>> SlurmctldHost=slurm-master<br>
>> MpiDefault=none<br>
>> ProctrackType=proctrack/cgroup<br>
>> ReturnToService=2<br>
>> SlurmctldPidFile=/var/run/slurmctld.pid<br>
>> SlurmctldPort=6817<br>
>> SlurmdPidFile=/var/run/slurmd.pid<br>
>> SlurmdPort=6818<br>
>> SlurmdSpoolDir=/var/spool/slurmd<br>
>> SlurmUser=slurm<br>
>> StateSaveLocation=/home/slurm/var/spool/slurmctld<br>
>> SwitchType=switch/none<br>
>> TaskPlugin=task/cgroup<br>
>> #<br>
>> # TIMERS<br>
>> InactiveLimit=0<br>
>> KillWait=30<br>
>> MinJobAge=300<br>
>> SlurmctldTimeout=120<br>
>> SlurmdTimeout=300<br>
>> Waittime=0<br>
>> #<br>
>> # SCHEDULING<br>
>> SchedulerType=sched/backfill<br>
>> SelectType=select/cons_tres<br>
>> SelectTypeParameters=CR_Core_Memory<br>
>> #<br>
>> # LOGGING AND ACCOUNTING<br>
>> JobAcctGatherFrequency=30<br>
>> JobAcctGatherType=jobacct_gather/cgroup<br>
>> SlurmctldDebug=info<br>
>> SlurmctldLogFile=/var/log/slurmctld.log<br>
>> SlurmdDebug=info<br>
>> SlurmdLogFile=/var/log/slurmd.log<br>
>> # COMPUTE NODES<br>
>> NodeName=DEFAULT CPUs=1 RealMemory=1000 State=UNKNOWN<br>
>> NodeName=compute001 CPUs=32<br>
>> NodeName=compute002 CPUs=2<br>
>> PartitionName=DEFAULT State=UP<br>
>> PartitionName=interactive Nodes=compute002 MaxTime=INFINITE OverSubscribe=FORCE<br>
>> PartitionName=simulation Nodes=compute001 MaxTime=30 OverSubscribe=FORCE<br>
> <br>
> <o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>