<div dir="ltr"><div>Hi, thanks for getting back to me.</div><div><br></div><div>I have been doing some more experimenting, and I think that the issue is because the Azure VMs for my nodes are HyperThreaded.</div><div><br></div><div>Slurm sees the cluser as 5 nodes with 1 CPU and seems to ignore the HyperThreading - so hence Slurm sees the cluster as a 5 CPU cluster (and not 10 as I thought) - so it is correct that it can't run a 10 cpu job.<br></div><div><br></div><div>Speaking with my CFD types - they say our code should not be run on HT nodes, so I have switched to a different Azure VM sku for the nodes without HT, and the CPU count in Slurm matches the count of those in the VMs.<br></div><div><br></div><div>So - does Slurm actually ignore HT cores, as I am supposing?</div><div><br></div><div>Regards</div><div>Gary<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 13 Dec 2022 at 15:52, Brian Andrus <<a href="mailto:toomuchit@gmail.com">toomuchit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <p>Gary,</p>
    <p>Well your first issue is using Cyclecloud, but that is mostly
      opinion :)<br>
    </p>
    <p>Your error states there aren't enough CPUs in the partition,
      which means we should take a look at the partition settings.</p>
    <p>Take a look at 'scontrol show partition hpc' and see how many
      nodes are assigned to it. Also check the state of the nodes with
      'sinfo'<br>
    </p>
    <p>It would also be good to ensure the node settings are right. Run
      'slurmd -C' on a node and see if the output matches what is in the
      config.</p>
    <p>Brian Andrus<br>
    </p>
    <div>On 12/13/2022 1:38 AM, Gary Mansell
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>Dear Slurm
            Users, perhaps you can
            help me with a problem that I am having using the Scheduler
            (I am new to this,
            so please forgive me for any stupid
            mistakes/misunderstandings).<span></span></span></p>
        <div><br>
        </div>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>I am not able to
            submit a Multi-Threaded
            MPI job on a small demo cluster that I have setup using
            Azure CycleCloud that
            uses all the 10x CPUs on my cluster, and I don’t understand
            why – perhaps you
            can explain why and how I can fix this to use all available
            CPUs?<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>The hpc
            partition that I have setup
            consists of 5 nodes (Azure VM type = Standard_F2s_v2), each
            with 2 cpu’s (I
            presume that these are Hyperthreaded cores, rather than 2
            cpus – but I am not
            certain of this)?<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">[azccadmin@ricslurm-hpc-pg0-1
            ~]$ <span style="background:yellow">cat /proc/cpuinfo</span><span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">processor      
            : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">vendor_id      
            :
            GenuineIntel<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpu
            family      : 6<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">model          
            : 106<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">model
            name      : Intel(R)
            Xeon(R) Platinum 8370C CPU @ 2.80GHz<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">stepping       
            :
            6<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">microcode      
            :
            0xffffffff<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpu
            MHz        
            : 2793.436<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cache
            size      : 49152 KB<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">physical
            id     : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">siblings       
            :
            2<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">core
            id         : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpu
            cores       : 1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">apicid  
                   : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">initial
            apicid  : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">fpu            
            : yes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">fpu_exception  
            : yes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpuid
            level     : 21<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">wp             
            : yes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">flags          
            : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
            cmov pat pse36
            clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm
            constant_tsc
            rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma
            cx16 pcid sse4_1
            sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor
            lahf_lm abm
            3dnowprefetch invpcid_single tpr_shadow vnmi ept vpid
            fsgsbase bmi1 hle avx2
            smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap
            clflushopt avx512cd
            avx512bw avx512vl xsaveopt xsavec md_clear<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">bogomips       
            :
            5586.87<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">clflush
            size    : 64<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cache_alignment
            : 64<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">address
            sizes   : 46 bits physical, 48
            bits virtual<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">power
            management:<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">processor      
            : 1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">vendor_id      
            :
            GenuineIntel<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpu
            family      : 6<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">model          
            : 106<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">model
            name      : Intel(R)
            Xeon(R) Platinum 8370C CPU @ 2.80GHz<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">stepping       
            :
            6<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">microcode      
            :
            0xffffffff<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpu
            MHz         : 2793.436<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cache
            size      : 49152 KB<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">physical
            id     : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">siblings       
            :
            2<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">core
            id         : 0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpu
            cores       : 1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">apicid         
            : 1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">initial
            apicid  : 1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">fpu            
            : yes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">fpu_exception  
            : yes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cpuid
            level     : 21<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">wp             
            : yes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">flags          
            : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
            cmov pat pse36
            clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm
            constant_tsc
            rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma
            cx16 pcid sse4_1
            sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor
            lahf_lm abm
            3dnowprefetch invpcid_single tpr_shadow vnmi ept vpid
            fsgsbase bmi1 hle avx2
            smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap
            clflushopt avx512cd
            avx512bw avx512vl xsaveopt xsavec md_clear<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">bogomips       
            :
            5586.87<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">clflush
            size    : 64<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">cache_alignment
            : 64<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">address
            sizes   : 46 bits physical, 48
            bits virtual<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">power
            management:</span><span style="font-size:8pt;font-family:"Courier New""><span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>This is how
            Slurm sees one of the
            nodes:<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">[azccadmin@ricslurm-scheduler
            LID_CAVITY]$ <span style="background:yellow">scontrol show
              nodes</span><span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">NodeName=ricslurm-hpc-pg0-1
            Arch=x86_64
            CoresPerSocket=1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            CPUAlloc=0 CPUEfctv=1 CPUTot=1
            CPULoad=0.88<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            AvailableFeatures=cloud<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            ActiveFeatures=cloud<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Gres=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            NodeAddr=ricslurm-hpc-pg0-1
            NodeHostName=ricslurm-hpc-pg0-1 Version=22.05.3<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            OS=Linux 3.10.0-1127.19.1.el7.x86_64 #1
            SMP Tue Aug 25 17:23:54 UTC 2020<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            RealMemory=3072 AllocMem=0 FreeMem=1854
            Sockets=1 Boards=1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            State=IDLE+CLOUD ThreadsPerCore=2
            TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Partitions=hpc<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            BootTime=2022-12-12T17:42:27
            SlurmdStartTime=2022-12-12T17:42:28<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            LastBusyTime=2022-12-12T17:52:29<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            CfgTRES=cpu=1,mem=3G,billing=1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            AllocTRES=<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            CapWatts=n/a<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            CurrentWatts=0 AveWatts=0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            ExtSensorsJoules=n/s ExtSensorsWatts=0
            ExtSensorsTemp=n/s<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>This is the
            Slurm Job Control
            Script I have come up with to run the Vectis Job (I have set
            5x Node, 1x CPU,
            and 2x Threads – is this right?):<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#!/bin/bash<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Job name<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --job-name=run-grma<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            File to write standard output and error<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --output=run-grma.out<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --error=run-grma.err<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Partition for the cluster (you might not need that)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --partition=hpc<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Number of nodes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --nodes=5<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Number of CPUs per nodes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --ntasks-per-node=1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Number
            of CPUs per task<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --cpus-per-task=2<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            General<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">module
            purge<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Initialise VECTIS 2022.3b4<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">if
            [
            -d /shared/apps/RealisSimulation/2022.3/bin ]<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">then<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">   
            export PATH=$PATH:/shared/apps/RealisSimulation/2022.3/bin<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">else<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">   
            echo "Failed to Initialise VECTIS"<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">fi<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Run<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">vpre
            -V 2022.3 -np $SLURM_NTASKS
            /shared/data/LID_CAVITY/files/lid.GRD<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">vsolve
            -V 2022.3 -np $SLURM_NTASKS -mpi intel_2018.4 -rdmu
            /shared/data/LID_CAVITY/files/lid_no_write.inp<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>But, the
            submitted job will not run as it says that there is not
            enough CPUs.<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>Here is the
            debug log from slurmctld – where you can see that it
            is saying the job has requested 10 CPUs (which is what I
            want), but the hpc
            partition only has 5 (which I think is wrong?):<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.177]
            debug2: Processing RPC:
            REQUEST_NODE_INFO from UID=0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.370]
            debug2: Processing RPC:
            REQUEST_SUBMIT_BATCH_JOB from UID=20001<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3: _set_hostname:
            Using auth hostname for alloc_node: ricslurm-scheduler<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3: JobDesc:
            user_id=20001 JobId=N/A partition=hpc name=run-grma<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            cpus=10-4294967294 pn_min_cpus=2 core_spec=-1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            Nodes=5-[5] Sock/Node=65534 Core/Sock=65534
            Thread/Core=65534<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            pn_min_memory_job=18446744073709551615 pn_min_tmp_disk=-1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            immediate=0 reservation=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            features=(null) batch_features=(null)
            cluster_features=(null) prefer=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            req_nodes=(null) exc_nodes=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            time_limit=15-15 priority=-1 contiguous=0 shared=-1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            kill_on_node_fail=-1 script=#!/bin/bash<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">##
            Job name<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">#SBATCH
            --job-n...<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            argv="/shared/data/LID_CAVITY/slurm-runit.sh"<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
environment=XDG_SESSION_ID=12,HOSTNAME=ricslurm-scheduler,SELINUX_ROLE_REQUESTED=,...<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.371]
            debug3:   
            stdin=/dev/null stdout=/shared/data/LID_CAVITY/run-grma.out
            stderr=/shared/data/LID_CAVITY/run-grma.err<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            work_dir=/shared/data/LID_CAVITY
            alloc_node:sid=ricslurm-scheduler:13464<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            power_flags=<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            resp_host=(null) alloc_resp_port=0 other_port=0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            dependency=(null) account=(null) qos=(null) comment=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            mail_type=0 mail_user=(null) nice=0 num_tasks=5 open_mode=0
            overcommit=-1
            acctg_freq=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            network=(null) begin=Unknown cpus_per_task=2 requeue=-1
            licenses=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            end_time= signal=0@0 wait_all_nodes=-1 cpu_freq=<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            ntasks_per_node=1 ntasks_per_socket=-1 ntasks_per_core=-1
            ntasks_per_tres=-1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            mem_bind=0:(null) plane_size:65534<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            array_inx=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            burst_buffer=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            mcs_label=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            deadline=Unknown<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3:   
            bitflags=0x1a00c000 delay_boot=4294967294<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug3: job_submit/lua:
            slurm_lua_loadscript: skipping loading Lua script:
            /etc/slurm/job_submit.lua<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            lua: Setting reqswitch to
            1.<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            lua: returning.<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.372]
            debug2:
            _part_access_check: <span style="background:yellow">Job
              requested too many CPUs (10) of partition hpc(5)</span><span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.373]
            debug2: _part_access_check:
            Job requested too many CPUs (10) of partition hpc(5)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">[2022-12-13T09:05:01.373]
            debug2: JobId=1 can't run
            in partition hpc: More processors requested than permitted<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>The job will run
            fine if I use the
            below settings (across 5 nodes, but only using one of the
            two CPUs on each
            node):<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Number of nodes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --nodes=5<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Number of CPUs per nodes<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --ntasks-per-node=1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">##
            Number
            of CPUs per task<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">#SBATCH
            --cpus-per-task=1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>
            </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>Here is the
            successfully submitted Job details showing it using 5
            CPU’s (only one CPU per node) across 5x Nodes:<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">[azccadmin@ricslurm-scheduler
            LID_CAVITY]$ <span style="background:yellow">scontrol show
              job 3</span><span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">JobId=3
            JobName=run-grma<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            UserId=azccadmin(20001)
            GroupId=azccadmin(20001) MCS_label=N/A<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Priority=4294901757 Nice=0
            Account=(null) QOS=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            JobState=RUNNING Reason=None
            Dependency=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Requeue=1 Restarts=0 BatchFlag=1
            Reboot=0 ExitCode=0:0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            RunTime=00:07:35 TimeLimit=00:15:00
            TimeMin=N/A<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            SubmitTime=2022-12-12T17:32:01
            EligibleTime=2022-12-12T17:32:01<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            AccrueTime=2022-12-12T17:32:01<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            StartTime=2022-12-12T17:42:46
            EndTime=2022-12-12T17:57:46 Deadline=N/A<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            SuspendTime=None SecsPreSuspend=0
            LastSchedEval=2022-12-12T17:32:01 Scheduler=Main<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Partition=hpc
            AllocNode:Sid=ricslurm-scheduler:11723<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            ReqNodeList=(null) ExcNodeList=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            NodeList=ricslurm-hpc-pg0-[1-5]<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            BatchHost=ricslurm-hpc-pg0-1<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            <span style="background:yellow">NumNodes=5 NumCPUs=5
              NumTasks=5 CPUs/Task=1</span>
            ReqB:S:C:T=0:0:*:*<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            TRES=cpu=5,mem=15G,node=5,billing=5<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Socks/Node=* NtasksPerN:B:S:C=1:0:*:*
            CoreSpec=*<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            MinCPUsNode=1 MinMemoryCPU=3G
            MinTmpDiskNode=0<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Features=(null) DelayBoot=00:00:00<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            OverSubscribe=OK Contiguous=0
            Licenses=(null) Network=(null)<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Command=/shared/data/LID_CAVITY/slurm-runit.sh<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            WorkDir=/shared/data/LID_CAVITY<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            StdErr=/shared/data/LID_CAVITY/run-grma.err<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            StdIn=/dev/null<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            StdOut=/shared/data/LID_CAVITY/run-grma.out<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Switches=1@00:00:24<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm 0cm 0cm 36pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:8pt;font-family:"Courier New"">  
            Power=</span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span><br>
            </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>What am I doing
            wrong here - how do
            I get it to run the job on both CPU’s on all 5 nodes (i.e.
            fully utilising the
            available cluster resources of 10x CPUs)?<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>Regards<span></span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span><span> </span></span></p>
        <p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span>Gary<span></span></span></p>
      </div>
    </blockquote>
  </div>

</blockquote></div>