[slurm-users] I can't seem to use all the CPUs in my Cluster?

Tue Dec 13 17:24:11 UTC 2022

You can use slurm with hyperthreaded cores. It takes awareness when 
configuring and requesting the resources.

The can of worms you are opening is the stance (in HPC) that 
hyperthreading is detrimental. If you are using HPC as intended, I 
completely agree with this stance. The objective is to be as efficient 
as possible with the resources. If you have 4 cores running at 100%, you 
will lose efficiency by turning it into 8 hyperthreads and doing the 
same work. So, rather than increase core count, strive for 100% utilization.

That being said, if you are running interactive jobs, those may well 
benefit having hyperthreaded cores.  I have users that insist on using 
an HPC node to run a Linux desktop (not what HPC is meant for, to be 
sure). They definitely come out ahead with hyperthreading enabled.

So, it depends on a number of variables, many of which there is 
disagreement on whether they should even be in the equation.

To really get a better understanding, I would steer you away from 
Cyclecloud and encourage you to do your own install so you can learn the 
knobs and gauges that are hidden from view by middleware. This list can 
be a great source of help as well as the many articles, wikis and videos 
out there.

TLDR; If you are going to be running efficient HPC jobs, you are indeed 
better off with HT turned off.

Brian Andrus

On 12/13/2022 8:03 AM, Gary Mansell wrote:
> Hi, thanks for getting back to me.
>
> I have been doing some more experimenting, and I think that the issue 
> is because the Azure VMs for my nodes are HyperThreaded.
>
> Slurm sees the cluser as 5 nodes with 1 CPU and seems to ignore the 
> HyperThreading - so hence Slurm sees the cluster as a 5 CPU cluster 
> (and not 10 as I thought) - so it is correct that it can't run a 10 
> cpu job.
>
> Speaking with my CFD types - they say our code should not be run on HT 
> nodes, so I have switched to a different Azure VM sku for the nodes 
> without HT, and the CPU count in Slurm matches the count of those in 
> the VMs.
>
> So - does Slurm actually ignore HT cores, as I am supposing?
>
> Regards
> Gary
>
>
> On Tue, 13 Dec 2022 at 15:52, Brian Andrus <toomuchit at gmail.com> wrote:
>
>     Gary,
>
>     Well your first issue is using Cyclecloud, but that is mostly
>     opinion :)
>
>     Your error states there aren't enough CPUs in the partition, which
>     means we should take a look at the partition settings.
>
>     Take a look at 'scontrol show partition hpc' and see how many
>     nodes are assigned to it. Also check the state of the nodes with
>     'sinfo'
>
>     It would also be good to ensure the node settings are right. Run
>     'slurmd -C' on a node and see if the output matches what is in the
>     config.
>
>     Brian Andrus
>
>     On 12/13/2022 1:38 AM, Gary Mansell wrote:
>>
>>     Dear Slurm Users, perhaps you can help me with a problem that I
>>     am having using the Scheduler (I am new to this, so please
>>     forgive me for any stupid mistakes/misunderstandings).
>>
>>
>>     I am not able to submit a Multi-Threaded MPI job on a small demo
>>     cluster that I have setup using Azure CycleCloud that uses all
>>     the 10x CPUs on my cluster, and I don’t understand why – perhaps
>>     you can explain why and how I can fix this to use all available CPUs?
>>
>>     The hpc partition that I have setup consists of 5 nodes (Azure VM
>>     type = Standard_F2s_v2), each with 2 cpu’s (I presume that these
>>     are Hyperthreaded cores, rather than 2 cpus – but I am not
>>     certain of this)?
>>
>>     [azccadmin at ricslurm-hpc-pg0-1 ~]$ cat /proc/cpuinfo
>>
>>     processor       : 0
>>
>>     vendor_id       : GenuineIntel
>>
>>     cpu family      : 6
>>
>>     model           : 106
>>
>>     model name      : Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
>>
>>     stepping        : 6
>>
>>     microcode       : 0xffffffff
>>
>>     cpu MHz         : 2793.436
>>
>>     cache size      : 49152 KB
>>
>>     physical id     : 0
>>
>>     siblings        : 2
>>
>>     core id         : 0
>>
>>     cpu cores       : 1
>>
>>     apicid          : 0
>>
>>     initial apicid  : 0
>>
>>     fpu             : yes
>>
>>     fpu_exception   : yes
>>
>>     cpuid level     : 21
>>
>>     wp              : yes
>>
>>     flags           : fpu vme de pse tsc msr pae mce cx8 apic sep
>>     mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht
>>     syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology
>>     eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2
>>     movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm
>>     3dnowprefetch invpcid_single tpr_shadow vnmi ept vpid fsgsbase
>>     bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed
>>     adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec
>>     md_clear
>>
>>     bogomips        : 5586.87
>>
>>     clflush size    : 64
>>
>>     cache_alignment : 64
>>
>>     address sizes   : 46 bits physical, 48 bits virtual
>>
>>     power management:
>>
>>     processor       : 1
>>
>>     vendor_id       : GenuineIntel
>>
>>     cpu family      : 6
>>
>>     model           : 106
>>
>>     model name      : Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
>>
>>     stepping        : 6
>>
>>     microcode       : 0xffffffff
>>
>>     cpu MHz         : 2793.436
>>
>>     cache size      : 49152 KB
>>
>>     physical id     : 0
>>
>>     siblings        : 2
>>
>>     core id         : 0
>>
>>     cpu cores       : 1
>>
>>     apicid          : 1
>>
>>     initial apicid  : 1
>>
>>     fpu             : yes
>>
>>     fpu_exception   : yes
>>
>>     cpuid level     : 21
>>
>>     wp              : yes
>>
>>     flags           : fpu vme de pse tsc msr pae mce cx8 apic sep
>>     mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht
>>     syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology
>>     eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2
>>     movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm
>>     3dnowprefetch invpcid_single tpr_shadow vnmi ept vpid fsgsbase
>>     bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed
>>     adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec
>>     md_clear
>>
>>     bogomips        : 5586.87
>>
>>     clflush size    : 64
>>
>>     cache_alignment : 64
>>
>>     address sizes   : 46 bits physical, 48 bits virtual
>>
>>     power management:
>>
>>     This is how Slurm sees one of the nodes:
>>
>>     [azccadmin at ricslurm-scheduler LID_CAVITY]$ scontrol show nodes
>>
>>     NodeName=ricslurm-hpc-pg0-1 Arch=x86_64 CoresPerSocket=1
>>
>>        CPUAlloc=0 CPUEfctv=1 CPUTot=1 CPULoad=0.88
>>
>>        AvailableFeatures=cloud
>>
>>        ActiveFeatures=cloud
>>
>>        Gres=(null)
>>
>>        NodeAddr=ricslurm-hpc-pg0-1 NodeHostName=ricslurm-hpc-pg0-1
>>     Version=22.05.3
>>
>>        OS=Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25
>>     17:23:54 UTC 2020
>>
>>        RealMemory=3072 AllocMem=0 FreeMem=1854 Sockets=1 Boards=1
>>
>>        State=IDLE+CLOUD ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A
>>     MCS_label=N/A
>>
>>        Partitions=hpc
>>
>>        BootTime=2022-12-12T17:42:27 SlurmdStartTime=2022-12-12T17:42:28
>>
>>        LastBusyTime=2022-12-12T17:52:29
>>
>>        CfgTRES=cpu=1,mem=3G,billing=1
>>
>>        AllocTRES=
>>
>>        CapWatts=n/a
>>
>>        CurrentWatts=0 AveWatts=0
>>
>>        ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
>>
>>     This is the Slurm Job Control Script I have come up with to run
>>     the Vectis Job (I have set 5x Node, 1x CPU, and 2x Threads – is
>>     this right?):
>>
>>     #!/bin/bash
>>
>>     ## Job name
>>
>>     #SBATCH --job-name=run-grma
>>
>>     #
>>
>>     ## File to write standard output and error
>>
>>     #SBATCH --output=run-grma.out
>>
>>     #SBATCH --error=run-grma.err
>>
>>     #
>>
>>     ## Partition for the cluster (you might not need that)
>>
>>     #SBATCH --partition=hpc
>>
>>     #
>>
>>     ## Number of nodes
>>
>>     #SBATCH --nodes=5
>>
>>     #
>>
>>     ## Number of CPUs per nodes
>>
>>     #SBATCH --ntasks-per-node=1
>>
>>     #
>>
>>     ## Number of CPUs per task
>>
>>     #SBATCH --cpus-per-task=2
>>
>>     #
>>
>>     ## General
>>
>>     module purge
>>
>>     ## Initialise VECTIS 2022.3b4
>>
>>     if [ -d /shared/apps/RealisSimulation/2022.3/bin ]
>>
>>     then
>>
>>         export PATH=$PATH:/shared/apps/RealisSimulation/2022.3/bin
>>
>>     else
>>
>>         echo "Failed to Initialise VECTIS"
>>
>>     fi
>>
>>     ## Run
>>
>>     vpre -V 2022.3 -np $SLURM_NTASKS
>>     /shared/data/LID_CAVITY/files/lid.GRD
>>
>>     vsolve -V 2022.3 -np $SLURM_NTASKS -mpi intel_2018.4 -rdmu
>>     /shared/data/LID_CAVITY/files/lid_no_write.inp
>>
>>     But, the submitted job will not run as it says that there is not
>>     enough CPUs.
>>
>>     Here is the debug log from slurmctld – where you can see that it
>>     is saying the job has requested 10 CPUs (which is what I want),
>>     but the hpc partition only has 5 (which I think is wrong?):
>>
>>     [2022-12-13T09:05:01.177] debug2: Processing RPC:
>>     REQUEST_NODE_INFO from UID=0
>>
>>     [2022-12-13T09:05:01.370] debug2: Processing RPC:
>>     REQUEST_SUBMIT_BATCH_JOB from UID=20001
>>
>>     [2022-12-13T09:05:01.371] debug3: _set_hostname: Using auth
>>     hostname for alloc_node: ricslurm-scheduler
>>
>>     [2022-12-13T09:05:01.371] debug3: JobDesc: user_id=20001
>>     JobId=N/A partition=hpc name=run-grma
>>
>>     [2022-12-13T09:05:01.371] debug3: cpus=10-4294967294
>>     pn_min_cpus=2 core_spec=-1
>>
>>     [2022-12-13T09:05:01.371] debug3: Nodes=5-[5] Sock/Node=65534
>>     Core/Sock=65534 Thread/Core=65534
>>
>>     [2022-12-13T09:05:01.371] debug3:
>>     pn_min_memory_job=18446744073709551615 pn_min_tmp_disk=-1
>>
>>     [2022-12-13T09:05:01.371] debug3: immediate=0 reservation=(null)
>>
>>     [2022-12-13T09:05:01.371] debug3: features=(null)
>>     batch_features=(null) cluster_features=(null) prefer=(null)
>>
>>     [2022-12-13T09:05:01.371] debug3: req_nodes=(null) exc_nodes=(null)
>>
>>     [2022-12-13T09:05:01.371] debug3: time_limit=15-15 priority=-1
>>     contiguous=0 shared=-1
>>
>>     [2022-12-13T09:05:01.371] debug3: kill_on_node_fail=-1
>>     script=#!/bin/bash
>>
>>     ## Job name
>>
>>     #SBATCH --job-n...
>>
>>     [2022-12-13T09:05:01.371] debug3:
>>     argv="/shared/data/LID_CAVITY/slurm-runit.sh"
>>
>>     [2022-12-13T09:05:01.371] debug3:
>>     environment=XDG_SESSION_ID=12,HOSTNAME=ricslurm-scheduler,SELINUX_ROLE_REQUESTED=,...
>>
>>     [2022-12-13T09:05:01.371] debug3: stdin=/dev/null
>>     stdout=/shared/data/LID_CAVITY/run-grma.out
>>     stderr=/shared/data/LID_CAVITY/run-grma.err
>>
>>     [2022-12-13T09:05:01.372] debug3:
>>     work_dir=/shared/data/LID_CAVITY
>>     alloc_node:sid=ricslurm-scheduler:13464
>>
>>     [2022-12-13T09:05:01.372] debug3: power_flags=
>>
>>     [2022-12-13T09:05:01.372] debug3: resp_host=(null)
>>     alloc_resp_port=0 other_port=0
>>
>>     [2022-12-13T09:05:01.372] debug3: dependency=(null)
>>     account=(null) qos=(null) comment=(null)
>>
>>     [2022-12-13T09:05:01.372] debug3: mail_type=0 mail_user=(null)
>>     nice=0 num_tasks=5 open_mode=0 overcommit=-1 acctg_freq=(null)
>>
>>     [2022-12-13T09:05:01.372] debug3: network=(null) begin=Unknown
>>     cpus_per_task=2 requeue=-1 licenses=(null)
>>
>>     [2022-12-13T09:05:01.372] debug3: end_time= signal=0 at 0
>>     wait_all_nodes=-1 cpu_freq=
>>
>>     [2022-12-13T09:05:01.372] debug3: ntasks_per_node=1
>>     ntasks_per_socket=-1 ntasks_per_core=-1 ntasks_per_tres=-1
>>
>>     [2022-12-13T09:05:01.372] debug3: mem_bind=0:(null) plane_size:65534
>>
>>     [2022-12-13T09:05:01.372] debug3: array_inx=(null)
>>
>>     [2022-12-13T09:05:01.372] debug3: burst_buffer=(null)
>>
>>     [2022-12-13T09:05:01.372] debug3: mcs_label=(null)
>>
>>     [2022-12-13T09:05:01.372] debug3: deadline=Unknown
>>
>>     [2022-12-13T09:05:01.372] debug3: bitflags=0x1a00c000
>>     delay_boot=4294967294
>>
>>     [2022-12-13T09:05:01.372] debug3: job_submit/lua:
>>     slurm_lua_loadscript: skipping loading Lua script:
>>     /etc/slurm/job_submit.lua
>>
>>     [2022-12-13T09:05:01.372] lua: Setting reqswitch to 1.
>>
>>     [2022-12-13T09:05:01.372] lua: returning.
>>
>>     [2022-12-13T09:05:01.372] debug2: _part_access_check: Job
>>     requested too many CPUs (10) of partition hpc(5)
>>
>>     [2022-12-13T09:05:01.373] debug2: _part_access_check: Job
>>     requested too many CPUs (10) of partition hpc(5)
>>
>>     [2022-12-13T09:05:01.373] debug2: JobId=1 can't run in partition
>>     hpc: More processors requested than permitted
>>
>>     The job will run fine if I use the below settings (across 5
>>     nodes, but only using one of the two CPUs on each node):
>>
>>     ## Number of nodes
>>
>>     #SBATCH --nodes=5
>>
>>     #
>>
>>     ## Number of CPUs per nodes
>>
>>     #SBATCH --ntasks-per-node=1
>>
>>     #
>>
>>     ## Number of CPUs per task
>>
>>     #SBATCH --cpus-per-task=1
>>
>>     Here is the successfully submitted Job details showing it using 5
>>     CPU’s (only one CPU per node) across 5x Nodes:
>>
>>     [azccadmin at ricslurm-scheduler LID_CAVITY]$ scontrol show job 3
>>
>>     JobId=3 JobName=run-grma
>>
>>        UserId=azccadmin(20001) GroupId=azccadmin(20001) MCS_label=N/A
>>
>>        Priority=4294901757 Nice=0 Account=(null) QOS=(null)
>>
>>        JobState=RUNNING Reason=None Dependency=(null)
>>
>>        Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>>
>>        RunTime=00:07:35 TimeLimit=00:15:00 TimeMin=N/A
>>
>>        SubmitTime=2022-12-12T17:32:01 EligibleTime=2022-12-12T17:32:01
>>
>>        AccrueTime=2022-12-12T17:32:01
>>
>>        StartTime=2022-12-12T17:42:46 EndTime=2022-12-12T17:57:46
>>     Deadline=N/A
>>
>>        SuspendTime=None SecsPreSuspend=0
>>     LastSchedEval=2022-12-12T17:32:01 Scheduler=Main
>>
>>        Partition=hpc AllocNode:Sid=ricslurm-scheduler:11723
>>
>>        ReqNodeList=(null) ExcNodeList=(null)
>>
>>        NodeList=ricslurm-hpc-pg0-[1-5]
>>
>>        BatchHost=ricslurm-hpc-pg0-1
>>
>>     NumNodes=5 NumCPUs=5 NumTasks=5 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>>
>>        TRES=cpu=5,mem=15G,node=5,billing=5
>>
>>        Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
>>
>>        MinCPUsNode=1 MinMemoryCPU=3G MinTmpDiskNode=0
>>
>>        Features=(null) DelayBoot=00:00:00
>>
>>        OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>>
>>     Command=/shared/data/LID_CAVITY/slurm-runit.sh
>>
>>        WorkDir=/shared/data/LID_CAVITY
>>
>>     StdErr=/shared/data/LID_CAVITY/run-grma.err
>>
>>        StdIn=/dev/null
>>
>>     StdOut=/shared/data/LID_CAVITY/run-grma.out
>>
>>        Switches=1 at 00:00:24
>>
>>        Power=
>>
>>
>>     What am I doing wrong here - how do I get it to run the job on
>>     both CPU’s on all 5 nodes (i.e. fully utilising the available
>>     cluster resources of 10x CPUs)?
>>
>>     Regards
>>
>>     Gary
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20221213/c9439906/attachment-0001.htm>