<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Xiang,</p>
<p>From what I've of the original question, gres.conf may be another place to verify the setup that only one core is being allocated per gpu request:
<a class="moz-txt-link-freetext" href="https://slurm.schedmd.com/gres.conf.html">
https://slurm.schedmd.com/gres.conf.html</a></p>
<p>Seeing the run submission line and gres.conf might help others give you further advise.</p>
<p>To Jeffrey's email: the concept of oversubscription may be beneficial versus changing resource inventories:
<a class="moz-txt-link-freetext" href="https://slurm.schedmd.com/cons_res_share.html">
https://slurm.schedmd.com/cons_res_share.html</a></p>
<p>Best,</p>
<p>Cyrus<br>
</p>
<div class="moz-cite-prefix">On 2/8/19 9:44 AM, Jeffrey Frey wrote:<br>
</div>
<blockquote type="cite" cite="mid:D842E036-79AF-46A9-9952-2E22C7F3B3AD@udel.edu">
Documentation for CR_CPU:
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">CR_CPU</div>
<div class="">CPUs are consumable resources. Configure the number of CPUs on each node, which may be equal to the count of cores or hyper-threads on the node depending upon the desired minimum resource allocation. The node's Boards, Sockets, CoresPerSocket andThreadsPerCore may
optionally be configured and result in job allocations which have improved locality;
<b class="">however doing so will prevent more than one job being from being allocated on each core.</b></div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">So once you're configured node(s) with ThreadsPerCore=N, the cons_res plugin still forces tasks to span all threads on a core. Elsewhere in the documentation it is stated:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class=""><b class="">Note that the Slurm can allocate resources to jobs down to the resolution of a core.</b></div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">So you MUST treat a thread as a core if you want to schedule individual threads. I can confirm this using the config:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">SelectTypeParameters = CR_CPU_MEMORY</div>
<div class="">NodeName=n[003,008] CPUS=16 Sockets=2 CoresPerSocket=4 ThreadsPerCore=2</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Submitting a 1-cpu job, if I check the cpuset assigned to a job on n003:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">$ cat /sys/fs/cgroup/cpuset/slurm/{uid}/{job}/cpuset.cpus</div>
4,12</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">If I instead configure as:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">SelectTypeParameters = CR_Core_Memory</div>
<div class="">NodeName=n[003,008] CPUS=16 Sockets=2 CoresPerSocket=8 ThreadsPerCore=1</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Slurm will schedule "cores" 0-15 to jobs, which the cpuset cgroup happily accepts. A 1-cpu job then shows:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">$ cat /sys/fs/cgroup/cpuset/slurm/{uid}/{job}/cpuset.cpus</div>
<div class="">2</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">and a 2-cpu job shows:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding:
0px;" class="">
<div class="">$ cat /sys/fs/cgroup/cpuset/slurm/{uid}/{job}/cpuset.cpus</div>
<div class="">4,12</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">On Feb 8, 2019, at 5:09 AM, Antony Cleave <<a href="mailto:antony.cleave@gmail.com" class="" moz-do-not-send="true">antony.cleave@gmail.com</a>> wrote:<br class="">
<br class="">
if you want slurm to just ignore the difference between physical and logical cores then you can change <br class="">
SelectTypeParameters=CR_Core<br class="">
to<br class="">
SelectTypeParameters=CR_CPU<br class="">
<br class="">
and then it will treat threads as CPUs and then it will let you start the number of tasks you expect<br class="">
<br class="">
Antony<br class="">
<br class="">
On Thu, 7 Feb 2019 at 18:04, Jeffrey Frey <<a href="mailto:frey@udel.edu" class="" moz-do-not-send="true">frey@udel.edu</a>> wrote:<br class="">
Your nodes are hyperthreaded (ThreadsPerCore=2). Slurm always allocates _all threads_ associated with a selected core to jobs. So you're being assigned both threads on core N.<br class="">
<br class="">
<br class="">
On our development-partition nodes we configure the threads as cores, e.g.<br class="">
<br class="">
<br class="">
NodeName=moria CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1<br class="">
<br class="">
<br class="">
to force Slurm to schedule the threads separately.<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On Feb 7, 2019, at 12:10 PM, Xiang Gao <a class="moz-txt-link-rfc2396E" href="mailto:qasdfgtyuiop@gmail.com">
<qasdfgtyuiop@gmail.com></a> wrote:<br class="">
<br class="">
Hi All,<br class="">
<br class="">
We configured slurm on a server with 8 GPU and 16 CPUs and want to use slurm to scheduler for both CPU and GPU jobs. We observed an unexpected behavior that, although there are 16 CPUs, slurm only schedule 8 jobs to run even if there are jobs not asking any
GPU. If I inspect detailed information using `scontrol show job`, I see some strange thing on some job that just ask for 1 CPU:<br class="">
<br class="">
NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1<br class="">
<br class="">
If I understand these concepts correctly, as the number of nodes is 1, number of tasks is 1, and number of cpus/task is 1, in principle there is no way that the final number of CPUs is 2. I'm not sure if I misunderstand the concepts, configure slurm wrongly,
or this is a bug. So I come for help.<br class="">
<br class="">
Some related config are:<br class="">
<br class="">
# COMPUTE NODES <br class="">
NodeName=moria CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=120000 Gres=gpu:gtx1080ti:2,gpu:titanv:3,gpu:v100:1,gpu:gp100:2<br class="">
State=UNKNOWN <br class="">
PartitionName=queue Nodes=moria Default=YES MaxTime=INFINITE State=UP<br class="">
<br class="">
# SCHEDULING <br class="">
FastSchedule=1 <br class="">
SchedulerType=sched/backfill <br class="">
GresTypes=gpu <br class="">
SelectType=select/cons_res <br class="">
SelectTypeParameters=CR_Core<br class="">
<br class="">
Best,<br class="">
Xiang Gao<br class="">
</blockquote>
<br class="">
<br class="">
::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">
Jeffrey T. Frey, Ph.D.<br class="">
Systems Programmer V / HPC Management<br class="">
Network & Systems Services / College of Engineering<br class="">
University of Delaware, Newark DE 19716<br class="">
Office: (302) 831-6034 Mobile: (302) 419-4976<br class="">
::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
<div class=""><br class="">
::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">
Jeffrey T. Frey, Ph.D.<br class="">
Systems Programmer V / HPC Management<br class="">
Network & Systems Services / College of Engineering<br class="">
University of Delaware, Newark DE 19716<br class="">
Office: (302) 831-6034 Mobile: (302) 419-4976<br class="">
::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">
<br class="">
<br class="">
<br class="">
</div>
<br class="">
</div>
</blockquote>
</body>
</html>