<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
I'm experiencing a strange issue related to a CPU swap (8352Y -> 6326) on two of our nodes. I adapted the slurm.conf to accommodate the new CPU:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted4">
<div class="ContentPasted4"><span style="font-family: "Courier New", monospace;">slurm.conf: NodeName=ice27[57-58] CPUs=64 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 Realmemory=257550 MemSpecLimit=12000</span></div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted4">
which is also what slurmd -C autodetects: <span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;" class="ContentPasted5">NodeName=ice2758 CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=257578</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted2 ContentPasted3">
Slurm 22.05.7 (compiled from source)<br class="ContentPasted2">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted2 ContentPasted3">
Kernel: <span style="">4.18.0-372.32.1.el8_6.x86_64</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted2 ContentPasted3">
<div class="ContentPasted3">OS: <span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">Rocky Linux release 8.6 (Green Obsidian)</span></div>
<div class="ContentPasted3"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;"><br>
</span></div>
<div class="ContentPasted3"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">All nodes boot the same OS image (PXE) and therefore have the same SW.</span></div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
When I try to run a simple single node job (exclusive) on ice2758, the job immediately fails and the nodes is drained with "batch job complete failure". <span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">From the nodes slurmd.log:</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;"><br>
</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted1">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted9"># grep 224313 slurmd.ice2758.log | grep -v debug<br class="ContentPasted9">
<span style="font-family: "Courier New", monospace; font-size: 11pt;">[2023-02-08T18:09:35.026] Launching batch job 224313 for UID 1234502026</span><br>
</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted1">
<div class="ContentPasted1 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:35.037] [224313.batch] task/affinity: init: task affinity plugin loaded
with CPU mask 0xffffffffffffffff</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted8">[2023-02-08T18:09:35.037] [224313.batch] cred/mCPUs=64unge: init: Munge credential
signature plugin loaded</span></div>
<div class="ContentPasted1 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:35.049] [224313.batch] error: xcpuinfo_abs_to_mac: failed</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:35.049] [224313.batch] error: unable to build job physical cores</span></div>
<div class="ContentPasted1 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:35.050] [224313.batch] task/cgroup: _memcg_initialize: job: alloc=245571MB
mem.limit=245571MB memsw.limit=unlimited</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:35.050] [224313.batch] task/cgroup: _memcg_initialize: step: alloc=245571MB mem.limit=245571MB
memsw.limit=unlimited</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:35.061] [224313.batch] starting 1 tasks</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:35.061] [224313.batch] task 0 (20552) started 2023-02-08T18:09:35</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:35.062] [224313.batch] error: common_file_write_uint32s: write pid 20552 to /sys/fs/cgroup/cpuset/slurm/uid_1234502026/job_224313/step_batch/cgroup.procs
failed: No space left on device</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:35.062] [224313.batch] error: unable to add pids to '/sys/fs/cgroup/cpuset/slurm/uid_1234502026/job_224313/step_batch'</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:35.062] [224313.batch] error: task_g_pre_set_affinity: No space left on device</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:35.062] [224313.batch] error: _exec_wait_child_wait_for_parent: failed: Resource temporarily
unavailable</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">[2023-02-08T18:09:36.065] [224313.batch] error: job_manager: exiting abnormally: Slurmd could not execve
job</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:36.065] [224313.batch] job 224313 completed with slurm_rc = 4020, job_rc = 0</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[2023-02-08T18:09:36.068] [224313.batch] done with job</span></div>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted10">
There is plenty of space (= memory, bc PXE boot) available. lscgroup and cat /proc/cgroups shows far less than 1000 cgroups.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
I then compared this to other nodes and what they are reporting when it comes to cgroups during job launch:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"># grep -i "job abstract" slurmd*log | grep 2023-02-08</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<div class="ContentPasted0"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">slurmd.banner2401.log:[2023-02-08T18:20:34.102] [224315.batch] debug: task/cgroup: task_cgroup_cpuset_create:
</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(128, 255, 128);">job abstract cores are '0-31'</span></div>
<div class="ContentPasted0 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> slurmd.ice2701.log:[2023-02-08T18:27:05.391] [224319.batch] debug: task/cgroup: task_cgroup_cpuset_create:
</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(128, 255, 128);">job abstract cores are '0-71'</span></div>
<div class="ContentPasted0 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> slurmd.ice2758.log:[2023-02-08T18:09:35.049] [224313.batch] debug: task/cgroup: task_cgroup_cpuset_create:
</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">job abstract cores are '0-63'</span></div>
<div><br>
</div>
<div class="ContentPasted0 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"># psh banner2401,ice2701,ice2758 slurmd -C | grep -vi uptime</span></div>
<div class="ContentPasted0 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">banner2401: NodeName=banner2401
</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(128, 255, 128);">CPUs=64</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=193090</span></div>
<div class="ContentPasted0 elementToProof"><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ice2701: NodeName=ice2701 </span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(128, 255, 128);">CPUs=144</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Boards=1 SocketsPerBoard=2 CoresPerSocket=36 ThreadsPerCore=2 RealMemory=257552</span></div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ice2758: NodeName=ice2758
</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 0);">CPUs=64</span><span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=257578</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted0" style="font-size: 11pt; margin: 0px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; margin: 0px; background-color: rgb(255, 255, 255);"><span class="ContentPasted6 ContentPasted7" style="font-family: Calibri, Helvetica, sans-serif; margin: 0px; background-color: rgb(255, 255, 255);">To
me, it looks like slurmd -C is correctly detecting the CPUs, but when it comes to cgroups, the plugin somehow addresses all cores, even the HT ones, whereas on the other two nodes shown, the cgroups plugin is only addressing half, the real cores, of the node.
A reboot does not fix this problem. We're happy with how slurm works for all the other nodes, just the two which had their CPUs changed are behaving differently. What am I missing here?</span></span></span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-size: 11pt; font-family: Calibri, Helvetica, sans-serif; margin: 0px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted6">Cheers,</span><br class="Apple-interchange-newline ContentPasted6">
</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-size: 11pt; font-family: Calibri, Helvetica, sans-serif; margin: 0px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted6">Florian</span></span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="font-family: "Courier New", monospace; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div class="elementToProof">
<div id="Signature">
<div style="">
<div style=""></div>
</div>
</div>
</div>
</body>
</html>