<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Hey guys,</span></div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">We noticed that Slurm memory constrain options (via cgroups) on CentOS 7 upstream kernel <= 4.5 breaks cgroup task plugin. Reproduced with Slurm 21.08.8.</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Jobs fail to start:</span></div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"># srun --mem=1MB hostname</span><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">srun: error: task 0 launch failed: Slurmd could not execve job</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">In slurmd log (with debug) we see:</span></div>
<div><br>
</div>
<div></div>
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.802] [6.0] debug2: setup for a launch_task</span>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.803] [6.0] debug: jobacct_gather/linux: init: Job accounting gather LINUX plugin loaded</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.804] debug2: Finish processing RPC: REQUEST_LAUNCH_TASKS</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.804] [6.0] debug2: profile signaling type Task</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.805] [6.0] debug: Message thread started pid = 3386</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.807] [6.0] debug2: hwloc_topology_init</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.807] [6.0] debug2: xcpuinfo_hwloc_topo_load: xml file (/cm/local/apps/slurm/var/spool/hwloc_topo_whole.xml) found</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.808] [6.0] debug: CPUs:2 Boards:1 Sockets:1 CoresPerSocket:1 ThreadsPerCore:2</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.810] [6.0] debug: cgroup/v1: init: Cgroup v1 plugin loaded</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.814] [6.0] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: total:1998M allowed:100%(enforced), swap:0%(permissive),
max:100%(1998M) max+swap:100%(3996M) min:25M kmem:100%(1998M enforced) min:25M swappiness:1(set)</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.814] [6.0] debug: task/cgroup: init: memory enforcement enabled</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.814] [6.0] debug: task/cgroup: init: Tasks containment cgroup plugin loaded</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.815] [6.0] cred/munge: init: Munge credential signature plugin loaded</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.817] [6.0] debug: job_container/none: init: job_container none plugin loaded</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.817] [6.0] debug: mpi type = none</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.819] [6.0] debug2: Before call to spank_init()</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.819] [6.0] debug: spank: opening plugin stack /cm/shared/apps/slurm/var/etc/slurm/plugstack.conf</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.819] [6.0] debug2: After call to spank_init()</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.819] [6.0] debug: mpi type = (null)</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.819] [6.0] debug: mpi/none: p_mpi_hook_slurmstepd_prefork: mpi/none: slurmstepd prefork</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.825] [6.0] task/cgroup: _memcg_initialize: job: alloc=0MB mem.limit=1998MB memsw.limit=unlimited</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.825] [6.0] debug: task_g_pre_setuid: task/cgroup: Unspecified error</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.825] [6.0] error: Failed to invoke task plugins: one of task_p_pre_setuid functions returned error</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.825] [6.0] debug: _fork_all_tasks failed</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.826] [6.0] debug: signaling condition</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.826] [6.0] debug2: step_terminate_monitor will run for 60 secs</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.826] [6.0] debug2: step_terminate_monitor is stopping</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.826] [6.0] debug2: _monitor exit code: 0</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.826] [6.0] debug2: switch/none: switch_p_job_postfini: Sending SIGKILL to pgid 3386</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.841] [6.0] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.841] [6.0] debug2: Before call to spank_fini()</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.841] [6.0] debug2: After call to spank_fini()</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.841] [6.0] error: job_manager: exiting abnormally: Slurmd could not execve job</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.841] [6.0] debug: Sending launch resp rc=4020</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.845] [6.0] debug2: Rank 0 has no children slurmstepd</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.845] [6.0] debug2: _one_step_complete_msg: first=0, last=0</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.851] [6.0] debug2: false, shutdown</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.851] [6.0] debug: Message thread exited</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.852] [6.0] done with job</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.865] debug2: Start processing RPC: REQUEST_TERMINATE_JOB</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.865] debug2: Processing RPC: REQUEST_TERMINATE_JOB</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.865] debug: _rpc_terminate_job: uid = 450 JobId=6</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.865] debug: credential for job 6 revoked</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.865] debug2: No steps in jobid 6 to send signal 18</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.866] debug2: No steps in jobid 6 to send signal 15</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.866] debug2: set revoke expiration for jobid 6 to 1660899711 UTS</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.866] debug: Waiting for job 6's prolog to complete</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.866] debug: Finished wait for job 6's prolog to complete</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.867] debug: [job 6] attempting to run epilog [/cm/local/apps/cmd/scripts/epilog]</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.880] debug: completed epilog for jobid 6</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.881] debug: JobId=6: sent epilog complete msg: rc = 0</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[2022-08-19T10:59:51.881] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB</span></div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">cgroups.conf:</span></div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf"</span>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">CgroupMountpoint="/sys/fs/cgroup"</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">CgroupAutomount=no</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">ConstrainCores=no</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">ConstrainRAMSpace=yes <---------------</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">ConstrainSwapSpace=no</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">ConstrainDevices=no</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">ConstrainKmemSpace=yes <--------------</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">AllowedRamSpace=100.00</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">AllowedSwapSpace=0.00</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">MinKmemSpace=25</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">MaxKmemPercent=100.00</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">MemorySwappiness=1</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">MaxRAMPercent=100.00</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">MaxSwapPercent=100.00</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">MinRAMSpace=25</span></div>
</div>
<div><br>
</div>
<div class="elementToProof"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">If only one of those 2 options are enabled, then the issue is gone. Updating to kernel >=4.6 fixes the plugin.</span></div>
<div><br>
</div>
<div class="elementToProof"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Has anyone faced this issue?</span></div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Best reagrds,</span></div>
<div><br>
</div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Taras</span></div>
</body>
</html>