<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">My site recently updated to Slurm 21.08.6 and for the most part everything went fine. Two Ubuntu nodes however are having issues. Slurmd cannot execve the jobs on the nodes. As an example:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[jrlang@tmgt1 ~]$ salloc -A ARCC --nodes=1 --ntasks=20 -t 1:00:00 --bell --nodelist=mdgx01 --partition=dgx /bin/bash<o:p></o:p></p>
<p class="MsoNormal">salloc: Granted job allocation 2328489<o:p></o:p></p>
<p class="MsoNormal">[jrlang@tmgt1 ~]$ srun hostname<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 0 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 1 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 2 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 3 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 4 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 5 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 6 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 7 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 8 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 9 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 10 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 11 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 12 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 13 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 14 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 15 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 16 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 17 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 18 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">srun: error: task 19 launch failed: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Looking in slurmd-mdgx01.log we only see<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2022-03-24T14:44:02.408] [2328501.interactive] error: Failed to invoke task plugins: one of task_p_pre_setuid functions returned error<o:p></o:p></p>
<p class="MsoNormal">[2022-03-24T14:44:02.409] [2328501.interactive] error: job_manager: exiting abnormally: Slurmd could not execve job<o:p></o:p></p>
<p class="MsoNormal">[2022-03-24T14:44:02.411] [2328501.interactive] done with job<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Note that this issues didn’t occure with Slurm 20.11.8. <o:p>
</o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any ideas what could be causing the issue, cause I’m stumped?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jeff<o:p></o:p></p>
</div>
</body>
</html>