<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">As of Slurm 21.08, you have to recompile your spank plugins when you upgrade e.g. from 20.11 to 21.08, or from 21.08 to 22.05.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://github.com/SchedMD/slurm/blob/slurm-21.08/RELEASE_NOTES#L18-L19">https://github.com/SchedMD/slurm/blob/slurm-21.08/RELEASE_NOTES#L18-L19</a><o:p></o:p></p>
<p class="MsoNormal"><a href="https://bugs.schedmd.com/show_bug.cgi?id=12318">https://bugs.schedmd.com/show_bug.cgi?id=12318</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Alan Orth<br>
<b>Sent:</b> Tuesday, August 16, 2022 10:36 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] "Incompatible plugin version" after upgrade<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellspacing="3" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Dear list,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Twice this month I've had jobs stuck in completing state (CG). When I go to the compute node and check slurmd.log I see a message about "incompatible plugin version", for example:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<pre>[2022-08-16T03:36:25.823] [748139.batch] done with job<o:p></o:p></pre>
<pre>[2022-08-16T12:54:21.404] [748139.extern] plugin_load_from_file: Incompatible Slurm plugin /usr/lib64/slurm/hash_k12.so version (22.05.3<o:p></o:p></pre>
<pre>)<o:p></o:p></pre>
<pre>[2022-08-16T12:54:21.404] [748139.extern] error: Couldn't load specified plugin name for hash/k12: Incompatible plugin version<o:p></o:p></pre>
<pre>[2022-08-16T12:54:21.404] [748139.extern] error: cannot create hash context for K12<o:p></o:p></pre>
<pre>[2022-08-16T12:54:21.404] [748139.extern] error: slurm_send_node_msg: hash_g_compute: REQUEST_STEP_COMPLETE has error<o:p></o:p></pre>
<pre>[2022-08-16T12:54:21.404] [748139.extern] error: Rank 0 failed sending step completion message directly to slurmctld, retrying<o:p></o:p></pre>
</div>
<div>
<p class="MsoNormal">For context, I did a minor upgrade of SLURM yesterday (22.05.2 to 22.05.3), so it's possible there is an incompatible version somewhere, but if I look up earlier in the log file I see that the running version of slurmd is correct and still
prints that error right after startup:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<pre>[2022-08-15T22:27:59.865] slurmd version 22.05.3 started<o:p></o:p></pre>
<pre>[2022-08-15T22:27:59.867] slurmd started on Mon, 15 Aug 2022 22:27:59 +0300<o:p></o:p></pre>
<pre>[2022-08-15T22:27:59.869] CPUs=48 Boards=1 Sockets=2 Cores=24 Threads=1 Memory=386525 TmpDisk=71645 Uptime=2679297 CPUSpecList=(null) Fe<o:p></o:p></pre>
<pre>aturesAvail=(null) FeaturesActive=(null)<o:p></o:p></pre>
<pre>[2022-08-16T02:36:10.020] [748139.batch] plugin_load_from_file: Incompatible Slurm plugin /usr/lib64/slurm/hash_k12.so version (22.05.3)<o:p></o:p></pre>
<pre>[2022-08-16T02:36:10.020] [748139.batch] error: Couldn't load specified plugin name for hash/k12: Incompatible plugin version<o:p></o:p></pre>
<pre>[2022-08-16T02:36:10.022] [748139.batch] error: cannot create hash context for K12<o:p></o:p></pre>
</div>
<div>
<p class="MsoNormal">I'm running SLURM 22.05.3. The slurmctld is running on CentOS 7, and compute nodes are on CentOS Stream 8 (not sure if this matters?).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks for any advice,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Alan Orth<br>
<a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpicturingjordan.com%2F&data=05%7C01%7Clyeager%40nvidia.com%7C7980b43c59e243a0b1da08da7f9d67fe%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637962611285459013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0CbO2KUBoi05CaMqMrZuUVa9JWso818sRYEHdHVk2ms%3D&reserved=0" target="_blank">https://picturingjordan.com</a><br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fenglishbulgaria.net%2F&data=05%7C01%7Clyeager%40nvidia.com%7C7980b43c59e243a0b1da08da7f9d67fe%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637962611285459013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NDOmhxZik8lYdkQaFEw1G1OKppLByvPcp6NQCgnIKtI%3D&reserved=0" target="_blank">https://englishbulgaria.net</a><br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmjanja.ch%2F&data=05%7C01%7Clyeager%40nvidia.com%7C7980b43c59e243a0b1da08da7f9d67fe%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637962611285459013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1D9bsDMfVb7Znuy%2FuERSXv5xnfAjLJ8Hkt7LiBQDZRA%3D&reserved=0" target="_blank">https://mjanja.ch</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>