<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="en-AT" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">We did a couple of major and minor SLURM upgrades without draining the compute nodes.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Once slurmdbd and slurmctld were updated to the new major version, we did a package update on the compute nodes and restarted slurmd on them.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">The existing running jobs continued to run fine and new jobs on the same compute started by the updated slurmd daemon and also worked fine.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><br>
So, for us this worked smoothly. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Best<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Ümit<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ole Holm Nielsen <Ole.H.Nielsen@fysik.dtu.dk><br>
<b>Date: </b>Monday, 30. May 2022 at 20:58<br>
<b>To: </b>slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] Rolling upgrade of compute nodes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">On 30-05-2022 19:34, Chris Samuel wrote:<br>
> On 30/5/22 10:06 am, Chris Samuel wrote:<br>
> <br>
>> If you switch that symlink those jobs will pick up the 20.11 srun <br>
>> binary and that's where you may come unstuck.<br>
> <br>
> Just to quickly fix that, srun talks to slurmctld (which would also be <br>
> 20.11 for you), slurmctld will talk to the slurmd's running the job <br>
> (which would be 19.05, so OK) but then the slurmd would try and launch a <br>
> 20.11 slurmstepd and that is where I suspect things could come undone.<br>
<br>
How about restarting all slurmd's at version 20.11 in one shot? No <br>
reboot will be required. There will be running 19.05 slurmstepd's for <br>
the running job steps, even though slurmd is at 20.11. You could <br>
perhaps restart 20.11 slurmd one partition at a time in order to see if <br>
it works correctly on a small partition of the cluster.<br>
<br>
I think we have done this successfully when we install new RPMs on *all* <br>
compute nodes in one shot, and I'm not aware of any job crashes. Your <br>
mileage may vary depending on job types!<br>
<br>
Question: Does anyone have bad experiences with upgrading slurmd while <br>
the cluster is running production?<br>
<br>
/Ole<o:p></o:p></span></p>
</div>
</div>
</body>
</html>