<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="PT" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello all<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Since yesterday we’ve been having some trouble with slurm where it crashes and isn’t able to recover.<br>
I’ve managed to track the fault to a zero sized file, launching slurmctld -Dvvvv<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Consolas">slurmctld: File /mnt/nfs/lobo/IMM-NFS/slurm/hash.4/job.2044004/environment has zero size</span><span lang="EN-US"><br>
<br>
That’s the StateSaveLocation, so the environment file for this particular job is not getting correctly created.<br>
I don’t believe it’s a space issue as there’s about 2TB of free space on this mountpoint.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Shouldn’t be permissions either, as other jobs run fine and get completed.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For now I’ve been launching slurmctld -i to work around this issue, killing the job in question.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This way slurm can still be running for our users.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Any ideas where I should look next to try and troubleshoot this issue?<br>
<br>
Thanks for all the help in advance.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:PT">Pedro Luiz de Castro</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:PT"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:8.0pt"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:PT">IT Support & System Administrator<br>
</span><span lang="EN-US" style="mso-fareast-language:PT">Information Systems</span><span lang="EN-US" style="mso-fareast-language:PT"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D;mso-fareast-language:PT"><img width="219" height="51" style="width:2.2812in;height:.5312in" id="Picture_x0020_1" src="cid:image001.jpg@01D7CC30.6858DE30" alt="iMM_JLA_horizontal_RGB_cor_positivo"></span><span lang="EN-US" style="color:#1F497D;mso-fareast-language:PT"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:8.0pt"><span style="color:black;mso-fareast-language:PT">Faculdade de Medicina, Universidade de Lisboa
<br>
Avenida Professor Egas Moniz, 1649-028, Lisboa, Portugal <br>
iMM Lisboa general contact (+351) 217 999 411 - ext: 47356<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#00C6A5;mso-fareast-language:PT">imm.medicina</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#00C6A5;mso-fareast-language:PT"><b>.ulisboa</b><b>.pt<o:p></o:p></b></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>