<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>You may have space, but do you have enough inodes?</p>
<p>Two different things to look at when trying to see why you cannot
write to a disk.</p>
<p>Also verify that it is writeable by SlurmUser.</p>
<p>If something happened and it automatically remounted itself as
read-only, that can do it too.</p>
<p>Brian Andrus<br>
</p>
<div class="moz-cite-prefix">On 10/28/2021 11:57 AM, Pedro Luiz de
Castro wrote:<br>
</div>
<blockquote type="cite"
cite="mid:ab10446b21084e098bee0c70ee6ff2b6@UL-MBX03.ul.pt">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello all<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Since yesterday we’ve
been having some trouble with slurm where it crashes and
isn’t able to recover.<br>
I’ve managed to track the fault to a zero sized file,
launching slurmctld -Dvvvv<br>
<br>
</span><span style="font-size:10.0pt;font-family:Consolas"
lang="EN-US">slurmctld: File
/mnt/nfs/lobo/IMM-NFS/slurm/hash.4/job.2044004/environment
has zero size</span><span lang="EN-US"><br>
<br>
That’s the StateSaveLocation, so the environment file for
this particular job is not getting correctly created.<br>
I don’t believe it’s a space issue as there’s about 2TB of
free space on this mountpoint.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Shouldn’t be permissions
either, as other jobs run fine and get completed.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For now I’ve been
launching slurmctld -i to work around this issue, killing
the job in question.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This way slurm can still
be running for our users.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Any ideas where I should
look next to try and troubleshoot this issue?<br>
<br>
Thanks for all the help in advance.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:PT"
lang="EN-US">Pedro Luiz de Castro</span></b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:PT"
lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:8.0pt"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:PT"
lang="EN-US">IT Support & System Administrator<br>
</span><span style="mso-fareast-language:PT" lang="EN-US">Information
Systems</span><span style="mso-fareast-language:PT"
lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="color:#1F497D;mso-fareast-language:PT"><img
style="width:2.2812in;height:.5312in" id="Picture_x0020_1"
src="cid:part1.lh05rTmL.qsrxEHzp@gmail.com"
alt="iMM_JLA_horizontal_RGB_cor_positivo" class=""
width="219" height="51"></span><span
style="color:#1F497D;mso-fareast-language:PT" lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:8.0pt"><span
style="color:black;mso-fareast-language:PT">Faculdade de
Medicina, Universidade de Lisboa
<br>
Avenida Professor Egas Moniz, 1649-028, Lisboa, Portugal <br>
iMM Lisboa general contact (+351) 217 999 411 - ext:
47356<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#00C6A5;mso-fareast-language:PT">imm.medicina</span></b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#00C6A5;mso-fareast-language:PT"><b>.ulisboa</b><b>.pt<o:p></o:p></b></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</body>
</html>