<div dir="auto">Brian, your prompt about the user not being present on the node was what I needed. To close the loop on this, the error was due to an expired vendor SSL cert for LDAP. This was causing sssd on the nodes to balk. Once patched, all is well again. </div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Jason</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 11, 2023 at 1:28 PM Jason Simms <<a href="mailto:jsimms1@swarthmore.edu">jsimms1@swarthmore.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr">Thanks, Brian, helpful as always. Yes, /opt/slurm/prolog.sh is mounted across IB on all nodes, so it's reachable from everywhere. And the slurmd user can execute it.<div><br></div><div>I'll keep mucking around with it...</div><div><br></div><div>Warmest regards,</div><div>Jason</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 11, 2023 at 12:57 PM Brian Andrus <<a href="mailto:toomuchit@gmail.com" target="_blank">toomuchit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<p>From the documentation:</p>
<p><br>
</p>
<table style="box-sizing:border-box;border-spacing:0px;margin-bottom:0px;font-size:20px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px;break-inside:avoid;font-family:Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);color:rgb(70,84,92)" width="100%" cellspacing="0" cellpadding="6" border="1">
<tbody style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:inherit;font-style:inherit;font-variant-caps:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline">
<tr style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif">
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" height="18" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">Parameter</b></font></p>
</td>
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" height="18" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">Location</b></font></p>
</td>
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">Invoked by</b></font></p>
</td>
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">User</b></font></p>
</td>
<td style="box-sizing:border-box;margin-bottom:0px;font-family:Arial,Helvetica,sans-serif" width="40%" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">When executed</b></font></p>
</td>
</tr>
<tr style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif">
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1">Prolog (from slurm.conf)</font></p>
</td>
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1">Compute or front end node</font></p>
</td>
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1">slurmd daemon</font></p>
</td>
<td style="box-sizing:border-box;font-family:Arial,Helvetica,sans-serif" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1">SlurmdUser (normally user
root)</font></p>
</td>
<td style="box-sizing:border-box;margin-bottom:0px;font-family:Arial,Helvetica,sans-serif" width="40%">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-variant-numeric:inherit;font-variant-alternates:inherit;font-variant-east-asian:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:Arial,Helvetica,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt;font-family:Arial,Helvetica,sans-serif;color:rgb(70,84,92)" size="1">First job or job step
initiation on that node (by default); PrologFlags=Alloc
will force the script to be executed at job allocation</font></p>
</td>
</tr>
</tbody>
</table>
<p></p>
<div>So ensure:</div>
<div>1) /opt/slurm/prolog.sh exists on the
node(s)</div>
<div>2) the slurmd user is able to execute
it</div>
<div><br>
</div>
<div>I would connect to the node and try to
run the command as the slurmd user.<br>
Also, ensure the user exists on the node, however you are
propagating the uids.</div>
<div><br>
</div>
<div>Brian ANdrus<br>
</div>
<div><br>
</div>
<div>On 4/11/2023 9:48 AM, Jason Simms
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello all,
<div><br>
</div>
<div>Regularly I'm seeing array jobs fail, and the only log info
from the compute node is this:</div>
<div><br>
</div>
<div><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">[2023-04-11T11:41:12.336] error:
/opt/slurm/prolog.sh: exited with status 0x0100<br>
[2023-04-11T11:41:12.336] error: [job 26090] prolog failed
status=1:0<br>
[2023-04-11T11:41:12.336] Job 26090 already killed, do not
launch batch job</font></div>
<div><br>
</div>
<div>The contents of prolog.sh are incredibly simple:</div>
<div><br>
</div>
<div><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">#!/bin/bash<br>
loginctl enable-linger $SLURM_JOB_USER</font></div>
<div><br>
</div>
<div>I can't sort out what may be going on here. An example
script from a job that can result in this error is here:</div>
<div><br>
</div>
<div><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">#!/bin/bash<br>
#SBATCH -t 2:00:00<br>
#SBATCH -n 1<br>
#SBATCH -N 1<br>
#SBATCH -p compute<br>
#SBATCH --array=1-100<br>
#SBATCH -o tempOut/MSO-%j-%a.log<br>
<br>
module load python3/python3<br>
python3 runVoltage.py $SLURM_ARRAY_TASK_ID</font></div>
<div><br>
</div>
<div>Any insight would be welcome! This is really frustrating
because it's constantly causing nodes to drain.</div>
<div><br>
</div>
<div>Warmest regards,</div>
<div>Jason<br clear="all">
<div><br>
</div>
<span>-- </span><br>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><span style="font-family:Helvetica;color:rgb(130,36,51)"><font face="Century
Gothic" style="font-family:"Century Gothic";color:rgb(130,36,51)"><b style="font-family:"Century Gothic"">Jason L. Simms, Ph.D., M.P.H.</b></font></span></div>
<font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)">Manager of Research
Computing</font><br>
</div>
<div dir="ltr">
<div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)"><span style="font-family:"Century Gothic";color:gray">Swarthmore
College<br>
Information Technology Services</span></font></div>
<div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)"><span style="font-family:"Century Gothic";color:gray">(610)
328-8102<br>
</span></font></div>
<div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)">Schedule a meeting: </font><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(32,33,36)"><a href="https://calendly.com/jlsimms" target="_blank" style="font-family:Arial,Helvetica,sans-serif">https://calendly.com/jlsimms</a></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div><span>-- </span><br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><span style="font-family:Helvetica;color:rgb(130,36,51)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(130,36,51)"><b style="font-family:"Century Gothic"">Jason L. Simms, Ph.D., M.P.H.</b></font></span></div><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)">Manager of Research Computing</font><br></div><div dir="ltr"><div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)"><span style="font-family:"Century Gothic";color:gray">Swarthmore College<br>Information Technology Services</span></font></div><div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)"><span style="font-family:"Century Gothic";color:gray">(610) 328-8102<br></span></font></div><div style="font-family:Helvetica;font-size:14px;margin:0px;color:rgb(0,0,0)"><font face="Century Gothic" style="font-family:"Century Gothic";color:rgb(0,0,0)">Schedule a meeting: </font><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(32,33,36)"><a href="https://calendly.com/jlsimms" target="_blank" style="font-family:Arial,Helvetica,sans-serif">https://calendly.com/jlsimms</a></span><br></div></div></div></div></div></div>
</blockquote></div></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><span style="color:rgb(130,36,51)"><font face="Century Gothic"><b>Jason L. Simms, Ph.D., M.P.H.</b></font></span></div><font face="Century Gothic">Manager of Research Computing</font><br></div><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic"><span style="color:gray">Swarthmore College<br>Information Technology Services</span></font></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic"><span style="color:gray">(610) 328-8102<br></span></font></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic">Schedule a meeting: </font><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(32,33,36)"><a href="https://calendly.com/jlsimms" target="_blank">https://calendly.com/jlsimms</a></span><br></div></div></div></div></div></div>