<div dir="ltr">Thanks, Brian, helpful as always. Yes, /opt/slurm/prolog.sh is mounted across IB on all nodes, so it's reachable from everywhere. And the slurmd user can execute it.<div><br></div><div>I'll keep mucking around with it...</div><div><br></div><div>Warmest regards,</div><div>Jason</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 11, 2023 at 12:57 PM Brian Andrus <<a href="mailto:toomuchit@gmail.com">toomuchit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>From the documentation:</p>
<p><br>
</p>
<table style="box-sizing:border-box;border-spacing:0px;margin-bottom:0px;color:rgb(70,84,92);font-size:20px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;break-inside:avoid;font-family:Arial,Helvetica,sans-serif" width="100%" cellspacing="0" cellpadding="6" border="1">
<tbody style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">
<tr style="box-sizing:border-box">
<td style="box-sizing:border-box" width="15%" height="18" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">Parameter</b></font></p>
</td>
<td style="box-sizing:border-box" width="15%" height="18" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">Location</b></font></p>
</td>
<td style="box-sizing:border-box" width="15%" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">Invoked by</b></font></p>
</td>
<td style="box-sizing:border-box" width="15%" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">User</b></font></p>
</td>
<td style="box-sizing:border-box;margin-bottom:0px" width="40%" bgcolor="#e0e0e0">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="CENTER"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1"><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:bold;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline">When executed</b></font></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td style="box-sizing:border-box" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1">Prolog (from slurm.conf)</font></p>
</td>
<td style="box-sizing:border-box" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1">Compute or front end node</font></p>
</td>
<td style="box-sizing:border-box" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1">slurmd daemon</font></p>
</td>
<td style="box-sizing:border-box" width="15%" height="18">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1">SlurmdUser (normally user
root)</font></p>
</td>
<td style="box-sizing:border-box;margin-bottom:0px" width="40%">
<p style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:1.5em;font-family:inherit;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline" align="LEFT"><font style="box-sizing:border-box;margin-bottom:0px;font-size:8pt" size="1">First job or job step
initiation on that node (by default); PrologFlags=Alloc
will force the script to be executed at job allocation</font></p>
</td>
</tr>
</tbody>
</table>
<p></p>
<div>So ensure:</div>
<div>1) /opt/slurm/prolog.sh exists on the
node(s)</div>
<div>2) the slurmd user is able to execute
it</div>
<div><br>
</div>
<div>I would connect to the node and try to
run the command as the slurmd user.<br>
Also, ensure the user exists on the node, however you are
propagating the uids.</div>
<div><br>
</div>
<div>Brian ANdrus<br>
</div>
<div><br>
</div>
<div>On 4/11/2023 9:48 AM, Jason Simms
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello all,
<div><br>
</div>
<div>Regularly I'm seeing array jobs fail, and the only log info
from the compute node is this:</div>
<div><br>
</div>
<div><font face="monospace">[2023-04-11T11:41:12.336] error:
/opt/slurm/prolog.sh: exited with status 0x0100<br>
[2023-04-11T11:41:12.336] error: [job 26090] prolog failed
status=1:0<br>
[2023-04-11T11:41:12.336] Job 26090 already killed, do not
launch batch job</font></div>
<div><br>
</div>
<div>The contents of prolog.sh are incredibly simple:</div>
<div><br>
</div>
<div><font face="monospace">#!/bin/bash<br>
loginctl enable-linger $SLURM_JOB_USER</font></div>
<div><br>
</div>
<div>I can't sort out what may be going on here. An example
script from a job that can result in this error is here:</div>
<div><br>
</div>
<div><font face="monospace">#!/bin/bash<br>
#SBATCH -t 2:00:00<br>
#SBATCH -n 1<br>
#SBATCH -N 1<br>
#SBATCH -p compute<br>
#SBATCH --array=1-100<br>
#SBATCH -o tempOut/MSO-%j-%a.log<br>
<br>
module load python3/python3<br>
python3 runVoltage.py $SLURM_ARRAY_TASK_ID</font></div>
<div><br>
</div>
<div>Any insight would be welcome! This is really frustrating
because it's constantly causing nodes to drain.</div>
<div><br>
</div>
<div>Warmest regards,</div>
<div>Jason<br clear="all">
<div><br>
</div>
<span>-- </span><br>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><span style="color:rgb(130,36,51)"><font face="Century
Gothic"><b>Jason L. Simms, Ph.D., M.P.H.</b></font></span></div>
<font face="Century Gothic">Manager of Research
Computing</font><br>
</div>
<div dir="ltr">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic"><span style="color:gray">Swarthmore
College<br>
Information Technology Services</span></font></div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic"><span style="color:gray">(610)
328-8102<br>
</span></font></div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic">Schedule a meeting: </font><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(32,33,36)"><a href="https://calendly.com/jlsimms" target="_blank">https://calendly.com/jlsimms</a></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><span style="color:rgb(130,36,51)"><font face="Century Gothic"><b>Jason L. Simms, Ph.D., M.P.H.</b></font></span></div><font face="Century Gothic">Manager of Research Computing</font><br></div><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic"><span style="color:gray">Swarthmore College<br>Information Technology Services</span></font></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic"><span style="color:gray">(610) 328-8102<br></span></font></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font face="Century Gothic">Schedule a meeting: </font><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(32,33,36)"><a href="https://calendly.com/jlsimms" target="_blank">https://calendly.com/jlsimms</a></span><br></div></div></div></div></div></div>