<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>From the documentation:</p>
<p><br>
</p>
<table style="box-sizing: border-box; border-spacing: 0px;
margin-bottom: 0px; color: rgb(70, 84, 92); font-size: 20px;
font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-thickness:
initial; text-decoration-style: initial; text-decoration-color:
initial; break-inside: avoid; font-family: Arial, Helvetica,
sans-serif;" width="100%" cellspacing="0" cellpadding="6"
border="1">
<tbody style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font: inherit; vertical-align: baseline;">
<tr style="box-sizing: border-box;">
<td style="box-sizing: border-box;" width="15%" height="18"
bgcolor="#e0e0e0">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="CENTER"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1"><b style="box-sizing:
border-box; margin: 0px; padding: 0px; border: 0px;
font-style: inherit; font-variant: inherit;
font-weight: bold; font-stretch: inherit; font-size:
inherit; line-height: inherit; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit;
font-variation-settings: inherit; vertical-align:
baseline;">Parameter</b></font></p>
</td>
<td style="box-sizing: border-box;" width="15%" height="18"
bgcolor="#e0e0e0">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="CENTER"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1"><b style="box-sizing:
border-box; margin: 0px; padding: 0px; border: 0px;
font-style: inherit; font-variant: inherit;
font-weight: bold; font-stretch: inherit; font-size:
inherit; line-height: inherit; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit;
font-variation-settings: inherit; vertical-align:
baseline;">Location</b></font></p>
</td>
<td style="box-sizing: border-box;" width="15%"
bgcolor="#e0e0e0">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="CENTER"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1"><b style="box-sizing:
border-box; margin: 0px; padding: 0px; border: 0px;
font-style: inherit; font-variant: inherit;
font-weight: bold; font-stretch: inherit; font-size:
inherit; line-height: inherit; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit;
font-variation-settings: inherit; vertical-align:
baseline;">Invoked by</b></font></p>
</td>
<td style="box-sizing: border-box;" width="15%"
bgcolor="#e0e0e0">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="CENTER"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1"><b style="box-sizing:
border-box; margin: 0px; padding: 0px; border: 0px;
font-style: inherit; font-variant: inherit;
font-weight: bold; font-stretch: inherit; font-size:
inherit; line-height: inherit; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit;
font-variation-settings: inherit; vertical-align:
baseline;">User</b></font></p>
</td>
<td style="box-sizing: border-box; margin-bottom: 0px;"
width="40%" bgcolor="#e0e0e0">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="CENTER"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1"><b style="box-sizing:
border-box; margin: 0px; padding: 0px; border: 0px;
font-style: inherit; font-variant: inherit;
font-weight: bold; font-stretch: inherit; font-size:
inherit; line-height: inherit; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit;
font-variation-settings: inherit; vertical-align:
baseline;">When executed</b></font></p>
</td>
</tr>
<tr style="box-sizing: border-box;">
<td style="box-sizing: border-box;" width="15%" height="18">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="LEFT"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1">Prolog (from slurm.conf)</font></p>
</td>
<td style="box-sizing: border-box;" width="15%" height="18">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="LEFT"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1">Compute or front end node</font></p>
</td>
<td style="box-sizing: border-box;" width="15%" height="18">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="LEFT"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1">slurmd daemon</font></p>
</td>
<td style="box-sizing: border-box;" width="15%" height="18">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="LEFT"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1">SlurmdUser (normally user
root)</font></p>
</td>
<td style="box-sizing: border-box; margin-bottom: 0px;"
width="40%">
<p style="box-sizing: border-box; margin: 0px; padding: 0px;
border: 0px; font-style: inherit; font-variant: inherit;
font-weight: inherit; font-stretch: inherit; font-size:
inherit; line-height: 1.5em; font-family: inherit;
font-optical-sizing: inherit; font-kerning: inherit;
font-feature-settings: inherit; font-variation-settings:
inherit; vertical-align: baseline;" align="LEFT"><font
style="box-sizing: border-box; margin-bottom: 0px;
font-size: 8pt;" size="1">First job or job step
initiation on that node (by default); PrologFlags=Alloc
will force the script to be executed at job allocation</font></p>
</td>
</tr>
</tbody>
</table>
<p></p>
<div class="moz-cite-prefix">So ensure:</div>
<div class="moz-cite-prefix">1) /opt/slurm/prolog.sh exists on the
node(s)</div>
<div class="moz-cite-prefix">2) the slurmd user is able to execute
it</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I would connect to the node and try to
run the command as the slurmd user.<br>
Also, ensure the user exists on the node, however you are
propagating the uids.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Brian ANdrus<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 4/11/2023 9:48 AM, Jason Simms
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAE9cfnCp_qMBKNc9ZqeLPbs5YvqaAizYUFMSb4FVaV=-Dv89AA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hello all,
<div><br>
</div>
<div>Regularly I'm seeing array jobs fail, and the only log info
from the compute node is this:</div>
<div><br>
</div>
<div><font face="monospace">[2023-04-11T11:41:12.336] error:
/opt/slurm/prolog.sh: exited with status 0x0100<br>
[2023-04-11T11:41:12.336] error: [job 26090] prolog failed
status=1:0<br>
[2023-04-11T11:41:12.336] Job 26090 already killed, do not
launch batch job</font></div>
<div><br>
</div>
<div>The contents of prolog.sh are incredibly simple:</div>
<div><br>
</div>
<div><font face="monospace">#!/bin/bash<br>
loginctl enable-linger $SLURM_JOB_USER</font></div>
<div><br>
</div>
<div>I can't sort out what may be going on here. An example
script from a job that can result in this error is here:</div>
<div><br>
</div>
<div><font face="monospace">#!/bin/bash<br>
#SBATCH -t 2:00:00<br>
#SBATCH -n 1<br>
#SBATCH -N 1<br>
#SBATCH -p compute<br>
#SBATCH --array=1-100<br>
#SBATCH -o tempOut/MSO-%j-%a.log<br>
<br>
module load python3/python3<br>
python3 runVoltage.py $SLURM_ARRAY_TASK_ID</font></div>
<div><br>
</div>
<div>Any insight would be welcome! This is really frustrating
because it's constantly causing nodes to drain.</div>
<div><br>
</div>
<div>Warmest regards,</div>
<div>Jason<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><span
style="color:rgb(130,36,51)"><font face="Century
Gothic"><b>Jason L. Simms, Ph.D., M.P.H.</b></font></span></div>
<font face="Century Gothic">Manager of Research
Computing</font><br>
</div>
<div dir="ltr">
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font
face="Century Gothic"><span style="color:gray">Swarthmore
College<br>
Information Technology Services</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font
face="Century Gothic"><span style="color:gray">(610)
328-8102<br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font
face="Century Gothic">Schedule a meeting: </font><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(32,33,36)"><a
href="https://calendly.com/jlsimms"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://calendly.com/jlsimms</a></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>