<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>We also call scontrol in our scripts (a little as we can manage)
and we run at the scale of 1500 nodes. It hasn't really caused
many issues, but we try to limit it as much as we possibly can.</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 9/16/22 9:41 AM, Sebastian Potthoff
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:9233C5FA-0CFE-4130-A371-DBC165975DFC@uni-muenster.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="">Hi Hermann,</div>
<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">So you both are happily(?)
ignoring this warning the "Prolog and Epilog Guide",<br
class="">
right? :-)<br class="">
<br class="">
"Prolog and Epilog scripts [...] should not call Slurm
commands (e.g. squeue,<br class="">
scontrol, sacctmgr, etc)."<br class="">
</blockquote>
<br class="">
<span class="" style="float: none; display: inline !important;">We
have probably been doing this since before the warning was
added to</span><br class="">
<span class="" style="float: none; display: inline !important;">the
documentation. So we are "ignorantly ignoring" the advice :-/</span></blockquote>
<div><br class="">
</div>
<div>Same here :) But if $SLURM_JOB_STDOUT is not defined as
documented … what can you do.</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<blockquote type="cite" class="">May I ask how big your
clusters are (number of nodes) and how heavily they are<br
class="">
used (submitted jobs per hour)?</blockquote>
</blockquote>
</div>
<div><br class="">
</div>
<div>We have around 500 nodes (mostly 2x18 cores). Jobs ending
(i.e. calling the epilog script) varies quite a lot between 1000
and 15k a day, so something in between 40 and 625 Jobs/hour.
During those peaks Slurm can become noticeably slower, however
usually it runs fine.</div>
<div><br class="">
</div>
<div>Sebastian </div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">Am 16.09.2022 um 15:15 schrieb Loris Bennett
<<a href="mailto:loris.bennett@fu-berlin.de"
class="moz-txt-link-freetext" moz-do-not-send="true">loris.bennett@fu-berlin.de</a>>:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta charset="UTF-8" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Hi Hermann,</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Hermann Schwärzler <</span><a
href="mailto:hermann.schwaerzler@uibk.ac.at"
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px;" class="moz-txt-link-freetext" moz-do-not-send="true">hermann.schwaerzler@uibk.ac.at</a><span
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
float: none; display: inline !important;" class="">>
writes:</span><br style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">Hi Loris,<br class="">
hi Sebastian,<br class="">
<br class="">
thanks for the information on how you are doing this.<br
class="">
So you both are happily(?) ignoring this warning the
"Prolog and Epilog Guide",<br class="">
right? :-)<br class="">
<br class="">
"Prolog and Epilog scripts [...] should not call Slurm
commands (e.g. squeue,<br class="">
scontrol, sacctmgr, etc)."<br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">We have probably been doing this
since before the warning was added to</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">the documentation. So we are
"ignorantly ignoring" the advice :-/</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">May I ask how big your clusters are (number of
nodes) and how heavily they are<br class="">
used (submitted jobs per hour)?<br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">We have around 190 32-core nodes. I
don't know how I would easily find</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">out the average number of jobs per
hour. The only problems we have had</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">with submission have been when
people have written their own mechanisms</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">for submitting thousands of jobs.
Once we get them to use job array,</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">such problems generally disappear.</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Cheers,</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Loris</span><br style="caret-color:
rgb(0, 0, 0); font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">Regards,<br class="">
Hermann<br class="">
<br class="">
On 9/16/22 9:09 AM, Loris Bennett wrote:<br class="">
<blockquote type="cite" class="">Hi Hermann,<br class="">
Sebastian Potthoff <<a
href="mailto:s.potthoff@uni-muenster.de"
class="moz-txt-link-freetext" moz-do-not-send="true">s.potthoff@uni-muenster.de</a>>
writes:<br class="">
<br class="">
<blockquote type="cite" class="">Hi Hermann,<br class="">
<br class="">
I happened to read along this conversation and was
just solving this issue today. I added this part to
the epilog script to make it work:<br class="">
<br class="">
# Add job report to stdout<br class="">
StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID |
/usr/bin/grep StdOut | /usr/bin/xargs | /usr/bin/awk
'BEGIN { FS = "=" } ; { print $2 }')<br class="">
<br class="">
NODELIST=($(/usr/bin/scontrol show hostnames))<br
class="">
<br class="">
# Only add to StdOut file if it exists and if we are
the first node<br class="">
if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a !
-z "${StdOut}" ]<br class="">
then<br class="">
echo "################################# JOB REPORT
##################################" >> $StdOut<br
class="">
/usr/bin/seff $SLURM_JOB_ID >> $StdOut<br
class="">
echo
"###############################################################################"
>> $StdOut<br class="">
fi<br class="">
</blockquote>
We do something similar. At the end of our script
pointed to by<br class="">
EpilogSlurmctld we have<br class="">
OUT=`scontrol show jobid ${job_id} | awk -F= '/
StdOut/{print $2}'`<br class="">
if [ ! -f "$OUT" ]; then<br class="">
exit<br class="">
fi<br class="">
printf "\n== Epilog Slurmctld<br class="">
==================================================\n\n"
>> ${OUT}<br class="">
seff ${SLURM_JOB_ID} >> ${OUT}<br class="">
printf<br class="">
"\n======================================================================\n"<br
class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">${OUT}<br class="">
</blockquote>
</blockquote>
chown ${user} ${OUT}<br class="">
Cheers,<br class="">
Loris<br class="">
<br class="">
<blockquote type="cite" class=""> Contrary to what it
says in the slurm docs <a
href="https://slurm.schedmd.com/prolog_epilog.html"
class="moz-txt-link-freetext" moz-do-not-send="true">https://slurm.schedmd.com/prolog_epilog.html</a>
I was not able to use the env var SLURM_JOB_STDOUT,
so I had to fetch it via scontrol. In addition I had
to<br class="">
make sure it is only called by the „leading“ node as
the epilog script will be called by ALL nodes of a
multinode job and they would all call seff and clutter
up the output. Last thing was to check if StdOut is<br
class="">
not of length zero (i.e. it exists). Interactive jobs
would otherwise cause the node to drain.<br class="">
<br class="">
Maybe this helps.<br class="">
<br class="">
Kind regards<br class="">
Sebastian<br class="">
<br class="">
PS: goslmailer looks quite nice with its
recommendations! Will definitely look into it.<br
class="">
<br class="">
--<br class="">
Westfälische Wilhelms-Universität (WWU) Münster<br
class="">
WWU IT<br class="">
Sebastian Potthoff (eScience / HPC)<br class="">
<br class="">
Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler
<<a href="mailto:hermann.schwaerzler@uibk.ac.at"
class="moz-txt-link-freetext" moz-do-not-send="true">hermann.schwaerzler@uibk.ac.at</a>>:<br
class="">
<br class="">
Hi Ole,<br class="">
<br class="">
On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:<br
class="">
<br class="">
On 15-09-2022 16:08, Hermann Schwärzler wrote:<br
class="">
<br class="">
Just out of curiosity: how do you insert the output
of seff into the out-file of a job?<br class="">
<br class="">
Use the "smail" tool from the slurm-contribs RPM and
set this in slurm.conf:<br class="">
MailProg=/usr/bin/smail<br class="">
<br class="">
Maybe I am missing something but from what I can tell
smail sends an email and does *not* change or append
to the .out file of a job...<br class="">
<br class="">
Regards,<br class="">
Hermann<br class="">
</blockquote>
<br class="">
</blockquote>
<br class="">
</blockquote>
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">--<span
class="Apple-converted-space"> </span></span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Dr. Loris Bennett (Herr/Mr)</span><br
style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;"
class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">ZEDAT, Freie Universität Berlin
Email<span class="Apple-converted-space"> </span></span><a
href="mailto:loris.bennett@fu-berlin.de"
style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px;" class="moz-txt-link-freetext" moz-do-not-send="true">loris.bennett@fu-berlin.de</a></div>
</blockquote>
</div>
<br class="">
</blockquote>
</body>
</html>