<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Sam,<br>
<br>
this is expected and how bash works.<br>
<br>
Regarding the #SBATCH --output problem this seems to be an error,
because only one output file is created (I just tested it myself).<br>
<br>
<br>
Regarding variable substitution:<br>
<br>
srun echo SLURMD_NODENAME:$SLURMD_NODENAME
SLURM_ARRAY_TASK_ID:$SLURM_ARRAY_TASK_ID
SLURM_ARRAY_JOB_ID:$SLURM_ARRAY_JOB_ID SLURM_JOB_ID:$SLURM_JOB_ID
SLURM_TASK_PID:$SLURM_TASK_PID<br>
<br>
bash evaluates the variables before the actual program is started,
otherwise e.g. "cd $HOME" would not work, because in most unixoid
systems $HOME never exist, but the variable HOME would point to the
user's home directory.<br>
<br>
So, in fact, here's what you're letting go:<br>
<br>
srun echo SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:644<br>
This is exactly the output you received.<br>
<br>
<br>
Here's what you could try:<br>
<br>
srun echo 'SLURMD_NODENAME:$SLURMD_NODENAME
SLURM_ARRAY_TASK_ID:$SLURM_ARRAY_TASK_ID
SLURM_ARRAY_JOB_ID:$SLURM_ARRAY_JOB_ID
SLURM_JOB_ID:$SLURM_JOB_ID_LURM_TASK_PID_TASK_PID'.<br>
<br>
the single quotes (no backticks!) should prevent bash from replacing
the variables.<br>
<br>
<br>
Best<br>
Marcus<br>
<br>
<br>
<div class="moz-cite-prefix">On 07/13/2018 06:54 PM, Sam wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANunXaxK1P6G-x6pPyyedpe-44Ww2qTkgy7b41Un480t1f_LBQ@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">
<p
style="margin-top:0px;margin-bottom:0px;font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">StackOverflow
Thread: <a
href="https://stackoverflow.com/questions/51328917/slurm-sbatch-multiple-nodes-same-slurmd-nodename"
moz-do-not-send="true">https://stackoverflow.com/questions/51328917/slurm-sbatch-multiple-nodes-same-slurmd-nodename</a></p>
<br>
possibly related to:<br>
<font face="Calibri, Helvetica, sans-serif, serif, EmojiFont"><span
style="font-size:16px"><a
href="https://groups.google.com/forum/#%21topic/slurm-users/suclnO2V0aA"
moz-do-not-send="true">https://groups.google.com/forum/#!topic/slurm-users/suclnO2V0aA</a><br>
</span></font><br
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> -
slurm-wlm 17.11.2 </div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> -
Installed from Ubuntu Apt repos, Ubuntu:18.04</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">We
have a cluster of 20 identical nodes.</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">Running
the simple script below give me a confusing problem.</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">All
the jobs think they are running on node3, while running the
hostname command gives the accurate answer. This is also a
problem for the output filename. I expected to have many
different outputs, but I get just one, with 'node3' in the
filename. This seems to be a Bash Eval() / Variable
substitution error.</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">Wrapping </div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
<span
style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android Emoji",EmojiSymbols">$</span>SLURMD_NODENAME</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">in
a </div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
bash -c "echo Bash3: \$SLURMD_NODENAME"</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">works.
But why did I have to do this? This workaround won't work for
the <span
style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android Emoji",EmojiSymbols">
#SBATCH --output</span></div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">cn.job:</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
#!/bin/bash</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
#SBATCH
--output=/share/output.txt.%j.%J.%a.%A.%n.%N.%s.%t.%x</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
#SBATCH --time=00:00:30</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
#SBATCH --tasks-per-node=2</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
#SBATCH --nodes=4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
srun hostname</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
srun bash -c "echo Bash2: \$(hostname)"</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
srun echo SLURMD_NODENAME:$SLURMD_NODENAME
SLURM_ARRAY_TASK_ID:$SLURM_ARRAY_TASK_ID
SLURM_ARRAY_JOB_ID:$SLURM_ARRAY_JOB_ID
SLURM_JOB_ID:$SLURM_JOB_ID SLURM_TASK_PID:$SLURM_TASK_PID</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
srun bash -c "echo Bash3: \$SLURMD_NODENAME"</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
srun sleep 20</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">Ran
like:</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
sbatch cn.job</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">produces
this output:</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">**/share/output.txt.2056.2056.4294967294.2056.0.node3.4294967294.0.cn.job**</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br>
</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node3</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node3</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node6</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node5</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node6</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
node5</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node3</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node6</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node5</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node3</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node6</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash2: node5</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID:
SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node3</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node5</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node3</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node6</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node4</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node6</div>
<div
style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">
Bash3: node5</div>
<br class="inbox-inbox-Apple-interchange-newline">
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>