<div dir="ltr"><p style="margin-top:0px;margin-bottom:0px;font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">StackOverflow Thread: <a href="https://stackoverflow.com/questions/51328917/slurm-sbatch-multiple-nodes-same-slurmd-nodename">https://stackoverflow.com/questions/51328917/slurm-sbatch-multiple-nodes-same-slurmd-nodename</a></p><br>possibly related to:<br><font face="Calibri, Helvetica, sans-serif, serif, EmojiFont"><span style="font-size:16px"><a href="https://groups.google.com/forum/#!topic/slurm-users/suclnO2V0aA">https://groups.google.com/forum/#!topic/slurm-users/suclnO2V0aA</a><br></span></font><br style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><p style="margin-top:0px;margin-bottom:0px;font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"></p><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> - slurm-wlm 17.11.2 </div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> - Installed from Ubuntu Apt repos, Ubuntu:18.04</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">We have a cluster of 20 identical nodes.</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">Running the simple script below give me a confusing problem.</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">All the jobs think they are running on node3, while running the hostname command gives the accurate answer. This is also a problem for the output filename. I expected to have many different outputs, but I get just one, with 'node3' in the filename. This seems to be a Bash Eval() / Variable substitution error.</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">Wrapping </div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">$</span>SLURMD_NODENAME</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">in a </div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    bash -c "echo Bash3: \$SLURMD_NODENAME"</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">works. But why did I have to do this? This workaround won't work for the <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">  #SBATCH --output</span></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">cn.job:</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    #!/bin/bash</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    #SBATCH --output=/share/output.txt.%j.%J.%a.%A.%n.%N.%s.%t.%x</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    </div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    #SBATCH --time=00:00:30</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    #SBATCH --tasks-per-node=2</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    #SBATCH --nodes=4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    </div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    srun hostname</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    srun bash -c "echo Bash2: \$(hostname)"</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    srun echo SLURMD_NODENAME:$SLURMD_NODENAME SLURM_ARRAY_TASK_ID:$SLURM_ARRAY_TASK_ID SLURM_ARRAY_JOB_ID:$SLURM_ARRAY_JOB_ID SLURM_JOB_ID:$SLURM_JOB_ID SLURM_TASK_PID:$SLURM_TASK_PID</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    srun bash -c "echo Bash3: \$SLURMD_NODENAME"</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    </div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    srun sleep 20</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">Ran like:</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    sbatch cn.job</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">produces this output:</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">**/share/output.txt.2056.2056.4294967294.2056.0.node3.4294967294.0.cn.job**</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><br></div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node3</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node3</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node6</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node5</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node6</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    node5</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node3</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node6</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node5</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node3</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node6</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash2: node5</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    SLURMD_NODENAME:node3 SLURM_ARRAY_TASK_ID: SLURM_ARRAY_JOB_ID: SLURM_JOB_ID:2056 SLURM_TASK_PID:6441</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node3</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node5</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node3</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node6</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node4</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node6</div><div style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px">    Bash3: node5</div><br class="inbox-inbox-Apple-interchange-newline"></div>