<div dir="ltr"><div>Does the GUI run as the user (e.g. the user starts the GUI, and so submitting process is owned by user), or is the GUI running as a daemon (in which case, is it submitting jobs as the user and if so how). And is the default shell of the user submitting the job tcsh (like the shebang in the script) or bash/something else?</div><div><br></div><div>Basically, only things I can think of that might be different between GUI submitted job and manually submitted job are:</div><div>1) the environment of the process running the sbatch command. <br></div><div>2) possibly the shell initialization scripts (if the GUI runs as a different user, and maybe if the job script is different shell than user's default shell)</div><div><br></div><div>As Christopher suggested, adding -x (and maybe -v also) to the submission script shebang might be helpful, at least might tell us if something funky is being missed in the script.</div><div>Also might be helpful to add a -V to the mpirun (did you ever specify which MPI library is being used. For OpenMPI at least that should print the version, I assume others have something similar. Anyway, at least if for some reason the mpirun is being executed but is exiting quickly and silently this will let us know that it ran). A printenv in the script might</div><div>be useful too.<br></div><div><br></div><div>Also, if you could try the manual run with '--export=NONE' added to the sbatch command, this should at least help let us see if it is the environment (or at least if there is something in the manual run environment which is missing in the GUI environment that makes things work).</div><div><br></div><div>I doubt it is a Python issue; all python is doing is generating the job submission script and invoking sbatch. You verified that the job was submitted and that the submitted job script</div><div>is as expected, so I think this lets python off the hook. That suggests either the MPI library or Slurm, or with how the Slurm library is integrated with the MPI library. Is it possible to try with another MPI library (either changing the code being run to something simple or using an incompatible MPI lib --- at this point a crash and burn from MPI would be an improvement:)</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 22, 2019 at 1:24 PM Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov">pbisbal@pppl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thomas, <br>
</p>
<p>The GUI app writes the script to the file slurm_script.sh in the
cwd. I did exactly what you suggested as my first step in
debugging check the Command= value from the output of 'scontrol
show job' to see what script was actually submitted, and it was
the slurm_script.sh in the cwd. <br>
</p>
<p>The user did provide me with some very useful information this
afternoon: The GUI app uses python to launch the job: Here's what
the user wrote to me. OMFIT is the name of the GUI application: <br>
</p>
<p><br>
</p>
<p>
</p><blockquote type="cite">New clue for the mpirun issue: The
following information might be helpful.
<ul style="box-sizing:border-box;margin-bottom:16px;margin-top:0px;padding-left:2em;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px;font-variant-ligatures:normal;background-color:rgb(255,255,255)">
<li style="box-sizing:border-box;margin-left:0px">I
modified the script to use <code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;background-color:rgba(27,31,35,0.05);border-radius:3px;margin:0px;padding:0.2em 0.4em">subprocess</code> submitting the job
directly. The job was submitted, but somehow it returned
NoneZeroError and the <code>mpiexec</code> line
was skipped.</li>
</ul>
<div class="gmail-m_2435371770098009432highlight gmail-m_2435371770098009432highlight-source-python" style="box-sizing:border-box;margin-bottom:16px;background-color:rgb(255,255,255);color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px;font-variant-ligatures:normal;overflow:visible">
<pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;margin-bottom:0px;margin-top:0px;background-color:rgb(246,248,250);border-radius:3px;line-height:1.45;overflow:auto;padding:16px;word-break:normal">OMFITx.executable(root,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">inputs</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>inputs,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">outputs</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>outputs,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">executable</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span><span class="gmail-m_2435371770098009432pl-s" style="box-sizing:border-box;color:rgb(3,47,98)"><span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span>echo <span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">%s</span><span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span></span>,<span class="gmail-m_2435371770098009432pl-c" style="box-sizing:border-box;color:rgb(106,115,125)"><span class="gmail-m_2435371770098009432pl-c" style="box-sizing:border-box">#</span>submit_command,</span>
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">script</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>(bashscript, <span class="gmail-m_2435371770098009432pl-s" style="box-sizing:border-box;color:rgb(3,47,98)"><span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span>slurm.script<span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span></span>),
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">clean</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span><span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">True</span>,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">std_out</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>std_out,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">remotedir</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>unique_remotedir,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">ignoreReturnCode</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span><span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">True</span>)
p<span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>subprocess.Popen(<span class="gmail-m_2435371770098009432pl-s" style="box-sizing:border-box;color:rgb(3,47,98)"><span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span>sbatch <span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span></span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">+</span>unique_remotedir<span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">+</span><span class="gmail-m_2435371770098009432pl-s" style="box-sizing:border-box;color:rgb(3,47,98)"><span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span>slurm.script<span class="gmail-m_2435371770098009432pl-pds" style="box-sizing:border-box">'</span></span>,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">shell</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span><span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">True</span>,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">stdout</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>subprocess.<span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">PIPE</span>,
<span class="gmail-m_2435371770098009432pl-v" style="box-sizing:border-box;color:rgb(227,98,9)">stderr</span><span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">=</span>subprocess.<span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">PIPE</span>)
std_out.append(p.stdout.read())
<span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">print</span>(std_out[<span class="gmail-m_2435371770098009432pl-k" style="box-sizing:border-box;color:rgb(215,58,73)">-</span><span class="gmail-m_2435371770098009432pl-c1" style="box-sizing:border-box;color:rgb(0,92,197)">1</span>], p.stderr.read())</pre>
</div>
<ul style="box-sizing:border-box;margin-bottom:16px;margin-top:0px;padding-left:2em;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px;font-variant-ligatures:normal;background-color:rgb(255,255,255)">
<li style="box-sizing:border-box;margin-left:0px">As
I mentioned above, my standalone python script can normally
submit jobs likewise using <code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;background-color:rgba(27,31,35,0.05);border-radius:3px;margin:0px;padding:0.2em 0.4em">subprocess.Popen</code> or <code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;background-color:rgba(27,31,35,0.05);border-radius:3px;margin:0px;padding:0.2em 0.4em">subprocess.call</code>.
I created the following script at the working directory and
executed it with the same python version as OMFIT. It works
without skip.</li>
</ul>
<pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;margin-bottom:16px;margin-top:0px;background-color:rgb(246,248,250);border-radius:3px;line-height:1.45;overflow:auto;padding:16px;color:rgb(36,41,46);font-variant-ligatures:normal"><code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;background-color:transparent;border-radius:3px;margin:0px;padding:0px;border:0px none;word-break:normal;display:inline;line-height:inherit;overflow:visible">import sys
import os.path
import subprocess
print(sys.version, sys.path, subprocess.__file__)
p = subprocess.Popen('sbatch slurm.script', shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
print(p.stdout.read(), p.stderr.read())
</code></pre>
<div>The question is why the same <code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;background-color:rgba(27,31,35,0.05);border-radius:3px;margin:0px;padding:0.2em 0.4em">subprocee.Popen</code> command works
differently in OMFIT and in the terminal, even if they are
called by the same version <code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;background-color:rgba(27,31,35,0.05);border-radius:3px;margin:0px;padding:0.2em 0.4em">python2.7</code>.</div>
</blockquote>
<p></p>
<p>So now it's unclear whether this is a bug in Python, or Slurm
18.06.6-2. Since the user can write a python script that does
work, I think this is something specific to the application's
environment, rather than an issue with the Python-Slurm
interaction. The main piece of evidence that this might be a bug
in Slurm is that this issue started after the upgrade from
18.08.5-2 to 18.08.6-2, but correlation doesn't necessarily mean
causation. <br>
</p>
<p>Prentice<br>
</p>
<p><br>
</p>
<div class="gmail-m_2435371770098009432moz-cite-prefix">On 3/22/19 12:48 PM, Thomas M. Payerle
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Assuming the GUI produced script is as you
indicated (I am not sure where you got the script you showed,
but if it is not the actual script used by a job it might be
worthwhile to examine the Command= file from scontrol show job
to verify), then the only thing that should be different from a
GUI submission and a manual submission is the submission
environment. Does the manual submission work if you add
--export=NONE to the sbatch command to prevent the exporting of
environment variables? And maybe add a printenv to the script
to see what environment is in both cases. Though I confess I am
unable to think of any reasonable environmental setting that
might cause the observed symptoms.<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Mar 22, 2019 at 11:23
AM Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov" target="_blank">pbisbal@pppl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On
3/21/19 6:56 PM, Reuti wrote:<br>
> Am 21.03.2019 um 23:43 schrieb Prentice Bisbal:<br>
><br>
>> Slurm-users,<br>
>><br>
>> My users here have developed a GUI application which
serves as a GUI interface to various physics codes they use.
From this GUI, they can submit jobs to Slurm. On Tuesday, we
upgraded Slurm from 18.08.5-2 to 18.08.6-2,and a user has
reported a problem when submitting Slurm jobs through this GUI
app that do not occur when the same sbatch script is submitted
from sbatch on the command-line.<br>
>><br>
>> […]<br>
>> When I replaced the mpirun command with an equivalent
srun command, everything works as desired, so the user can get
back to work and be productive.<br>
>><br>
>> While srun is a suitable workaround, and is arguably
the correct way to run an MPI job, I'd like to understand what
is going on here. Any idea what is going wrong, or additional
steps I can take to get more debug information?<br>
> Was an alias to `mpirun` introduced? It may cover the
real application and even the `which mpirun` will return the
correct value, but never be executed.<br>
><br>
> $ type mpirun<br>
> $ alias mpirun<br>
><br>
> may tell in the jobscript.<br>
><br>
Unfortunately, the script is in tcsh, so the 'type' command
doesn't work <br>
since, it's a bash built-in function. I did use the 'alias'
command to <br>
see all the defined aliases, and mpirun and mpiexec are not
aliased. Any <br>
other ideas?<br>
<br>
Prentice<br>
<br>
<br>
<br>
<br>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail-m_2435371770098009432gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">Tom Payerle <br>
DIT-ACIGS/Mid-Atlantic Crossroads <a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a><br>
</div>
<div>5825 University Research Park (301)
405-6135<br>
</div>
<div dir="ltr">University of Maryland<br>
College Park, MD 20740-3831<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Tom Payerle <br>DIT-ACIGS/Mid-Atlantic Crossroads <a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a><br></div><div>5825 University Research Park (301) 405-6135<br></div><div dir="ltr">University of Maryland<br>College Park, MD 20740-3831<br></div></div></div></div></div></div>