<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p style="margin-top:0; margin-bottom:0"></p>
<pre class="bz_comment_text" id="comment_text_0" style="white-space:pre-wrap; width:50em">Hello,
I wondered if someone could please help us to understand why the PrologFlags=contain flag is causing jobs to fail and draining compute nodes. We are, by the way, using slurm 18.08.0. Has anyone else seem this behaviour?
I'm currently experimenting with PrologFlags=contain. I've found that the addition of this flag in the slurm.conf radically changes the behaviour of jobs on the compute nodes.
When PrologFlags=contain is <b>commented out</b> in the slurm.conf jobs are assigned to the compute node and start/execute as expected. Here is the relevant extract from the slurmd logs on that node..
[2018-12-12T09:51:40.748] _run_prolog: run job script took usec=4
[2018-12-12T09:51:40.748] _run_prolog: prolog with lock for job 243317 ran for 0 seconds
[2018-12-12T09:51:40.748] Launching batch job 243317 for UID 57337
[2018-12-12T09:51:40.762] [243317.batch] task/cgroup: /slurm/uid_57337/job_243317: alloc=0MB mem.limit=193080MB memsw.limit=unlimited
[2018-12-12T09:51:40.763] [243317.batch] task/cgroup: /slurm/uid_57337/job_243317/step_batch: alloc=0MB mem.limit=193080MB memsw.limit=unlimited
When PrologFlags=contain is <b>activated</b> I find the following...
-- I don't see the "_run_prolog" and the "task/cgroup" messages in the slurmd logs
-- The job prolog fails, the job fails and the job output is owned by root
-- The compute node is drained.
sinfo -lN | grep red017 ....
red017 1 batch* drained 40 2:20:1 190000 0 1 (null) batch job complete f
Here is the extract from the slurmd logs
[2018-12-12T09:56:54.564] error: Waiting for JobId=243321 prolog has failed, giving up after 50 sec
[2018-12-12T09:56:54.565] Could not launch job 243321 and not able to requeue it, cancelling job</pre>
Best regards,
<p></p>
<p style="margin-top:0; margin-bottom:0">David</p>
</div>
</body>
</html>