<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Greetings All<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am struggling a bit with how TaskProlog works with srun. We have our TaskProlog set up to create a TMPDIR on local compute node scratch space and export the path in a variable called $TMPDIR. Our TaskEpilog
deletes TMPDIR. This is working great for jobs submitted with sbatch. If I start an srun job with –ntasks=1, then everything works the same as with sbatch. Namely, the $TMPDIR variable is set and the directory is created on local scratch. However, if I use
–ntasks=n where n > 1, we still get the $TMPDIR variable created but the directory itself is not created. Key files and examples:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">slurm.conf (relevant entries):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#Prolog=/opt/slurm/prolog.bash<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#PrologFlags=Alloc,NoHold<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#Epilog=/opt/slurm/epilog.bash<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#SrunProlog=/opt/slurm/srun_prolog<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#SrunEpilog=/opt/slurm/srun_epilog<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">TaskProlog=/opt/slurm/task_prolog<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">TaskEpilog=/opt/slurm/task_epilog<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">/opt/slurm/task_prolog:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">mytmpdir=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">mkdir -p $mytmpdir<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">echo export TMPDIR=$mytmpdir<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">exit;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">/opt/slurm/task_epilog<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">mytmpdir=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">rm -Rf $mytmpdir<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">exit;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Run Example –ntasks=1:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">$ srun --pty --mem=16g --ntasks=1 --time 0-08:00 --gres=scratch:20g --partition=cbc --nodelist=c4-n13 $SHELL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">[hputnam@c4-n13:job=421362 ~]$ echo $TMPDIR<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">/scratch/hputnam/421362<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">[hputnam@c4-n13:job=421362 ~]$ ls $TMPDIR<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">[hputnam@c4-n13:job=421362 ~]$<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Run Example –ntasks=2 $TMPDIR variable is set but the directory is not created:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">$ srun --pty --mem=16g --ntasks=2 --time 0-08:00 --gres=scratch:20g --partition=cbc --nodelist=c4-n13 $SHELL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">[hputnam@c4-n13:job=421370 ~]$ echo $TMPDIR<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">/scratch/hputnam/421370<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">[hputnam@c4-n13:job=421370 ~]$ ls $TMPDIR<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier">ls: cannot access /scratch/hputnam/421370: No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Courier"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am quite confused by this. I read this:
<a href="https://slurm.schedmd.com/prolog_epilog.html">https://slurm.schedmd.com/prolog_epilog.html</a> which says TaskProlog is run by the user executing srun prior to lunching job step. I am not sure I understand what constitutes a job step. I do see a stepd
process launched on the compute node each time I execute srun. That seems independent of –ntasks, I get one process per srun regardless of what –ntasks is set to.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks in advance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-Harry<o:p></o:p></span></p>
</div>
</body>
</html>