<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Thanks for your reply Bjorn-Helge<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This cleared things up for me. I had not understood that we need to use Prolog and Epilog for the TMPDIR stuff because that guarantees it is created at the very beginning of the job and deleted at the very end. Everything now works as expected,
thanks so much for your help.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Harry<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
On 2/11/22, 1:19 AM, "slurm-users" <slurm-users-bounces@lists.schedmd.com> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">"Putnam, Harry" <<a href="mailto:Harry.Putnam@ucsf.edu">Harry.Putnam@ucsf.edu</a>> writes:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> /opt/slurm/task_epilog<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> #!/bin/bash<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> mytmpdir=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> rm -Rf $mytmpdir<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> exit;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">This might not be the reason for what you observe, but I believe<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">deleting the scratch dir in the task epilog is not a good idea. The<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">task epilog is run after every "srun" or "mpirun" inside a job, which<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">means that the scratch dir will be created and deleted for each job<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">step. On our systems, we create the scratch dir in the (slurmd) Prolog,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">set the environment variable in the TaskProlog, and delete the dir in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">the (slurmd) Epilog. That way the dir is just created and deleted once.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> I am not sure I understand what constitutes a job step.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">In practice, every run of srun or mpirun creates a job step, and the job<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">script itself counts as a job step.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">B/H<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</body>
</html>