<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>I think it isn't running how you think or there is something not
provided in the description.</p>
<p><br>
</p>
<p>You have:</p>
<blockquote>
<div> export
TMP="/local/scratch/${SLURM_ARRAY_JOB_ID}.${SLURM_ARRAY_TASK_ID}"</div>
</blockquote>
<div><br>
</div>
<div>Notice that period in there.</div>
<div>Then you have:</div>
<div>
<blockquote>
<div>node001::~$ echo $TMP</div>
<div>/local/scratch/80472</div>
</blockquote>
<div>There is no period.</div>
<div>In fact, SLURM_ARRAY_JOB_ID should be blank too if you are
not running as an array session.</div>
<div><br>
</div>
<div>However, to your desire for a job-specific tmp directory:</div>
<div>Check out the mktemp command. It should do just what you
want. I use it for interactive desktop sessions for users to
create the temp directory that is used for X sessions.</div>
<div>You just need to make sure the user has write access to the
directory you are creating the directory in (chmod 1777 for the
parent directory is good)<br>
</div>
<div><br>
</div>
<div>Brian Andrus<br>
</div>
<div><br>
</div>
</div>
<div class="moz-cite-prefix">On 3/4/2021 9:03 AM, Chin,David wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BN7PR01MB3858A5FA06B614B7BBFF923ED6979@BN7PR01MB3858.prod.exchangelabs.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
Hi, Brian:</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
So, this is my SrunProlog script -- I want a job-specific tmp
dir, which makes for easy cleanup at end of job:</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<blockquote style="margin-top:0;margin-bottom:0">#!/bin/bash
<div>if [[ -z ${SLURM_ARRAY_JOB_ID+x} ]]</div>
<div>then</div>
<div> export TMP="/local/scratch/${SLURM_JOB_ID}"</div>
<div> export TMPDIR="${TMP}"</div>
<div> export LOCAL_TMPDIR="${TMP}"</div>
<div> export
BEEGFS_TMPDIR="/beegfs/scratch/${SLURM_JOB_ID}"</div>
<div>else</div>
<div> export
TMP="/local/scratch/${SLURM_ARRAY_JOB_ID}.${SLURM_ARRAY_TASK_ID}"</div>
<div> export TMPDIR="${TMP}"</div>
<div> export LOCAL_TMPDIR="${TMP}"</div>
<div> export
BEEGFS_TMPDIR="/beegfs/scratch/${SLURM_ARRAY_JOB_ID}.${SLURM_ARRAY_TASK_ID}"</div>
<div>fi</div>
<div><br>
</div>
<div>echo DEBUG srun_set_tmp.sh</div>
<div>echo I am `whoami`</div>
<div><br>
</div>
<div>/usr/bin/mkdir -p ${TMP}</div>
<div>chmod 700 ${TMP}</div>
<div>/usr/bin/mkdir -p ${BEEGFS_TMPDIR}</div>
chmod 700 ${BEEGFS_TMPDIR}<br>
</blockquote>
</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
And this is my srun session:</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
picotte001::~$ whoami</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
dwc62<br>
</div>
<div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
picotte001::~$ srun -p def --mem 1000 -n 4 -t 600 --pty
/bin/bash
<div>DEBUG srun_set_tmp.sh</div>
<div>I am dwc62</div>
<div>node001::~$ echo $TMP</div>
<div>/local/scratch/80472</div>
<div>node001::~$ ll !$</div>
<div>ll $TMP</div>
<div>/bin/ls: cannot access '/local/scratch/80472': No such
file or directory</div>
<div>node001::~$ mkdir $TMP</div>
<div>node001::~$ ll -d !$</div>
<div>ll -d $TMP</div>
<div>drwxrwxr-x 2 dwc62 dwc62 6 Mar 4 11:52
/local/scratch/80472/</div>
node001::~$ exit<br>
</div>
</div>
</blockquote>
<div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
So, the "echo" and "whoami" statements are executed by the
prolog script, as expected, but the mkdir commands are not?</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
Dave</div>
<div style="font-family: "Courier New", monospace;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr"
style="font-size:12pt; color:#000000; font-family:'Courier
New',monospace">
<div class="BodyFragment"><font size="2"><span
style="font-size:10pt">
<div class="PlainText"
style="font-family:"Courier
New",monospace; font-size:13.3333px">
</div>
<span id="ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin,
URCF, Drexel</div>
<div><a class="moz-txt-link-abbreviated" href="mailto:dwc62@drexel.edu">dwc62@drexel.edu</a>
215.571.4335 (o)</div>
<div>For URCF support: <a class="moz-txt-link-abbreviated" href="mailto:urcf-support@drexel.edu">urcf-support@drexel.edu</a></div>
<div><a class="moz-txt-link-freetext" href="https://proteusmaster.urcf.drexel.edu/urcfwiki">https://proteusmaster.urcf.drexel.edu/urcfwiki</a></div>
<div>github:prehensilecode</div>
</div>
<span id="ms-rterangepaste-end"></span>
<div class="PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on
behalf of Brian Andrus <a class="moz-txt-link-rfc2396E" href="mailto:toomuchit@gmail.com"><toomuchit@gmail.com></a><br>
<b>Sent:</b> Thursday, March 4, 2021 10:12<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] prolog not passing env var
to job</font>
<div> </div>
</div>
<div>
<table width="100%">
<tbody>
<tr>
<td style="border-left:4px solid goldenrod;
background:cornsilk; padding:0 3pt">
<p style="font:small-caps bold 100% sans-serif">External.</p>
</td>
</tr>
</tbody>
</table>
<div>
<p><br>
</p>
<p>It seems to me, if you are using srun directly to get an
interactive shell, you can just run the script once you get
your shell.</p>
<p><br>
</p>
<p>You can set the variables and then run srun. It
automatically exports the environment.</p>
<p>If you want to change a particular one (or more), use
something like --export=ALL,MYVAR=othervalue</p>
<p>do 'man srun' and look at the --export option<br>
</p>
<p><br>
</p>
<p>Brian Andrus<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 3/3/2021 9:28 PM, Chin,David
wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style>
<div style="font-family:"Courier New",monospace;
font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier
New",monospace"><a
class="x_moz-txt-link-abbreviated"
href="mailto:ahmet.mercan@uhem.itu.edu.tr"
moz-do-not-send="true">ahmet.mercan@uhem.itu.edu.tr</a>
wrote:</span></div>
<div style="font-family:"Courier New",monospace;
font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier
New",monospace">> </span><span
style="font-family:"Courier New",monospace;
font-size:14.6667px; background-color:rgb(255,255,255);
display:inline!important">Prolog and TaskProlog are
different parameters and scripts. You should</span><br>
<span style="font-family:"Courier
New",monospace; font-size:14.6667px;
background-color:rgb(255,255,255);
display:inline!important">> use the TaskProlog script
to set env. variables.</span></div>
<div style="font-family:"Courier New",monospace;
font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier
New",monospace; font-size:14.6667px;
background-color:rgb(255,255,255);
display:inline!important"><br>
</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">Can
you tell me how to do this for srun? E.g. users request
an interactive shell:</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">
srun -n 1 -t 600 --pty /bin/bash</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">but
the shell on the compute node does not have the env
variables set.</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">I use
the same prolog script as TaskProlog, which sets it
properly for jobs submitted</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">with
sbatch.</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">Thanks
in advance,</span></div>
<div style="font-family:"Courier New",monospace;
color:rgb(0,0,0)"><span style="font-size:14.6667px">
Dave Chin</span></div>
<div>
<div style="font-family:"Courier New",monospace;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="">
<div class="x_BodyFragment"><font size="2"><span
style="font-size:10pt">
<span id="x_ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr.
SysAdmin, URCF, Drexel</div>
<div><a class="x_moz-txt-link-abbreviated"
href="mailto:dwc62@drexel.edu"
moz-do-not-send="true">dwc62@drexel.edu</a>
215.571.4335 (o)</div>
<div>For URCF support: <a
class="x_moz-txt-link-abbreviated"
href="mailto:urcf-support@drexel.edu"
moz-do-not-send="true">
urcf-support@drexel.edu</a></div>
<div><a class="x_moz-txt-link-freetext"
href="https://proteusmaster.urcf.drexel.edu/urcfwiki"
moz-do-not-send="true">https://proteusmaster.urcf.drexel.edu/urcfwiki</a></div>
<div>github:prehensilecode</div>
</div>
<span id="x_ms-rterangepaste-end"></span>
<div class="x_PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<div style="font-family:"Courier New",monospace;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> slurm-users
<a class="x_moz-txt-link-rfc2396E"
href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true">
<slurm-users-bounces@lists.schedmd.com></a> on
behalf of mercan <a class="x_moz-txt-link-rfc2396E"
href="mailto:ahmet.mercan@uhem.itu.edu.tr"
moz-do-not-send="true">
<ahmet.mercan@uhem.itu.edu.tr></a><br>
<b>Sent:</b> Friday, February 12, 2021 16:27<br>
<b>To:</b> Slurm User Community List <a
class="x_moz-txt-link-rfc2396E"
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true">
<slurm-users@lists.schedmd.com></a>; Herc
Silverstein <a class="x_moz-txt-link-rfc2396E"
href="mailto:herc.silverstein@schrodinger.com"
moz-do-not-send="true">
<herc.silverstein@schrodinger.com></a>; <a
class="x_moz-txt-link-abbreviated"
href="mailto:slurm-users@schedmd.com"
moz-do-not-send="true">
slurm-users@schedmd.com</a> <a
class="x_moz-txt-link-rfc2396E"
href="mailto:slurm-users@schedmd.com"
moz-do-not-send="true">
<slurm-users@schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] prolog not passing env
var to job</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span
style="font-size:11pt">
<div class="x_PlainText">External.<br>
<br>
Hi;<br>
<br>
Prolog and TaskProlog are different parameters and
scripts. You should<br>
use the TaskProlog script to set env. variables.<br>
<br>
Regards;<br>
<br>
Ahmet M.<br>
<br>
<br>
</div>
</span></font></div>
<br>
<p style="font-family:Calibri; font-size:10pt;
color:#000000; margin:5pt" align="Left">
Drexel Internal Data<br>
</p>
</blockquote>
</div>
</div>
<br>
<p
style="font-family:Calibri;font-size:10pt;color:#000000;margin:5pt;"
align="Left">
Drexel Internal Data<br>
</p>
</blockquote>
</body>
</html>