<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Hi, Brian:</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
So, this is my SrunProlog script -- I want a job-specific tmp dir, which makes for easy cleanup at end of job:</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<blockquote style="margin-top:0;margin-bottom:0">#!/bin/bash
<div>if [[ -z ${SLURM_ARRAY_JOB_ID+x} ]]</div>
<div>then</div>
<div> export TMP="/local/scratch/${SLURM_JOB_ID}"</div>
<div> export TMPDIR="${TMP}"</div>
<div> export LOCAL_TMPDIR="${TMP}"</div>
<div> export BEEGFS_TMPDIR="/beegfs/scratch/${SLURM_JOB_ID}"</div>
<div>else</div>
<div> export TMP="/local/scratch/${SLURM_ARRAY_JOB_ID}.${SLURM_ARRAY_TASK_ID}"</div>
<div> export TMPDIR="${TMP}"</div>
<div> export LOCAL_TMPDIR="${TMP}"</div>
<div> export BEEGFS_TMPDIR="/beegfs/scratch/${SLURM_ARRAY_JOB_ID}.${SLURM_ARRAY_TASK_ID}"</div>
<div>fi</div>
<div><br>
</div>
<div>echo DEBUG srun_set_tmp.sh</div>
<div>echo I am `whoami`</div>
<div><br>
</div>
<div>/usr/bin/mkdir -p ${TMP}</div>
<div>chmod 700 ${TMP}</div>
<div>/usr/bin/mkdir -p ${BEEGFS_TMPDIR}</div>
chmod 700 ${BEEGFS_TMPDIR}<br>
</blockquote>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
And this is my srun session:</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
picotte001::~$ whoami</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
dwc62<br>
</div>
<div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
picotte001::~$ srun -p def --mem 1000 -n 4 -t 600 --pty /bin/bash
<div>DEBUG srun_set_tmp.sh</div>
<div>I am dwc62</div>
<div>node001::~$ echo $TMP</div>
<div>/local/scratch/80472</div>
<div>node001::~$ ll !$</div>
<div>ll $TMP</div>
<div>/bin/ls: cannot access '/local/scratch/80472': No such file or directory</div>
<div>node001::~$ mkdir $TMP</div>
<div>node001::~$ ll -d !$</div>
<div>ll -d $TMP</div>
<div>drwxrwxr-x 2 dwc62 dwc62 6 Mar 4 11:52 /local/scratch/80472/</div>
node001::~$ exit<br>
</div>
</div>
</blockquote>
<div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
So, the "echo" and "whoami" statements are executed by the prolog script, as expected, but the mkdir commands are not?</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Dave</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:'Courier New',monospace">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText"></div>
<div class="PlainText" style="font-family:"Courier New",monospace; font-size:13.3333px">
</div>
<span id="ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div>dwc62@drexel.edu 215.571.4335 (o)</div>
<div>For URCF support: urcf-support@drexel.edu</div>
<div>https://proteusmaster.urcf.drexel.edu/urcfwiki</div>
<div>github:prehensilecode</div>
</div>
<span id="ms-rterangepaste-end"></span>
<div class="PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Brian Andrus <toomuchit@gmail.com><br>
<b>Sent:</b> Thursday, March 4, 2021 10:12<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] prolog not passing env var to job</font>
<div> </div>
</div>
<div>
<table width="100%">
<tbody>
<tr>
<td style="border-left:4px solid goldenrod; background:cornsilk; padding:0 3pt">
<p style="font:small-caps bold 100% sans-serif">External.</p>
</td>
</tr>
</tbody>
</table>
<div>
<p><br>
</p>
<p>It seems to me, if you are using srun directly to get an interactive shell, you can just run the script once you get your shell.</p>
<p><br>
</p>
<p>You can set the variables and then run srun. It automatically exports the environment.</p>
<p>If you want to change a particular one (or more), use something like --export=ALL,MYVAR=othervalue</p>
<p>do 'man srun' and look at the --export option<br>
</p>
<p><br>
</p>
<p>Brian Andrus<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 3/3/2021 9:28 PM, Chin,David wrote:<br>
</div>
<blockquote type="cite"><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace"><a class="x_moz-txt-link-abbreviated" href="mailto:ahmet.mercan@uhem.itu.edu.tr">ahmet.mercan@uhem.itu.edu.tr</a> wrote:</span></div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace">> </span><span style="font-family:"Courier New",monospace; font-size:14.6667px; background-color:rgb(255,255,255); display:inline!important">Prolog and TaskProlog are different parameters and scripts. You should</span><br>
<span style="font-family:"Courier New",monospace; font-size:14.6667px; background-color:rgb(255,255,255); display:inline!important">> use the TaskProlog script to set env. variables.</span></div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace; font-size:14.6667px; background-color:rgb(255,255,255); display:inline!important"><br>
</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px">Can you tell me how to do this for srun? E.g. users request an interactive shell:</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px"> srun -n 1 -t 600 --pty /bin/bash</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px">but the shell on the compute node does not have the env variables set.</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px">I use the same prolog script as TaskProlog, which sets it properly for jobs submitted</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px">with sbatch.</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px"><br>
</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px">Thanks in advance,</span></div>
<div style="font-family:"Courier New",monospace; color:rgb(0,0,0)"><span style="font-size:14.6667px"> Dave Chin</span></div>
<div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="">
<div class="x_BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="x_PlainText" style=""></div>
<span id="x_ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div><a class="x_moz-txt-link-abbreviated" href="mailto:dwc62@drexel.edu">dwc62@drexel.edu</a> 215.571.4335 (o)</div>
<div>For URCF support: <a class="x_moz-txt-link-abbreviated" href="mailto:urcf-support@drexel.edu">
urcf-support@drexel.edu</a></div>
<div><a class="x_moz-txt-link-freetext" href="https://proteusmaster.urcf.drexel.edu/urcfwiki">https://proteusmaster.urcf.drexel.edu/urcfwiki</a></div>
<div>github:prehensilecode</div>
</div>
<span id="x_ms-rterangepaste-end"></span>
<div class="x_PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users
<a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of mercan <a class="x_moz-txt-link-rfc2396E" href="mailto:ahmet.mercan@uhem.itu.edu.tr">
<ahmet.mercan@uhem.itu.edu.tr></a><br>
<b>Sent:</b> Friday, February 12, 2021 16:27<br>
<b>To:</b> Slurm User Community List <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a>; Herc Silverstein <a class="x_moz-txt-link-rfc2396E" href="mailto:herc.silverstein@schrodinger.com">
<herc.silverstein@schrodinger.com></a>; <a class="x_moz-txt-link-abbreviated" href="mailto:slurm-users@schedmd.com">
slurm-users@schedmd.com</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@schedmd.com">
<slurm-users@schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] prolog not passing env var to job</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_PlainText">External.<br>
<br>
Hi;<br>
<br>
Prolog and TaskProlog are different parameters and scripts. You should<br>
use the TaskProlog script to set env. variables.<br>
<br>
Regards;<br>
<br>
Ahmet M.<br>
<br>
<br>
</div>
</span></font></div>
<br>
<p align="Left" style="font-family:Calibri; font-size:10pt; color:#000000; margin:5pt">
Drexel Internal Data<br>
</p>
</blockquote>
</div>
</div>
<br>
<p style="font-family:Calibri;font-size:10pt;color:#000000;margin:5pt;" align="Left">
Drexel Internal Data<br>
</p>
</body>
</html>