<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">TmpFS in slurm.conf wasn’t being honored from my experience from at least v16.05.10. When I initially configured Slurm, I noticed this myself. As with the user below, we are also just setting this elsewhere.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks!<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">John </span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Shenglong Wang <sw77@nyu.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Wednesday, November 21, 2018 at 8:55 AM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] $TMPDIR does not honor "TmpFS"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We have TMPDIR setup inside prolog file. Hope users do not have absolute path /tmp inside their scripts.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">#!/bin/bash<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">SLURM_BIN="/opt/slurm/bin"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">SLURM_job_tmp=/state/partition1/job-${SLURM_JOB_ID}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">mkdir -m 700 -p $SLURM_job_tmp<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">chown $SLURM_JOB_USER $SLURM_job_tmp<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">echo "export SLURM_JOBTMP=$SLURM_job_tmp"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo "export SLURM_JOB_TMP=$SLURM_job_tmp"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo "export SLURM_JOB_TMPDIR=$SLURM_job_tmp"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">echo "export TMPDIR=$SLURM_job_tmp”<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Shenglong<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Nov 21, 2018, at 9:44 AM, Roger Moye <<a href="mailto:rmoye@quantlab.com">rmoye@quantlab.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">We are having the exact same problem with $TMPDIR. I wonder if a bug has crept in? I spoke to the SchedMD guys at SC18 last week and they were not aware of a bug but since more than one person is having
this difficulty something must be wrong somewhere.</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">-Roger</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>slurm-users [<a href="mailto:slurm-users-bounces@lists.schedmd.com">mailto:slurm-users-bounces@lists.schedmd.com</a>]<span class="apple-converted-space"> </span><b>On Behalf Of<span class="apple-converted-space"> </span></b>Douglas
Duckworth<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Wednesday, November 21, 2018 7:38 AM<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>[slurm-users] $TMPDIR does not honor "TmpFS"<span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Hi<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">We are setting TmpFS=/scratchLocal in /etc/slurm/slurm.conf on nodes and controller. However $TMPDIR value seems to be /tmp not /scratchLocal. As a result users are writing
to /tmp which we do not want.<br>
<br>
We are not setting $TMPDIR anywhere else such as /etc/profile.d nor do users have it defined in their ~/.bashrc or ~/.bash_profile. <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">We do not see any error messages anywhere which could indicate why the default value of /tmp overrides our value of of TmpFS. <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">As I understand prolog scripts can change this value though, if that's the case, then what's the purpose of setting TmpFS in /etc/slurm/slurm.conf?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><br clear="all">
<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Thanks,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><br>
Douglas Duckworth, MSc, LFCS<br>
HPC System Administrator<br>
</span><span style="font-size:9.5pt;font-family:"Times New Roman",serif"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__scu.med.cornell.edu&d=DwMGaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=gI7YPDDMIksUN29Y65_UCg&m=VfqOl-yXqx9492N0dCQZ7TafygrL8Ev0zem-MEqEYEY&s=H8jTI5dijpEftNyF3D1oZZOBrn4sRgY0wXSxgqUgD2U&e=" target="_blank"><span style="color:purple">Scientific
Computing Unit</span></a></span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Weill Cornell Medicine<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">1300 York Avenue<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">New York, NY 10065<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">E:<span class="apple-converted-space"> </span><a href="mailto:doug@med.cornell.edu" target="_blank"><span style="color:purple">doug@med.cornell.edu</span></a><br>
O: 212-746-6305<br>
F: 212-746-8690<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:8.5pt;font-family:"Verdana",sans-serif"> -----------------------------------------------------------------------------------</span><span style="font-size:9.0pt;font-family:Helvetica"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:8.5pt;font-family:"Verdana",sans-serif">The information in this communication and any attachment is confidential and intended solely for the attention and use of the named addressee(s). All information and opinions expressed herein are
subject to change without notice. This communication is not to be construed as an offer to sell or the solicitation of an offer to buy any security. Any such offer or solicitation can only be made by means of the delivery of a confidential private offering
memorandum (which should be carefully reviewed for a complete description of investment strategies and risks). Any reliance one may place on the accuracy or validity of this information is at their own risk. Past performance is not necessarily indicative of
the future results of an investment. All figures are estimated and unaudited unless otherwise noted. If you are not the intended recipient, or a person responsible for delivering this to the intended recipient, you are not authorized to and must not disclose,
copy, distribute, or retain this message or any part of it. In this case, please notify the sender immediately at<span class="apple-converted-space"> </span><a href="tel:713-333-5440"><span style="color:purple">713-333-5440</span></a></span><span style="font-size:9.0pt;font-family:Helvetica"><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>