<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">We have TMPDIR setup inside prolog file. Hope users do not have absolute path /tmp inside their scripts.</div><div class=""><br class=""></div><div class=""><div class="">#!/bin/bash</div><div class=""><br class=""></div><div class="">SLURM_BIN="/opt/slurm/bin"</div><div class=""><br class=""></div><div class="">SLURM_job_tmp=/state/partition1/job-${SLURM_JOB_ID}</div><div class=""><br class=""></div><div class="">mkdir -m 700 -p $SLURM_job_tmp</div><div class="">chown $SLURM_JOB_USER $SLURM_job_tmp</div><div class=""><br class=""></div><div class="">echo "export SLURM_JOBTMP=$SLURM_job_tmp"</div><div class="">echo "export SLURM_JOB_TMP=$SLURM_job_tmp"</div><div class="">echo "export SLURM_JOB_TMPDIR=$SLURM_job_tmp"</div><div class="">echo "export TMPDIR=$SLURM_job_tmp”</div><div class=""><br class=""></div><div class="">Best.</div><div class="">Shenglong</div></div><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 21, 2018, at 9:44 AM, Roger Moye <<a href="mailto:rmoye@quantlab.com" class="">rmoye@quantlab.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">We are having the exact same problem with $TMPDIR. I wonder if a bug has crept in? I spoke to the SchedMD guys at SC18 last week and they were not aware of a bug but since more than one person is having this difficulty something must be wrong somewhere.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">-Roger<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>slurm-users [<a href="mailto:slurm-users-bounces@lists.schedmd.com" class="">mailto:slurm-users-bounces@lists.schedmd.com</a>]<span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Douglas Duckworth<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, November 21, 2018 7:38 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:slurm-users@lists.schedmd.com" class="">slurm-users@lists.schedmd.com</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>[slurm-users] $TMPDIR does not honor "TmpFS"<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Hi<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">We are setting TmpFS=/scratchLocal in /etc/slurm/slurm.conf on nodes and controller. However $TMPDIR value seems to be /tmp not /scratchLocal. As a result users are writing to /tmp which we do not want.<br class=""><br class="">We are not setting $TMPDIR anywhere else such as /etc/profile.d nor do users have it defined in their ~/.bashrc or ~/.bash_profile. <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">We do not see any error messages anywhere which could indicate why the default value of /tmp overrides our value of of TmpFS. <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">As I understand prolog scripts can change this value though, if that's the case, then what's the purpose of setting TmpFS in /etc/slurm/slurm.conf?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br clear="all" class=""><o:p class=""></o:p></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class="">Douglas Duckworth, MSc, LFCS<br class="">HPC System Administrator<br class=""><span style="font-size: 9.5pt;" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__scu.med.cornell.edu&d=DwMGaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=gI7YPDDMIksUN29Y65_UCg&m=VfqOl-yXqx9492N0dCQZ7TafygrL8Ev0zem-MEqEYEY&s=H8jTI5dijpEftNyF3D1oZZOBrn4sRgY0wXSxgqUgD2U&e=" target="_blank" style="color: purple; text-decoration: underline;" class="">Scientific Computing Unit</a></span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Weill Cornell Medicine<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">1300 York Avenue<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">New York, NY 10065<o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">E:<span class="Apple-converted-space"> </span><a href="mailto:doug@med.cornell.edu" target="_blank" style="color: purple; text-decoration: underline;" class="">doug@med.cornell.edu</a><br class="">O: 212-746-6305<br class="">F: 212-746-8690<o:p class=""></o:p></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font style="font-family: Verdana; font-size: 11px;" class=""><span style="font-family: Verdana;" class=""> </span>-----------------------------------------------------------------------------------</font></p><font style="caret-color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Verdana; font-size: 11px;" class=""></font><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font style="font-family: Verdana; font-size: 11px;" class="">The information in this communication and any attachment is confidential and intended solely for the attention and use of the named addressee(s). All information and opinions expressed herein are subject to change without notice. This communication is not to be construed as an offer to sell or the solicitation of an offer to buy any security. Any such offer or solicitation can only be made by means of the delivery of a confidential private offering memorandum (which should be carefully reviewed for a complete description of investment strategies and risks). Any reliance one may place on the accuracy or validity of this information is at their own risk. Past performance is not necessarily indicative of the future results of an investment. All figures are estimated and unaudited unless otherwise noted. If you are not the intended recipient, or a person responsible for delivering this to the intended recipient, you are not authorized to and must not disclose, copy, distribute, or retain this message or any part of it. In this case, please notify the sender immediately at<span class="Apple-converted-space"> </span><a href="tel:713-333-5440" style="color: purple; text-decoration: underline;" class="">713-333-5440</a></font></p></div></blockquote></div><br class=""></body></html>