<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Sorry, obvs wasn’t ready to send that last message yet…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our issue is the shared storage is via NFS, and the “fast storage in limited supply” is only local on each node. Hence the need to copy it over from NFS (and then remove it when finished with it.)<br>
<br>
I also wanted the copy & remove to be different jobs, because the main processing job usually requires GPU gres, which is a time-limited resource on the partition. I don’t want to tie up the allocation of GPUs while the data is staged (and removed), and if
the data copy fails, don’t want to even progress to the job where the compute happens (so like, copy_data_locally && process_data)
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:5.0pt;margin-right:0in;margin-bottom:5.0pt;margin-left:0in">
<span style="color:black">If you've got other fast storage in limited supply that can be used for data that can be staged, then by all means use it, but consider whether you want batch cpu cores tied up with the wall time of transferring the data. This could
easily be done on a time-shared frontend login node from which the users could then submit (via script) jobs after the data was staged. Most of the transfer wallclock is in network wait, so don't waste dedicated cores for it.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:5.0pt;margin-right:0in;margin-bottom:5.0pt;margin-left:0in">
<span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>