<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi all,</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have various NFS servers that contain the data that our researchers want to process. These are mounted on our Slurm clusters on well-known paths. Also, the nodes have local fast scratch disk on another well-known path. We do not have
any distributed file systems in use (Our Slurm clusters are basically just collections of hetero nodes of differing types, not a traditional HPC setup by any means.)</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In most cases, the researchers can process the data directly off the NFS mounts without it causing any issues, but in some cases, this slows down the computation unacceptably. They could manually copy the data to the local drive using an
allocation & srun commands, but I am wondering if there is a way to do this in sbatch?
</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I tried this method:</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">wdennis@submit01 ~> sbatch transfer.sbatch<o:p></o:p></p>
<p class="MsoNormal">Submitted batch job 329572<o:p></o:p></p>
<p class="MsoNormal">wdennis@submit01 ~> sbatch --dependency=afterok:329572 test_job.sbatch<o:p></o:p></p>
<p class="MsoNormal">Submitted batch job 329573<o:p></o:p></p>
<p class="MsoNormal">wdennis@submit01 ~> sbatch --dependency=afterok:329573 rm_data.sbatch<o:p></o:p></p>
<p class="MsoNormal">Submitted batch job 329574<o:p></o:p></p>
<p class="MsoNormal">wdennis@submit01 ~><o:p></o:p></p>
<p class="MsoNormal">wdennis@submit01 ~> squeue<o:p></o:p></p>
<p class="MsoNormal"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<o:p></o:p></p>
<p class="MsoNormal"> 329573 gpu wdennis_ wdennis PD 0:00 1 (Dependency)<o:p></o:p></p>
<p class="MsoNormal"> 329574 gpu wdennis_ wdennis PD 0:00 1 (Dependency)<o:p></o:p></p>
<p class="MsoNormal"> 329572 gpu wdennis_ wdennis R 0:23 1 compute-gpu02</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But it seems to not preserve the node allocated with the --dependency jobs:</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">JobID|JobName|User|Partition|NodeList|AllocCPUS|ReqMem|CPUTime|QOS|State|ExitCode|AllocTRES|<o:p></o:p></p>
<p class="MsoNormal">329572|wdennis_data_transfer|wdennis|gpu|compute-gpu02|1|2Gc|00:02:01|normal|COMPLETED|0:0|cpu=1,mem=2G,node=1|<o:p></o:p></p>
<p class="MsoNormal">329573|wdennis_compute_job|wdennis|gpu|compute-gpu05|1|128Gn|00:03:00|normal|COMPLETED|0:0|cpu=1,mem=128G,node=1,gres/gpu=1|<o:p></o:p></p>
<p class="MsoNormal">329574|wdennis_data_removal|wdennis|gpu|compute-gpu02|1|2Gc|00:00:01|normal|COMPLETED|0:0|cpu=1,mem=2G,node=1|</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What is the best way to do something like “stage the data on a local path / run computation using the local copy / remove the locally staged data when complete”?</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!</p>
<p class="MsoNormal">Will</p>
</div>
</body>
</html>