<div dir="ltr"><div><br></div><div>Unfortunately this is not a good workflow.</div><div><br></div><div>You would submit a staging job with a dependency for the compute job; however, in the meantime, the scheduler might launch higher-priority jobs that would want the scratch space, and cause it to be scrubbed.</div><div><br></div><div>In a rational process, the scratch space would be scrubbed for the higher-priority jobs. I'm now thinking of a way that the scheduler could consider data turds left by previous jobs, but that's not currently a scheduling feature in SLURM multi-factor or any other scheduler I know.</div><div><br></div><div>The best current workflow is to stage data into fast local persistent storage, and then to schedule jobs, or schedule a job that does it synchronously (TImeLimit=Stage+Compute). The latter is pretty unsocial and wastes cycles.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Apr 3, 2021 at 3:45 PM Will Dennis <<a href="mailto:wdennis@nec-labs.com">wdennis@nec-labs.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="gmail-m_2433803683187651096WordSection1">
<p class="MsoNormal">Hi all,</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We have various NFS servers that contain the data that our researchers want to process. These are mounted on our Slurm clusters on well-known paths. Also, the nodes have local fast scratch disk on another well-known path. We do not have
any distributed file systems in use (Our Slurm clusters are basically just collections of hetero nodes of differing types, not a traditional HPC setup by any means.)</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">In most cases, the researchers can process the data directly off the NFS mounts without it causing any issues, but in some cases, this slows down the computation unacceptably. They could manually copy the data to the local drive using an
allocation & srun commands, but I am wondering if there is a way to do this in sbatch?
</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I tried this method:</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">wdennis@submit01 ~> sbatch transfer.sbatch<u></u><u></u></p>
<p class="MsoNormal">Submitted batch job 329572<u></u><u></u></p>
<p class="MsoNormal">wdennis@submit01 ~> sbatch --dependency=afterok:329572 test_job.sbatch<u></u><u></u></p>
<p class="MsoNormal">Submitted batch job 329573<u></u><u></u></p>
<p class="MsoNormal">wdennis@submit01 ~> sbatch --dependency=afterok:329573 rm_data.sbatch<u></u><u></u></p>
<p class="MsoNormal">Submitted batch job 329574<u></u><u></u></p>
<p class="MsoNormal">wdennis@submit01 ~><u></u><u></u></p>
<p class="MsoNormal">wdennis@submit01 ~> squeue<u></u><u></u></p>
<p class="MsoNormal"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<u></u><u></u></p>
<p class="MsoNormal"> 329573 gpu wdennis_ wdennis PD 0:00 1 (Dependency)<u></u><u></u></p>
<p class="MsoNormal"> 329574 gpu wdennis_ wdennis PD 0:00 1 (Dependency)<u></u><u></u></p>
<p class="MsoNormal"> 329572 gpu wdennis_ wdennis R 0:23 1 compute-gpu02</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">But it seems to not preserve the node allocated with the --dependency jobs:</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">JobID|JobName|User|Partition|NodeList|AllocCPUS|ReqMem|CPUTime|QOS|State|ExitCode|AllocTRES|<u></u><u></u></p>
<p class="MsoNormal">329572|wdennis_data_transfer|wdennis|gpu|compute-gpu02|1|2Gc|00:02:01|normal|COMPLETED|0:0|cpu=1,mem=2G,node=1|<u></u><u></u></p>
<p class="MsoNormal">329573|wdennis_compute_job|wdennis|gpu|compute-gpu05|1|128Gn|00:03:00|normal|COMPLETED|0:0|cpu=1,mem=128G,node=1,gres/gpu=1|<u></u><u></u></p>
<p class="MsoNormal">329574|wdennis_data_removal|wdennis|gpu|compute-gpu02|1|2Gc|00:00:01|normal|COMPLETED|0:0|cpu=1,mem=2G,node=1|</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">What is the best way to do something like “stage the data on a local path / run computation using the local copy / remove the locally staged data when complete”?</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks!</p>
<p class="MsoNormal">Will</p>
</div>
</div>
</blockquote></div>