[slurm-users] Staging data on the nodes one will be processing on via sbatch

[Will Dennis]

What I mean by “scratch” space is indeed local persistent storage in our case; sorry if my use of “scratch space” is already a generally-known Slurm concept I don’t understand, or something like /tmp… That’s why my desired workflow is to “copy data locally / use data from copy / remove local copy” in separate steps.


From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Fulcomer, Samuel <samuel_fulcomer at brown.edu>
Date: Saturday, April 3, 2021 at 4:00 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] Staging data on the nodes one will be processing on via sbatch
[…]
The best current workflow is to stage data into fast local persistent storage, and then to schedule jobs, or schedule a job that does it synchronously (TImeLimit=Stage+Compute). The latter is pretty unsocial and wastes cycles.
[…]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210403/da629d2e/attachment-0001.htm>