[slurm-users] Staging data on the nodes one will be processing on via sbatch
wdennis at nec-labs.com
Sat Apr 3 20:10:53 UTC 2021
What I mean by “scratch” space is indeed local persistent storage in our case; sorry if my use of “scratch space” is already a generally-known Slurm concept I don’t understand, or something like /tmp… That’s why my desired workflow is to “copy data locally / use data from copy / remove local copy” in separate steps.
From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Fulcomer, Samuel <samuel_fulcomer at brown.edu>
Date: Saturday, April 3, 2021 at 4:00 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] Staging data on the nodes one will be processing on via sbatch
The best current workflow is to stage data into fast local persistent storage, and then to schedule jobs, or schedule a job that does it synchronously (TImeLimit=Stage+Compute). The latter is pretty unsocial and wastes cycles.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the slurm-users