[slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

[Bjørn-Helge Mevik]

Juergen Salk <juergen.salk at uni-ulm.de> writes:

> We are also going to implement disk quotas for the amount of local
> scratch space that has been allocated for the job by means of generic
> resources (e.g. `--gres=scratch:100´ for 100GB). This is especially
> important when several users share a node.

Indeed.

> This leads me to ask how you plan to determine the amount of local
> scratch space allocated for the job from within its prolog and epilog
> scripts.
[...]
> I already thought about running `scontrol show job $SLURM_JOB_ID´ from
> within the prolog/epilog scripts in order to get that piece of
> information.

This is exactly what we do. :)

> This line could eventually be parsed to get the amount of scratch
> allocated for this job (and then further used to increase/decrease the
> quota limits for the corresponding $SLURM_JOB_USER in the
> prolog/epilog scripts).

If you use separate directories for each job, and use "project" quotas
(a.k.a folder quotas), then you don't have to adjust the quota when a
new job arrives, even if it is from the same user.

> However, this still looks kind of clumsily to me and I wonder, if I
> have just overlooked a more obvious, cleaner or more robust solution.

Nope.    (We could have done something more elegant like writing a
small Perl utility that extracted just the needed parts, but never got
around to it.)

I _think_ another option would be to write a SPANK plugin for the gres,
and let that create/remove the scratch directory and set the quota, but
I haven't looked into that.  That would probably count as a more elegant
solution.

> Since this is probably not an unusual requirement, I suppose this is
> something that many other sites have already solved for
> themselves. No?

Yes, please, let us know how you've solved this!

-- 
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190903/09b3845c/attachment.sig>