[slurm-users] $TMPDIR does not honor "TmpFS"

[Christoph Brüning]

Hi Chris,

I actually had the same idea concerning XFS project quotas when I came 
across that private-tmp SPANK plugin recently. Haven't had the 
opportunity to make a test implementation yet, though.

I was wondering if constantly making and deleting XFS projects has a 
considerable impact on performance and stability. So I'd be glad if you 
could share some of your experience with that setup.
Also, would you mind providing access to your prolog and epilog scripts?

Greetings from Germany,
Christoph




On 22/11/2018 00.37, Christopher Samuel wrote:
> On 22/11/18 12:38 am, Douglas Duckworth wrote:
> 
>> We are setting TmpFS=/scratchLocal in /etc/slurm/slurm.conf on nodes
>> and controller.  However $TMPDIR value seems to be /tmp not
>> /scratchLocal. As a result users are writing to /tmp which we do not
>> want.
> 
> Our solution to that was to use a plugin that creates a per-job 
> directory and then maps it into the job as /tmp via namespaces.
> 
> The advantage of this is that it works for applications & runtimes that
> don't honour ${TMPDIR} (yes, Java, I'm looking at you).
> 
> https://github.com/chrissamuel/spank-private-tmp
> 
> I've got a branch there that does the same for /dev/shm (which relies on 
> /dev/shm/slurm existing).
> 
> Because our local scratch space is using XFS our prolog sets an XFS 
> project quota using the job ID to be the requested space via --tmp (and 
> our submit filter converts --tmp requests into --gres:tmp:${TMP}M 
> requests so they are used for scheduling so we don't overcommit space).
> 
> Our epilog then cleans both the per job temporary space and per job 
> /dev/shm up at the end.
> 
> All the best,
> Chris

-- 
Dr. Christoph Brüning
Universität Würzburg
Rechenzentrum
Am Hubland
D-97074 Würzburg
Tel.: +49 931 31-80499