[slurm-users] $TMPDIR does not honor "TmpFS"

[Christopher Samuel]

On 22/11/18 12:38 am, Douglas Duckworth wrote:

> We are setting TmpFS=/scratchLocal in /etc/slurm/slurm.conf on nodes
> and controller.  However $TMPDIR value seems to be /tmp not
> /scratchLocal. As a result users are writing to /tmp which we do not
> want.

Our solution to that was to use a plugin that creates a per-job 
directory and then maps it into the job as /tmp via namespaces.

The advantage of this is that it works for applications & runtimes that
don't honour ${TMPDIR} (yes, Java, I'm looking at you).

https://github.com/chrissamuel/spank-private-tmp

I've got a branch there that does the same for /dev/shm (which relies on 
/dev/shm/slurm existing).

Because our local scratch space is using XFS our prolog sets an XFS 
project quota using the job ID to be the requested space via --tmp (and 
our submit filter converts --tmp requests into --gres:tmp:${TMP}M 
requests so they are used for scheduling so we don't overcommit space).

Our epilog then cleans both the per job temporary space and per job 
/dev/shm up at the end.

All the best,
Chris
-- 
  Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC