[slurm-users] Reset TMPDIR for All Jobs

[Ellestad, Erik]

I get it working with a slurm Prolog, but because Prolog runs on every job, it creates a TMPDIR every time it slurm runs a job on any node.

Ideally, I’d like to create the TMPDIR ONLY if the job requested –tmp.

I’ve only perused the code a bit, to see how it works, but does this Spank plugin only create a TMPDIR with all jobs, or only if a job requests –tmp?

Erik

--
Erik Ellestad
Wynton Cluster SysAdmin
UCSF

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Greg Wickham
Sent: Monday, May 18, 2020 11:59 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] Reset TMPDIR for All Jobs

Hi Erik,

We use a private fork of https://github.com/hpc2n/spank-private-tmp<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hpc2n_spank-2Dprivate-2Dtmp&d=DwMGaQ&c=iORugZls2LlYyCAZRB3XLg&r=Ct3zEADMmPgyUYfpHDJQWaWsE9mNbEHEhGxpsYoThbE&m=z5SoPrb3-WhcufGXKtxYDuOeJKDLyoBXvnVUERecyh8&s=hD4WqyRTUkz00jF4NHFZwprPai2j3OaROq7PsOD1Vxg&e=>

It has worked quite well for us - jobs (or steps) don’t share a /tmp and during the prolog all files created for the job/step are deleted.

Users absolutely cannot see each others temporary files so there’s no issue even if they happen to have a hard corded path ie: /tmp/myfiles.txt

   -Greg


On 12 May 2020, at 18:40, Ellestad, Erik <Erik.Ellestad at ucsf.edu<mailto:Erik.Ellestad at ucsf.edu>> wrote:

I was wanted to set TMPDIR from /tmp to a per job directory I create in local /scratch/$SLURM_JOB_ID (for example)

This bug suggests I should be able to do this in a task-prolog.

https://bugs.schedmd.com/show_bug.cgi?id=2664<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.schedmd.com_show-5Fbug.cgi-3Fid-3D2664&d=DwMGaQ&c=iORugZls2LlYyCAZRB3XLg&r=Ct3zEADMmPgyUYfpHDJQWaWsE9mNbEHEhGxpsYoThbE&m=z5SoPrb3-WhcufGXKtxYDuOeJKDLyoBXvnVUERecyh8&s=0beKr3hsB8aMTqZ0pXDEggBEIVjhxrvwXlJoFLmobi8&e=>

However adding the following to task-prolog doesn’t seem to affect the variables the job script is running with.

unset TMPDIR
export TMPDIR=/scratch/$SLURM_JOB_ID

It does work if it is done in the job script, rather than the task-prolog.

Am I missing something?

Erik

--
Erik Ellestad
Wynton Cluster SysAdmin
UCSF

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200519/8d112ab1/attachment-0001.htm>