<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
On one of our clusters the <span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">"Epilog" setup in slurm.conf, (<span style="display: inline !important; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted6">Epilog=/etc/slurm/slurm.epilog.local) </span></span><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">call's
the following to run the tmpwatch utility with a tiny access time on /tmp. I think tmpwatch can be run on specified paths and not just /tmp. </span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
#####################################
<div class="ContentPasted0"># First, clean out /tmp, ruthlessly #</div>
<div class="ContentPasted0">#####################################</div>
<div class="ContentPasted0"># 2022-04-20, smcgrat@tcd.ie, RT#25239, RT#25262</div>
<div class="ContentPasted0"># this will delete anything in /tmp older than a minute</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">outfile=/tmp/epilog.$CLUSTER.$(date +%Y%m%d%H%M%S).txt</div>
<div class="ContentPasted0">{</div>
<div class="ContentPasted0">echo "------------------------------------------------------------------"</div>
<div class="ContentPasted0">echo -n "* /tmp maintenance - "</div>
<div class="ContentPasted0">date</div>
<div class="ContentPasted0">echo -n "* Current usage"</div>
<div class="ContentPasted0">echo ""</div>
<div class="ContentPasted0">df -h /tmp</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">echo -n "* Running tmpwatch to delete anything in /tmp older than a minute"</div>
<div class="ContentPasted0">/usr/sbin/tmpwatch --atime 1m /tmp</div>
<div class="ContentPasted0">echo ""</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">echo -n "* /tmp usage now"</div>
<div class="ContentPasted0">echo ""</div>
<div class="ContentPasted0">df -h /tmp</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">echo ""</div>
<div class="ContentPasted0">} >> "${outfile}"</div>
<br>
</div>
<div class="elementToProof">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="ContentPasted3 ContentPasted4 ContentPasted5">
The last thing our epilog setup does is run the standard /etc/slurm/slurm.epilog.clean.dist. Looking at that it uses the SLURM_UID and SLURM_JOB_ID variables. I would guess that it also has access to the other variables in the context.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="ContentPasted3 ContentPasted4 ContentPasted5">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="ContentPasted3 ContentPasted4 ContentPasted5">
Best</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="ContentPasted3 ContentPasted4 ContentPasted5">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="ContentPasted3 ContentPasted4 ContentPasted5">
Sean</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="ContentPasted3 ContentPasted4 ContentPasted5">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
---</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Sean McGrath</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Senior Systems Administrator, IT Services</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Jason Simms <jsimms1@swarthmore.edu><br>
<b>Sent:</b> Tuesday 10 October 2023 16:59<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] Clean Up Scratch After Failed Job</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hello all,
<div><br>
</div>
<div>Our template scripts for Slurm include a workflow to copy files to a scratch space prior to running a job, and then copying any output files, etc. back to the original submit directory on job completion, and then finally cleaning up (deleting) the scratch
space before exiting. This works great until a job fails or is requeued, in which case the scratch space isn't cleaned up.</div>
<div><br>
</div>
<div>In the past, I've run a cron job that deletes any material in scratch that hasn't been modified in any number of days beyond the max length of a job, but that can still allow "zombie" material to remain in scratch for quite a while. I'm intrigued by using
an epilog script that is triggered after each job completes (whether normally or due to failure, requeuing, etc.) to accomplish the same task more efficiently and consistently.</div>
<div><br>
</div>
<div>The first question is in which context would I run the epilog. I presume I'd want to run it after a job completes entirely, so looking at the table, I think I'd want an Epilog script to run on the compute node. Reading the documentation, however, it is
unclear to me that all variables I would need will be available in such a script. We use the variables $USER, $SLURM_JOB_NAME, and $SLURM_JOB_ID to create a path within scratch unique to each job.</div>
<div><br>
</div>
<div>Specifically, however, the documentation for $SLURM_JOB_NAME says:</div>
<div><br>
</div>
<div>"SLURM_JOB_NAME Name of the job. Available in PrologSlurmctld, SrunProlog, TaskProlog, EpilogSlurmctld, SrunEpilog and TaskEpilog."</div>
<div><br>
</div>
<div>So it doesn't seem to be available in the appropriate context. Thinking about it, however, I presume if I only use the $SLURM_JOB_ID and $USER (and then $SLURM_JOB_USER in the epilog script) that the path would still be unique; meaning, I could just not
use the job name.</div>
<div><br>
</div>
<div>Anyway, if anyone has any thoughts or examples of setting up something like this, I'd appreciate it!</div>
<div><br>
</div>
<div>Warmest regards,</div>
<div>Jason</div>
<div>
<div><br>
</div>
<span class="x_gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:14px; margin:0px">
<span style="color:rgb(130,36,51)"><font face="Century Gothic"><b>Jason L. Simms, Ph.D., M.P.H.</b></font></span></div>
<font face="Century Gothic">Manager of Research Computing</font><br>
</div>
<div dir="ltr">
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:14px; margin:0px">
<font face="Century Gothic"><span style="color:gray">Swarthmore College<br>
Information Technology Services</span></font></div>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:14px; margin:0px">
<font face="Century Gothic"><span style="color:gray">(610) 328-8102<br>
</span></font></div>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:14px; margin:0px">
<font face="Century Gothic">Schedule a meeting: </font><span style="font-family:Arial,Helvetica,sans-serif; font-size:small; color:rgb(32,33,36)"><a href="https://calendly.com/jlsimms" target="_blank">https://calendly.com/jlsimms</a></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>