[slurm-users] Providing users with info on wait time vs. run time

[Sebastian Potthoff]

Hi Hermann,

I happened to read along this conversation and was just solving this issue today. I added this part to the epilog script to make it work:


# Add job report to stdout
StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')

NODELIST=($(/usr/bin/scontrol show hostnames))

# Only add to StdOut file if it exists and if we are the first node
if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
then
  echo "################################# JOB REPORT ##################################" >> $StdOut
  /usr/bin/seff $SLURM_JOB_ID >> $StdOut
  echo "###############################################################################" >> $StdOut
fi


  Contrary to what it says in the slurm docs https://slurm.schedmd.com/prolog_epilog.html <https://slurm.schedmd.com/prolog_epilog.html>  I was not able to use the env var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I had to make sure it is only called by the „leading“ node as the epilog script will be called by ALL nodes of a multinode job and they would all call seff and clutter up the output. Last thing was to check if StdOut is not of length zero (i.e. it exists). Interactive jobs would otherwise cause the node to drain.

Maybe this helps. 

Kind regards
Sebastian

PS: goslmailer looks quite nice with its recommendations! Will definitely look into it.

--
Westfälische Wilhelms-Universität (WWU) Münster
WWU IT
Sebastian Potthoff (eScience / HPC)

> Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler <hermann.schwaerzler at uibk.ac.at>:
> 
> Hi Ole,
> 
> On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>> On 15-09-2022 16:08, Hermann Schwärzler wrote:
>>> Just out of curiosity: how do you insert the output of seff into the out-file of a job?
>> Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf:
>> MailProg=/usr/bin/smail
> 
> Maybe I am missing something but from what I can tell smail sends an email and does *not* change or append to the .out file of a job...
> 
> Regards,
> Hermann
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220915/296695f3/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4935 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220915/296695f3/attachment-0001.bin>