[slurm-users] Providing users with info on wait time vs. run time

[Paul Edmon]

We also call scontrol in our scripts (a little as we can manage) and we 
run at the scale of 1500 nodes.  It hasn't really caused many issues, 
but we try to limit it as much as we possibly can.

-Paul Edmon-

On 9/16/22 9:41 AM, Sebastian Potthoff wrote:
> Hi Hermann,
>
>>> So you both are happily(?) ignoring this warning the "Prolog and 
>>> Epilog Guide",
>>> right? :-)
>>>
>>> "Prolog and Epilog scripts [...] should not call Slurm commands 
>>> (e.g. squeue,
>>> scontrol, sacctmgr, etc)."
>>
>> We have probably been doing this since before the warning was added to
>> the documentation.  So we are "ignorantly ignoring" the advice :-/
>
> Same here :) But if $SLURM_JOB_STDOUT is not defined as documented … 
> what can you do.
>
>>> May I ask how big your clusters are (number of nodes) and how 
>>> heavily they are
>>> used (submitted jobs per hour)?
>
> We have around 500 nodes (mostly 2x18 cores). Jobs ending (i.e. 
> calling the epilog script) varies quite a lot between 1000 and 15k a 
> day, so something in between 40 and 625 Jobs/hour. During those peaks 
> Slurm can become noticeably slower, however usually it runs fine.
>
> Sebastian
>
>> Am 16.09.2022 um 15:15 schrieb Loris Bennett 
>> <loris.bennett at fu-berlin.de>:
>>
>> Hi Hermann,
>>
>> Hermann Schwärzler <hermann.schwaerzler at uibk.ac.at> writes:
>>
>>> Hi Loris,
>>> hi Sebastian,
>>>
>>> thanks for the information on how you are doing this.
>>> So you both are happily(?) ignoring this warning the "Prolog and 
>>> Epilog Guide",
>>> right? :-)
>>>
>>> "Prolog and Epilog scripts [...] should not call Slurm commands 
>>> (e.g. squeue,
>>> scontrol, sacctmgr, etc)."
>>
>> We have probably been doing this since before the warning was added to
>> the documentation.  So we are "ignorantly ignoring" the advice :-/
>>
>>> May I ask how big your clusters are (number of nodes) and how 
>>> heavily they are
>>> used (submitted jobs per hour)?
>>
>> We have around 190 32-core nodes.  I don't know how I would easily find
>> out the average number of jobs per hour.  The only problems we have had
>> with submission have been when people have written their own mechanisms
>> for submitting thousands of jobs.  Once we get them to use job array,
>> such problems generally disappear.
>>
>> Cheers,
>>
>> Loris
>>
>>> Regards,
>>> Hermann
>>>
>>> On 9/16/22 9:09 AM, Loris Bennett wrote:
>>>> Hi Hermann,
>>>> Sebastian Potthoff <s.potthoff at uni-muenster.de> writes:
>>>>
>>>>> Hi Hermann,
>>>>>
>>>>> I happened to read along this conversation and was just solving 
>>>>> this issue today. I added this part to the epilog script to make 
>>>>> it work:
>>>>>
>>>>> # Add job report to stdout
>>>>> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep 
>>>>> StdOut | /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { 
>>>>> print $2 }')
>>>>>
>>>>> NODELIST=($(/usr/bin/scontrol show hostnames))
>>>>>
>>>>> # Only add to StdOut file if it exists and if we are the first node
>>>>> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z 
>>>>> "${StdOut}" ]
>>>>> then
>>>>>   echo "################################# JOB REPORT 
>>>>> ##################################" >> $StdOut
>>>>>   /usr/bin/seff $SLURM_JOB_ID >> $StdOut
>>>>>   echo 
>>>>> "###############################################################################" 
>>>>> >> $StdOut
>>>>> fi
>>>> We do something similar.  At the end of our script pointed to by
>>>> EpilogSlurmctld we have
>>>>   OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
>>>>   if [ ! -f "$OUT" ]; then
>>>>     exit
>>>>   fi
>>>>   printf "\n== Epilog Slurmctld
>>>> ==================================================\n\n" >>  ${OUT}
>>>>   seff ${SLURM_JOB_ID} >> ${OUT}
>>>>   printf
>>>> "\n======================================================================\n"
>>>>>> ${OUT}
>>>>   chown ${user} ${OUT}
>>>> Cheers,
>>>> Loris
>>>>
>>>>>   Contrary to what it says in the slurm docs 
>>>>> https://slurm.schedmd.com/prolog_epilog.html  I was not able to 
>>>>> use the env var SLURM_JOB_STDOUT, so I had to fetch it via 
>>>>> scontrol. In addition I had to
>>>>> make sure it is only called by the „leading“ node as the epilog 
>>>>> script will be called by ALL nodes of a multinode job and they 
>>>>> would all call seff and clutter up the output. Last thing was to 
>>>>> check if StdOut is
>>>>> not of length zero (i.e. it exists). Interactive jobs would 
>>>>> otherwise cause the node to drain.
>>>>>
>>>>> Maybe this helps.
>>>>>
>>>>> Kind regards
>>>>> Sebastian
>>>>>
>>>>> PS: goslmailer looks quite nice with its recommendations! Will 
>>>>> definitely look into it.
>>>>>
>>>>> --
>>>>> Westfälische Wilhelms-Universität (WWU) Münster
>>>>> WWU IT
>>>>> Sebastian Potthoff (eScience / HPC)
>>>>>
>>>>>  Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
>>>>> <hermann.schwaerzler at uibk.ac.at>:
>>>>>
>>>>>  Hi Ole,
>>>>>
>>>>>  On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>>>>>
>>>>>  On 15-09-2022 16:08, Hermann Schwärzler wrote:
>>>>>
>>>>>  Just out of curiosity: how do you insert the output of seff into 
>>>>> the out-file of a job?
>>>>>
>>>>>  Use the "smail" tool from the slurm-contribs RPM and set this in 
>>>>> slurm.conf:
>>>>>  MailProg=/usr/bin/smail
>>>>>
>>>>>  Maybe I am missing something but from what I can tell smail sends 
>>>>> an email and does *not* change or append to the .out file of a job...
>>>>>
>>>>>  Regards,
>>>>>  Hermann
>>>>
>>>
>> --
>> Dr. Loris Bennett (Herr/Mr)
>> ZEDAT, Freie Universität Berlin         Emailloris.bennett at fu-berlin.de
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220916/793cba61/attachment-0001.htm>