Happy Monday everybody,
I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that.
As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-when-job-begi... Is that indeed the case, as searching the web indicates?
If Slurm does not support this, is there a workaround? For example, I'm thinking of installing a local SMTP server, or an alternative/dummy mailx program which instead of relaying emails as requested would post an encrypted web url thing using the information from the email. I am sure I could actually write such a software myself, but I don't have enough time to dedicate to the design, maintenance and debugging of such, so I am looking for something decent already in existence. A cursory web search did not find anything suitable, but perhaps I did not look in the appropriate places, because my gut feeling is that somebody must have already had such an itch to scratch!
Any other ideas about alternative ways to accomplish this?
Thanks
Hi Davide,
Two things you may want to look into:
1. some (most?) web services have "email-to-service" mechanisms of some sort: for instance, you can send an email to a Slack channel, which will create a message from it: https://slack.com/help/articles/206819278-Send-emails-to-Slack
2. Slurm has a trigger mechanism, which can act upon a variety of events: https://slurm.schedmd.com/strigger.html "Triggers include events such as a node failing, a job reaching its time limit or a job terminating." Registering a trigger for each job will probably be a bit on the heavy-side on busy environments, but at least, Slurm can do it.
Cheers, -- Kilian
On Mon, Apr 21, 2025 at 11:46 AM Davide DelVento via slurm-users slurm-users@lists.schedmd.com wrote:
Happy Monday everybody,
I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that.
As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-when-job-begi... Is that indeed the case, as searching the web indicates?
If Slurm does not support this, is there a workaround? For example, I'm thinking of installing a local SMTP server, or an alternative/dummy mailx program which instead of relaying emails as requested would post an encrypted web url thing using the information from the email. I am sure I could actually write such a software myself, but I don't have enough time to dedicate to the design, maintenance and debugging of such, so I am looking for something decent already in existence. A cursory web search did not find anything suitable, but perhaps I did not look in the appropriate places, because my gut feeling is that somebody must have already had such an itch to scratch!
Any other ideas about alternative ways to accomplish this?
Thanks
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Davide DelVento via slurm-users slurm-users@lists.schedmd.com writes:
I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that.
We are just in the process of implementing this on one of our clusters. (The ReST API is already there, what we are implementing is Slurm using it instead of sending emails.) For us, it is quite easy: Simply write a bash script that uses SLURM_* environment variables to get information about the message and user, and then uses curl to issue the required ReST API calls. Then we set the MailProg parameter in slurm.conf to point to this script.
Here is our current test version for this script (so far, it simply logs what it would do instead of actually contacting the ReST API, together with some debug output):
#!/bin/bash
exec &>> /tmp/mail.log
echo $(date +%FT%T): Starting SUBJECT="$2" echo Args: while [[ $# > 0 ]]; do echo "$1" shift done echo echo Envs: env | grep SLURM | sort echo
case $SLURM_JOB_MAIL_TYPE in Began) ACTION="started";; Ended) if [[ $SLURM_JOB_STATE == COMPLETED ]]; then ACTION="completed" elif [[ $SLURM_JOB_STATE == CANCELLED ]]; then ACTION="been cancelled" else ACTION="ended" fi;; Requeued) ACTION="been requeued";; *) ACTION="unknwon action";; esac
BODY="Your job $SLURM_JOB_ID ($SLURM_JOB_NAME) on $SLURM_CLUSTER_NAME has $ACTION. "
echo Recipient: $SLURM_JOB_USER echo Subject: $SUBJECT echo Body: echo $BODY echo echo Done.
Thank you all. I had thought of writing my own, but I suspected it would be too large of a time sink. Your nudges (and example script) have convinced me otherwise, and in fact this is what I will do! Thanks again!
On Tue, Apr 22, 2025 at 3:12 AM Bjørn-Helge Mevik via slurm-users < slurm-users@lists.schedmd.com> wrote:
Davide DelVento via slurm-users slurm-users@lists.schedmd.com writes:
I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that.
We are just in the process of implementing this on one of our clusters. (The ReST API is already there, what we are implementing is Slurm using it instead of sending emails.) For us, it is quite easy: Simply write a bash script that uses SLURM_* environment variables to get information about the message and user, and then uses curl to issue the required ReST API calls. Then we set the MailProg parameter in slurm.conf to point to this script.
Here is our current test version for this script (so far, it simply logs what it would do instead of actually contacting the ReST API, together with some debug output):
#!/bin/bash
exec &>> /tmp/mail.log
echo $(date +%FT%T): Starting SUBJECT="$2" echo Args: while [[ $# > 0 ]]; do echo "$1" shift done echo echo Envs: env | grep SLURM | sort echo
case $SLURM_JOB_MAIL_TYPE in Began) ACTION="started";; Ended) if [[ $SLURM_JOB_STATE == COMPLETED ]]; then ACTION="completed" elif [[ $SLURM_JOB_STATE == CANCELLED ]]; then ACTION="been cancelled" else ACTION="ended" fi;; Requeued) ACTION="been requeued";; *) ACTION="unknwon action";; esac
BODY="Your job $SLURM_JOB_ID ($SLURM_JOB_NAME) on $SLURM_CLUSTER_NAME has $ACTION. "
echo Recipient: $SLURM_JOB_USER echo Subject: $SUBJECT echo Body: echo $BODY echo echo Done.
-- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Bjørn-Helge Mevik -
Davide DelVento -
Kilian Cavalotti