Davide DelVento via slurm-users slurm-users@lists.schedmd.com writes:
I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that.
We are just in the process of implementing this on one of our clusters. (The ReST API is already there, what we are implementing is Slurm using it instead of sending emails.) For us, it is quite easy: Simply write a bash script that uses SLURM_* environment variables to get information about the message and user, and then uses curl to issue the required ReST API calls. Then we set the MailProg parameter in slurm.conf to point to this script.
Here is our current test version for this script (so far, it simply logs what it would do instead of actually contacting the ReST API, together with some debug output):
#!/bin/bash
exec &>> /tmp/mail.log
echo $(date +%FT%T): Starting SUBJECT="$2" echo Args: while [[ $# > 0 ]]; do echo "$1" shift done echo echo Envs: env | grep SLURM | sort echo
case $SLURM_JOB_MAIL_TYPE in Began) ACTION="started";; Ended) if [[ $SLURM_JOB_STATE == COMPLETED ]]; then ACTION="completed" elif [[ $SLURM_JOB_STATE == CANCELLED ]]; then ACTION="been cancelled" else ACTION="ended" fi;; Requeued) ACTION="been requeued";; *) ACTION="unknwon action";; esac
BODY="Your job $SLURM_JOB_ID ($SLURM_JOB_NAME) on $SLURM_CLUSTER_NAME has $ACTION. "
echo Recipient: $SLURM_JOB_USER echo Subject: $SUBJECT echo Body: echo $BODY echo echo Done.