Generally, the troubleshooting steps which you should take for Slurm are:

squeue to look at the list of running/queued or held jobs

sinfo to show which nodes are idle, busy or down

scontrol show node to get more detailed information on a node

For problem nodes - indeed just log into any node to see what a healthy node looks like systemctl status slurmd cat /var/log/slurm/slurmd.log

On your slurm controller look at the slurmctld and slurmdbd logs

On Tue, 7 Jan 2025 at 11:30, sportlecon sportlecon via slurm-users < slurm-users@lists.schedmd.com> wrote:

slurm 24.11 - squeue displays reason "launch failed requeued held"

-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com