Hi all, first of all, sorry for my English, it's not my native language.
We are currently experiencing an issue with srun and salloc on our
login nodes, while sbatch works properly.
slurm version 23.11.4.
slurmctld runs on management node mmgt01.
srun and salloc fail intermittently on login node, that means
we can successfully use srun on login node from time to time, but it
stops working for a while without us changing any configuration.
login nodes reports the following to the user:
$ srun -N 1 -n 64 --partition=gpunode --pty /bin/bash
srun: job 4872 queued and waiting for resources
srun: error: Security violation, slurm message from uid 202
srun: error: Security violation, slurm message from uid 202
srun: error: Security violation, slurm message from uid 202
srun: error: Task launch for StepId=4872.0 failed on node cn065: Invalid
job
credential
srun: error: Application launch failed: Invalid job credential
srun: Job step aborted
uid 202 is the slurm user.
On the server side, slurmctld logs show:
sched: _slurm_rpc_allocate_resources JobId=4872 NodeList=(null) usec=228
sched: Allocate JobId=4872 NodeList=cn065 #CPUs=64 Partition=gpunode
error: slurm_receive_msgs: [[snmgt01]:38727] failed: Zero Bytes were
transmitted or received
Killing interactive JobId=4872: Communication connection failure
_job_complete: JobId=4872 WEXITSTATUS 1
_job_complete: JobId=4872 done
step_partial_comp: JobId=4872 StepID=0 invalid; this step may have already
completed
_slurm_rpc_complete_job_allocation: JobId=4872 error Job/step already
completing or completed
We suspect it is a network issue regarding the Zero Bytes were transmitted
or received.
The configless system is working properly. Slurmd on login
node can read changes made at slurm.conf after a scontrol reconfig.
srun runs successfully from management nodes and from compute nodes.
The issue is from the login node.
scontrol ping always shows DOWN from login node, even when we can
successfully
run srun or salloc.
$ scontrol ping
Slurmctld(primary) at mmgt01 is DOWN
We checked as well for munge consistency.
mmgt and login nodes have the hostnames of their respective others on
/etc/hosts. They can communicate.
We would really appreciate some tips on what we could be missing.
Best regards,
Bruno Bruzzo
System Administrator - Clementina XXI
