Hi Em, Let me guess – a naïve nextflow workflow?
😊
I don’t know how much difference it’ll make to reducing load on SLURM, but Seqera did recently accept a small patch from me that allows nextflow to support using the --only-job-state option in squeue.
I don’t know exactly when they’ll release it, it was scheduled for Nextflow 25.12.0-edge or something like that.
It’s no substitute for actually writing efficient nextflow workflows, making use in particular of nextflow’s buffer approach, which is really great for grouping lots of small tasks into a single job.
Tim
From:
Emyr James via slurm-users <slurm-users@lists.schedmd.com>
Date: Monday, 12 January 2026 at 14:15
To: Patryk Bełzak <patryk.belzak@pwr.edu.pl>, Jason Ellul via slurm-users <slurm-users@lists.schedmd.com>, Jason Ellul <Jason.Ellul@petermac.org>
Subject: [slurm-users] Re: slurmctld hourly: Unexpected missing socket error
Hi,
We had the same error message. We see this happen when we get users submitting job arrays with lots of short jobs and then you get a storm of job starts and job completions within a short space of time.
For every start of a job or end of a job, there are multiple messages that go back and forth between the compute node, the slurm controller, the database daemon and the database itself and multiple rows are inserted
and updated within the database. When there is a high turnover of jobs in a short amount of time the slurm system is unable to keep up and it effectively becomes a denial-of-service attack against slurm.
This command allowed me to see users that had a lot of jobs completing within the last day or so (you can add start and end filters for more precision)...
sacct --allusers -o jobid%20,user%20,jobname,state,exitcode,elapsedraw | grep -v batch |grep -v extern | grep -v RUNNING | awk '{print $2}' | sort | uniq -c | sort -nr| head
(note because we are using pam_slurm_adopt we get 3 rows appearing for each job so I filter out the batch and extern rows)
For the users showing up here I then did
sacct --allusers -o jobid%20,user%20,jobname,state,exitcode,elapsedraw | grep -v batch |grep -v extern | grep -v RUNNING | grep <username> |awk '{print $6}' | sort -n | uniq -c
substituting in the usernames. This shows the number of jobs with elapsed time of 0, 1, 2, 3 etc. seconds. If you see multiple users with high numbers in the sub 30 second range then this could be the reason. Submitting
lots of short jobs (<30s runtime) is an anti-pattern.
Instead of submitting e.g. a 1000 element job array of 5 second jobs, the user could repackage this up into 10 jobs that do a for-loop over 100 of the individual jobs so that you get 10 jobs each with 100 subjobs
with a runtime of 500 seconds. This will avoid the message storms. You could even ask for multiple cores here and use gnu parallel to run the 100 jobs across a few cores. This would make the individual jobs run quicker than the expected 500 seconds and you
may get much higher cpu efficiency doing this if these jobs are bottlenecked on IO.
Emyr James
Head of Scientific IT
CRG - Centre for Genomic Regulation
From: Jason Ellul via slurm-users <slurm-users@lists.schedmd.com>
Sent: 30 July 2024 01:46
To: Patryk Bełzak <patryk.belzak@pwr.edu.pl>; Jason Ellul via slurm-users <slurm-users@lists.schedmd.com>
Subject: [slurm-users] Re: slurmctld hourly: Unexpected missing socket error
Thanks again Patryk,
For your insights, we have implemented many of the same things, but the socket errors are still occurring regularly.
If we find a solution that works I will be sure to add it to this thread.
Many thanks
Jason
Jason Ellul
Head - Research Computing Facility
Office of Cancer Research
My onsite days are Mon, alt Wed and Friday.
Phone +61 3 8559 6546
Email Jason.Ellul@petermac.org
305 Grattan Street
Melbourne, Victoria
3000 Australia
www.petermac.org
