- slurm-users - lists.schedmd.com

Can't specify multiple partitions when submitting GPU jobs
by milad＠earthspecies.org 29 Apr '25

29 Apr '25

Hi, I have defined a partition for each GPU type we have in the cluster. This was mainly because I have different Node types for each GPU type and I want to set `DefCpuPerGPU` `DefMemPerGPU` for each of them. Unfortunately one can't set them per node but can do that per partition. Now sometimes people don't care about the GPU type and would like any of the partitions to pick up the job. The `--partition` in `sbatch` does allow specifying multiple partitions and this works fine when I'm not specifying `--gpu`. However when I add do something like `sbatch -p A,B --gpus 1 script.sh` then I get "srun: job 6279 queued and waiting for resources" even though partition B does have a GPU to offer. Strangely if the first partition specified (i.e. A) had a free GPU it would allocate the GPU and run the job. Is this a bug? Perhaps related to this: https://groups.google.com/g/slurm-users/c/UOUVfkajUBQ

3 4

Re: Error "_check_core_range_matches_sock" when
by Gestió Servidors 29 Apr '25

29 Apr '25

Hello, Yes, I know system is single-socket, but in a similar computer, this configuration works fine... After some tests, I have realized that with SLURM 23.11.0 I can assign CPUs in "gres.conf" (and, then, use half CPU for one GPU and other half for other GPU), but with SLURM 24.11.4 I receive " _check_core_range_matches_sock". I have read "NEWS" file inside sources files in 24.11.4 tgz and it says: […] * Changes in Slurm 23.11.7 ========================== […] -- Fix issue that only left one thread on each core available when "CPUs=" is configured to total thread count on multi-threaded hardware and no other topology info ("Sockets=", "CoresPerSocket", etc.) is configured. […] So, I suppose, in version 23.11.7 SLURM corrected that behaviour. Could someone confirm that? Thanks.

1 0

Error "_check_core_range_matches_sock" when starting "slurmctld"
by Gestió Servidors 28 Apr '25

28 Apr '25

Hello, I have compiled SLURM-24.11.3 and I have configured two GPUs in my system (slurmctld and slurmd are running in the same computer). Computes has a old processor Intel i7 with 4 cores and 4 hyperthreading. Node is configured with "NodeName=mysystem CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7940 Gres=gpu:geforce_gtx_titan_x:1,gpu:geforce_gtx_titan_black:1". "lscpu" command returns: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz BIOS Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz File gres.conf is: NodeName=mysystem Autodetect=off Name=gpu Type=geforce_gtx_titan_x File=/dev/nvidia0 CPUs=0-1 NodeName=mysystem Autodetect=off Name=gpu Type=geforce_gtx_titan_black File=/dev/nvidia1 CPUs=2-3 However, when I start daemon "slurmctld", system returns this error: [2025-04-28T09:35:41.003] error: _check_core_range_matches_sock: gres/gpu GRES core specification 0-1 for node aopcvis5 doesn't match socket boundaries. (Socket 0 is cores 0-3) [2025-04-28T09:35:41.003] error: Setting node aopcvis5 state to INVAL with reason:gres/gpu GRES core specification 0-1 for node aopcvis5 doesn't match socket boundaries. (Socket 0 is cores 0-3) Where is my configuration error? Thanks.

2 1

How can we put limits on interactive jobs?
by Ole Holm Nielsen 25 Apr '25

25 Apr '25

We would like to put limits on interactive jobs (started by salloc) so that users don't leave unused interactive jobs behind on the cluster by mistake. I can't offhand find any configurations that limit interactive jobs, such as enforcing a timelimit. Perhaps this could be done in job_submit.lua, but I couldn't find any job_desc parameters in the source code which would indicate if a job is interactive or not. Question: How do people limit interactive jobs, or identify orphaned jobs and kill them? Thanks a lot, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

4 4

Setting QoS with slurm 24.05.7
by Patrick Begou 25 Apr '25

25 Apr '25

Hi all, I'm trying to setup a QoS on a small 5 nodes cluster running slurm 24.05.7. My goal is to limit the resources on a (time x number of cores) strategy to avoid one large job requesting all the resources for too long time. I've read from https://slurm.schedmd.com/qos.html and some discussion but my setup is still not working. I think I need to set these informations: MaxCPUsPerJob=172800 MaxWallDurationPerJob=48:00:00 Flags=DenyOnLimit,OverPartQOS for: 12h max for 240 cores => (12*240*60=172800mn) no job can exceed 2 days do not accept jobs out of these limits. What I've done: 1) create the QoS: sudo sacctmgr add qos workflowlimit \ MaxWallDurationPerJob=48:00:00 \ MaxCPUsPerJob=172800 \ Flags=DenyOnLimit,OverPartQOS 2) Check sacctmgr show qos Name=workflowlimit format=Name%16,MaxTRES,MaxWall Name MaxTRES MaxWall ---------------- ------------- ----------- workflowlimit cpu=172800 2-00:00:00 3) Set the QoS for the account "most" which is the default account for the users: sudo sacctmgr modify account name=most set qos=workflowlimit 4) Check $ sacctmgr show assoc format=account,cluster,user,qos Account Cluster User QOS ---------- ---------- ---------- -------------------- root osorno normal root osorno root normal legi osorno normal most osorno workflowlimit most osorno begou workflowlimit 5) Modifiy slurm.conf with: AccountingStorageEnforce=limits,qos and propagate on the 5 nodes and the front end (done via Ansible) 6) Check clush -b -w osorno-fe,osorno,osorno-0-[0-4] 'grep AccountingStorageEnforce /etc/slurm/slurm.conf' --------------- osorno,osorno-0-[0-4],osorno-fe (7) --------------- AccountingStorageEnforce=limits,qos 7) restart slurmd on all the compute nodes and slurmctld + slurmdbd on the management node. But I can still request 400 cores for 24 hours: [begou@osorno ~]$ srun -n 400 -t 24:0:0 --pty bash bash-5.1$ squeue JOBID PARTITION NAME USER ST TIME START_TIME TIME_LIMIT CPUS NODELIST(REASON) 147 genoa bash begou R 0:03 2025-04-18T16:52:11 1-00:00:00 400 osorno-0-[0-4] So I must have missed something ? My partition (I've only one) in slurm.conf is: PartitionName=genoa State=UP Default=YES MaxTime=48:00:00 DefaultTime=24:00:00 Shared=YES OverSubscribe=NO Nodes=osorno-0-[0-4] Thanks Patrick

2 4

Job running slower when using Slurm
by Jeffrey Layton 24 Apr '25

24 Apr '25

Good morning, I'm running an NPB test, bt.C that is OpenMP and built using NV HPC SDK (version 25.1). I run it on a compute node by ssh-ing to the node. It runs in about 19.6 seconds. Then I run the code using a simple job: Command to submit job: sbatch --nodes=1 run-npb-omp The script run-npb-omp is the following: #!/bin/bash cd /home/.../NPB3.4-OMP/bin ./bt.C.x When I use Slurm, the job takes 482 seconds. Nothing really appears in the logs. It doesn't do any IO. No data is copied anywhere. I'm king of at a loss to figure out why. Any suggestions of where to look? Thanks! Jeff

4 9

Strange output of sshare
by frsc＠adm.ku.dk 23 Apr '25

23 Apr '25

Hi all, I'm trying to clean up and reconfigure fair share on a slurm 20.11.9 production cluster after some trial and error before I started looking into it. I don't know the full story and need to pick up here. Fair share is enabled with default settings and not customised. It looks a bit like it was enabled by accident and relevant options are undefined/at default. As a first step I would like to understand the current state. Executing sshare includes output like this (output shortened but complete for the two relevant accounts): Account User Partition RawShares NormShares RawUsage EffectvUsage FairShare -------- ---------- ------------ ---------- ----------- ----------- ------------- ---------- root 0.000000 1002501 1.000000 .root root 1 0.083333 0 0.000000 1.000000 .A1 1 0.083333 0 0.000000 ..A1 U1 P1 1 0.166667 0 0.000000 0.873684 ..A1 U2 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P2 1 0.166667 0 0.000000 0.873684 ..A1 U4 P3 1 0.166667 0 0.000000 0.873684 ..A1 U4 P1 1 0.166667 0 1.000000 0.821053 <== What is going on here?? .A2 1 0.083333 0 0.000000 ..A2 U5 P2 1 0.142857 0 0.000000 1.000000 ..A2 U5 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P4 1 0.142857 0 0.000000 1.000000 ..A2 U6 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P2 1 0.142857 0 0.000000 1.000000 ..A2 U7 P1 1 0.142857 0 0.000000 1.000000 ..A2 U7 P2 1 0.142857 0 0.000000 1.000000 User U4 is not a member of any other account. I understand everything about this output except the line I marked. Both accounts A1 and A2 have zero usage, yet, for user U4 on partition P1 we have effective usage 1.0, screwing the fair share factors up for everyone in this account. As far as I can tell, both accounts should look identical with a fair share factor of 1 for every user. I'm grateful for any pointer for what to look for. Best regards, Frank

2 1

Slurm webhooks
by Davide DelVento 22 Apr '25

22 Apr '25

Happy Monday everybody, I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that. As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-when-job-beg… Is that indeed the case, as searching the web indicates? If Slurm does not support this, is there a workaround? For example, I'm thinking of installing a local SMTP server, or an alternative/dummy mailx program which instead of relaying emails as requested would post an encrypted web url thing using the information from the email. I am sure I could actually write such a software myself, but I don't have enough time to dedicate to the design, maintenance and debugging of such, so I am looking for something decent already in existence. A cursory web search did not find anything suitable, but perhaps I did not look in the appropriate places, because my gut feeling is that somebody must have already had such an itch to scratch! Any other ideas about alternative ways to accomplish this? Thanks

3 3

problem with slurmd dynamic mode with slurmctld in docker
by stef53864 22 Apr '25

22 Apr '25

hi everybody, i try to user dynamic mode with configless mode with slurm 24.11.3 and upgrade to 24.11.4 with slurmd and i found a problem. slurmctld is a container with docker, and my node is outside the container network . slurmctld register my ip with a function getpeeraddr on the slurmctld socket. but my ip connected to the socket come from the docker nat/bridge so slurmctld register my ip bridged ( not my real ip ) that is to say the docker gateway (172.20.0.1) *scontrol show node* *-------------------------* *NodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=172.20.0.1 NodeHostName=ltlsbubble1 Version=24.11.4* so the node go down after the "not pinging it" timeout i try to update the config *scontrol uupdate NodeName=ltlsbubble1* *NodeAddr=xx.xx.xx.xx* but a the first *scontrol reconfigure * it comes back to : *NodeAddr=172.20.0.1* in normal mode ------------------- *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.11.4* in normal mode NodeAddr is the same than NodeName , so it use DNS resolution for communication. to verify my hypothesis, i go to the c code of slurm, identify the register function and replace it with the same mechanism than normal node in src/slurmctld/node_mgr.c i replace : set_node_comm_name(node_ptr, *comm_name*, reg_msg->hostname); by set_node_comm_name(node_ptr, NULL, reg_msg->hostname); i rebuild slutmctld with this patch and try it with dynamic mode , it works like expected *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.1* no ip in nodeAddr , but only the nodename, so it use DNS resolution . the node works fine and no goes down for timeout ping so my question : can we have an option to force DNS resolution instead ip discover in Dynamic mode ? ( i try the option cloud_dns, but it not seems the purpose of this option) best regard, Stephane

1 0

pam_slurm_adopt ambiguity in instructions and with ssd
by Robert Kudyba 21 Apr '25

21 Apr '25

In the instructions for pam_slurm_adopt <https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: > > Add the following line to the appropriate file in /etc/pam.d, such as > system-auth or sshd (you may use either the "required" or "sufficient" > PAM control flag): This module is configurable. Add these options to the end of the > pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd > or system-auth): Assuming an OS like CentOS does this mean it should be put in both? slurm.conf on the node has: UsePAM yes slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets: Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user(a)ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth] Am I missing something?

3 5

Issue with Enforcing GPU Usage Limits in Slurm
by lyz＠simplehpc.com 17 Apr '25

17 Apr '25

Hi, I am currently encountering an issue with Slurm's GPU resource limitation. I have attempted to restrict the number of GPUs a user can utilize by executing the following command: sacctmgr modify user lyz set MaxTRES=gres/gpu=2 This command is intended to limit user 'lyz' to using a maximum of 2 GPUs. However, when the user submits a job using srun, specifying CUDA 0, 1, 2, and 3 in the job script, or os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3", the job still utilizes all 4 GPUs during execution. This indicates that the GPU usage limit is not being enforced as expected. How can I resolve this situation.

4 14

pam_slurm_adopt and multiple jobs on the same worker node
by Massimo Sgaravatto 14 Apr '25

14 Apr '25

Dear all With the pam_slurm_adopt module as far as I understand you can ssh to a worker node if there is at least a job running on the node by that user. If there are multiple jobs, if I am not wrong you will be "mapped" to the last job started on the node. And, if you are using cgroups, you will be "confined" to the resources assigned to this "last" job. Is it possible in some way to specify the job to be mapped, in case there are multiple jobs for that user on the same node ? Thanks, Massimo

2 2

sllurmrestd via unix socket
by Brian Andrus 10 Apr '25

10 Apr '25

All, Maybe someone has seen this. I have slurmrestd running listening on port 8081 as well as a unix socket at /run/slurmrestd/slurmrestd.sock I am able to query the port with curl and do a ping. Everything seems fine. Other commands work as well. "pings": [ { "hostname": "head01", "pinged": "UP", "responding": true, "latency": 1605, "mode": "primary", "primary": true }, { "hostname": "head02", "pinged": "UP", "responding": true, "latency": 2353, "mode": "backup", "primary": false } When I try doing so via socket, however, my two head nodes show 'DOWN' : "pings": [ { "hostname": "head01", "pinged": "DOWN", "latency": 11784, "mode": "primary" }, { "hostname": "head02", "pinged": "DOWN", "latency": 12668, "mode": "backup" } Other commands fail with: "error_number": 1007, "error": "Protocol authentication error", I'll admit, I don't usually use sockets, so I could easily be overlooking something there. Permissions on the socket look right. I am getting json back, so it is connecting. Note: slurmrestd is running under it's own user (not root and not slurmuser). Any ideas? Thanks in advance, Brian Andrus

1 0

Minimum cpu cores per node partition level configuration
by Jeherul Islam 10 Apr '25

10 Apr '25

Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam

4 4

weird sacct behavior?
by Pierre Abele 10 Apr '25

10 Apr '25

Hi everyone, I am currently stuck with an sacct issue and would appreciate any help/hints/ideas: My users cannot retrieve job data from their currently running jobs through sacct anymore. Running sacct -a as root also reproduces this issue: It does not show running jobs, but both sacct -j <JobID> and squeue -j <JobID> do. AFAICT, this is not intended behavior (?). Also including longer time windows witch sacct -S ... -E did not help. root@slurmmaster:~# sacct -a | grep 154415 # this returns nothing root@slurmmaster:~# sacct -j 154415 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 154415 allocation primevo 0 PENDING 0:0 154415.batch batch primevo 2 RUNNING 0:0 154415.exte+ extern primevo 2 RUNNING 0:0 root@slurmmaster:~# squeue -j 154415 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 154415 standard genedrop username R 1:31 1 hpc020 Also, possibly related, we had a slurmdbd crash before this changed. We run Ubuntu Server 24.04 LTS with Slurm 24.05.4, using a MariaDB accounting database hosted on the same machine as the Slurm controller. Does anyone here have any ideas? Best, Pierre -- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele(a)eva.mpg.de Phone: +49 (0) 341 3550 245

1 0

Re: slurmctld HA ; backup controller doesn't schedule and start any job
by Hiromasa Watanabe 10 Apr '25

10 Apr '25

Hi all, Fortunately I solved this problem by changing Slurm version from 22.05.9 to 23.11.10. With Slurm 23.11.10, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job becomes scheduled and runs. I don't know why, but this is OK. Thanks, Hiro

1 0

errors while trying to setup slurmdbd.
by Steven Jones 10 Apr '25

10 Apr '25

root@vuwunicohpcdbp1 ~]# systemctl status slurmdbd × slurmdbd.service - Slurm DBD accounting daemon Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-04-10 10:28:52 NZST; 2min 33s ago Duration: 85ms Process: 2413 ExecStart=/usr/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 2413 (code=exited, status=1/FAILURE) CPU: 11ms Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: Started Slurm DBD accounting daemon. Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version > Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin loaded Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: fatal: This host not configured to run SlurmDBD ((vuwunicohpcdbp1 or vuwunicohp> Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Main process exited, code=exited, status=1/FAILURE Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Failed with result 'exit-code'. lines 1-14/14 (END) This is a current RHEL9.5 [root@vuwunicohpcdbp1 ~]# rpm -qi mariadb Name : mariadb Epoch : 3 Version : 10.5.27 Release : 1.el9_5 Architecture: x86_64 Install Date: Fri 04 Apr 2025 11:13:14 AM NZDT Group : Unspecified Size : 18784786 License : GPLv2 and LGPLv2 Signature : RSA/SHA256, Fri 06 Dec 2024 10:47:26 PM NZDT, Key ID 199e2f91fd431d51 Source RPM : mariadb-10.5.27-1.el9_5.src.rpm Build Date : Thu 05 Dec 2024 03:22:03 PM NZDT Build Host : x86-64-04.build.eng.rdu2.redhat.com Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : http://mariadb.org Summary : A very fast and robust SQL database server Description : MariaDB is a community developed fork from MySQL - a multi-user, multi-threaded SQL database server. It is a client/server implementation consisting of a server daemon (mariadbd) and many different client programs and libraries. The base package contains the standard MariaDB/MySQL client programs and utilities. [root@vuwunicohpcdbp1 ~]# ========= # Example slurmdbd.conf file. # # See the slurmdbd.conf man page for more information. # # Archive info #ArchiveJobs=yes #ArchiveDir="/tmp" #ArchiveSteps=yes #ArchiveScript= #JobPurge=12 #StepPurge=1 # # Authentication info AuthType=auth/munge #AuthInfo=/var/run/munge/munge.socket.2 # # slurmDBD info DbdAddr=127.0.0.1 #DbdAddr=localhost DbdHost=vuwunicoslurmrp1.ods.vuw.ac.nz #DbdHost=localhost DbdPort=6819 SlurmUser=slurm #MessageTimeout=300 DebugLevel=verbose #DefaultQOS=normal,standby LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd/slurmdbd.pid #PluginDir=/usr/lib/slurm #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql #StorageHost=localhost StorageHost=127.0.0.1 #StoragePort=1234 StoragePort=3306 #StoragePort=6819 #StoragePass=xxxxx StoragePass=xxxxxx #StoragePass=aPIkaEqxLrA StorageUser=slurm StorageLoc=slurm_acct_db #ssj # ##ssj fini ============ This ran in test just fine but the slurm version is slightly newer in this case, [root@vuwunicohpcdbp1 admjonesst1]# rpm -qa |grep slurm slurm-24.11.3-1.el9.x86_64 slurm-slurmdbd-24.11.3-1.el9.x86_64 [root@vuwunicohpcdbp1 admjonesst1]# rpm -qi slurm-slurmdbd-24.11.3-1.el9.x86_64 Name : slurm-slurmdbd Version : 24.11.3 Release : 1.el9 Architecture: x86_64 Install Date: Mon 07 Apr 2025 01:58:32 PM NZST Group : System Environment/Base Size : 2959288 License : GPLv2+ Signature : (none) Source RPM : slurm-24.11.3-1.el9.src.rpm Build Date : Mon 07 Apr 2025 12:54:21 PM NZST Build Host : vuwunicoslurmp1.ods.vuw.ac.nz URL : https://slurm.schedmd.com/ Summary : Slurm database daemon Description : Slurm database daemon. Used to accept and process database RPCs and upload database changes to slurmctld daemons on each cluster [root@vuwunicohpcdbp1 admjonesst1]# regards Steven

2 2

pam error, related to accounting?
by David Bremner 09 Apr '25

09 Apr '25

Recently I enabled accounting on my tiny (1 compute node, one head node) slurm cluster. slurmdbd.conf looks like AuthType=auth/munge DbdHost=vertex DbdPort=6819 SlurmUser=slurm StorageHost=localhost StorageType=accounting_storage/mysql StorageUser=slurm StoragePass=#elided# StoragePort=3306 StorageLoc=slurm_acct_db PurgeEventAfter=30days PurgeJobAfter=30days PurgeResvAfter=30days PurgeStepAfter=30days PurgeSuspendAfter=30days Some potentially relevant parts of slurmd.conf are: UsePAM=1 ProctrackType=proctrack/cgroup SelectTypeParameters=CR_Core_Memory AccountingStorageType=accounting_storage/slurmdbd ClusterName=simplex JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup Since I enabled accounting the output from every job (whether started by srun or sbatch) finishes with slurmstepd-simplex: error: pam_close_session: Cannot make/remove an entry for the specified session /etc/pamd.d/slurm has the following content auth required pam_localuser.so auth required pam_shells.so account required pam_unix.so account required pam_access.so session required pam_unix.so session optional pam_systemd.so Accounting seems to be working OK. Does anyone know what PAM related code paths could be triggered by enabling accounting? d

1 0

recommended freeIPMI version
by Heckes, Frank 09 Apr '25

09 Apr '25

Hi all, I’d like to update to SLURM version 24.11.4. I was searching for a recommendation for freeIPMI (power measurement), but couldn’t find one on the schedMD web-pages (sorry, if I overlooked it). The latest available freeIPMI is 1.6.15 (https://ftp.gnu.org/gnu/freeipmi/freeipmi-1.6.15.tar.gz). Does anyone knows whether this is the best choice? Many thanks in advance for any advice. Cheers, -Frank

3 4

slurmctld HA ; backup controller doesn't schedule and start any job
by hiromasa.watanabe＠gmail.com 09 Apr '25

09 Apr '25

slurmctld HA ; backup controller doesn't schedule and start any job Hi all, I am trying a slurmctld HA configuration on two servers, using slurm version 22.05.9 of AlmaLinux 9.4. The problem is, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job will be pending with Reason=None, and it will not be scheduled and will not start running. $ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 43 cpu_multi job hpc PD 0:00 1 (None) Why won't my job start running? What should I change to get the job to start running? The slurmctld.log and configuration files are shown below. Backup's slurmctld.log： [2025-04-09T15:31:17.000] debug3: Heartbeat at 1744180276 [2025-04-09T15:31:18.000] debug3: Heartbeat at 1744180277 [2025-04-09T15:31:19.022] debug3: Heartbeat at 1744180279 [2025-04-09T15:31:19.605] debug2: Processing RPC: REQUEST_SUBMIT_BATCH_JOB from UID=1000 [2025-04-09T15:31:19.605] debug3: _set_hostname: Using auth hostname for alloc_node: compute1 [2025-04-09T15:31:19.605] debug3: JobDesc: user_id=1000 JobId=N/A partition=cpu_multi name=job [2025-04-09T15:31:19.605] debug3: cpus=2-4294967294 pn_min_cpus=1 core_spec=-1 [2025-04-09T15:31:19.605] debug3: Nodes=1-[1] Sock/Node=65534 Core/Sock=65534 Thread/Core=65534 [2025-04-09T15:31:19.605] debug3: pn_min_memory_job=18446744073709551615 pn_min_tmp_disk=-1 [2025-04-09T15:31:19.605] debug3: immediate=0 reservation=(null) [2025-04-09T15:31:19.605] debug3: features=(null) batch_features=(null) cluster_features=(null) prefer=(null) [2025-04-09T15:31:19.605] debug3: req_nodes=(null) exc_nodes=(null) [2025-04-09T15:31:19.605] debug3: time_limit=-1--1 priority=-1 contiguous=0 shared=-1 [2025-04-09T15:31:19.605] debug3: kill_on_node_fail=-1 script=#! /bin/bash #SBATCH -p cpu_multi #SBATC... [2025-04-09T15:31:19.605] debug3: argv="./twocore.sh" [2025-04-09T15:31:19.605] debug3: environment=SHELL=/bin/bash,PYENV_SHELL=bash,HISTCONTROL=ignoredups,... [2025-04-09T15:31:19.605] debug3: stdin=/dev/null stdout=/misc/home/hpc/slurmtest/twocore_%J.out stderr=(null) [2025-04-09T15:31:19.605] debug3: work_dir=/misc/home/hpc/slurmtest alloc_node:sid=compute1:281600 [2025-04-09T15:31:19.605] debug3: power_flags= [2025-04-09T15:31:19.605] debug3: resp_host=(null) alloc_resp_port=0 other_port=0 [2025-04-09T15:31:19.605] debug3: dependency=(null) account=(null) qos=(null) comment=(null) [2025-04-09T15:31:19.605] debug3: mail_type=0 mail_user=(null) nice=0 num_tasks=2 open_mode=0 overcommit=-1 acctg_freq=(null) [2025-04-09T15:31:19.605] debug3: network=(null) begin=Unknown cpus_per_task=1 requeue=-1 licenses=(null) [2025-04-09T15:31:19.605] debug3: end_time= signal=0@0 wait_all_nodes=-1 cpu_freq= [2025-04-09T15:31:19.605] debug3: ntasks_per_node=-1 ntasks_per_socket=-1 ntasks_per_core=-1 ntasks_per_tres=-1 [2025-04-09T15:31:19.605] debug3: mem_bind=0:(null) plane_size:65534 [2025-04-09T15:31:19.605] debug3: array_inx=(null) [2025-04-09T15:31:19.605] debug3: burst_buffer=(null) [2025-04-09T15:31:19.605] debug3: mcs_label=(null) [2025-04-09T15:31:19.605] debug3: deadline=Unknown [2025-04-09T15:31:19.605] debug3: bitflags=0x1a00c000 delay_boot=4294967294 [2025-04-09T15:31:19.605] debug3: assoc_mgr_fill_in_user: found correct user: hpc(1000) [2025-04-09T15:31:19.605] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=hpc(1000), acct=hpc, cluster=cluster, partition=cpu_multi [2025-04-09T15:31:19.605] debug3: assoc_mgr_fill_in_assoc: found correct association of user=hpc(1000), acct=hpc, cluster=cluster, partition=cpu_multi to assoc=16 acct=hpc [2025-04-09T15:31:19.605] debug3: found correct qos [2025-04-09T15:31:19.607] debug2: priority/multifactor: priority_p_set: initial priority for job 44 is 33 [2025-04-09T15:31:19.607] debug2: found 1 usable nodes from config containing compute1 [2025-04-09T15:31:19.607] debug2: found 1 usable nodes from config containing compute2 [2025-04-09T15:31:19.607] debug3: _pick_best_nodes: JobId=44 idle_nodes 2 share_nodes 2 [2025-04-09T15:31:19.607] debug2: select/cons_tres: select_p_job_test: evaluating JobId=44 [2025-04-09T15:31:19.607] debug2: sched: JobId=44 allocated resources: NodeList=(null) [2025-04-09T15:31:19.607] _slurm_rpc_submit_batch_job: JobId=44 InitPrio=33 usec=2490 [2025-04-09T15:31:19.608] debug3: create_mmap_buf: loaded file `/var/spool/slurm/ctld/job_state` as buf_t [2025-04-09T15:31:19.609] debug3: Writing job id 45 to header record of job_state file [2025-04-09T15:31:21.000] debug3: Heartbeat at 1744180280 [2025-04-09T15:31:21.257] debug2: _slurm_connect: failed to connect to 192.168.56.11:6817: Connection refused [2025-04-09T15:31:21.257] debug2: Error connecting slurm stream socket at 192.168.56.11:6817: Connection refused [2025-04-09T15:31:22.000] debug3: Heartbeat at 1744180282 [2025-04-09T15:31:24.000] debug3: Heartbeat at 1744180283 [2025-04-09T15:31:25.004] debug3: Heartbeat at 1744180285 [2025-04-09T15:31:27.001] debug3: Heartbeat at 1744180287 [2025-04-09T15:31:29.000] debug3: Heartbeat at 1744180288 [2025-04-09T15:31:30.000] debug3: Heartbeat at 1744180289 [2025-04-09T15:31:31.006] debug3: Heartbeat at 1744180291 [2025-04-09T15:31:32.822] debug2: _slurm_connect: failed to connect to 192.168.56.11:6817: Connection refused [2025-04-09T15:31:32.822] debug2: Error connecting slurm stream socket at 192.168.56.11:6817: Connection refused [2025-04-09T15:31:33.007] debug3: Heartbeat at 1744180293 [2025-04-09T15:31:35.000] debug3: Heartbeat at 1744180294 [2025-04-09T15:31:36.002] debug3: Heartbeat at 1744180296 [2025-04-09T15:31:36.395] debug2: select/cons_tres: select_p_job_test: evaluating JobId=43 [2025-04-09T15:31:36.395] debug2: select/cons_tres: select_p_job_test: evaluating JobId=44 [2025-04-09T15:31:38.000] debug3: Heartbeat at 1744180297 [2025-04-09T15:31:38.497] debug2: Performing purge of old job records [2025-04-09T15:31:39.000] debug3: Heartbeat at 1744180298 [2025-04-09T15:31:40.000] debug3: Heartbeat at 1744180300 [2025-04-09T15:31:40.655] debug2: Testing job time limits and checkpoints [2025-04-09T15:31:42.000] debug3: Heartbeat at 1744180301 [2025-04-09T15:31:43.000] debug3: Heartbeat at 1744180302 slurm.conf: ClusterName=cluster SlurmctldHost=gateway1 #Primary(192.168.56.11) SlurmctldHost=gateway2 #Backup(192.168.56.12) MpiDefault=pmix ProctrackType=proctrack/cgroup PrologFlags=Contain ReturnToService=0 SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurm/d SlurmUser=slurm StateSaveLocation=/var/spool/slurm/ctld SwitchType=switch/none TaskEpilog=/etc/slurm/taskepilog.sh TaskPlugin=task/cgroup,task/affinity TaskProlog=/etc/slurm/taskprolog.sh InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=10 SlurmdTimeout=300 Waittime=0 DefMemPerCPU=32 SchedulerType=sched/builtin SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory PriorityType=priority/multifactor PriorityWeightPartition=1000 AccountingStorageHost=gateway1 AccountingStorageBackupHost=gateway2 AccountingStorageType=accounting_storage/slurmdbd AccountingStoreFlags=job_comment JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=debug5 SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log NodeName=compute1 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=3900 Weight=1 NodeName=compute2 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=3900 Weight=1 PartitionName=gpu_single Nodes=ALL PriorityJobFactor=30 MaxTime=INFINITE State=UP Default=YES PartitionName=cpu_single Nodes=ALL PriorityJobFactor=10 MaxTime=INFINITE State=UP PartitionName=cpu_multi Nodes=ALL MaxTime=INFINITE State=UP slurmdbd.conf: AuthType=auth/munge DebugLevel=4 DbdHost=gateway1 DbdBackupHost=gateway2 LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurm/slurmdbd.pid PurgeEventAfter=1month PurgeJobAfter=1month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=1month PurgeUsageAfter=1month SlurmUser=slurm StorageType=accounting_storage/mysql StorageHost=gateway1 StoragePass=mypassword StorageUser=slurm Best regards, Hiro

1 0

Slurm version 24.11.4 is now available
by Marshall Garey 08 Apr '25

08 Apr '25

We are pleased to announce the availability of Slurm version 24.11.4. This release fixes a variety of major to minor severity bugs. Some edge cases that caused jobs to pend forever are fixed. Notable stability issues that are fixed include: * slurmctld crashing upon receiving a certain heterogeneous job submission. * slurmd crashing after a communications failure with a slurmstepd. * A variety of race conditions related to receiving and processing connections, including one that resulted in the slurmd ignoring new RPC connections. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > -- slurmctld,slurmrestd - Avoid possible race condition that could have caused > process to crash when listener socket was closed while accepting a new > connection. > -- slurmrestd - Avoid race condition that could have resulted in address > logged for a UNIX socket to be incorrect. > -- slurmrestd - Fix parameters in OpenAPI specification for the following > endpoints to have "job_id" field: > GET /slurm/v0.0.40/jobs/state/ > GET /slurm/v0.0.41/jobs/state/ > GET /slurm/v0.0.42/jobs/state/ > GET /slurm/v0.0.43/jobs/state/ > -- slurmd - Fix tracking of thread counts that could cause incoming > connections to be ignored after burst of simultaneous incoming connections > that trigger delayed response logic. > -- Stepmgr - Avoid unnecessary SRUN_TIMEOUT forwarding to stepmgr. > -- Fix jobs being scheduled on higher weighted powered down nodes. > -- Fix how backfill scheduler filters nodes from the available nodes based on > exclusive user and mcs_label requirements. > -- acct_gather_energy/{gpu,ipmi} - Fix potential energy consumption adjustment > calculation underflow. > -- acct_gather_energy/ipmi - Fix regression introduced in 24.05.5 (which > introduced the new way of preserving energy measurements through slurmd > restarts) when EnergyIPMICalcAdjustment=yes. > -- Prevent slurmctld deadlock in the assoc mgr. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Fix preemptor jobs not entering execution due to wrong calculation of > accounting policy limits. > -- Fix certain job requests that were incorrectly denied with node > configuration unavailable error. > -- slurmd - Avoid crash due when slurmd has a communications failure with > slurmstepd. > -- Fix memory leak when parsing yaml input. > -- Prevent slurmctld from showing error message about PreemptMode=GANG being a > cluster-wide option for `scontrol update part` calls that don't attempt to > modify partition PreemptMode. > -- Fix setting GANG preemption on partition when updating PreemptMode with > scontrol. > -- Fix CoreSpec and MemSpec limits not being removed from previously > configured slurmd. > -- Avoid race condition that could lead to a deadlock when slurmd, slurmstepd, > slurmctld, slurmrestd or sackd have a fatal event. > -- Fix jobs using --ntasks-per-node and --mem keep pending forever when the > requested mem divided by the number of cpus will surpass the configured > MaxMemPerCPU. > -- slurmd - Fix address logged upon new incoming RPC connection from "INVALID" > to IP address. > -- Fix memory leak when retrieving reservations. This affects scontrol, sinfo, > sview, and the following slurmrestd endpoints: > 'GET /slurm/{any_data_parser}/reservation/{reservation_name}' > 'GET /slurm/{any_data_parser}/reservations' > -- Log warning instead of debuflags=conmgr gated log when deferring new > incoming connections when number of active connections exceed > conmgr_max_connections. > -- Avoid race condition that could result in worker thread pool not activating > all threads at once after a reconfigure resulting in lower utilization of > available CPU threads until enough internal activity wakes up all threads > in the worker pool. > -- Avoid theoretical race condition that could result in new incoming RPC > socket connections being ignored after reconfigure. > -- slurmd - Avoid race condition that could result in a state where new > incoming RPC connections will always be ignored. > -- Add ReconfigFlags=KeepNodeStateFuture to restore saved FUTURE node state on > restart and reconfig instead of reverting to FUTURE state. This will be > made the default in 25.05. > -- Fix case where hetjob submit would cause slurmctld to crash. > -- Fix jobs using --cpus-per-gpu and --mem keep pending forever when the > requested mem divided by the number of cpus will surpass the configured > MaxMemPerCPU. > -- Enforce that jobs using --mem and several --*-per-* options do not violate > the MaxMemPerCPU in place. > -- slurmctld - Fix use-cases of jobs incorrectly pending held when --prefer > features are not initially satisfied. > -- slurmctld - Fix jobs incorrectly held when --prefer not satisfied in some > use-cases. > -- Ensure RestrictedCoresPerGPU and CoreSpecCount don't overlap.

1 0

slurm send email status with no details
by Oren 07 Apr '25

07 Apr '25

Hi, We set up a slurm system with email notification, this is the slurm.conf `MailProg=/usr/sbin/sendmail` But the email that I get has not status, just an empty message: [image: image.png] no subject, no info, what are we missing? Thanks~

2 1

Run a command in Slurm with all streams and signals connected to the submitting command
by Michael Milton 07 Apr '25

07 Apr '25

I'm helping with a workflow manager that needs to submit Slurm jobs. For logging and management reasons, the job (e.g. srun python) needs to be run as though it were a regular subprocess (python): - stdin, stdout and stderr for the command should be connected to process inside the job - signals sent to the command should be sent to the job process - We don't want to use the existing job allocation, if this is run from a Slurm job - The command should only terminate when the job is finished, to avoid us needing to poll Slurm We've tried: - sbatch --wait, but then SIGTERM'ing the process doesn't kill the job - salloc, but that requires a TTY process to control it (?) - salloc srun seems to mess with the terminal when it's killed, likely because of being "designed to be executed in the foreground" - Plain srun re-uses the existing Slurm allocation, and specifying resources like --mem will just request then from the current job rather than submitting a new one What is the best solution here?

2 2

Re: Run a command in Slurm with all streams and signals connected to the submitting command
by Michael Milton 05 Apr '25

05 Apr '25

Thanks Davide, It's true that srun will create an allocation if you aren't inside a job, but if you are inside a job and you request more resources than it has, then srun will just fail. This is the key issue that I want to avoid. On Sat, Apr 5, 2025 at 11:48 AM Davide DelVento <davide.quantum(a)gmail.com> wrote: > The plain srun is probably the best bet, and if you really need the thing > to be started from another slurm job (rather than the login node) you will > need to exploit the fact that > > > If necessary, srun will first create a resource allocation in which to > run the parallel job. > > AFAIK, there is no option to for the "create a resource allocation" even > if it's not necessary. But you may try to request something that is "above > and beyond" what the current allocation provides, and that might solve your > problem. > Looking at the srun man page, I could speculate that --clusters > or --cluster-constraint might help in that regard (but I am not sure). > > Have a nice weekend > > > On Fri, Apr 4, 2025 at 6:27 AM Michael Milton via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > >> I'm helping with a workflow manager that needs to submit Slurm jobs. For >> logging and management reasons, the job (e.g. srun python) needs to be run >> as though it were a regular subprocess (python): >> >> - stdin, stdout and stderr for the command should be connected to >> process inside the job >> - signals sent to the command should be sent to the job process >> - We don't want to use the existing job allocation, if this is run >> from a Slurm job >> - The command should only terminate when the job is finished, to >> avoid us needing to poll Slurm >> >> We've tried: >> >> - sbatch --wait, but then SIGTERM'ing the process doesn't kill the job >> - salloc, but that requires a TTY process to control it (?) >> - salloc srun seems to mess with the terminal when it's killed, >> likely because of being "designed to be executed in the foreground" >> - Plain srun re-uses the existing Slurm allocation, and specifying >> resources like --mem will just request then from the current job rather >> than submitting a new one >> >> What is the best solution here? >> >> -- >> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >> >

1 0

Best Way to See GPUs in Use?
by Jason Simms 02 Apr '25

02 Apr '25

Hello all, Apologies for the basic question, but is there a straightforward, best-accepted method for using Slurm to report on which GPUs are currently in use? I've done some searching and people recommend all sorts of methods, including parsing the output of nvidia-smi (seems inefficient, especially across multiple GPU nodes), as well as using other tools such as Grafana, XDMoD, etc. We do track GPUs as a resource, so I'd expect I could get at the info with sreport or something like that, but before trying to craft my own from scratch, I'm hoping someone has something working already. Ultimately I'd like to see either which cards are available by node, or the reverse (which are in use by node). I know recent versions of Slurm supposedly added tighter integration in some way with NVIDIA cards, but I can't seem to find definitive docs on what, exactly, changed or what is now possible as a result. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Research Computing Manager Swarthmore College Information Technology Services (610) 328-8102

3 2

slum job sumisison using different UID/GID
by navin srivastava 02 Apr '25

02 Apr '25

Hi, Can munge depend upon the UID/GID for the job submission? There is a scenario where the cluster is running with the NIS authentication mechanism. There is a login Node created which is part of LDAP and we installed the slurm to interact with slurm master. I am able to run jobs using root but for users it is not working as the UID/GID differ. Is there a way we can overcome this issue? and be able to run jobs. Rehards Navin

1 0

slurm releases
by Patrick Begou 01 Apr '25

01 Apr '25

Hi slurm team, I would ask some clarifications with slurm releases. Why two versions of slurm are available ? I speak of 24.05.7 versus 24.11.3 on https://www.schedmd.com/slurm-support/release-announcements and announces made on this list ? I'm managing small clusters in a french public research lab and I've successfully updated (lets says re-installed as it was very old) an old one with slurm-24.05.3 some month ago. Now I'm trying to deploy a second cluster with the same approach and I've selected 24.11.3-1 (but why not using 24.05.7 ?). Moreover I have some difficulties for deploying 24.11.3 : I've built the rpm with: rpmbuild \ --define '_prefix /usr' \ --with pmix --with ucx --with mysql --with hwloc \ -ta slurm-24.11.3.tar.bz2 but some packages wont install: $ dnf list --showduplicates slurm-slurmd Last metadata expiration check: 0:33:15 ago on Tue Apr 1 18:03:07 2025. Available Packages slurm-slurmd.x86_64 24.11.3-1.el9 bb-local $ sudo dnf install slurm-slurmd Last metadata expiration check: 0:33:03 ago on Tue Apr 1 18:03:25 2025. Error: Problem: conflicting requests - nothing provides is needed by slurm-slurmd-24.11.3-1.el9.x86_64 from bb-local - nothing provides installed needed by slurm-slurmd-24.11.3-1.el9.x86_64 from bb-local - nothing provides not needed by slurm-slurmd-24.11.3-1.el9.x86_64 from bb-local (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) I think I will try to build 24.05.7 or 24.05.3 as a next try but I'm interested in any advices. Thank you Patrick

2 1

cpus and gpus partitions and how to optimize the resource usage
by Massimo Sgaravatto 31 Mar '25

31 Mar '25

Dear all We have just installed a small SLURM cluster composed of 12 nodes: - 6 CPU only nodes: 2 Sockets=2, 96 CoresPerSocket 2, ThreadsPerCore=2, 1.5 TB of RAM - 6 nodes with also GPUS: same conf of the CPU-only node + 4 H100 per node We started with a setup with 2 partitions: - a 'onlycpus' partition which sees all the cpu-only nodes - a 'gpus' partition which sees the nodes with gpus and asked users to use the 'gpus' partition only for jobs that need gpus (for the time being we are not technically enforced that). The problem is that a job requiring a GPU usually needs only a few cores and only a few GB of RAM, which means wasting a lot of CPU cores. And having all nodes in the same partition would mean that there is the risk that a job requiring a GPU can't start if all CPU cores and/or all memory is used by CPU only jobs I went through the mailing list archive and I think that "splitting" a GPU node into two logical nodes (one to be used in the 'gpus' partition and one to be used in the 'onlycpus' partition) as discussed in [*] would help. Since that proposed solution is considered by his author a "bit of a kludge" and since I read that splitting a node into multiple logical nodes is in a general a bad idea, I'd like to understand if you could suggest other/best options. I also found this [**] thread, but I don't like too much that approach (i.e. relying on MaxCPUsPerNode) because it would mean having 3 partition (if I have got it right): two partitions for cpu only jobs and 1 partition for gpu jobs Many thanks, Massimo [*] https://groups.google.com/g/slurm-users/c/IUd7jLKME3M [**] https://groups.google.com/g/slurm-users/c/o7AiYAQ1YJ0

4 5

Preemption question
by Kamil Wilczek 31 Mar '25

31 Mar '25

Dear All, I would like to be able to preempt (SUSPEND) a single QoS of a user that blocks the queue for several days. Currently I have about 100 users on the cluster and it seems that setting the "Preempt" option to each QoS (we have personal QoSes) is not optimal. https://slurm.schedmd.com/sacctmgr.html#OPT_Preempt Is there a way to to set an option to this single problematic QoS, saying that the QoS can be preempted by any other QoS? It would be much more administrator-friendly solution ;) Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

3 4

Slurm 24.05 and OpenMPI
by Matthias Leopold 28 Mar '25

28 Mar '25

Hi, I built a small Slurm 21.08 cluster with NVIDIA GPU hardware and NVIDIA deepops framework a couple of years ago. It is based on Ubuntu 20.04 and makes use of the NVIDIA pyxis/enroot container solution. For operational validation I used the nccl-tests application in a container. nccl-tests is compiled with MPI support (OpenMPI 4.1.6 or 4.1.7) and I used it also for validation of MPI jobs. Slurm jobs use "pmix" and tasks are launched via srun (not mpirun). Some of the GPUs can talk to each other via Infiniband, but MPI is rarely used at our site and I'm fully aware that my MPI knowledge is very limited. Still it worked with Slurm 21.08. Now I built a Slurm 24.05 cluster based on Ubuntu 24.04 and started to move hardware there. When I run my nccl-tests container (also with newer software) I see error messages like this: [node1:21437] OPAL ERROR: Unreachable in file ext3x_client.c at line 111 -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using: version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix. Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location. Please configure as appropriate and try again. -------------------------------------------------------------------------- *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [node1:21437] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! One simple question: Is this related to https://github.com/open-mpi/ompi/issues/12471? If so: is there some workaround? I'm very grateful for any comments. I know that a lot of detail information is missing, but maybe someone can still already give me a hint where to look. Thanks a lot Matthias

3 10

Doubt with SelectTypeParameters in slurm.conf
by Gestió Servidors 28 Mar '25

28 Mar '25

Hello, I'm running some tests in a very small testing environment (before applying in the real scenario). My environment is only a computer with a old Intel i4 with this "lscpu" configuration: Architecture: x86_64 CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz "slurmd -C" returns: "CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2". My slurm.conf is configured with: SelectType=select/cons_tres SelectTypeParameters=CR_Core AccountingStorageTRES=cpu,mem,node,billing,fs/disk,vmem,pages After running a simple "helloworld" test, I have noticed that if SelectTypeParameters=CR_Core, system always reserves me an even number of CPUs (during "pending" time, I can see the real number I have requested, but when job starts "running", that number is increased to the next even number). Howerver, if I reconfigure slurm with SelectTypeParameters=CR_CPU, system reserves me the correct number of CPUs (during "pending" time and, also, during "running" time). I suppose that my system must be configured with CR_CPU, according https://slurm.schedmd.com/cons_tres.html, but could someone confirm that? Thanks.

2 1

bit_cache_init failure on the second time backup controller tries to take control
by Safdar Iqbal 28 Mar '25

28 Mar '25

Hi, We're running into an issue where slurmctld core-dumps with the following error. This happens on the backup controller, if it needs to take over from the primary, _for a second time_. slurmctld: fatal: bit_cache_init: cannot change size once set Has anyone seen this error before? Also if there are any existing discussions and/or tickets related to this, please let me know. Our slurm version is 24.11.1. ________________ Steps to reproduce: 1. On a healthy cluster, we make the primary controller unavailable. Since we're running our cluster on cloud VMs, we cause this by stopping the primary controller VM. 2. From the logs we can see the backup controller take over, and log the message "Running as primary controller" 3. We then start the primary again, making sure the IP addresses and hostnames stay consistent. Once slurmctld on the primary has started and taken back control, we can see the log "Running as primary controller" on that VM. 4. We then stop the primary controller VM again, causing the backup to try taking the control a second time. This time however the slurmctld on the backup coredumps, with following log entries from journalctl -u slurmctld: slurmctld: fatal: bit_cache_init: cannot change size once set slurmctld.service: Main process exited, code=dumped, status=6/ABRT slurmctld.service: Failed with result 'core-dump'. Thanks! - Safdar

1 0

How limit CPUs per node in a partition
by Gestió Servidors 27 Mar '25

27 Mar '25

Hello, I have a testing partition with only a node. That server has 12 CPUs (it's a very old server) (2 sockets, 6 cores per socket, 1 thread per core). That partition, called "test.q" only has that node, so by default, partition test.q has 12 CPUs (all from testing node). However, now I would like to decrease that CPU number from 12 to 4, for example. "slurmd -C" returns "CPUs=12 Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1" as I have said to you before and if in slurm.conf I configure partition in this way "PartitionName=test.q Nodes=test-node MaxTime=10:00 State=UP AllocNodes=clus-login MaxCPUsPerNode=4", partition continues accepting jobs with 12 CPUs. Is it any wrong in my configuration? How could I get my purpose? Thanks. -- [cid:image001.jpg@01DB9F0E.FDD9A7A0]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image002.jpg@01DB9F0E.FDD9A7A0]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

1 0

Re: Using more cores/CPUs that requested with
by Shunran Zhang 26 Mar '25

26 Mar '25

Ugh I think I did not catch up with the docs. I started with a system that defaults to cgroup v1 but the Slurm doc for that plugin is NOT available at that time. Thus I converted everything to cgroup v2. It appears that they are both supported and that documentation issue is more on the dev side than admin side. Thanks for pointing that out. I misinterpreted the "coming soon" part of cgroup v1 plugin and the "legacy" naming for "do not use". It should be fine. 2025年3月27日(木) 0:48 Williams, Jenny Avis <jenny_williams(a)unc.edu>: > “ … As cgroup is likely not supposed to be used in newer deployments of > Slurm.” > > > > I am curious about this statement. Would someone expand on this, to either > support or counter it? > > > > Jenny Williams > > UNC Chapel Hill > > > > > > *From:* Shunran Zhang via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Wednesday, March 26, 2025 10:52 AM > *To:* Gestió Servidors <sysadmin.caos(a)uab.cat> > *Cc:* Slurm User Community List <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Re: Using more cores/CPUs that requested with > > > > If you are letting systemd taking most things over, you got systemd-cgtop > that work better than top for your case. There is also systemd-cgls for > non-interactive listing. > > > > Also mind to check if you are using cgroup2? A mount to check your cgroup > would suffice. As cgroup is likely not supposed to be used in newer > deployments of Slurm. > > > > > > 2025年3月26日(水) 17:14 Gestió Servidors via slurm-users < > slurm-users(a)lists.schedmd.com>: > > Hello, > > > > Thanks for your answers. I will try now!! One more question: is there any > way to check if Cgroups restrictions is working fine during a “running” job > or during SLURM scheduling process? > > > > Thanks again! > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > >

1 0

Re: Using more cores/CPUs that requested with
by Gestió Servidors 26 Mar '25

26 Mar '25

Hello, Thanks for your answers. I will try now!! One more question: is there any way to check if Cgroups restrictions is working fine during a "running" job or during SLURM scheduling process? Thanks again!

4 3

Using more cores/CPUs that requested with sbatch
by Gestió Servidors 25 Mar '25

25 Mar '25

Hello, I would like to know if there is any mechanism to avoid a user to do "cheating" when he submits a job. For example, a user submits a OpenMP job requesting 1 node (-N 1) and 2 tasks (-n 2). However, inside his script (or compiled in his binany), user uses more than 2 ntasks, for example, all free cores. For SLURM, job is only using 2, but really, node is 100% assigned to that job. SLURM can control that "abuse" in some way? Cgroups? Thanks.

3 2

Re: Using more cores/CPUs that requested with sbatch
by Megan 25 Mar '25

25 Mar '25

Slurm can restrict jobs to their allocated resources using cgroups. To use cgroups you will have to configure the TaskPlugin cgroup option and specify how the cgroups should behave in cgroup.conf. slurm.conf TaskPlugin=task/cgroup Slurm Workload Manager - slurm.conf <https://slurm.schedmd.com/slurm.conf.html#OPT_task/cgroup> cgroup.conf ConstrainCores=yes Slurm Workload Manager - cgroup.conf <https://slurm.schedmd.com/cgroup.conf.html#OPT_ConstrainCores> Regards, --Megan

1 0

slurmrestd equivalent to "srun -n 10 echo HELLO"
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Community, I'm starting to experiment with slurmrestd for a new app we're writing. I'm having trouble understanding one aspect of submitting jobs. When I run something like `srun -n 10 echo HELLO', I get HELLO returned to my console/stdout 10x. When I submit this command as a script to the /jobs/submit route, I get success/200, but *I cannot determine how to get the console output of HELLO 10x in any form*. It's not in my stdout log for that job even though I can verify that the job ran successfully. Any suggestions? -- Thanks, Daniel Healy

3 2

Priority/Top seems to not be working
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Users, I have a newer install (23.11.3) and the priority of most all jobs is 1 and other small numbers. In previous versions, I would see numbers like 2^32. I have the multifactor plugin configured and confirmed it's in-use when I show the config. When I run `scontrol top` for a given job, the priority number doesn't change from 1. I have lots of jobs running at the moment and many queued as well. I've been using `scontrol top` for years and have seen the priority successfully change. I'm repeating this now with the newer version of Slurm and now seeing it work the same. Are you seeing this too? -- Thanks, Daniel Healy

1 0

missing jobs from queue
by Felix 14 Mar '25

14 Mar '25

Hello at site RO-14-ITIM, we have a very strange problem. A ticket was raised with the following: "Hi, The new ticket (Pilots disappearing from local batch system at ITIM) has been created by "*Timo Wilken*". Group: NGIs › NGI_RO Owner: - State: in progress Information: Hi, Pilots seem to be lost by the local batch system at ITIM. >50% of pilots fail: https://apfmon.lancs.ac.uk/CERN_central_A:RO-14-ITIM-arcn-node?state=fault Example log: https://aipanda156.cern.ch/condor_logs_2/25-03-12_13/grid.15478691.1.log Error message: "ARC job failed: LRMS error: (-1) Job missing from SLURM" Cheers, Timo" The problem is I can not find any of the jobs that are failing on the site: for example: the last job: https://aipanda023.cern.ch/condor_logs_2/25-03-14_09/grid.19602200.0.log with stdlog: 000 (19602200.000.000) 2025-03-14 09:28:03 Job submitted from host: <137.138.31.125:38090?addrs=137.138.31.125-38090+[2001-1458-d00-19--75]-38090&alias=aipanda023.cern.ch> ... 027 (19602200.000.000) 2025-03-14 09:28:13 Job submitted to grid resource GridResource: arc arcn-node.itim-cj.ro:443 GridJobId: arc arcn-node.itim-cj.ro:443 PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n ... 001 (19602200.000.000) 2025-03-14 09:40:51 Job executing on host: arc arcn-node.itim-cj.ro:443 ... 012 (19602200.000.000) 2025-03-14 09:40:58 Job was held. ARC job failed: LRMS error: (-1) Job missing from SLURM Code 0 Subcode 0 ... 009 (19602200.000.000) 2025-03-14 11:03:03 Job was aborted. Python-initiated action. (by user atlpan) ... 009 (19602200.000.000) 2025-03-14 11:03:17 Job was aborted. Python-initiated action. (by user atlpan) This name: PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n is nowhere to find on my server: [root@arcn-node log]# updatedb [root@arcn-node log]# locate PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n [root@arcn-node log]# I have on my site this kind of error The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE)It became Critical*at*2025-03-14T09:16:44Z due to *METRIC*org.nordugrid.ARC-CE-sw-gcc *Summary:*Script exited with code 127. Could not match GCC version. See /etc/arc/nagios/20-dist.ini for debugging hints. *Message:*'' which is going to be ok in the messag, one minute later The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE) It became Ok *at* 2025-03-14T09:17:53Z due to *METRIC* org.nordugrid.ARC-CE-sw-gcc *Summary: *Found GCC version 11.5.0. *Message: *'' Can you please advice where to look for any clue in solving this mystery? Thank you Felix :-- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Slurm version 24.05.7 is now available
by Marshall Garey 13 Mar '25

13 Mar '25

We are pleased to announce the availability of Slurm version 24.05.7. This release fixes some stability issues in 24.05, including a crash in slurmctld after updating a reservation with an empty nodelist. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.7 > ========================== > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix jobs being scheduled on higher weighted powered down > nodes. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Prevent slurmctld deadlock in the assoc mgr.

1 0

SLURM_JOB_ACCOUNT var missing in prolog
by Jonás Arce 13 Mar '25

13 Mar '25

Hi everyone, The Slurm variable SLURM_JOB_ACCOUNT is blank in my prolog, currently i have a TaskProlog and a Prolog in my system, in TaskProlog this variable works just fine (I use TaskProlog to set env variables), but in my Prolog (which I use to create temporal directories), it's just blank, it's strange beacuse other variables such as SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT or SLURM_JOBID work just fine in both Prolog and TaskProlog, I looked into the Slurm doc and it seems that this var should work everywhere, if someone could shed some light into this or tell me another equivalent var i'd appreciate it a lot. I need it because I need to make this type of temporal directories with my Prolog: /scratch-global/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOBID.

2 1

Job Env Vars in Slurm Core
by Bhaskar Chakraborty 13 Mar '25

13 Mar '25

Hi everyone, I have tried my best to extract custom job environment variables inside a custom select plugin.However, I am not able to get the same. I am able to get the env in a custom client filter plugin during job submission (sbatch --export) but Ineed the same in slurmctld as well. Is there any way this can be achieved. I have looked over few of the internal DS pointers (job_record, job_details etc)but none seem to provide the required functionality. Any insight is appreciated. Regards,Bhaskar.

2 1

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 12 Mar '25

12 Mar '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

3 5

SLURM queue/partition/qos configuration
by vittorio.confuorto＠gmail.com 12 Mar '25

12 Mar '25

Hi everyone, I need some advice on properly configuring my HPC cluster, which consists of three compute nodes (each with 256 cores and 800GB of RAM). The end users belong to three distinct groups: internal, external, and collaborator. I would like the internal users to have full (and exclusive) access to the first two compute nodes. That is, by default, if they submit a job, it should be scheduled on Node1 or Node2, depending on availability (or placed in the queue if necessary). Internal users should also have access to 100 cores and 200GB of RAM on Node3, with this limit applying as a hard cap across all their jobs. The remaining 156 cores and 600GB of RAM on Node3 should be exclusively available to external and collaborator users. Specifically, I would like the following setup: - The resources on Node3 should be split as follows: - 100 cores and 400GB of RAM for external users. - 56 cores and 200GB of RAM for collaborator users. - External users should be able to use the resources dedicated to collaborator users but with lowest priority. This means an external user can submit a job requesting the resources reserved for collaborators, but the job will only start if no collaborator jobs are in the queue. Any job submitted by a collaborator—even after an external user's job—should take priority. The same rule should apply in reverse for collaborators trying to use external resources. - There should be a group/partition/queue that includes all the remaining 156 cores and 600GB of RAM on Node3. Both external and collaborator users should be able to submit jobs to this queue, but a job submitted here should only start if there are no pending jobs in the two dedicated partitions. What is the best way to implement such an architecture? Thanks, everyone, for your support!

1 0

Slurm Job Kill intimation
by Bhaskar Chakraborty 10 Mar '25

10 Mar '25

Hello All, We wish to do some cleanup when user kills a pending slurm job using scancel. Our setup has a client filter plugin which sets some internal variables while interacting to a remote process.This needs to be undone during job termination. Is there a way to do this (using submitter credentials) ? Also, if one is to use a custom select plugin is there a way to inform select that the job is terminated.Presently, we don't find that slurmctld informs select plugin when a pending job is killed. Thanks in advance for any pointers ! Regards,Bhaskar.

1 0

Slurm upgrade using Debian packages
by Matthias Leopold 10 Mar '25

10 Mar '25

Hi, I'm building Slurm Debian packages from SchedMD sources using this tutorial https://www.schedmd.com/slurm/installation-tutorial/. Now I tried upgrading (minor release upgrade within 24.05) using these packages. https://slurm.schedmd.com/upgrades.html tells me to upgrade (a) slurmdbd (b) slurmctld (c) slurmd separately in this order, stopping each service for the upgrade. How can I follow this when the Debian packages have a dependency between slurmdbd + slurmctld that upgrades both packages at the same time? thx Matthias

4 7

allowing spyder-kernels to an interactive session without pam_slurm_adopt and older version of Slurm from OpenHPC repo; parmiko?
by Robert Kudyba 06 Mar '25

06 Mar '25

So the Spyder IDE has this great feature to connect a local/laptop/workstation to a spyder-kernels session <https://docs.spyder-ide.org/current/panes/ipythonconsole.html#using-externa…>. If a user starts an interactive, e.g., srun session, they end up on a compute node which is on one of our older clusters without the pam_slurm_adopt module. Is there a way with a library like Parmiko or this slurmjob <https://github.com/daangeijs/deepops-slurmjob> package? I see for vscode an admin posted the following option <https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment…> : if [[ -v SSH_AUTH_SOCK ]]; then if [[ ${SSH_AUTH_SOCK} =~ vscode ]]; then [ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash fi fi Is there a way to work around the restriction of non admins to connect to the spyder-kernel session? Is there anything similar to batchspawner for Jupyterhub <https://github.com/jupyterhub/batchspawner>? Unless anyone knows of an archive for Slurm slurm-ohpc-18.08 that includes pam_adopt?

1 0

Slurm versions 24.11.3 is now available
by Marshall Garey 06 Mar '25

06 Mar '25

We are pleased to announce the availability of Slurm version 24.11.3. 24.11.3 fixes the database cluster ID generation not being random, a regression in which slurmd -G gave no output, a long-standing crash in slurmctld after updating a reservation with an empty nodelist, and some other minor to moderate bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.3 > ========================== > -- Fix race condition in slurmrestd that resulted in "Requested > data_parser plugin does not support OpenAPI plugin" error being returned > for valid endpoints. > -- If multiple partitions are requested, set the SLURM_JOB_PARTITION > output environment variable to the partition in which the job is running > for salloc and srun in order to match the documentation and the behavior of > sbatch. > -- Fix regression where slurmd -G gives no output. > -- Don't print misleading errors for stepmgr enabled steps. > -- slurmrestd - Avoid connection to slurmdbd for the following > endpoints: > GET /slurm/v0.0.41/jobs > GET /slurm/v0.0.41/job/{job_id} > -- slurmrestd - Avoid connection to slurmdbd for the following > endpoints: > GET /slurm/v0.0.40/jobs > GET /slurm/v0.0.40/job/{job_id} > -- Significantly increase entropy of clusterid portion of the > sluid by seeding the random number generator > -- Avoid changing process name to "watch" from original daemon name. > This could potentially breaking some monitoring scripts. > -- Avoid slurmctld being killed by SIGALRM due to race condition > at startup. > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix race between task/cgroup cpuset and jobacctgather/cgroup. > The first was removing the pid from task_X cgroup directory causing > memory limits to not be applied. > -- srun - Fixed wrongly constructed SLURM_CPU_BIND env variable > that could get propagated to downward srun calls in certain mpi > environments, causing launch failures. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.40. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.41. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.42.

1 0

broken SLURM-PMIX out-of-band communication on v24.11.0 with PMIx v5
by Bertini, Denis Dr. 06 Mar '25

06 Mar '25

Hi there, Using slurm v24.11.0 together with openMPI 5.0.7 built with openpmix v5.0.6 i am facing a systematical crash at process wiring-up phase when launching standard MPI job (OSU benchmarks ) on our new AMD compute nodes ( amd-epyc 9654, 192 phys. cores +HT ) running Rocky Linux 9.4 OS The typical error reads: slurmstepd: error: mpi/pmix_v5: pmixp_p2p_send: ccexe0094 [4]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit slurmstepd: error: mpi/pmix_v5: _slurm_send: ccexe0094 [4]: pmixp_server.c:1581: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.656.0, size = 46979, hostlist: (null) srun: error: Node failure on ccexe0091 after such a error as you can see the node move to state down It looks like the slurmstep pmix_server can not use the local socket at var/spool/slurmd/stepd.slurm.pmix.job_id.0 for inter-node communication . * On one AMD node ( same SLURM version, same cluster setup ) wiring up works smoothly even at core satuation (192 cores used) * On Intel node (intel,xeon,gold6248r, 48 cores ) wiring-up works even with multiple node without any problem * When the problematic AMD nodes are setup as dynamic node<https://slurm.schedmd.com/dynamic_nodes.html> the wiring-up phase with multiple nodes works perfectly, without any issue Has anybody experienced this kind of problem? Any idea what could be the reason for that? I also add that when the problematic AMD nodes are setup as dynamic node<https://slurm.schedmd.com/dynamic_nodes.html> the wiring-up phase with multiple nodes works perfectly, without any issue Cheers, Denis * --------- Denis Bertini Abteilung: CIT Ort: SB3 2.265a Tel: +49 6159 71 2240 Fax: +49 6159 71 2986 E-Mail: d.bertini(a)gsi.de GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing Directors / Geschäftsführung: Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: Ministerialdirigent Dr. Volkmar Dietz

1 0

2025

2024

slurm-users ----- 2025 ----- July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users