- slurm-users - lists.schedmd.com

Can't specify multiple partitions when submitting GPU jobs
by milad＠earthspecies.org 29 Apr '25

29 Apr '25

Hi, I have defined a partition for each GPU type we have in the cluster. This was mainly because I have different Node types for each GPU type and I want to set `DefCpuPerGPU` `DefMemPerGPU` for each of them. Unfortunately one can't set them per node but can do that per partition. Now sometimes people don't care about the GPU type and would like any of the partitions to pick up the job. The `--partition` in `sbatch` does allow specifying multiple partitions and this works fine when I'm not specifying `--gpu`. However when I add do something like `sbatch -p A,B --gpus 1 script.sh` then I get "srun: job 6279 queued and waiting for resources" even though partition B does have a GPU to offer. Strangely if the first partition specified (i.e. A) had a free GPU it would allocate the GPU and run the job. Is this a bug? Perhaps related to this: https://groups.google.com/g/slurm-users/c/UOUVfkajUBQ

3 4

Re: Error "_check_core_range_matches_sock" when
by Gestió Servidors 29 Apr '25

29 Apr '25

Hello, Yes, I know system is single-socket, but in a similar computer, this configuration works fine... After some tests, I have realized that with SLURM 23.11.0 I can assign CPUs in "gres.conf" (and, then, use half CPU for one GPU and other half for other GPU), but with SLURM 24.11.4 I receive " _check_core_range_matches_sock". I have read "NEWS" file inside sources files in 24.11.4 tgz and it says: […] * Changes in Slurm 23.11.7 ========================== […] -- Fix issue that only left one thread on each core available when "CPUs=" is configured to total thread count on multi-threaded hardware and no other topology info ("Sockets=", "CoresPerSocket", etc.) is configured. […] So, I suppose, in version 23.11.7 SLURM corrected that behaviour. Could someone confirm that? Thanks.

1 0

Error "_check_core_range_matches_sock" when starting "slurmctld"
by Gestió Servidors 28 Apr '25

28 Apr '25

Hello, I have compiled SLURM-24.11.3 and I have configured two GPUs in my system (slurmctld and slurmd are running in the same computer). Computes has a old processor Intel i7 with 4 cores and 4 hyperthreading. Node is configured with "NodeName=mysystem CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7940 Gres=gpu:geforce_gtx_titan_x:1,gpu:geforce_gtx_titan_black:1". "lscpu" command returns: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz BIOS Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz File gres.conf is: NodeName=mysystem Autodetect=off Name=gpu Type=geforce_gtx_titan_x File=/dev/nvidia0 CPUs=0-1 NodeName=mysystem Autodetect=off Name=gpu Type=geforce_gtx_titan_black File=/dev/nvidia1 CPUs=2-3 However, when I start daemon "slurmctld", system returns this error: [2025-04-28T09:35:41.003] error: _check_core_range_matches_sock: gres/gpu GRES core specification 0-1 for node aopcvis5 doesn't match socket boundaries. (Socket 0 is cores 0-3) [2025-04-28T09:35:41.003] error: Setting node aopcvis5 state to INVAL with reason:gres/gpu GRES core specification 0-1 for node aopcvis5 doesn't match socket boundaries. (Socket 0 is cores 0-3) Where is my configuration error? Thanks.

2 1

How can we put limits on interactive jobs?
by Ole Holm Nielsen 25 Apr '25

25 Apr '25

We would like to put limits on interactive jobs (started by salloc) so that users don't leave unused interactive jobs behind on the cluster by mistake. I can't offhand find any configurations that limit interactive jobs, such as enforcing a timelimit. Perhaps this could be done in job_submit.lua, but I couldn't find any job_desc parameters in the source code which would indicate if a job is interactive or not. Question: How do people limit interactive jobs, or identify orphaned jobs and kill them? Thanks a lot, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

4 4

Setting QoS with slurm 24.05.7
by Patrick Begou 25 Apr '25

25 Apr '25

Hi all, I'm trying to setup a QoS on a small 5 nodes cluster running slurm 24.05.7. My goal is to limit the resources on a (time x number of cores) strategy to avoid one large job requesting all the resources for too long time. I've read from https://slurm.schedmd.com/qos.html and some discussion but my setup is still not working. I think I need to set these informations: MaxCPUsPerJob=172800 MaxWallDurationPerJob=48:00:00 Flags=DenyOnLimit,OverPartQOS for: 12h max for 240 cores => (12*240*60=172800mn) no job can exceed 2 days do not accept jobs out of these limits. What I've done: 1) create the QoS: sudo sacctmgr add qos workflowlimit \ MaxWallDurationPerJob=48:00:00 \ MaxCPUsPerJob=172800 \ Flags=DenyOnLimit,OverPartQOS 2) Check sacctmgr show qos Name=workflowlimit format=Name%16,MaxTRES,MaxWall Name MaxTRES MaxWall ---------------- ------------- ----------- workflowlimit cpu=172800 2-00:00:00 3) Set the QoS for the account "most" which is the default account for the users: sudo sacctmgr modify account name=most set qos=workflowlimit 4) Check $ sacctmgr show assoc format=account,cluster,user,qos Account Cluster User QOS ---------- ---------- ---------- -------------------- root osorno normal root osorno root normal legi osorno normal most osorno workflowlimit most osorno begou workflowlimit 5) Modifiy slurm.conf with: AccountingStorageEnforce=limits,qos and propagate on the 5 nodes and the front end (done via Ansible) 6) Check clush -b -w osorno-fe,osorno,osorno-0-[0-4] 'grep AccountingStorageEnforce /etc/slurm/slurm.conf' --------------- osorno,osorno-0-[0-4],osorno-fe (7) --------------- AccountingStorageEnforce=limits,qos 7) restart slurmd on all the compute nodes and slurmctld + slurmdbd on the management node. But I can still request 400 cores for 24 hours: [begou@osorno ~]$ srun -n 400 -t 24:0:0 --pty bash bash-5.1$ squeue JOBID PARTITION NAME USER ST TIME START_TIME TIME_LIMIT CPUS NODELIST(REASON) 147 genoa bash begou R 0:03 2025-04-18T16:52:11 1-00:00:00 400 osorno-0-[0-4] So I must have missed something ? My partition (I've only one) in slurm.conf is: PartitionName=genoa State=UP Default=YES MaxTime=48:00:00 DefaultTime=24:00:00 Shared=YES OverSubscribe=NO Nodes=osorno-0-[0-4] Thanks Patrick

2 4

Job running slower when using Slurm
by Jeffrey Layton 24 Apr '25

24 Apr '25

Good morning, I'm running an NPB test, bt.C that is OpenMP and built using NV HPC SDK (version 25.1). I run it on a compute node by ssh-ing to the node. It runs in about 19.6 seconds. Then I run the code using a simple job: Command to submit job: sbatch --nodes=1 run-npb-omp The script run-npb-omp is the following: #!/bin/bash cd /home/.../NPB3.4-OMP/bin ./bt.C.x When I use Slurm, the job takes 482 seconds. Nothing really appears in the logs. It doesn't do any IO. No data is copied anywhere. I'm king of at a loss to figure out why. Any suggestions of where to look? Thanks! Jeff

4 9

Strange output of sshare
by frsc＠adm.ku.dk 23 Apr '25

23 Apr '25

Hi all, I'm trying to clean up and reconfigure fair share on a slurm 20.11.9 production cluster after some trial and error before I started looking into it. I don't know the full story and need to pick up here. Fair share is enabled with default settings and not customised. It looks a bit like it was enabled by accident and relevant options are undefined/at default. As a first step I would like to understand the current state. Executing sshare includes output like this (output shortened but complete for the two relevant accounts): Account User Partition RawShares NormShares RawUsage EffectvUsage FairShare -------- ---------- ------------ ---------- ----------- ----------- ------------- ---------- root 0.000000 1002501 1.000000 .root root 1 0.083333 0 0.000000 1.000000 .A1 1 0.083333 0 0.000000 ..A1 U1 P1 1 0.166667 0 0.000000 0.873684 ..A1 U2 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P1 1 0.166667 0 0.000000 0.873684 ..A1 U3 P2 1 0.166667 0 0.000000 0.873684 ..A1 U4 P3 1 0.166667 0 0.000000 0.873684 ..A1 U4 P1 1 0.166667 0 1.000000 0.821053 <== What is going on here?? .A2 1 0.083333 0 0.000000 ..A2 U5 P2 1 0.142857 0 0.000000 1.000000 ..A2 U5 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P4 1 0.142857 0 0.000000 1.000000 ..A2 U6 P1 1 0.142857 0 0.000000 1.000000 ..A2 U6 P2 1 0.142857 0 0.000000 1.000000 ..A2 U7 P1 1 0.142857 0 0.000000 1.000000 ..A2 U7 P2 1 0.142857 0 0.000000 1.000000 User U4 is not a member of any other account. I understand everything about this output except the line I marked. Both accounts A1 and A2 have zero usage, yet, for user U4 on partition P1 we have effective usage 1.0, screwing the fair share factors up for everyone in this account. As far as I can tell, both accounts should look identical with a fair share factor of 1 for every user. I'm grateful for any pointer for what to look for. Best regards, Frank

2 1

Slurm webhooks
by Davide DelVento 22 Apr '25

22 Apr '25

Happy Monday everybody, I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that. As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-when-job-beg… Is that indeed the case, as searching the web indicates? If Slurm does not support this, is there a workaround? For example, I'm thinking of installing a local SMTP server, or an alternative/dummy mailx program which instead of relaying emails as requested would post an encrypted web url thing using the information from the email. I am sure I could actually write such a software myself, but I don't have enough time to dedicate to the design, maintenance and debugging of such, so I am looking for something decent already in existence. A cursory web search did not find anything suitable, but perhaps I did not look in the appropriate places, because my gut feeling is that somebody must have already had such an itch to scratch! Any other ideas about alternative ways to accomplish this? Thanks

3 3

problem with slurmd dynamic mode with slurmctld in docker
by stef53864 22 Apr '25

22 Apr '25

hi everybody, i try to user dynamic mode with configless mode with slurm 24.11.3 and upgrade to 24.11.4 with slurmd and i found a problem. slurmctld is a container with docker, and my node is outside the container network . slurmctld register my ip with a function getpeeraddr on the slurmctld socket. but my ip connected to the socket come from the docker nat/bridge so slurmctld register my ip bridged ( not my real ip ) that is to say the docker gateway (172.20.0.1) *scontrol show node* *-------------------------* *NodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=172.20.0.1 NodeHostName=ltlsbubble1 Version=24.11.4* so the node go down after the "not pinging it" timeout i try to update the config *scontrol uupdate NodeName=ltlsbubble1* *NodeAddr=xx.xx.xx.xx* but a the first *scontrol reconfigure * it comes back to : *NodeAddr=172.20.0.1* in normal mode ------------------- *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.11.4* in normal mode NodeAddr is the same than NodeName , so it use DNS resolution for communication. to verify my hypothesis, i go to the c code of slurm, identify the register function and replace it with the same mechanism than normal node in src/slurmctld/node_mgr.c i replace : set_node_comm_name(node_ptr, *comm_name*, reg_msg->hostname); by set_node_comm_name(node_ptr, NULL, reg_msg->hostname); i rebuild slutmctld with this patch and try it with dynamic mode , it works like expected *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.1* no ip in nodeAddr , but only the nodename, so it use DNS resolution . the node works fine and no goes down for timeout ping so my question : can we have an option to force DNS resolution instead ip discover in Dynamic mode ? ( i try the option cloud_dns, but it not seems the purpose of this option) best regard, Stephane

1 0

pam_slurm_adopt ambiguity in instructions and with ssd
by Robert Kudyba 21 Apr '25

21 Apr '25

In the instructions for pam_slurm_adopt <https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: > > Add the following line to the appropriate file in /etc/pam.d, such as > system-auth or sshd (you may use either the "required" or "sufficient" > PAM control flag): This module is configurable. Add these options to the end of the > pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd > or system-auth): Assuming an OS like CentOS does this mean it should be put in both? slurm.conf on the node has: UsePAM yes slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets: Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user(a)ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth] Am I missing something?

3 5