- slurm-users - lists.schedmd.com

Issues with Slurm 23.11.1
by Fokke Dijkstra 15 Mar '24

15 Mar '24

Dear all, Since the upgrade from Slurm 22.05 to 23.11.1 we are having problems with the communication between the slurmctld and slurmd processes. We are running a cluster with 183 nodes and almost 19000 cores. Unfortunately some nodes are in a different network preventing full internode communication. A network topology and setting TopologyParam RouteTree have been used to make sure no slurmd communication happens between nodes on different networks. In the new Slurm version we see the following issues, which did not appear in 22.05: 1. slurmd processes acquire many network connections in CLOSE-WAIT (or CLOSE_WAIT depending on the tool used) causing the processes to hang, when trying to restart slurmd. When checking for CLOSE-WAIT processes we see the following behaviour: Recv-Q Send-Q Local Address:Port Peer Address:Port Process 1 0 10.5.2.40:6818 10.5.0.43:58572 users:(("slurmd",pid=1930095,fd=72),("slurmd",pid=1930067,fd=72)) 1 0 10.5.2.40:6818 10.5.0.43:58284 users:(("slurmd",pid=1930095,fd=8),("slurmd",pid=1930067,fd=8)) 1 0 10.5.2.40:6818 10.5.0.43:58186 users:(("slurmd",pid=1930095,fd=22),("slurmd",pid=1930067,fd=22)) 1 0 10.5.2.40:6818 10.5.0.43:58592 users:(("slurmd",pid=1930095,fd=76),("slurmd",pid=1930067,fd=76)) 1 0 10.5.2.40:6818 10.5.0.43:58338 users:(("slurmd",pid=1930095,fd=19),("slurmd",pid=1930067,fd=19)) 1 0 10.5.2.40:6818 10.5.0.43:58568 users:(("slurmd",pid=1930095,fd=68),("slurmd",pid=1930067,fd=68)) 1 0 10.5.2.40:6818 10.5.0.43:58472 users:(("slurmd",pid=1930095,fd=69),("slurmd",pid=1930067,fd=69)) 1 0 10.5.2.40:6818 10.5.0.43:58486 users:(("slurmd",pid=1930095,fd=38),("slurmd",pid=1930067,fd=38)) 1 0 10.5.2.40:6818 10.5.0.43:58316 users:(("slurmd",pid=1930095,fd=29),("slurmd",pid=1930067,fd=29)) The first IP address is that of the compute node, the second that of the node running slurmctld. The nodes can communicate using these IP addresses just fine. 2. slurmd cannot be properly restarted [2024-01-18T10:45:26.589] slurmd version 23.11.1 started [2024-01-18T10:45:26.593] error: Error binding slurm stream socket: Address already in use [2024-01-18T10:45:26.593] fatal: Unable to bind listen port (6818): Address already in use This is probably because of the processes being in CLOSE-WAIT, which can only be killed using signal -9. 3. We see jobs stuck in completing CG state, probably due to communication issues between slurmctld and slurmd. The slurmctld sends repeated kill requests but those do not seem to be acknowledged by the client. This happens more often in large job arrays, or generally when many jobs start at the same time. However, this could be just a biased observation (i.e., it is more noticeable on large job arrays because there are more jobs to fail in the first place). 4. Since the new version we also see messages like: [2024-01-17T09:58:48.589] error: Failed to kill program loading user environment [2024-01-17T09:58:48.590] error: Failed to load current user environment variables [2024-01-17T09:58:48.590] error: _get_user_env: Unable to get user's local environment, running only with passed environment The effect of this is that the users run with the wrong environment and can’t load the modules for the software that is needed by their jobs. This leads to many job failures. The issue appears to be somewhat similar to the one described at: https://bugs.schedmd.com/show_bug.cgi?id=18561 In that case the site downgraded the slurmd clients to 22.05 which got rid of the problems. We’ve now downgraded the slurmd on the compute nodes to 23.02.7 which also seems to be a workaround for the issue. Does anyone know of a better solution? Kind regards, Fokke Dijkstra -- Fokke Dijkstra <f.dijkstra(a)rug.nl> <f.dijkstra(a)rug.nl> Team High Performance Computing Center for Information Technology, University of Groningen Postbus 11044, 9700 CA Groningen, The Netherlands

2 3

Overhead of multiple concurrent job steps
by Mehta, Kshitij 15 Mar '24

15 Mar '24

Hello, We have a use case in which we need to launch multiple concurrently running MPI applications inside a job allocation. Most supercomputing facilities limit the number of concurrent job steps as they incur an overhead with the global Slurm scheduler. Some frameworks, such as the Flux framework from LLNL, claim to mitigate this issue by starting an instance of their own scheduler inside an allocation, which then acts as the resource manager for the compute nodes in the allocation. Out of curiosity, I was wondering if there is a fundamental reason behind having a single global scheduler that the srun launch commands must contact to launch job steps. Perhaps it was overkill to develop a ‘hierarhical’ design in which Slurm launches a local job daemon for every allocation that manages resources for that allocation? I would appreciate your insight in understanding more about Slurm’s core design. Thanks and regards, Kshitij Mehta Oak Ridge National Laboratory

1 0

Jobs being denied for GrpCpuLimit despite having enough resource
by Simon Andrews 14 Mar '24

14 Mar '24

Our cluster has developed a strange intermittent behaviour where jobs are being put into a pending state because they aren't passing the AssocGrpCpuLimit, even though the user submitting has enough cpus for the job to run. For example: $ squeue -o "%.6i %.9P %.8j %.8u %.2t %.10M %.7m %.7c %.20R" JOBID PARTITION NAME USER ST TIME MIN_MEM MIN_CPU NODELIST(REASON) 799 normal hostname andrewss PD 0:00 2G 5 (AssocGrpCpuLimit) ..so the job isn't running, and it's the only job in the queue, but: $ sacctmgr list associations part=normal user=andrewss format=Account,User,Partition,Share,GrpTRES Account User Partition Share GrpTRES ---------- ---------- ---------- --------- ------------- andrewss andrewss normal 1 cpu=5 That user has a limit of 5 CPUs so the job should run. The weird thing is that this effect is intermittent. A job can hang and the queue will stall for ages but will then suddenly start working and you can submit several jobs and they all work, until one fails again. The cluster has active nodes and plenty of resource: $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST normal* up infinite 2 idle compute-0-[6-7] interactive up 1-12:00:00 3 idle compute-1-[0-1,3] The slurmctld log just says: [2024-03-14T16:21:41.275] _slurm_rpc_submit_batch_job: JobId=799 InitPrio=4294901720 usec=259 Whilst it's in this state I can run other jobs with core requests of up to 4 and they work, but not 5. It's like slurm is adding one CPU to the request and then denying it. I'm sure I'm missing something fundamental but would appreciate it if someone could point out what it is! Thanks Simon.

3 3

We're Live! Check out the new SchedMD.com now!
by Victoria Hobson 13 Mar '24

13 Mar '24

Our website has gone through some much needed change and we'd love for you to explore it! The new SchedMD.com is equipped with the latest information about Slurm, your favorite workload manager, and details about SchedMD services, support, and training offerings. Toggle through our Industries pages (https://www.schedmd.com/slurm-industries/) to learn more about how Slurm can service your specific site needs. Why Slurm? (https://www.schedmd.com/slurm/why-slurm/) gives you all the basics around our market-leading scheduler and SchedMD Services (https://www.schedmd.com/slurm-support/our-services/) addresses all the ways we can help you optimize your site. These new web pages also feature access to our Documentation Site, Bug Site, and Installation Guide. Browse our Events tab to see where we'll be when, and be sure to register for our Slurm User Group (SLUG) in Oslo, Norway this fall! (https://www.schedmd.com/about-schedmd/events/) SchedMD.com, your one stop shop for all things Slurm. Check it out now! -- Victoria Hobson SchedMD LLC Vice President of Marketing

2 1

Slurm Utilities
by Paul Edmon 13 Mar '24

13 Mar '24

Just wanted to share some slurm utilities that we've written at Harvard FASRC that maybe useful to the community. seff-account: https://github.com/fasrc/seff-account Creates job statistics summaries for users and accounts similar to what seff and seff-array does. showq: https://github.com/fasrc/slurm_showq A slurm version of the Moab showq command lsload: https://github.com/fasrc/lsload A slurm version of the LSF lsload command scalc: https://github.com/fasrc/scalc A calculator for various fairshare related things spart: https://github.com/fasrc/spart A simplified output for slurm partition information stdg: https://github.com/fasrc/stdg Slurm test deck generator prometheus-slurm-exporter: https://github.com/fasrc/prometheus-slurm-exporter Slurm exporters for prometheus Hopefully people find these useful. Pull requests are always appreciated. -Paul Edmon-

1 0

Re: SLURM in K8s, any advice?
by LEAVY Alan 13 Mar '24

13 Mar '24

I'm a little late to this party but would love to establish contact with others using slurm in Kubernetes. I recently joined a research institute in Vienna (IIASA) and I'm getting to grips with slurm and Kubernetes (my previous role was data engineering / fintech). My current setup sounds like what Urban described in this thread, back in Nov 22. It has some rough edges though. Right now, I'm trying to upgrade to slurm-23.11.4 in Ubuntu 23.10 containers. I'm having trouble with the cgroup/v2 plugin. Are you still using slurm on K8s Urban? How did your installation work out Hans? Would either of you be willing to share your experiences? Regards, Alan.

4 3

Shards distribution over multiple GPUs
by Francois Broquedis 12 Mar '24

12 Mar '24

Hi all, We're trying to enable sharding on our compute cluster. On this cluster: - ensicompute-1 comes with 1 NVIDIA V100 GPU ; - ensicompute-13 comes with 3 NVIDIA A40 GPUs ; - all other nodes (for now, ensicompute-11 and ensicompute-12, but several others will come) come with 3 NVIDIA RTX 6000 GPUs. To enable sharding, I followed these steps: 1. [slurm.conf] Add "shard" to GresTypes ; 2. [slurm.conf] Add "shard:N" to Gres for each node. For testing purposes, I have set N to 9, so each GPU can execute up to 3 jobs concurrently: NodeName=ensicompute-[11-12] Gres=gpu:Quadro:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht 3. [gres.conf] Declare the shards after the definition of the GPUs GRES. For step 3, I tried different things, leading to different outcomes: a. Define a global number of shards, for the entire host: Name=shard Count=9 ==> This way, sharding seems to work ok, but all the jobs are executed on GPU#0. If running 12 jobs for example, 9 of them are assigned to GPU#0 and start executing, while 3 of them remain in a pending state. No job is assigned to GPU#1 or GPU#2. b. Define a per-GPU number of shards, associated to the device file representing the GPU: Name=shard Count=3 File=/dev/nvidia0 Name=shard Count=3 File=/dev/nvidia1 Name=shard Count=3 File=/dev/nvidia2 ==> In this case, the slurmd service fails to start on the compute node. The error message found in /var/log/slurmd.log is "fatal: Invalid GRES record for shard, count does not match File value". c. Don't define anything about shards in gres.conf. ==> Same behavior than in a.: all jobs are executed on GPU#0. I attach to this message the full content of the slurm.conf and gres.conf files. What is the proper way to configure sharding in a cluster with several GPUs per node? Is there a way to specify how many shards should be allocated to each GPU? Cheers, François === slurm.conf === # slurm.conf file generated by configurator.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=ensimag SlurmctldHost=nash ProctrackType=proctrack/cgroup SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup ReturnToService=2 # # # TIMERS InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES GresTypes=gpu,shard NodeName=ensicompute-1 Gres=gpu:Tesla:1,shard:3 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht NodeName=ensicompute-13 Gres=gpu:A40:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht NodeName=ensicompute-[11-12] Gres=gpu:Quadro:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht PartitionName=compute Nodes=ALL Default=YES MaxTime=INFINITE State=UP === gres.conf === AutoDetect=off # ensicompute-1 NodeName=ensicompute-1 Name=gpu Type=Tesla File=/dev/nvidia0 NodeName=ensicompute-1 Name=shard Count=3 File=/dev/nvidia0 # ensicompute-11 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia0 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia1 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia2 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia2 # ensicompute-12 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia0 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia1 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia2 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia2 # ensicompute-13 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia0 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia1 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia2 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia2 -- François Broquedis, Ingénieur Service Informatique Grenoble INP - Ensimag, bureau E208 681 rue de la Passerelle BP 72, 38402 Saint Martin d'Hères CEDEX Tél.: +33 (0)4 76 82 72 78

1 0

Re: user_name attribute missing in job response from slurm rest api method jobs
by Acartürk, Ayhan 11 Mar '24

11 Mar '24

Sorry... my previous message should have been a reply to the post by Filip Holka from March, 25th 2022: https://lists.schedmd.com/pipermail/slurm-users/2022-March/008543.html Best wishes, Ayhan Acartürk

1 0

Re: user_name attribute missing in job response from slurm rest api method jobs
by Acartürk, Ayhan 11 Mar '24

11 Mar '24

Hello together, we have the same problem here. All jobs delivered by the api endpoint /slurm/v0.0.36/jobs have a user_id attached but no user_name. Currently, we solve this issue by an extra call for each job_id to the api endpoint /slurmdb/v.0.0.36/job/{job_id}. Any idea where to search for a possible solution is highly appreciated. Best wishes, Ayhan Acartürk

1 0

Problem building slurm with PMIx
by Patrick Begou 11 Mar '24

11 Mar '24

Hi ! I manage a small CentOS8 cluster using slurm slurm-20.11.7-1 and OpenMPI built from sources. - I know this OS is not maintained any more and I need to negotiate downtime to reinstall - I know Slurm 20.11.7 has security issue (I've built it from source some years ago with rpmbuild -ta --with mysql --with hwloc slurm-20.11.7.tar.bz) and I should update. All was running fine until I add a GPU Node and Nvidia sdk. This SDK provides an openMPI3 implementation GPU aware but I'm unable to launch an intranode parallel job with it using srun: -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using: version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix. Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location. Please configure as appropriate and try again. -------------------------------------------------------------------------- I check with "srun --mpi=list" and got no pmx. srun: MPI types are... srun: pmi2 srun: cray_shasta srun: none So I decide to build the rpms from slurm-20.11.9.tar.bz2 as I had done previously for 20.11.7 and update. I've first installed pmix-2.1.1-1 from src as I had no pmix-devel rpm in my local CentOS8 repo: rpm --rebuild pmix-2.1.1-1.el8.src.rpm dnf install pmix-devel-2.1.1-1.el8.x86_64.rpm pmix-2.1.1-1.el8.x86_64.rpm Then build Slurm from slurm-20.11.9.tar.bz2 (just changing python3 to python38 in the spec file) rpmbuild -ta --with mysql --with hwloc --with pmix slurm-20.11.9.tar.bz2 And the try to install these package on the GPU node dnf install slurm-slurmd-20.11.9-1.el8.x86_64.rpm slurm-20.11.9-1.el8.x86_64.rpm slurm-devel-20.11.9-1.el8.x86_64.rpm slurm-libpmi-20.11.9-1.el8.x86_64.rpm But I get this strange error: Error: Problem: conflicting requests - nothing provides pmix = 20.11.9 needed by slurm-slurmd-20.11.9-1.el8.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Why this request on PMIX with the slurm version number ? Am I wrong somewhere ? Thanks for your help Patrick

2 1

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users