- slurm-users - lists.schedmd.com

Slurm version 23.11.7 is now available
by Marshall Garey 21 May '24

21 May '24

We are pleased to announce the availability of Slurm version 23.11.7. The 23.11.7 release fixes a few potential crashes in slurmctld when using less common options on job submission, slurmrestd compatibility with auth/slurm, and some additional minor and moderate severity bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- slurmrestd - Correct OpenAPI specification for > 'GET /slurm/v0.0.40/jobs/state' having response as null. > -- Allow running jobs on overlapping partitions if jobs don't specify -s. > -- Fix segfault when requesting a shared gres along with an exclusive > allocation. > -- Fix regression in 23.02 where afternotok and afterok dependencies were > rejected for federated jobs not running on the origin cluster of the > submitting job. > -- slurmctld - Disable job table locking while job state cache is active when > replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`. > -- Fix sanity check when setting tres-per-task on the job allocation as well as > the step. > -- slurmrestd - Fix compatiblity with auth/slurm. > -- Fix issue where TRESRunMins gets off correct value if using > QOS UsageFactor != 1. > -- slurmrestd - Require `user` and `association_condition` fields to be > populated for requests to 'POST /slurmdb/v0.0.40/users_association'. > -- Avoid a slurmctld crash with extra_constraints enabled when a job requests > certain invalid --extra values. > -- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job > array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array > task (e.g. 1_10), which previously signaled the whole array job instead. > -- Fix a possible slurmctld segfault when at some point we failed to create an > external launcher step. > -- Allow the slurmctld to open a connection to the slurmdbd if the first > attempt fails due to a protocol error. > -- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob. > -- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units. > -- Fix a rare deadlock on slurmctld shutdown or reconfigure. > -- Fix issue that only left one thread on each core available when "CPUs=" is > configured to total thread count on multi-threaded hardware and no other > topology info ("Sockets=", "CoresPerSocket", etc.) is configured. > -- Fix the external launcher step not being allocated a VNI when requested. > -- jobcomp/kafka - Fix payload length when producing and sending a message. > -- scrun - Avoid a crash if RunTimeDelete is called before the container > finishes. > -- Save the slurmd's cred_state while reconfiguring to prevent the loss job > credentials.

1 0

SLUG Call for Papers Deadline
by Victoria Hobson 21 May '24

21 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The deadline to submit a presentation abstract is Friday, May 31st. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 Notifications of final presentations accepted will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Slurm not allocating correct cgroup cpu ids in srun step (possible bug)
by Ashley Wright 21 May '24

21 May '24

Hi, At our site we have recently upgraded to Slurm 23.11.5 and are having trouble with MPI jobs doing srun inside a sbatch'ed script. The cgroup does not appear to be setup correctly for the srun (step_0). As an example $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../cpuset.cpus 0,2-3,68-69,96,98-99,164-165 $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../step_0/cpuset.cpus 0,2,68,96,98,164 The sbatch is allocated a range of cpus in the cgroup. However, when step_0 is run, only some of those CPUs are in the group. I have noticed that it is always the range which is missing, ie 2-5 only 2 is included, 3,4,5 are missing. This also only happens if there are multiple groups of cpus in the allocations. ie only 1-12 would be fine, however 1-12,15-20 would result in 1,15 only. The sbatch also seems fine, with step_batch and step_extern being allocated correctly. This causes numerous issues with MPI jobs as they end up overloading cpus. We are running our nodes with threading enabled on the CPUs, and with cgroups and affinity plugins. I have attached our slurm.conf to show our settings. Our /etc/slurm/cgroup.conf is ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes We have turned on logging at debug2 level, but I haven't yet found anything useful. Happy for a suggestion on what to look for. Is anyone able to provide any advice on where to go next to try and identify the issue? Regards, Ashley Wright

1 0

Running slurm on alternate ports
by Alan Stange 20 May '24

20 May '24

Hello all, for testing purposes, we would like to run slurm on ports different from the default values. No problems in setting this up. But how does one tell srun/sbatch/etc what the different port numbers are? I see no command line options to specify a port or an alternate configuration file. Thank you, Alan

2 2

Invalid/incorrect gres.conf syntax
by Gestió Servidors 20 May '24

20 May '24

Hello, I have configured my "gres.conf" in this way: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 node-gpu-1 and node-gpu-2 are two systems with two sockets; node-gpu-3 and node-gpu-4 have only one socket. In my "slurm.conf" I have these lines: AccountingStorageTRES=gres/gpu SelectType=select/cons_tres GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 However, slurmctld reports warning logs about "error syntax in Cores attribute" in gres.conf. Where is the syntax error? Thanks a lot!

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 20 May '24

20 May '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks.

1 0

srun weirdness
by Dj Merrill 17 May '24

17 May '24

I'm running into a strange issue and I'm hoping another set of brains looking at this might help. I would appreciate any feedback. I have two Slurm Clusters. The first cluster is running Slurm 21.08.8 on Rocky Linux 8.9 machines. The second cluster is running Slurm 23.11.6 on Rocky Linux 9.4 machines. This works perfectly fine on the first cluster: $ srun --mem=32G --pty /bin/bash srun: job 93911 queued and waiting for resources srun: job 93911 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help and the ollama help message appears as expected. However, on the second cluster: $ srun --mem=32G --pty /bin/bash srun: job 3 queued and waiting for resources srun: job 3 has been allocated resources and on the resulting shell on the compute node: $ /mnt/local/ollama/ollama help fatal error: failed to reserve page summary memory runtime stack: runtime.throw({0x1240c66?, 0x154fa39a1008?}) runtime/panic.go:1023 +0x5c fp=0x7ffe6be32648 sp=0x7ffe6be32618 pc=0x4605dc runtime.(*pageAlloc).sysInit(0x127b47e8, 0xf8?) runtime/mpagealloc_64bit.go:81 +0x11c fp=0x7ffe6be326b8 sp=0x7ffe6be32648 pc=0x456b7c runtime.(*pageAlloc).init(0x127b47e8, 0x127b47e0, 0x128d88f8, 0x0) runtime/mpagealloc.go:320 +0x85 fp=0x7ffe6be326e8 sp=0x7ffe6be326b8 pc=0x454565 runtime.(*mheap).init(0x127b47e0) runtime/mheap.go:769 +0x165 fp=0x7ffe6be32720 sp=0x7ffe6be326e8 pc=0x451885 runtime.mallocinit() runtime/malloc.go:454 +0xd7 fp=0x7ffe6be32758 sp=0x7ffe6be32720 pc=0x434f97 runtime.schedinit() runtime/proc.go:785 +0xb7 fp=0x7ffe6be327d0 sp=0x7ffe6be32758 pc=0x464397 runtime.rt0_go() runtime/asm_amd64.s:349 +0x11c fp=0x7ffe6be327d8 sp=0x7ffe6be327d0 pc=0x49421c If I ssh directly to the same node on that second cluster (skipping Slurm entirely), and run the same "/mnt/local/ollama/ollama help" command, it works perfectly fine. My first thought was that it might be related to cgroups. I switched the second cluster from cgroups v2 to v1 and tried again, no difference. I tried disabling cgroups on the second cluster by removing all cgroups references in the slurm.conf file but that also made no difference. My guess is something changed with regards to srun between these two Slurm versions, but I'm not sure what. Any thoughts on what might be happening and/or a way to get this to work on the second cluster? Essentially I need a way to request an interactive shell through Slurm that is associated with the requested resources. Should we be using something other than srun for this? Thank you, -Dj

6 12

Slurm DB upgrade failure behavior
by Yuengling, Philip J. 17 May '24

17 May '24

Hi everyone, I'm writing up some Ansible code to manage Slurm software updates, and I haven't found any documentation about slurmdbd behavior if the mysql/mariadb database doesn't upgrade successfully. What I do know is that if it is sucessful I can expect to see "Conversion done: success!" in the slurmdbd log. This is good, but minor updates do not update the database as far as I know. If the Slurm database cannot upgrade upon an update, does it always shut down with a fatal error? What other behaviors should I look for if there is a failure? Cheers, Phil Y

2 1

Removing safely a node
by Ratnasamy, Fritz 17 May '24

17 May '24

Hi, What is the "official" process to remove nodes safely? I have drained the nodes so jobs are completed and put them in down state after they are completely drained. I edited the slurm.conf file to remove the nodes. After some time, I can see that the nodes were removed from the partition with the command sinfo However, I was told I might need to restart the service slurmctld, do you know if it is necessary? Should I also run scontrol reconfig? Best, *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Best practice for jobs resuming from suspended state
by Paul Jones 17 May '24

17 May '24

Hi, We use PreemptMode and PriorityTier within Slurm to suspend low priority jobs when more urgent work needs to be done. This generally works well, but on occasion resumed jobs fail to restart - which is to say Slurm sets the job status to running but the actual code doesn't recover from being suspended. Technically everything is working as expected, but I wondered if there was any best practice to pass onto users about how to cope with this state? Obviously not a direct Slurm question, but wondered if others had experience with this and any advice on how best to limit the impact? Thanks, Paul --

2 1