- slurm-users - lists.schedmd.com

Replacing MUNGE with SACK (auth/slurm)
by Daniel Letai 11 Jul '24

11 Jul '24

Does SACK replace MUNGE? As in - MUNGE is not required when building Slurm or on compute? If so, can the Requires and BuildRequires for munge be made optional on bcond_without_munge in the spec file? Or is there a reason MUNGE must remain a hard require for Slurm? Thanks, --Dani_L.

1 0

Job submitted to multiple partitions not running when any partition is full
by Paul Raines 09 Jul '24

09 Jul '24

I have a job 465072 submitted to multiple partitions (rtx6000,rtx8000,pubgpu) JOBID PARTITION PENDING PRIORITY TRES_ALLOC|REASON 4650727 rtx6000 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4650727 rtx8000 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4650727 pubgpu 47970 0.00367972 cpu=5,mem=400G,node=1,gpu=1|Priority 4646926 rtx6000 487048 0.00121987 cpu=10,mem=32G,node=1,gpu=1|Priority,Resources 4650186 rtx8000 56979 0.00000000 cpu=4,mem=10G,node=1,gpu=1|Priority,Resources We see the two partitions rtx6000 and rtx8000 are full and two other jobs are at the top of the queue waiting to run on those. But partition pubgpu is NOT full and you can see here a node leo with resources to run the 4650727 job HOST PARTITION CORES MEMORY GPUS leo pubgpu 48/ 64 12288/1030994 0/ 1 leo pubcpu 48/ 64 12288/1030994 0/ 1 The node leo is NOT part of the rtx6000 or rtx8000 partitions and there are no other pending jobs waiting on either the pubgpu or pubcpu partition that leo is part of So why is 4650727 not running on the pubgpu partition? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

2 2

Temporarily bypassing pam_slurm_adopt.so
by Daniel L'Hommedieu 09 Jul '24

09 Jul '24

Hi, all. We have a use case where we need to allow a group of users (members of an LDAP group, which I can easily add to a Linux group) to SSH to a compute node, without disabling pam_slurm_adopt.so. Is there a way to do this? We can add users to the sudo group, which will bypass pam_slurm_adopt.so, but we do not want to grant sudo access to these users. Is there any way to bypass for a group of users, pam_slurm_adopt.so without removing it from /etc/pam.d/sshd, and without adding them to the sudo group? Thanks. Daniel

5 4

vers 23: slurmctld pb (memory leak and response time)
by LEROY Christine 208562 09 Jul '24

09 Jul '24

Hello Everyone, We tried several version of slurm 23-x (till the last one: 23.11.8), and we observe a memory leak in slurmctld in addition to a bad response time (sometimes). It seems to be due to a slurm.conf configuration with Partition containing Allowgroups option. Is it a known problem? Is it recommended to use the version 22, or 24 ? Thanks in advance, Christine Leroy

1 1

Can SLURM queue different jobs to start concurrently?
by Dan Healy 08 Jul '24

08 Jul '24

Hi there, I've received a question from an end user, which I presume the answer is "No", but would like to ask the community first. Scenario: The user wants to create a series of jobs that all need to start at the same time. Example: there are 10 different executable applications which have varying CPU and RAM constraints, all of which need to communicate via TCP/IP. Of course the user could design some type of idle/statusing mechanism to wait until all jobs are *randomly *started, then begin execution, but this feels like a waste of resources. The complete execution of these 10 applications would be considered a single simulation. The goal would be to distribute these 10 applications across the cluster and not necessarily require them all to execute on a single node. Is there a good architecture for this using SLURM? If so, please kindly point me in the right direction. -- Thanks, Daniel Healy

3 2

Re: Can SLURM queue different jobs to start concurrently?
by Lloyd Brown 08 Jul '24

08 Jul '24

I'm confused. Why can't they just use a multi-node job, and have the job script farm out the individual tasks to the various workers through some mechanism (srun, mpirun, ssh, etc.)? AFAIK, there's nothing preventing a job from using resources on multiple hosts. The job just needs to have some way of pushing the work out to those hosts. Lloyd On 7/8/24 14:17, Dan Healy via slurm-users wrote: > Hi there, > > I've received a question from an end user, which I presume the answer > is "No", but would like to ask the community first. > > Scenario: The user wants to create a series of jobs that all need to > start at the same time. Example: there are 10 different executable > applications which have varying CPU and RAM constraints, all of which > need to communicate via TCP/IP. Of course the user could design some > type of idle/statusing mechanism to wait until all jobs are /randomly > /started, then begin execution, but this feels like a waste of > resources. The complete execution of these 10 applications would be > considered a single simulation. The goal would be to distribute these > 10 applications across the cluster and not necessarily require them > all to execute on a single node. > > Is there a good architecture for this using SLURM? If so, please > kindly point me in the right direction. > > -- > Thanks, > > Daniel Healy -- Lloyd Brown HPC Systems Administrator Office of Research Computing Brigham Young University http://rc.byu.edu

1 0

cgroups/v2 plugin rpmbuild issue
by Chris Taylor 06 Jul '24

06 Jul '24

Trying to use rpmbuild on Rocky9 Linux, Slurm 21.08 - I want to build with cgroups/v2 support and have these installed: libbpf-devel.x86_64 2:1.3.0-2.el9 dbus-devel.x86_64 1:1.12.20-8.el9 kernel-headers.x86_64 5.14.0-427.22.1.el9_4 hwloc-devel.x86_64 2.4.1-5.el9 In https://slurm.schedmd.com/cgroup_v2.html it says: Requirements For building cgroup/v2 there are two required libraries checked at configure time... Look at your config.log when configuring to see if they were correctly detected on your system. I don't see any mention of 'ebpf', 'bpf', or 'dbus' when I grep through ~/rpmbuild/BUILD/slurm-21.08.8-2/config.log. Before I noticed that, I installed the RPMS on my cluster and get cgroup errors when I try to run a job or get an allocation. The slurmctld and slurmd processes start up fine on the servers, and I have a generic /etc/slurm/cgroup.conf. It seems like the cgroups plugin isn't getting built right with rpmbuild- how do I troubleshoot? Thanks Chris

2 3

Qlustar HPC Core Stack 24.06
by Roland Fehrenbacher 06 Jul '24

06 Jul '24

Hi all, the Qlustar HPC Core Stack 24.06 update is available. It includes an update to Slurm 23.11.8. Find further details at https://qlustar.com/news/2024/2024-06-29-hpc-core-stack-2406 Enjoy, Roland ------- https://qlustar.com -- 100% Open Source HPC / AI / Storage / Cloud Linux Cluster OS --

1 0

Run a program via Strigger when a node joins the cluster
by Karri Vrkreddy 06 Jul '24

06 Jul '24

Hi, We have a requirement to run a specific program whenever any new node joins the slurm cluster. For this, we have tried using strigger with the following options : "strigger --set --node --up --flags=perm --program=<program>" We see that the trigger is not getting activated always. Checked slurmctld logs for "type=node:up" and noticed it is not always happening. Based on documentation at https://slurm.schedmd.com/strigger.html , "up" flag would mean Trigger an event if the specified node is returned to service from a DOWN state 1. From Slurm Workload Manager - Slurm Power Saving Guide , and the above message does it mean that a node has to be already in DOWN state in the cluster for a trigger with "--up" to get activated ? 2. Are there any other ways beyond strigger to achieve this ? | | | | | | | | | | | Slurm Workload Manager - Slurm Power Saving Guide | | | -- Thanks Karri

1 0

Using sharding
by Ricardo Cruz 05 Jul '24

05 Jul '24

Greetings, There are not many questions regarding GPU sharding here, and I am unsure if I am using it correctly... I have configured it according to the instructions <https://slurm.schedmd.com/gres.html>, and it seems to be configured properly: $ scontrol show node compute01 NodeName=compute01 Arch=x86_64 CoresPerSocket=32 CPUAlloc=48 CPUEfctv=128 CPUTot=128 CPULoad=10.95 AvailableFeatures=(null) ActiveFeatures=(null) * Gres=gpu:8,shard:32* [truncated] When running with gres:gpu everything works perfectly: $ /usr/bin/srun --gres=gpu:2 ls srun: job 192 queued and waiting for resources srun: job 192 has been allocated resources (...) However, when using sharding, it just stays waiting indefinitely: $ /usr/bin/srun --gres=shard:2 ls srun: job 193 queued and waiting for resources The reason it gives for pending is just "Resources": $ scontrol show job 193 JobId=193 JobName=ls UserId=rpcruz(1000) GroupId=rpcruz(1000) MCS_label=N/A Priority=1 Nice=0 Account=account QOS=normal * JobState=PENDING Reason=Resources Dependency=(null)* Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2024-06-28T05:36:51 EligibleTime=2024-06-28T05:36:51 AccrueTime=2024-06-28T05:36:51 StartTime=2024-06-29T18:13:22 EndTime=2024-07-01T18:13:22 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-06-28T05:37:20 Scheduler=Backfill:* Partition=partition AllocNode:Sid=localhost:47757 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=1031887M,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=ls WorkDir=/home/rpcruz Power= * TresPerNode=gres/shard:2* Again, I think I have configured it properly - it shows up correctly in scontrol (as shown above). Our setup is pretty simple - I just added shard to /etc/slurm/slurm.conf: GresTypes=gpu,shard NodeName=compute01 Gres=gpu:8,shard:32 [truncated] Our /etc/slurm/gres.conf is also straight-forward: (it works fine for --gres=gpu:1) Name=gpu File=/dev/nvidia[0-7] Name=shard Count=32 Maybe I am just running srun improperly? Shouldn't it just be srun --gres= shard:2 to allocate half of a GPU? (since I am using 32 shards for the 8 gpus, so it's 4 shards per gpu) Thank you very much for your attention, -- Ricardo Cruz - https://rpmcruz.github.io

5 7

2025

2024

slurm-users ----- 2025 ----- July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users