July 2024 - slurm-users - lists.schedmd.com

cgroups/v2 plugin rpmbuild issue
by Chris Taylor 06 Jul '24

06 Jul '24

Trying to use rpmbuild on Rocky9 Linux, Slurm 21.08 - I want to build with cgroups/v2 support and have these installed: libbpf-devel.x86_64 2:1.3.0-2.el9 dbus-devel.x86_64 1:1.12.20-8.el9 kernel-headers.x86_64 5.14.0-427.22.1.el9_4 hwloc-devel.x86_64 2.4.1-5.el9 In https://slurm.schedmd.com/cgroup_v2.html it says: Requirements For building cgroup/v2 there are two required libraries checked at configure time... Look at your config.log when configuring to see if they were correctly detected on your system. I don't see any mention of 'ebpf', 'bpf', or 'dbus' when I grep through ~/rpmbuild/BUILD/slurm-21.08.8-2/config.log. Before I noticed that, I installed the RPMS on my cluster and get cgroup errors when I try to run a job or get an allocation. The slurmctld and slurmd processes start up fine on the servers, and I have a generic /etc/slurm/cgroup.conf. It seems like the cgroups plugin isn't getting built right with rpmbuild- how do I troubleshoot? Thanks Chris

2 3

Qlustar HPC Core Stack 24.06
by Roland Fehrenbacher 06 Jul '24

06 Jul '24

Hi all, the Qlustar HPC Core Stack 24.06 update is available. It includes an update to Slurm 23.11.8. Find further details at https://qlustar.com/news/2024/2024-06-29-hpc-core-stack-2406 Enjoy, Roland ------- https://qlustar.com -- 100% Open Source HPC / AI / Storage / Cloud Linux Cluster OS --

1 0

Run a program via Strigger when a node joins the cluster
by Karri Vrkreddy 05 Jul '24

05 Jul '24

Hi, We have a requirement to run a specific program whenever any new node joins the slurm cluster. For this, we have tried using strigger with the following options : "strigger --set --node --up --flags=perm --program=<program>" We see that the trigger is not getting activated always. Checked slurmctld logs for "type=node:up" and noticed it is not always happening. Based on documentation at https://slurm.schedmd.com/strigger.html , "up" flag would mean Trigger an event if the specified node is returned to service from a DOWN state 1. From Slurm Workload Manager - Slurm Power Saving Guide , and the above message does it mean that a node has to be already in DOWN state in the cluster for a trigger with "--up" to get activated ? 2. Are there any other ways beyond strigger to achieve this ? | | | | | | | | | | | Slurm Workload Manager - Slurm Power Saving Guide | | | -- Thanks Karri

1 0

Using sharding
by Ricardo Cruz 05 Jul '24

05 Jul '24

Greetings, There are not many questions regarding GPU sharding here, and I am unsure if I am using it correctly... I have configured it according to the instructions <https://slurm.schedmd.com/gres.html>, and it seems to be configured properly: $ scontrol show node compute01 NodeName=compute01 Arch=x86_64 CoresPerSocket=32 CPUAlloc=48 CPUEfctv=128 CPUTot=128 CPULoad=10.95 AvailableFeatures=(null) ActiveFeatures=(null) * Gres=gpu:8,shard:32* [truncated] When running with gres:gpu everything works perfectly: $ /usr/bin/srun --gres=gpu:2 ls srun: job 192 queued and waiting for resources srun: job 192 has been allocated resources (...) However, when using sharding, it just stays waiting indefinitely: $ /usr/bin/srun --gres=shard:2 ls srun: job 193 queued and waiting for resources The reason it gives for pending is just "Resources": $ scontrol show job 193 JobId=193 JobName=ls UserId=rpcruz(1000) GroupId=rpcruz(1000) MCS_label=N/A Priority=1 Nice=0 Account=account QOS=normal * JobState=PENDING Reason=Resources Dependency=(null)* Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2024-06-28T05:36:51 EligibleTime=2024-06-28T05:36:51 AccrueTime=2024-06-28T05:36:51 StartTime=2024-06-29T18:13:22 EndTime=2024-07-01T18:13:22 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-06-28T05:37:20 Scheduler=Backfill:* Partition=partition AllocNode:Sid=localhost:47757 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=1031887M,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=ls WorkDir=/home/rpcruz Power= * TresPerNode=gres/shard:2* Again, I think I have configured it properly - it shows up correctly in scontrol (as shown above). Our setup is pretty simple - I just added shard to /etc/slurm/slurm.conf: GresTypes=gpu,shard NodeName=compute01 Gres=gpu:8,shard:32 [truncated] Our /etc/slurm/gres.conf is also straight-forward: (it works fine for --gres=gpu:1) Name=gpu File=/dev/nvidia[0-7] Name=shard Count=32 Maybe I am just running srun improperly? Shouldn't it just be srun --gres= shard:2 to allocate half of a GPU? (since I am using 32 shards for the 8 gpus, so it's 4 shards per gpu) Thank you very much for your attention, -- Ricardo Cruz - https://rpmcruz.github.io

5 7

Re: Slurm commands fail when run in Singularity container with the error "Invalid user for SlurmUser slurm, SINGULARITYENV_SLURM_CONF
by Robert Kudyba 03 Jul '24

03 Jul '24

In https://support.schedmd.com/show_bug.cgi?id=9282#c6 Tim mentioned this env variable SINGULARITYENV_SLURM_CONF, what is the usage/syntax for it? I can't find any reference to this. I'm running into the same issue mentioned there. Thanks in advance!

2 2

2024

slurm-users July 2024 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users July 2024