- slurm-users - lists.schedmd.com

Shards distribution over multiple GPUs
by Francois Broquedis 12 Mar '24

12 Mar '24

Hi all, We're trying to enable sharding on our compute cluster. On this cluster: - ensicompute-1 comes with 1 NVIDIA V100 GPU ; - ensicompute-13 comes with 3 NVIDIA A40 GPUs ; - all other nodes (for now, ensicompute-11 and ensicompute-12, but several others will come) come with 3 NVIDIA RTX 6000 GPUs. To enable sharding, I followed these steps: 1. [slurm.conf] Add "shard" to GresTypes ; 2. [slurm.conf] Add "shard:N" to Gres for each node. For testing purposes, I have set N to 9, so each GPU can execute up to 3 jobs concurrently: NodeName=ensicompute-[11-12] Gres=gpu:Quadro:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht 3. [gres.conf] Declare the shards after the definition of the GPUs GRES. For step 3, I tried different things, leading to different outcomes: a. Define a global number of shards, for the entire host: Name=shard Count=9 ==> This way, sharding seems to work ok, but all the jobs are executed on GPU#0. If running 12 jobs for example, 9 of them are assigned to GPU#0 and start executing, while 3 of them remain in a pending state. No job is assigned to GPU#1 or GPU#2. b. Define a per-GPU number of shards, associated to the device file representing the GPU: Name=shard Count=3 File=/dev/nvidia0 Name=shard Count=3 File=/dev/nvidia1 Name=shard Count=3 File=/dev/nvidia2 ==> In this case, the slurmd service fails to start on the compute node. The error message found in /var/log/slurmd.log is "fatal: Invalid GRES record for shard, count does not match File value". c. Don't define anything about shards in gres.conf. ==> Same behavior than in a.: all jobs are executed on GPU#0. I attach to this message the full content of the slurm.conf and gres.conf files. What is the proper way to configure sharding in a cluster with several GPUs per node? Is there a way to specify how many shards should be allocated to each GPU? Cheers, François === slurm.conf === # slurm.conf file generated by configurator.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=ensimag SlurmctldHost=nash ProctrackType=proctrack/cgroup SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup ReturnToService=2 # # # TIMERS InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES GresTypes=gpu,shard NodeName=ensicompute-1 Gres=gpu:Tesla:1,shard:3 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht NodeName=ensicompute-13 Gres=gpu:A40:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht NodeName=ensicompute-[11-12] Gres=gpu:Quadro:3,shard:9 CPUs=40 RealMemory=128520 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Feature=gpu,ht PartitionName=compute Nodes=ALL Default=YES MaxTime=INFINITE State=UP === gres.conf === AutoDetect=off # ensicompute-1 NodeName=ensicompute-1 Name=gpu Type=Tesla File=/dev/nvidia0 NodeName=ensicompute-1 Name=shard Count=3 File=/dev/nvidia0 # ensicompute-11 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia0 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia1 NodeName=ensicompute-11 Name=gpu Type=Quadro File=/dev/nvidia2 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-11 Name=shard Count=3 File=/dev/nvidia2 # ensicompute-12 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia0 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia1 NodeName=ensicompute-12 Name=gpu Type=Quadro File=/dev/nvidia2 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-12 Name=shard Count=3 File=/dev/nvidia2 # ensicompute-13 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia0 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia1 NodeName=ensicompute-13 Name=gpu Type=A40 File=/dev/nvidia2 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia0 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia1 NodeName=ensicompute-13 Name=shard Count=3 File=/dev/nvidia2 -- François Broquedis, Ingénieur Service Informatique Grenoble INP - Ensimag, bureau E208 681 rue de la Passerelle BP 72, 38402 Saint Martin d'Hères CEDEX Tél.: +33 (0)4 76 82 72 78

1 0

Re: user_name attribute missing in job response from slurm rest api method jobs
by Acartürk, Ayhan 11 Mar '24

11 Mar '24

Sorry... my previous message should have been a reply to the post by Filip Holka from March, 25th 2022: https://lists.schedmd.com/pipermail/slurm-users/2022-March/008543.html Best wishes, Ayhan Acartürk

1 0

Re: user_name attribute missing in job response from slurm rest api method jobs
by Acartürk, Ayhan 11 Mar '24

11 Mar '24

Hello together, we have the same problem here. All jobs delivered by the api endpoint /slurm/v0.0.36/jobs have a user_id attached but no user_name. Currently, we solve this issue by an extra call for each job_id to the api endpoint /slurmdb/v.0.0.36/job/{job_id}. Any idea where to search for a possible solution is highly appreciated. Best wishes, Ayhan Acartürk

1 0

Problem building slurm with PMIx
by Patrick Begou 11 Mar '24

11 Mar '24

Hi ! I manage a small CentOS8 cluster using slurm slurm-20.11.7-1 and OpenMPI built from sources. - I know this OS is not maintained any more and I need to negotiate downtime to reinstall - I know Slurm 20.11.7 has security issue (I've built it from source some years ago with rpmbuild -ta --with mysql --with hwloc slurm-20.11.7.tar.bz) and I should update. All was running fine until I add a GPU Node and Nvidia sdk. This SDK provides an openMPI3 implementation GPU aware but I'm unable to launch an intranode parallel job with it using srun: -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using: version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix. Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location. Please configure as appropriate and try again. -------------------------------------------------------------------------- I check with "srun --mpi=list" and got no pmx. srun: MPI types are... srun: pmi2 srun: cray_shasta srun: none So I decide to build the rpms from slurm-20.11.9.tar.bz2 as I had done previously for 20.11.7 and update. I've first installed pmix-2.1.1-1 from src as I had no pmix-devel rpm in my local CentOS8 repo: rpm --rebuild pmix-2.1.1-1.el8.src.rpm dnf install pmix-devel-2.1.1-1.el8.x86_64.rpm pmix-2.1.1-1.el8.x86_64.rpm Then build Slurm from slurm-20.11.9.tar.bz2 (just changing python3 to python38 in the spec file) rpmbuild -ta --with mysql --with hwloc --with pmix slurm-20.11.9.tar.bz2 And the try to install these package on the GPU node dnf install slurm-slurmd-20.11.9-1.el8.x86_64.rpm slurm-20.11.9-1.el8.x86_64.rpm slurm-devel-20.11.9-1.el8.x86_64.rpm slurm-libpmi-20.11.9-1.el8.x86_64.rpm But I get this strange error: Error: Problem: conflicting requests - nothing provides pmix = 20.11.9 needed by slurm-slurmd-20.11.9-1.el8.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Why this request on PMIX with the slurm version number ? Am I wrong somewhere ? Thanks for your help Patrick

2 1

FairShare value is always 0.
by Zacarias Benta 07 Mar '24

07 Mar '24

Hi guys, We've just setup our new cluster and are facing some issues regading fairshare calculation. Our slurm directive regarding priority calculation are defines as follows: PriorityType=priority/multifactor PriorityFlags=MAX_TRES PriorityDecayHalfLife=14-0 PriorityFavorSmall=NO PriorityMaxAge=14-0 PriorityWeightAge=1000 PriorityWeightJobSize=1000 PriorityWeightPartition=10000000 PriorityWeightQOS=10000000 PriorityWeightTRES=CPU=2000,Mem=4000 PriorityWeightFairshare=100000 The partition we are submitinh out jobs to is setup as follows: PartitionName=mypartPriority=1000TRESBillingWeights="CPU=1.0,Mem=0.25G"Default=YESMaxTime=96:0:0DefMemPerCPU=5333Nodes=node[001-036] MaxNodes=20 Whenever we take a look at the fairshare value using sshare -l we see the following output: AccountUserRawSharesNormSharesRawUsageNormUsageEffectvUsageFairShareLevelFSGrpTRESMinsTRESRunMins ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- root10.0000002687245970.0000000.000000cpu=1098201,mem=5856709132,en+ rootroot10.10000000.0000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group110.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group210.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group310.1000002687245970.0000000.000000cpu=1098201,mem=5856709132,en+ group410.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group510.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group610.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group710.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group810.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ group910.10000000.0000000.000000cpu=0,mem=0,energy=0,node=0,b+ We think it is really weird that the FairShare value is 0 for the root account and "NULL" for all other groups, even the one who had the greatest raw usage. While taking a look at the data for our users we see the following: AccountUserRawSharesNormSharesRawUsageEffectvUsageFairShare ------------------------------------------------------------------------------------- root10.0000002689837210.000000 rootroot10.10000000.0000000.000000 group310.1000002689837210.000000 group3user110.090909121093740.0000000.000000 group3user210.09090900.0000000.000000 group3user310.09090900.0000000.000000 group3user410.09090900.0000000.000000 group3user510.09090900.0000000.000000 group3user610.09090900.0000000.000000 group3user710.09090900.0000000.000000 group3user810.0909092088245970.0000000.000000 group3user910.09090900.0000000.000000 group3user1010.09090900.0000000.000000 group3user1110.090909480497500.0000000.000000 group410.10000000.000000 group4user1310.0000004994520.0000000.000000 group510.10000000.000000 group5user1410.00000015396030.0000000.000000 This is a weird behavior, since user1, user8, user11, user13 and user14 are the ones who have more RawUsage and the FairShare value is the same for all of them, including the users that have no yet submited any job. We also noticed that in the slurmctld log there is the fillowing error message that appears with some regularity [2024-03-07T16:38:13.260] error: _append_list_to_array: unable to append NULL list to assoc list. [2024-03-07T16:38:13.260] error: _calc_tree_fs: unable to calculate fairshare on empty tree The error above looks like it is coming from: https://github.com/SchedMD/slurm/blob/b11bf689b270f1f5dfe4b0cd54c4fa84b4af3… Are we missing any setting on slurm.conf? This is kind of strange, because we have another cluster with pretty much the same configuration and the FairShare is calculated without any problems. Any help would be appreciated. -- Cumprimentos / Best Regards, Zacarias Benta LIP/INCD @ UMINHO ---------------------------------------------- / Use linux, and may the source be with you. / ---------------------------------------------- \ __ -=(o '. '.-.\ /| \\ '| || _\_):,_

1 0

Slurm management of dual-node server trays?
by Ole Holm Nielsen 07 Mar '24

07 Mar '24

We're in the process of installing some racks with Lenovo SD665 V3 [1] water-cooled servers. A Lenovo DW612S chassis contains 6 1U trays with 2 SD665 V3 servers mounted side-by-side in each tray. Lenovo delivers SD665 V3 servers including water-cooled NVIDIA InfiniBand "SharedIO" adapters [2] so that one node is the Primary including a PCIe adapter, and the other is Auxiliary with just a cable to the Primary's adapter. Obviously, servicing 2 "Siamese twin" Slurm nodes requires a bit of care and planning. What is worse is that when the Primary node is rebooted or powered down, the Auxiliary node will lose its Infiniband connection and may have a PCIe fault or an NMI as documented in [3]. And when nodes are powered up, the Primary must have completed POST before the Auxiliary gets started. I wonder how to best deal with power failures? It seems that when Slurm jobs are running on Auxiliary nodes, these jobs are going to crash when the possibly unrelated Primary node goes down. This looks like a pretty bad system design on the part of Lenovo :-( The goal was apparently to same some money on IB adapters and having fewer IB cables. Question: Do any Slurm sites out there already have experiences with Lenovo "Siamese twin" nodes with SharedIO IB? Have you developed some operational strategies, for example dealing with node pairs as a single entity for job scheduling? Thanks for sharing any ideas and insights! Ole [1] https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server [2] https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-… [3] https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-c… -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

3 3

Re: Lua script
by Paul Raines 06 Mar '24

06 Mar '24

Alternativelly consider setting EnforcePartLimits=ALL in slurm.conf -- Paul Raines (http://help.nmr.mgh.harvard.edu) The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Re: Lua script
by Gestió Servidors 06 Mar '24

06 Mar '24

And how can I reject the job inside the lua script?

2 1

Lua script
by Gestió Servidors 06 Mar '24

06 Mar '24

Hello, I'm writing a small lua script that for modify "TimeLimit" of a submited job if user has configured a TimeLimit bigger that configured in the partition. So, is TimeLimit for partition is, for example, 4 hours (04:00:00) and user submit his/her job with a TimeLimit of 5 hours, lua script modify this submit and forces "TimeLimit" to 4 hours for avoiding that job remains at queue in state "pending" with reason "Partition Time Limit...". I have written these lines: function slurm_job_submit(job_desc, part_list, submit_uid) --[[ my id is 1008; for testing purposes, lua only applies to me --]] if (job_desc.user_id == 1008) then slurm.log_info("Submit job by me") --[[ I check partition --]] if (job_desc['partition'] == "nodo.q") then --[[ if (job_desc['time_limit'] > 14400) job_desc['time_limit'] = 14400 end end end return slurm.SUCCESS end However, if I submit a job with TimeLimit of 5 hours, lua script doesn't modify submit and job remains "pending"... What am I doing wrong? Thanks.

2 1

Slurm billback and sreport
by Chip Seraphine 05 Mar '24

05 Mar '24

Hello, I am attempting to implement a billback model and finding myself stymied by the way that sreport handles job arrays. Basically, when a user submits a large array, their usage includes time that jobs in the back of the array spend waiting their turn. (My #1 user in “sreport user topusage” shows more “used” cpu*minutes than the cluster physically _has_ during that interval.) However, jobs that are idle pending resources are simply regarded as pending; as a result, a “polite” user who submits an array of 1000 jobs running N at a time is penalized over a user who just dumps 1000 loose jobs into the queue. This incentives my users to do exactly what I do not want! Has anyone tried to bill their users based on the results of sreport? If so, how did you work around this problem? What did you use to determine the # of CPU*Minutes that a user actually allocated on during a given interval? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

3 3

Is SWAP memory mandatory for SLURM
by John Joseph 04 Mar '24

04 Mar '24

Dear All, Good morning I do have a 4 node SLURM instance up and running. Like to know if I disable the SWAP memory, will it effect the SLURM performance Is SWAP a mandatory requirement, I have each node more RAM, if my phsicall RAM is more, is there any need for the SWAP thanks Joseph John

4 3

What is "eligible time"
by Oren Shani 04 Mar '24

04 Mar '24

Hi All, Can someone shed some light on the exact meaning of the "eligible time" attribute in the slurm accounter, or to be more exact on the time between when the jobs becomes "eligible" to the time it actually starts to run? I understand that "eligible time" refers to the time in which the job becomes a candidate for running and that one case in which it will still not run, is that it waits for resources to become available. But are there any other conditions for a job to keep waiting at this point? For example, is fair-share scheduling considered only at this point? Many thanks, Oren

1 0

Question about IB and Ethernet networks
by Dan Healy 04 Mar '24

04 Mar '24

Hi Fellow Slurm Users, This question is not slurm-specific, but it might develop into that. My question relates to understanding how *typical* HPCs are designed in terms of networking. To start, is it typical for there to be a high speed Ethernet *and* Infiniband networks (meaning separate switches, NICs)? I know you can easily setup IP over IB, but is IB usually fully reserved for MPI messages? I’m tempted to spec all new HPCs with only a high speed (200Gbps) IB network, and use IPoIB for all slurm comms with compute nodes. I plan on using BeeGFS for the file system with RDMA. Just looking for some feedback, please. Is this OK? Is there a better way? If yes, please share why it’s better. Thanks, Daniel Healy

6 6

How to get usage data for a QOS
by thomas.hartmann＠th-ht.de 01 Mar '24

01 Mar '24

Hi, so, I figured out that I can give some users priority access for a specific amount of TRES by creating a qos with the GrpTRESMins property and the DenyOnLimit,NoDecay flags. This works nicely. However, I would like to know, how much of this has already been consumed and I have not yet found a way to do this. Like: How can I get the amount of TRES/TRES minutes consumed for a certain QOS? Thanks a lot! Thomas

3 2

MinTRES in QoS and power saving
by Stefan Walter 01 Mar '24

01 Mar '24

I am running a GPU cluster where nodes are mostly off to save on electricity. I have run into the problem that if I set 'MinTRES=gres/gpu=1' in the QoS for user-account associations, waking up nodes on-demand stops working for these users. Jobs are allocated on all running nodes but if a user starts a job that would require a node to be woken up, it remains pending with 'QOSMinGRES'. In the verbose logs, such a job will show this: debug3: TRES Weight: cpu = 1.000000 * 0.000000 = 0.000000 debug3: TRES Weight: mem = 6000.000000 * 0.000000 = 0.000000 debug3: TRES Weight: energy = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: node = 1.000000 * 0.000000 = 0.000000 debug3: TRES Weight: fs/disk = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: vmem = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: pages = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpu = 0.000000 * 1.000000 = 0.000000 debug3: TRES Weight: gres/gpu:nvidia_geforce_gtx_1080ti = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpumem = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpuutil = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weighted: SUM(TRES) = 0.000000 debug2: JobId=4256 is being held, QOS local818-test min tres(gres/gpu) per job request 0 exceeds min tres limit 1 The job was started with '-G 1' or '--gres=gpu:1' but notably the TRES value for 'gres/gpu' becomes 0. For jobs that fit on the running nodes the TRES weight remains correct: debug3: TRES Weight: gres/gpu = 1.000000 * 1.000000 = 1.000000 The quick fix was to unset MinTRES for all QoS but I still would force user in some accounts to use GPUs because that is what is billed for them. Is this by design? I was not successful to find any hint that this is expected behavior. Thanks, Stefan

1 0

pty jobs are killed when another job on the same node terminates
by Thomas Hartmann 01 Mar '24

01 Mar '24

Hi, when I start an interactive job like this: srun --pty --mem=3G -c2 bash And then I schedule and run other jobs (can be interactive or non interactive) and one of these jobs that runs on the same node terminates, the interactive job gets killed with this message: srun: error: node01.abc.at: task 0: Killed I attached our slurm config. Does anybody have an idea what is going on here or where I could look to debug? I'm quite new to slurm, so I don't know all the places to look... Thanks a lot in advance! Thomas

4 3

GPU shards not exclusive
by Reed Dier 29 Feb '24

29 Feb '24

I seem to have run into an edge case where I’m able to oversubscribe a specific subset of GPUs on one host in particular. Slurm 22.05.8 Ubuntu 20.04 cgroups v1 (ProctrackType=proctrack/cgroup) It seems to be partly a corner case with a couple of caveats. This host has 2 different GPU types in the same host. All GPUs are configured with 6 shards/gpu. > NodeName=gpu03 CPUs=88 RealMemory=768000 Sockets=2 CoresPerSocket=22 ThreadsPerCore=2 State=UNKNOWN Feature=avx512 Gres=gpu:rtx:2,gpu:p40:6,shard:48 My gres.conf is auto-nvml, with nothing else in the file. > AutoDetect=nvml Schedule and select from slurm.conf > SchedulerType=sched/backfill > SelectType=select/cons_tres > SelectTypeParameters=CR_CPU_Memory > GresTypes=gpu,shard Here’s a copy of slurmd’s log starting up with debug logging: https://pastebin.com/fbriTAZD <https://pastebin.com/fbriTAZD> I checked there to make sure that it was allocating an even 6 shards per GPU as intended, and it did. The only thing odd about the rtx cards is that they have an NVLink bridge between them, but I wouldn’t expect that to have any impact with respect to shards. So far it only appears to manifest if anything is using the named gpu (--gres gpu:rtx:1), but not as a general gres (--gres gpu:1). It also doesn’t occur with any count of named p40 gres either, only with the rtx gres. The below was as “high” as I was able to capture it oversubscribed in very quick testing. You can see there are 2 rtx and 43 shards allocated, which should be an impossible number, if we correctly assume an rtx==6 shards, and with 48 total shards, that should reduce the available pool of shards to 36 shards. > NodeName=gpu03 Arch=x86_64 CoresPerSocket=22 > CPUAlloc=88 CPUEfctv=88 CPUTot=88 CPULoad=0.27 > AvailableFeatures=avx512 > ActiveFeatures=avx512 > Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8 > OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 > RealMemory=768000 AllocMem=45056 FreeMem=563310 Sockets=2 Boards=1 > State=ALLOCATED+RESERVED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A > Partitions=gpu,gpu-prod > BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-14T16:28:17 > LastBusyTime=2024-02-14T16:38:03 > CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48 > AllocTRES=cpu=88,mem=44G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=43 > CapWatts=n/a > CurrentWatts=0 AveWatts=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s The above is 1 rtx:2 jobs, and 43 shard:1 jobs, which is as many as I can seem to push through with the given cpu cores on the system, I expect if there were 98-100 cores I could completely oversubscribe it. If I do (1) rtx:2 + (24) shard:2 it actually limits it correctly to 36 shards, which is odd because the case that caused us to discover this was when 2 rtx:1 jobs were running for multiple days, and somehow shards got scheduled on those gpus. > AllocTRES=cpu=40,mem=20G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=36 If I flip it around to shards before rtx, then I’m unable to oversubscribe it. If I add any P40 jobs before the rtx and then shard jobs, also unable to oversubscribe. This sounded familiar to Bug 16484 <https://bugs.schedmd.com/show_bug.cgi?id=16484> although they claimed it was resolved by using auto-nvml which would preclude this node. Here’s my very simple bash script that just srun's a sleep 20 so that it runs long enough to observe it. > #!/bin/bash > #rtx 2 > for((i=1;i<=1;++i)) ; > do > srun --gres gpu:rtx:2 --reservation=shardtest -J "RTX $i" sleep 20 & > done > sleep 1 > #shard 48 > for((i=1;i<=43;++i)) ; > do > srun --gres shard:1 --reservation=shardtest -J "shard $i" sleep 20 & > done > sleep 5 ; scontrol show node gpu03 Curious if anyone has any ideas beyond try a newer release? Thanks, Reed

2 2

sbatch and cgroup v2
by Dietmar Rieder 29 Feb '24

29 Feb '24

Hi, I'm new to slrum, but maybe someone can help me: I'm trying to restrict the CPU usage to the actually requested/allocated resources using cgroup v2. For this I made the following settings in slurmd.conf: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity And in cgroup.conf CgroupPlugin=cgroup/v2 CgroupAutomount=yes ConstrainCores=yes ConstrainRAMSpace=yes ConstrainDevices=yes AllowedRAMSpace=98 cgroup v2 seems to be active on the compute node: # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) # cat /sys/fs/cgroup/cgroup.subtree_control cpuset cpu io memory pids # cat /sys/fs/cgroup/system.slice/cgroup.subtree_control cpuset cpu io memory pids Now, when I use sbatch to submit the following test script, the python script which is started from the batch script is utilizing all CPUs (96) at 100% on the allocated node, although I only ask for 4 cpus (--cpus-per-task=4). I'd expect that the task can not use more that these 4. #!/bin/bash #SBATCH --output=/local/users/appadmin/test-%j.log #SBATCH --job-name=test #SBATCH --chdir=/local/users/appadmin #SBATCH --cpus-per-task=4 #SBATCH --ntasks=1 #SBATCH --nodes=1 #SBATCH --mem=64gb #SBATCH --time=4:00:00 #SBATCH --partition=standard #SBATCH --gpus=0 #SBATCH --export #SBATCH --get-user-env=L export PATH=/usr/local/bioinf/jupyterhub/bin:/usr/local/bioinf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/bioinf/miniforge/condabin source .bashrc conda activate test python test.py The python code in test.py is the following using the cpu_load_generator package from [1]: #!/usr/bin/env python import sys from cpu_load_generator import load_single_core, load_all_cores, from_profile load_all_cores(duration_s=120, target_load=1) # generates load on all cores Interestingly, when I use srun to launch an interactive job, and run the python script manually, I see with top that only 4 cpus are running at 100%. And I also python errors thrown when the script tries to start the 5th process (which makes sense): File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/cpu_load_generator/_interface.py", line 24, in load_single_core process.cpu_affinity([core_num]) File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/psutil/__init__.py", line 867, in cpu_affinity self._proc.cpu_affinity_set(list(set(cpus))) File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/psutil/_pslinux.py", line 1714, in wrapper return fun(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/psutil/_pslinux.py", line 2213, in cpu_affinity_set cext.proc_cpu_affinity_set(self.pid, cpus) OSError: [Errno 22] Invalid argument What am I missing, why are the CPU resources not restricted when I use sbatch? Thanks for any input or hint Dietmar [1]: https://pypi.org/project/cpu-load-generator/

3 7

slurmdbd error - Symbol `slurm_conf' has different size in shared object
by Miriam Olmi 29 Feb '24

29 Feb '24

Hi all, I am having some issue with the new version of slurm 23.11.0-1. I had already installed and configured slurm 23.02.3-1 on my cluster and all the services were active and running properly. Following the instructions of the official SLURM webpage, for the moment I upgrated only the slurmdbd service. In principle the cluster should be able to work properly if the slurmdbd has a higher version with respect to slurmctld and slurmd. Unfortunately the slurmdbd service fails to start with the following status: slurmdbd.service - Slurm DBD accounting daemon Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; vendor preset: disabled) Active: failed (Result: core-dump) since Wed 2024-02-28 10:05:53 CET; 9min ago Process: 534938 ExecStart=/usr/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS (code=dumped, signal=SEGV) Main PID: 534938 (code=dumped, signal=SEGV) Feb 28 10:05:53 slurm-db systemd[1]: Started Slurm DBD accounting daemon. Feb 28 10:05:53 slurm-db slurmdbd[534938]: /usr/sbin/slurmdbd: Symbol `slurm_conf' has different size in shared object, consider re-linking Feb 28 10:05:53 slurm-db systemd[1]: slurmdbd.service: Main process exited, code=dumped, status=11/SEGV Feb 28 10:05:53 slurm-db systemd[1]: slurmdbd.service: Failed with result 'core-dump'. Can anyone help me? Thanks in advance, Miriam

2 4

salloc+srun vs just srun
by wdennis＠nec-labs.com 29 Feb '24

29 Feb '24

Hi list, In our institution, our instructions to users who want to spawn an interactive job (for us, a bash shell) have always been to do "srun ..." from the login node, which has always been working well for us. But when we had a recent Slurm training, the SchedMD folks advised us to use "salloc" and then "srun" to do interactive jobs. I tried this today, "salloc" gave me a shell on a server, the same as srun does, but then when I tried to "srun [programname]" it hung there with no output. Of course when I tried "srun [programname] &" it spawned the background job, and gave me back a prompt. Either time I had to Ctrl-C the running srun job, and got no output other than the srun/slurmstepd termination output. I think I read somewhere that directly invoking srun creates an allocation; why then would I want to do an initial salloc, and then srun? (i the case that I want a foreground program, such as a bash shell) I have surveyed some other institution's Slurm interactive jobs documentation for users, I see both examples of advice to run srun directly, or salloc and then srun. Please help me to understand how this is intended to work, and if we are "doing it wrong" :) Thanks, Will

3 4

canonical way to run longer shell/bash interactive job (instead of srun inside of screen/tmux at front-end)?
by Josef Dvoracek 28 Feb '24

28 Feb '24

What is the recommended way to run longer interactive job at your systems? Our how-to includes starting screen at front-end node and running srun with bash/zsh inside, but that indeed brings dependency between login node (with screen) and the compute node job. On systems with multiple front-ends users need to remember the login node where they have their screen session.. Are you anybody using something more advanced and still understandable by casual user of HPC? (I know Open On Demand, but often the use of native console has certain benefits. ) cheers josef

7 9

Enforcing relative resource restrictions in submission script
by Matthew R. Baney 28 Feb '24

28 Feb '24

Hello Slurm users, I'm trying to write a check in our job_submit.lua script that enforces relative resource requirements such as disallowing more than 4 CPUs or 48GB of memory per GPU. The QOS itself has a MaxTRESPerJob of cpu=32,gres/gpu=8,mem=384G (roughly one full node), but we're looking to prevent jobs from "stranding" GPUs, e.g., a 32 CPU/384GB memory job with only 1 GPU. I might be missing something obvious, but the rabbit hole I'm going down at the moment is trying to check all of the different ways job arguments could be set in the job descriptor. i.e., the following should all be disallowed: srun --gres=gpu:1 --mem=49G ... (tres_per_node, mem_per_node set in the descriptor) srun --gpus=1 --mem-per-gpu=49G ... (tres_per_job, mem_per_tres) srun --gres=gpu:1 --ntasks-per-gpu=5 ... (tres_per_node, num_tasks, ntasks_per_tres) srun --gpus=1 --ntasks=2 --mem-per-cpu=25G ... (tres_per_job, num_tasks, mem_per_cpu) ... Essentially what I'm looking for is a way to access the ReqTRES string from the job record before it exists, and then run some logic against that i.e., if (CPU count / GPU count) > 4 or (mem count / GPU count) > 48G, error out. Is something like this possible? Thanks, Matthew -- Matthew Baney Assistant Director of Computational Systems mbaney(a)umd.edu | (301) 405-6756 University of Maryland Institute for Advanced Computer Studies 3154 Brendan Iribe Center 8125 Paint Branch Dr. College Park, MD 20742

2 1

User-facing documentation on shard use
by wdennis＠nec-labs.com 28 Feb '24

28 Feb '24

Hello list, We have just enabled "gres/shard" in order to enable sharing of GPUs on our cluster. I am now looking for examples of user-facing documentation on this feature. If anyone has something, and can send a URL or other example, I'd appreciate it. Thanks, Will

1 0

Partition, Qos Limits & Scheduling of large jobs
by Muck, Katrin 28 Feb '24

28 Feb '24

Hi everyone! I read the slurm documentation about qos, resource limits, scheduling and priority now multiple times and even looked into the slurm source but I'm still not sure if I got everything correctly, so this is why I decided to ask here ... The problem: we see the effect that sometimes larger jobs with e.g. 16 gpus in our (small) gpu queue get delayed and shifted back without any reason that is apparent to us. small jobs that only use e.g. 1 or 2 gpus get scheduled much quicker even though they have a runtime of 3 days ... What we want to do: - We have a number of nodes with 2x gpus that are usable by the users of our cluster - Some of these nodes belong to so called 'private projects'. Private projects have higher priority than other projects. Attached to that is a contingent of nodes & "guaranteed" nodes e.g. they could have a contingent of 4 nodes (8 gpus) and e.g. 2 "guaranteed" nodes (4 gpus) - Guaranteed nodes are nodes that should always be kept idle for the private project, so users of the private project can immediately schedule work on those nodes - The other nodes are shared with other projects in general if they are not "in use" How we are currently doing this (it has history): Lets assume we have 50 nodes and 100 gpus. - We have a single partition for all gpu nodes (e.g. 50 nodes) - Private projects have private queues with a very high priority and a gres limit of the number of gpus they reserved (e.g. 10 nodes -> 20 gpus) - Normal projects only have access to the public queue and schedule work there. - This public queue has an upper gres limit of "total number of gpus" - "guaranteed gpus of all private projects" (e.g. 50 - 10 nodes -> 40 nodes -> 80 gpus). Regarding the scheduler, we currently use the following settings: SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_CORE_MEMORY,CR_CORE_DEFAULT_DIST_BLOCK,CR_ONE_TASK_PER_CORE SchedulerParameters=defer,max_sched_time=4,default_queue_depth=1000,partition_job_depth=500,enable_user_top,bf_max_job_user=20,bf_interval=120,bf_window=4320,bf_resolution=1800,bf_continue Partition/queue depth is deliberately set high at the moment to avoid problems with jobs not even being examined. The problem in more detail: One of the last jobs (16 gpus needed) we diagnosed had a approximate start time that was beyond all end times of running/scheduled jobs - like jobs on feb 22 would release more than enough gpus so the job could have been immediately scheduled afterwards, but the start time was still feb 23. Priority wise the job had the highest priority of all pending jobs for the partition. When we turned on scheduler debugging and increased log levels, we observed the following messages for this job: JobId=xxxx being held, if allowed the job request will exceed QOS xxxxx group max tres(gres/gpu) limit yy with already used yy + requested 16 followed by sched: JobId=2796696 delayed for accounting policy So to us this meant that the scheduler was always hitting the qos limits, which makes sense because the usage is always very high in the gpu queue and thus the job wasn't scheduled ... At first we were worried, that this meant that "held"/"delayed" jobs like this would never actually get scheduled when contention is high enough e.g. small jobs getting backfilled in and thus qos limits stay at max for a long time. But for some reason we could not determine the job eventually got scheduled at one point and then ran at the scheduled start time. Open Questions: - why it couldn't be scheduled in the first place. initially we thought (from the source code i looked into) the "delayed for accounting policy" prevents further scheduling in general, but since it was scheduled this assumption must be wrong? - why it was scheduled at some point. when it was scheduled, contention was still high and the qos limits definitely still applied - how we could modify the current setup so that the scheduling of larger jobs becomes "better" and more reproducible/explainable Apart from all of this I'm also asking myself if there is maybe a better way to setup a system that works the way we want? This got a bit long but I hope its clear enough :) Kind regards, Katrin

1 0

"Optimal" slurm configuration
by Max Grönke 26 Feb '24

26 Feb '24

Hello! In our current cluster the workflows are quite diverse (bunch of large, long (24-72h) jobs; medium size <4h job; and many small 1 node jobs). The current priority is fair share only (averaged on a ~few months timescale). For the new setup we would like to (1) discourage the 1 node jobs [especially the long ones] (2) have a fair use of the machine for the different cases. In particular, we found that the waiting time / runtime ratio is very large for the ~1-4h jobs. For this reason, I have been trying to read up on strategies in terms of partition / priority politics but could not find any good documentation in this regard (as https://slurm.schedmd.com/documentation.html is more focused on the technical aspects). Does anyone know any good references for this? In particular, we were thinking about (a) somehow have a long-term as well as shorter term fairshare count, i.e., lowering the priority if a user has something running already or (b) introduce a "small" partition for the <4h jobs with higher priority but we're unsure if this will block all the larger jobs to run... Any help or reading material is highly appreciated. Many thanks, Max

2 1

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users