- slurm-users - lists.schedmd.com

Could not find group with gid even when they exist
by Nic Lewis 09 Feb '24

09 Feb '24

After upgrading to version 23.11.3 we started to get slammed with the following log messages from slurmctld "error: validate_group: Could not find group with gid <id>" This spans a handful of groups and repeats constantly, drowning out just about everything else. Attempting to do a lookup on the group shows that they exist on the scheduler node, same for all the submission and compute nodes. As far as I can tell, slurm should be able to locate the group in question. Jobs submitted from users within those groups go through just fine. They get scheduled, run, and clean up no problem. I'm at a loss on where to look next.

1 1

Relative QOS limits in sacctmgr?
by Chip Seraphine 09 Feb '24

09 Feb '24

Hello, TL,DR: How does the relative QOS flag work? I have a QOS and I want it to be collectively restricted to 50% of the reachable cores in the cluster. I’ve been managing this by dividing my core count to 2 to get N, and doing ‘sacctmgr update qos foobar set MaxTRES=cpu=N’. That’s fine, except that N changes frequently as worker nodes come and go, so I have an hourly cronjob that enforces this. The problem with *that* is the values you back out with ‘sacctmgr show qos’ are often in different units than you input (e.g. gigs instead of megs of ram), which means I need to do some annoying parsing, will miss minor rounding errors, etc. (Using JSON solves that problem but introduces others that I do not want to deal with.) So while looking for a better way I noticed that QOSs support a flag called “relative”, which says “the QOS limits will be treated as percentages of a cluster/partition instead of absolutes”. Perfect! However, it’s not very clear to me what that means in practice. For example, with the “relative” flag set, the MaxTRES units still seem to work the same, expect metric suffices (“G”, ”M”, etc) and won’t accept periods. Is anyone actually using this flag, and if so can you tell me what attributes it affects, and in what way? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

Is there a way to list allocated/unallocated resources defined in a QoS?
by Alastair Neil 09 Feb '24

09 Feb '24

Slurm version 23.02.07 If I have a QoS defined that has a set number of say GPU devices set in the GrpTRES. Is there an easy way to generate a list of how much of the defined quota is allocated or conversely un-allocated? e.g.: Name|Priority|GraceTime|Preempt|PreemptExemptTime|PreemptMode|Flags|UsageThres|UsageFactor|GrpTRES|GrpTRESMins|GrpTRESRunMins|GrpJobs|GrpSubmit|GrpWall|MaxTRES|MaxTRESPerNode|MaxTRESMins|MaxWall|MaxTRESPU|MaxJobsPU|MaxSubmitPU|MaxTRESPA|MaxJobsPA|MaxSubmitPA|MinTRES| normal|0|00:00:00|||cluster|||1.000000|||||||||||cpu=3000,gres/gpu=20||||||| dept1|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=16,gres/gpu:2g.20gb=8,gres/gpu:3g.40gb=8,gres/gpu:a100.80gb=8||||||||||||||||| dept2|1|00:00:00|||cluster|||1.000000|cpu=256,gres/gpu:1g.10gb=0,gres/gpu:2g.20gb=0,gres/gpu:3g.40gb=0,gres/gpu:a100.80gb=16||||||||||||||||| So dept1 and dept2 qos are set on the same partition. How can a user with access to one or other see if there are available resources in the partition?

2 1

Errors upgrading to 23.11.0
by Xaver Stiensmeier 08 Feb '24

08 Feb '24

Dear slurm-user list, I got this error: Unable to start service slurmctld: Job for slurmctld.service failed because the control process exited with error code.\nSee \"systemctl status slurmctld.service\" and \"journalctl -xeu slurmctld.service\" for details. but in slurmctld.service I see nothing suspicious: slurmctld.service - Slurm controller daemon Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/slurmctld.service.d └─override.conf Active: active (running) since Wed 2024-02-07 15:50:56 UTC; 19min ago Main PID: 51552 (slurmctld) Tasks: 21 (limit: 9363) Memory: 10.4M CPU: 1min 16.088s CGroup: /system.slice/slurmctld.service ├─51552 /usr/sbin/slurmctld --systemd └─51553 "slurmctld: slurmscriptd" "" "" "" "" "" "" Feb 07 15:58:21 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=3 NodeList=(null) usec=959 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 WTERMSIG 2 Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 cancelled by interactive user Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=3 done Feb 07 15:58:23 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=3 error Job/step already completing or completed Feb 07 15:58:42 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: sched: _slurm_rpc_allocate_resources JobId=4 NodeList=cluster-master-2vt2bqh7ahec04c,cluster-worker-2vt2bqh7ahec04c-2 usec=512 Feb 07 16:06:04 cluster-master-2vt2bqh7ahec04c slurmctld[51553]: slurmctld: error: _run_script: JobId=0 resumeprog exit status 1:0 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 WTERMSIG 2 Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _job_complete: JobId=4 done Feb 07 16:09:33 cluster-master-2vt2bqh7ahec04c slurmctld[51552]: slurmctld: _slurm_rpc_complete_job_allocation: JobId=4 error Job/step already completing or completed I am unsure how to debug this further. It might be coming from a previous problem I tried to fix (basically a few deprecated keys in the configuration). I will try to restart the entire cluster with the added changes to rule out any follow up errors, but maybe it's something obvious a fellow list user can see. Best regards, Xaver

2 2

sinfo gresused shard count wrong/incomplete
by Reed Dier 07 Feb '24

07 Feb '24

I have a bash script that grabs current statistics from sinfo to ship into a time series database to use for Grafana dashboards. We recently began using shards with our gpus, and I’m seeing some unexpected behavior with the data reported from sinfo. > $ sinfo -h -O "NodeHost:5 ,GresUsed:100 ,Gres:100" | grep gpu03 > gpu03 gpu:p40:0(IDX:N/A),gpu:rtx:0(IDX:N/A),shard:p40:1(IDX:1),shard:rtx:0(IDX:N/A) gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > > $ scontrol show node gpu03 > NodeName=gpu03 Arch=x86_64 CoresPerSocket=22 > CPUAlloc=52 CPUEfctv=88 CPUTot=88 CPULoad=43.67 > AvailableFeatures=avx,avx512,largeMemory,matlab > ActiveFeatures=avx,avx512,largeMemory,matlab > Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8 > OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 > RealMemory=768000 AllocMem=425984 FreeMem=116850 Sockets=2 Boards=1 > State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A > Partitions=gpu > BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-05T12:33:50 > LastBusyTime=2024-02-07T09:48:50 > CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48 > AllocTRES=cpu=52,mem=416G,gres/shard=21 > CapWatts=n/a > CurrentWatts=0 AveWatts=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Specifically, we can see there are 48 total shards on this specific node, 36 of type p40, 12 of type rtx. From control, I can see that 21 shards are in use, and from nvidia-smi I can see that the breakdown of shards per gpu index is: [0]p40:4/6 [1]p40:6/6 [2]rtx:4/6 [3]rtx:3/6 [4]p40:4/6 [5]p40:0/6 [6]p40:0/6 [7]p40:0/6 So, I can tell that what “GresUsed” is reporting is actually an entire gpu’s worth of shards used, and the index of that specific entire GPU consumed. However, what it doesn’t show is the amount of specific gres/shard actually used, or more succinctly, I can’t seem to extrapolate that there are 21 shards in use. Right now my script says there is 1 shard out of 48 used, because I’m scraping for the num of shards in use, which is reporting the value of entire GPUs used by shards, and anything less than 100% is rounded to 0.. So, given that I’m on 22.05.8, maybe its better in 23.02.X which I’m hoping to move to within the next month. However I can’t seem to find anything in the release notes for 23.02 or 23.11 that would imply that sinfo reports (my) expected value of the actual count of shards used, Does anyone have any ideas for how I might be able to achieve what I’m looking for using sinfo, or should I instead try to use the sinfo json parser which has a “tres_used” field, that doesn’t appear to be accessible outside of the json output? > { > "architecture": "x86_64", > "burstbuffer_network_address": "", > "boards": 1, > "boot_time": 1706901428, > "comment": "", > "cores": 22, > "cpu_binding": 0, > [SNIP] > "tres": "cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48", > "slurmd_version": "22.05.8", > "alloc_memory": 284380, > "alloc_cpus": 62, > "idle_cpus": 26, > "tres_used": "cpu=62,mem=284380M,gres/gpu=7,gres/gpu:p40=5,gres/gpu:rtx=2,gres/shard=17", > "tres_weighted": 62 > }, Any ideas appreciated. Thanks, Reed

1 0

Re: Restricting local disk storage of jobs
by Tim Schneider 07 Feb '24

07 Feb '24

Hey Jeffrey, thanks for this suggestion! This is probably the way to go if one can find a way to access GRES in the prolog. I read somewhere that people were calling scontrol to get this information, but this seems a bit unclean. Anyway, if I find some time I will try it out. Best, Tim On 2/6/24 16:30, Jeffrey T Frey wrote: > Most of my ideas have revolved around creating file systems on-the-fly > as part of the job prolog and destroying them in the epilog. The > issue with that mechanism is that formatting a file system (e.g. > mkfs.<type>) can be time-consuming. E.g. formatting your local > scratch SSD as an LVM PV+VG and allocating per-job volumes, you'd > still need to run a e.g. mkfs.xfs and mount the new file system. > > > ZFS file system creation is much quicker (basically combines the LVM + > mkfs steps above) but I don't know of any clusters using ZFS to manage > local file systems on the compute nodes :-) > > > One /could/ leverage XFS project quotas. E.g. for Slurm job 2147483647: > > > *[root@r00n00 /]# mkdir /tmp-alloc/slurm-2147483647* > *[root@r00n00 /]# xfs_quota -x -c 'project -s -p > /tmp-alloc/slurm-2147483647 2147483647' /tmp-alloc* > Setting up project 2147483647 (path /tmp-alloc/slurm-2147483647)... > Processed 1 (/etc/projects and cmdline) paths for project > 2147483647 with recursion depth infinite (-1). > *[root@r00n00 /]# xfs_quota -x -c 'limit -p bhard=1g 2147483647' > /tmp-alloc* > *[root@r00n00 /]# cd /tmp-alloc/slurm-2147483647* > *[root@r00n00 slurm-2147483647]# dd if=/dev/zero of=zeroes bs=5M > count=1000* > dd: error writing ‘zeroes’: No space left on device > 205+0 records in > 204+0 records out > 1073741824 bytes (1.1 GB) copied, 2.92232 s, 367 MB/s > > : > > [root@r00n00 /]# rm -rf /tmp-alloc/*slurm-2147483647* > [root@r00n00 /]# *xfs_quota -x -c 'limit -p bhard=0 2147483647' > /tmp-alloc* > > > Since Slurm jobids max out at 0x03FFFFFF (and 2147483647 = 0x7FFFFFFF) > we have an easy on-demand project id to use on the file system. Slurm > tmpfs plugins have to do a mkdir to create the per-job directory, > adding two xfs_quota commands (which run in more or less O(1) time) > won't extend the prolog by much. Likewise, Slurm tmpfs plugins have to > scrub the directory at job cleanup, so adding another xfs_quota > command will not do much to change their epilog execution times. The > main question is "where does the tmpfs plugin find the quota limit for > the job?" > > > > > >> On Feb 6, 2024, at 08:39, Tim Schneider via slurm-users >> <slurm-users(a)lists.schedmd.com> wrote: >> >> Hi, >> >> In our SLURM cluster, we are using the job_container/tmpfs plugin to >> ensure that each user can use /tmp and it gets cleaned up after them. >> Currently, we are mapping /tmp into the nodes RAM, which means that >> the cgroups make sure that users can only use a certain amount of >> storage inside /tmp. >> >> Now we would like to use of the node's local SSD instead of its RAM >> to hold the files in /tmp. I have seen people define local storage as >> GRES, but I am wondering how to make sure that users do not exceed >> the storage space they requested in a job. Does anyone have an idea >> how to configure local storage as a proper tracked resource? >> >> Thanks a lot in advance! >> >> Best, >> >> Tim >> >> >> -- >> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >

2 1

scheme for protected GPU jobs from preemption
by Paul Raines 06 Feb '24

06 Feb '24

After using just Fairshare for over a year on our GPU cluster, we have decided it is not working for us for what we really want to achieve among our groups. We have decided to look at preemption. What we want is for users to NOT have a #job/GPU maximum (if they are only person on the cluster they should be able to use it all), but if another user comes to the "full" cluster they should immediately be able to run some jobs. Thus preemption is needed. In our scheme we want * users to have N protected GPU jobs that cannot be preempted where N is the number of GPUs allocated. * N may not be the same for all users. Some priviledged users get more. * jobs pending in the queue will have lower priority dependent on the number of GPUs allocated to running jobs. Maybe doable somehow with PriorityWeightJobSize though not sure how. * Jobs over N are subject to preemption (and requeued if --requeue is given) with shortest running jobs of the user with most unprotected GPUs preempted first. * another complication is we have a variety of different GPUs and users may ask for specific ones which can limit what unprotected GPU jobs are available for preemption My first attempt to do this in SLURM was to just create two partitions, GPU and GPU-req, with different PriorityTier values and the later partition have PreemptMode=REQUEUE. But N would be set by a MaxTRES on the first partition and be the same for everyone and we need it to be INDEPENDENT for each user. Also users would have to "think" about which partition to submit jobs to. And users want their longest running "unprotected" job to be able to be PROMOTED to a "protected" jobs automatically when a "protected" job finishes. However slurm does not allow running jobs to move between partitions. I am trying to figure out QOS pre-emption which might solve the independent N per user issue but I don't think it will solve the promotion issue. Any ideas how this scheme might be possible in SLURM? Otherwise I might have to write a complicated cron job that tries to do it all "outside" of SLURM issuing scontrol commands. --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi, In our SLURM cluster, we are using the job_container/tmpfs plugin to ensure that each user can use /tmp and it gets cleaned up after them. Currently, we are mapping /tmp into the nodes RAM, which means that the cgroups make sure that users can only use a certain amount of storage inside /tmp. Now we would like to use of the node's local SSD instead of its RAM to hold the files in /tmp. I have seen people define local storage as GRES, but I am wondering how to make sure that users do not exceed the storage space they requested in a job. Does anyone have an idea how to configure local storage as a proper tracked resource? Thanks a lot in advance! Best, Tim

3 2

Re: [ext] Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi Magnus, I understand. Thanks a lot for your suggestion. Best, Tim On 06.02.24 15:34, Hagdorn, Magnus Karl Moritz wrote: > Hi Tim, > in the end the InitScript didn't contain anything useful because > > slurmd: error: _parse_next_key: Parsing error at unrecognized key: > InitScript > > At this stage I gave up. This was with SLURM 23.02. My plan was to > setup the local scratch directory with XFS and then get the script to > apply a project quota, ie quota attached to the directory. > > I would start by checking if slurm recognises the InitScript option. > > Regards > magnus > > On Tue, 2024-02-06 at 15:24 +0100, Tim Schneider wrote: >> Hi Magnus, >> >> thanks for your reply! If you can, would you mind sharing the >> InitScript >> of your attempt at getting it to work? >> >> Best, >> >> Tim >> >> On 06.02.24 15:19, Hagdorn, Magnus Karl Moritz wrote: >>> Hi Tim, >>> we are using the container/tmpfs plugin to map /tmp to a local NVMe >>> drive which works great. I did consider setting up directory >>> quotas. I >>> thought the InitScript [1] option should do the trick. Alas, I >>> didn't >>> get it to work. If I remember correctly, slurm complained about the >>> option being present. In the end we recommend our users to make >>> exclusive use a node if they are going to use a lot of local >>> scratch >>> space. I don't think this happens very often if at all. >>> Regards >>> magnus >>> >>> [1] >>> https://slurm.schedmd.com/job_container.conf.html#OPT_InitScript >>> >>> >>> On Tue, 2024-02-06 at 14:39 +0100, Tim Schneider via slurm-users >>> wrote: >>>> Hi, >>>> >>>> In our SLURM cluster, we are using the job_container/tmpfs plugin >>>> to >>>> ensure that each user can use /tmp and it gets cleaned up after >>>> them. >>>> Currently, we are mapping /tmp into the nodes RAM, which means >>>> that >>>> the >>>> cgroups make sure that users can only use a certain amount of >>>> storage >>>> inside /tmp. >>>> >>>> Now we would like to use of the node's local SSD instead of its >>>> RAM >>>> to >>>> hold the files in /tmp. I have seen people define local storage >>>> as >>>> GRES, >>>> but I am wondering how to make sure that users do not exceed the >>>> storage >>>> space they requested in a job. Does anyone have an idea how to >>>> configure >>>> local storage as a proper tracked resource? >>>> >>>> Thanks a lot in advance! >>>> >>>> Best, >>>> >>>> Tim >>>> >>>>

1 0

Re: [ext] Restricting local disk storage of jobs
by Tim Schneider 06 Feb '24

06 Feb '24

Hi Magnus, thanks for your reply! If you can, would you mind sharing the InitScript of your attempt at getting it to work? Best, Tim On 06.02.24 15:19, Hagdorn, Magnus Karl Moritz wrote: > Hi Tim, > we are using the container/tmpfs plugin to map /tmp to a local NVMe > drive which works great. I did consider setting up directory quotas. I > thought the InitScript [1] option should do the trick. Alas, I didn't > get it to work. If I remember correctly, slurm complained about the > option being present. In the end we recommend our users to make > exclusive use a node if they are going to use a lot of local scratch > space. I don't think this happens very often if at all. > Regards > magnus > > [1] > https://slurm.schedmd.com/job_container.conf.html#OPT_InitScript > > > On Tue, 2024-02-06 at 14:39 +0100, Tim Schneider via slurm-users wrote: >> Hi, >> >> In our SLURM cluster, we are using the job_container/tmpfs plugin to >> ensure that each user can use /tmp and it gets cleaned up after them. >> Currently, we are mapping /tmp into the nodes RAM, which means that >> the >> cgroups make sure that users can only use a certain amount of storage >> inside /tmp. >> >> Now we would like to use of the node's local SSD instead of its RAM >> to >> hold the files in /tmp. I have seen people define local storage as >> GRES, >> but I am wondering how to make sure that users do not exceed the >> storage >> space they requested in a job. Does anyone have an idea how to >> configure >> local storage as a proper tracked resource? >> >> Thanks a lot in advance! >> >> Best, >> >> Tim >> >>

2 1

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users