- slurm-users - lists.schedmd.com

SLURM GRES reservation not working properly on 24.05.1
by Minulakshmi S 20 Sep '24

20 Sep '24

Hello, *Issue 1:* I am using slurm version 24.05.1 , my slurmd has a single node where I connect multiple gres by enabling the overscribe feature. I am able to use the advance reservation of gres only using *gres** name* (tres=gres/gpu:*SYSTEM12*). i.e while in reservation period , if other users submits job with SYSTEM12 , then slurm places this job in queue *user1@host$ srun --gres=gpu:SYSTEM12:1 hostname* *srun: job 333 queued and waiting for resources * but when other users just submit a job without any system name , slurm jobs goes through on that gres immediately even though it is reserved. *user1@host$ srun --gres=gpu:1 hostname * *mylinux.wbi.com <http://mylinux.wbi.com> * Also I can see GresUsed in busy mode using "*scontrol show node -d*" , this means the job is running on Gres/GPU and not on cpu etc. Same way , job submission based on Feature "rev1 in my case" is also going through even though it is reserved for other users in multiple partition slurm. *snippet of slurm.conf file* NodeName=cluster01 NodeAddr=cluster Port=6002CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 Feature="rev1" Gres=gpu:SYSTEM12:1 RealMemory=64171 State=IDLE *Issue 2:* while execution , Slurm o/p's some extra prints in the srun output user1@host$ srun --gres=gpu:1 hostname srun: error: extract_net_cred: net_cred not provided srun: error: Malformed RPC of type RESPONSE_NODE_ALIAS_ADDRS(3017) received srun: error: slurm_unpack_received_msg: [[inv1715771615.nxdi.us-aus01.nxp.com]:41242] Header lengths are longer than data received *mylinux.wbi.com <http://mylinux.wbi.com>* Regards, MS

1 0

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

SlurmDBD errors
by Sajesh Singh 18 Sep '24

18 Sep '24

OS: CentOS 8.5 Slurm: 22.05 Recently upgraded to 22.05. Upgrade was successful, but after a while I started to see the following messages in the slurmdbd.log file: error: We have more time than is possible (9344745+7524000+0)(16868745) > 12362400 for cluster CLUSTERNAME(3434) from 2024-09-18T13:00:00 - 2024-09-18T14:00:00 tres 1 (this may happen if oversubscription of resources is allowed without Gang) We do have partitions with overlapping nodes, but do not have "Suspend,Gang" set as the global PreemptMode mode. It is currently set to requeue. I have also check sacct and there are no runaway jobs listed. Oversubscription is not enabled on any of the queues as well. Do I need to modify my slurm config to address or is this an error condition caused by the upgrade? Thank you, SS

2 2

Change a job from --exclusive to --exclusive=user
by Gerhard Strangar 18 Sep '24

18 Sep '24

Hello, is it possible to change a pending job from --exclusive to --exclusive=user? I tried scontrol update jobid=... oversubscribe=user, but it seems to only accept yes or no. Gerhard

1 0

Feature request: Max Jobs Per Minute
by Ransom, Geoffrey M. 16 Sep '24

16 Sep '24

Hello We have another batch of new users and some more batches of large array jobs with very short runtimes due to errors in the jobs or just by design. Trying to deal with these issues, Setting ArrayTaskThrottle and user education, I had a thought that it would be very nice to have a limit on how many jobs can start in a given minute for users, so if they posted a 200000 array job with 15 second tasks then the scheduler wouldn't launch more than a 100 or 200 per minute and be less likely to bog down, but if they had longer runtimes (1 hour +) it would take a few extra minutes to start using all the resources they are allowed to but not add much overall delay to the whole set of jobs. I thought about adding something to our CLI filter, but usually these jobs are asking for a runtime of 3-4 hours even though they run for <30 seconds so the submit options don't indicate the problem jobs ahead of time. We currently limit our users to %80 of the available resources which is way more than slurm needs to bog down with fast turnover jobs, but we have users who complain that they can't use that other 20% when the cluster is not busy so putting in lower default restrictions is not currently an option. Has this already been discussed and isn't feasible for technical reasons? (Not finding anything like this yet searching the archives) I think slurm used have a feature request severity on their bug submission site. Is there a severity level they prefer to have suggested requests like this? Thanks

2 2

Detailed locations for SLUG'24
by Bjørn-Helge Mevik 10 Sep '24

10 Sep '24

Dear all SLUG attendees! The information about which buildings/addresses the SLUG reception and presentations are to be held is not very visible on the https://slug24.splashthat.com. There is a map there with all locations (https://www.google.com/maps/d/u/0/edit?mid=1bcGaTiW0TNB5noQsjQ3ulctzKuqlGrQ…), but I've gotten questions about it, so: The reception on Wednesday will be held in the top floor of Oslo Science Park (Forskningsparken). Address: Gaustadalléen 21. There will be someone in the reception who can point you in the right direction. The presentations will be held in auditorium 3 in Helga Engs Hus ("Helga Eng's House"). Address: Sem Sælands vei 7. Lunch will be in the canteen in the same building. The closest subway station to both these buildings is Blindern Subway Station (Blindern T-banestasjon). Looking forward to see you there! -- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo

1 1

Issue with cgroup v2 when IgnoreSystemd=yes
by Honoré Bergeron 09 Sep '24

09 Sep '24

Hi, This is a follow-up from https://groups.google.com/g/slurm-users/c/JI3UkbCtj3U, but as I could not find any progress, I am opening a new thread. When setting IgnoreSystemd=yes in cgroup.conf, I have the error: error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable cpuset in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable memory in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable cpu in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: Could not create scope directory /sys/fs/cgroup/system.slice/slurmstepd.scope: No such file or directory error: Couldn't load specified plugin name for cgroup/v2: Plugin init() callback failed error: cannot create cgroup context for cgroup/v2 error: Unable to initialize cgroup plugin error: slurmd initialization failed I wrote a patch that solves this issue: --- cgroup_v2.c.orig 2024-09-02 13:18:21.376312875 +0200 +++ cgroup_v2.c 2024-09-02 13:22:00.516986953 +0200 @@ -43,6 +43,7 @@ #include <sys/inotify.h> #include <poll.h> #include <unistd.h> +#include <libgen.h> #include "slurm/slurm.h" #include "slurm/slurm_errno.h" @@ -743,11 +744,33 @@ return SLURM_SUCCESS; } +static int _mkdir(const char *path, mode_t mode) +{ + int rc; + char *dir, *pdir; + + dir = strdup(path); + if (dir == NULL) { + return ENOMEM; + } + pdir = dirname(dir); + if (strcmp(pdir, path) != 0) { + rc = _mkdir(pdir, mode); + if (rc && (errno != EEXIST)) { + free(dir); + return rc; + } + } + rc = mkdir(path, mode); + free(dir); + return rc; +} + static int _init_new_scope(char *scope_path) { int rc; - rc = mkdir(scope_path, 0755); + rc = _mkdir(scope_path, 0755); if (rc && (errno != EEXIST)) { error("Could not create scope directory %s: %m", scope_path); return SLURM_ERROR; This patch concerns the file src/plugins/cgroup/v2/cgroup_v2.c. Am I missing something ? Cheers, Honoré.

1 0

Configuration for nodes with different TmpFs locations and TmpDisk sizes
by Jake Longo 06 Sep '24

06 Sep '24

Hi, We have a number of machines in our compute cluster that have larger disks available for local data. I would like to add them to the same partition as the rest of the nodes but assign them a larger TmpDisk value which would allow users to request a larger tmp and land on those machines. The main hurdle is that (for reasons beyond my control) the larger local disks are on a special mount point /largertmp whereas the rest of the compute cluster uses the vanilla /tmp. I can't see an obvious way to make this work as the TmpFs value appears to be global only and attempting to set TmpDisk to a value larger than TmpFs for those nodes will put the machine into an invalid state. I couldn't see any similar support tickets or anything in the mail archive but I wouldn't have thought it would be that unusual to do this. Thanks in advance! Jake

2 1

salloc not starting shell despite LaunchParameters=use_interactive_step
by Loris Bennett 06 Sep '24

06 Sep '24

Hi, With $ salloc --version slurm 23.11.10 and $ grep LaunchParameters /etc/slurm/slurm.conf LaunchParameters=use_interactive_step the following $ salloc --partition=interactive --ntasks=1 --time=00:03:00 --mem=1000 --qos=standard salloc: Granted job allocation 18928869 salloc: Nodes c001 are ready for job creates a job $ squeue --me JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 18928779 interacti interact loris R 1:05 1 c001 but causes the terminal to block. From a second terminal I can log into the compute node: $ ssh c001 [13:39:36] loris@c001 (1000) ~ Is that the expected behaviour or should salloc return a shell directly on the compute node (like srun --pty /bin/bash -l used to do)? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

5 8

Nodelist syntax and semantics
by Jackson, Gary L. 05 Sep '24

05 Sep '24

Is there a description of the “nodelist” syntax and semantics somewhere other than the source code? By “nodelist” I mean expressions like “name[000,099-100]” and how this one, for example, expands to “name000, name099, name100”. -- Gary

2 1

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users