- slurm-users - lists.schedmd.com

Implementing a "soft" wall clock limit
by Davide DelVento 12 Jun '25

12 Jun '25

In the institution where I work, so far we have managed to live without mandatory wallclock limits (a policy decided well before I joined the organization), and that has been possible because the cluster was not very much utilized. Now that is changing, with more jobs being submitted and those being larger ones. As such I would like to introduce wallclock limits to allow slurm to be more efficient in scheduling jobs, including with backfill. My concern is that this user base is not used to it and therefore I want to make it easier for them, and avoid common complaints. I anticipate one of them would be "my job was cancelled even though there were enough nodes idle and no other job in line after mine" (since the cluster utilization is increasing, but not yet always full like it has been at most other places I know). So my question is: is it possible to implement "soft" wallclock limits in slurm, namely ones which would not be enforced unless necessary to run more jobs? In other words, is it possible to change the pre-emptability of a job only after some time has passed? I can think of some ways to hack this functionality myself with some cron or at jobs, and that might be easy enough to do, but I am not sure I can make it robust enough to cover all situations, so I'm looking for something either slurm-native or (if external solution) field-tested by someone else already, so that at least the worst kinks have been already ironed out. Thanks in advance for any suggestions you may provide!

3 7

MIG H100 with xeon Intel
by Richard Lefebvre 12 Jun '25

12 Jun '25

I'm having problems with Autodetect=nvml in gres.conf. I get on the controller log the following: error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 16-31 on node node001 doesn't match socket boundaries. (Socket 0 is cores 0-31). Consider setting SlurmdParameters=l3cache_as_socket (recommended) or override this by manually specifying core affinity in gres.conf. I did set l3cache_as_socket in the slurm.conf of the node, but I still get the error on the slurm controler I'm running 24.11.5 on AlmaLinux 9.5 Richard

1 0

How are the results produced by 'seff'?
by Loris Bennett 12 Jun '25

12 Jun '25

Hi, With Slurm 24.11.5 for some jobs I am seeing differences between the memory usage reported by 'seff' and that shown by Prometheus as 'cgroup_memory_rss_bytes' (and ultimately reported by 'jobstats' [1]). Certainly at the University of Delft they seem to feel that the memory usage reported by 'seff' is unreliable [2]. Is that indeed the case? Cheers, Loris Footnotes: [1] https://github.com/PrincetonUniversity/jobstats [2] https://doc.dhpc.tudelft.nl/delftblue/Slurm-trouble-shooting/ -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

slurm_pam_adopt module not working
by Ratnasamy, Fritz 11 Jun '25

11 Jun '25

Hi, We wanted to block users from ssh to a node where there are no jobs running however it looks like users are able to do so. We have installed the slurm_pam_adopt_module and set up the slurm.conf accordingly (the same way we did on our first cluster where the pam module denies ssh access correctly). We have PrologFlags=contain in slurm.conf and inside /etc/pam.d/sshd, we set the last row to : account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny Are we missing anything? Thanks *Fritz Ratnasamy*Data Scientist Information Technology

3 2

CR_CPU used but only cores used
by Adrian Sevcenco 11 Jun '25

11 Jun '25

Hi! i have a weird situation in which only cores are used instead of CPUs this is Alma9/slurm 22.05.9 (the last one from epel) I have: conf.d/resources.conf 9:SelectType=select/cons_res conf.d/resources.conf 11:SelectTypeParameters=CR_CPU conf.d/resources.conf 5:TaskPluginParam=autobind=threads conf.d/resources.conf 2:TaskPlugin=task/affinity,task/cgroup cgroup.conf 2:ConstrainCores=no conf.d/parts_issaf.conf 8:PartitionName=CLUSTER DEFAULT=YES State=UP nodes=DEFAULT DefaultTime=240:00:00 MaxTime=480:00:00 LLN=yes MaxNodes=1 What am i missing? Why are only cores used? Thanks a lot!! Adrian

2 2

Restrict and prioritize usage of certain nodes according to accounts
by thomas.hartmann＠th-ht.de 09 Jun '25

09 Jun '25

Hi, I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account. I have two sets of physical nodes: 1. node01 - nodeXX: General Purpose Nodes. Everybody can use them 2. project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them. Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts. I have three sets of accounts (each can have child accounts): 1. "General" accounts: These are allowed to use all physical nodes. 2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes 3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes. My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need. In the future, there might be more project specific nodes for other projects. I would be grateful for any alternative solution or hints, how this could be solved optimally. Thanks a lot! Thomas

5 4

Problem querying slurm batch script archive
by Davide DelVento 09 Jun '25

09 Jun '25

I configured slurm.conf on our machine with AccountingStoreFlags=job_script,job_comment So that people can run sacct -Bj <job_ID> and check what the batch script was for the specified job_ID. This works fine for root and works great for many users (who can see their own job scripts, but not those of other users, as it ought to be). Unfortunately, for several users (but not even close to half of the users of the machine), the sacct -Bj command returns nothing, like if they queried someone else's job, even when querying their own. Does anybody have any clue on what might be wrong and triggering this odd behavior? Thanks!

1 0

seff command not found
by Ratnasamy, Fritz 09 Jun '25

09 Jun '25

Hi, We installed a new slurm version and it returns "command not found" for seff. I do not remember doing any manual installation for the previous versions, I thought it was coming with sacct, sbatch, ect... Any idea how I would need to set it up? I read online seff is actually a perl script. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Re: Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi Brian, sorry for breaking the email thread, my subscription settings were not set correctly and I didn't receive your response as email. Thanks for the tips! I'll give that a try. Best, Nick > Nick, > Presuming you have followed the SchedMD instructions, you should be able > to get a session in your login container: > kubectl --namespace=slurm exec -it statefulsets/slurm-controller -- bash > --login > From there, you can do any standard testing you like. Simple 'srun > hostname' should work to let you know slurm itself is doing it's part. > You can also do commands such as 'scontrol show nodes' to see what and > how many resources you have configured. > Outside that, you need to ensure you have configured your slurm > containers to request the resources you plan on using (eg: gpus) and > that you have enough of them for the script(s) you wish to run. > Brian Andrus

1 0

X11 Forwarding Crashes X Session for too large windows
by Felix Winterhalter 06 Jun '25

06 Jun '25

Hello everyone, we have run into a peculiar issue with our setup. First and foremost let me say that X forwarding using ssh -X or ssh -Y works fine and doesn't show any irregularities. Trying to use the X forwarding of slurm however results in a weird situation. We are using it like: srun --pty --x11 bash And then run something like glxgears. The program starts up normal however whenever you increase the window size beyond a certain size the connection to the X server gets broken and glxgears crashes. The same is true for any other program using X, some don't even start as their inital window size is too large, others crash as soon as you resize them beyond a certain point. We have no clue what might be the cause only that it seems to be related to how slurm does the XForwarding. We've tried X11Parameters both with and without host_authority. We're running 25.05.0 Any help or idea would be very appreciated. Cheers, Felix This message and any attachments are for the sole use of the intended recipient(s). It may contain information that is confidential and privileged. If you are not the intended recipient of this message, you are prohibited from printing, copying, forwarding or saving it. Please delete the message and attachments and notify the sender immediately.

1 1

2025

2024

slurm-users

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users