- slurm-users - lists.schedmd.com

Re: Using more cores/CPUs that requested with sbatch
by Megan 25 Mar '25

25 Mar '25

Slurm can restrict jobs to their allocated resources using cgroups. To use cgroups you will have to configure the TaskPlugin cgroup option and specify how the cgroups should behave in cgroup.conf. slurm.conf TaskPlugin=task/cgroup Slurm Workload Manager - slurm.conf <https://slurm.schedmd.com/slurm.conf.html#OPT_task/cgroup> cgroup.conf ConstrainCores=yes Slurm Workload Manager - cgroup.conf <https://slurm.schedmd.com/cgroup.conf.html#OPT_ConstrainCores> Regards, --Megan

1 0

slurmrestd equivalent to "srun -n 10 echo HELLO"
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Community, I'm starting to experiment with slurmrestd for a new app we're writing. I'm having trouble understanding one aspect of submitting jobs. When I run something like `srun -n 10 echo HELLO', I get HELLO returned to my console/stdout 10x. When I submit this command as a script to the /jobs/submit route, I get success/200, but *I cannot determine how to get the console output of HELLO 10x in any form*. It's not in my stdout log for that job even though I can verify that the job ran successfully. Any suggestions? -- Thanks, Daniel Healy

3 2

Priority/Top seems to not be working
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Users, I have a newer install (23.11.3) and the priority of most all jobs is 1 and other small numbers. In previous versions, I would see numbers like 2^32. I have the multifactor plugin configured and confirmed it's in-use when I show the config. When I run `scontrol top` for a given job, the priority number doesn't change from 1. I have lots of jobs running at the moment and many queued as well. I've been using `scontrol top` for years and have seen the priority successfully change. I'm repeating this now with the newer version of Slurm and now seeing it work the same. Are you seeing this too? -- Thanks, Daniel Healy

1 0

missing jobs from queue
by Felix 14 Mar '25

14 Mar '25

Hello at site RO-14-ITIM, we have a very strange problem. A ticket was raised with the following: "Hi, The new ticket (Pilots disappearing from local batch system at ITIM) has been created by "*Timo Wilken*". Group: NGIs › NGI_RO Owner: - State: in progress Information: Hi, Pilots seem to be lost by the local batch system at ITIM. >50% of pilots fail: https://apfmon.lancs.ac.uk/CERN_central_A:RO-14-ITIM-arcn-node?state=fault Example log: https://aipanda156.cern.ch/condor_logs_2/25-03-12_13/grid.15478691.1.log Error message: "ARC job failed: LRMS error: (-1) Job missing from SLURM" Cheers, Timo" The problem is I can not find any of the jobs that are failing on the site: for example: the last job: https://aipanda023.cern.ch/condor_logs_2/25-03-14_09/grid.19602200.0.log with stdlog: 000 (19602200.000.000) 2025-03-14 09:28:03 Job submitted from host: <137.138.31.125:38090?addrs=137.138.31.125-38090+[2001-1458-d00-19--75]-38090&alias=aipanda023.cern.ch> ... 027 (19602200.000.000) 2025-03-14 09:28:13 Job submitted to grid resource GridResource: arc arcn-node.itim-cj.ro:443 GridJobId: arc arcn-node.itim-cj.ro:443 PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n ... 001 (19602200.000.000) 2025-03-14 09:40:51 Job executing on host: arc arcn-node.itim-cj.ro:443 ... 012 (19602200.000.000) 2025-03-14 09:40:58 Job was held. ARC job failed: LRMS error: (-1) Job missing from SLURM Code 0 Subcode 0 ... 009 (19602200.000.000) 2025-03-14 11:03:03 Job was aborted. Python-initiated action. (by user atlpan) ... 009 (19602200.000.000) 2025-03-14 11:03:17 Job was aborted. Python-initiated action. (by user atlpan) This name: PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n is nowhere to find on my server: [root@arcn-node log]# updatedb [root@arcn-node log]# locate PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n [root@arcn-node log]# I have on my site this kind of error The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE)It became Critical*at*2025-03-14T09:16:44Z due to *METRIC*org.nordugrid.ARC-CE-sw-gcc *Summary:*Script exited with code 127. Could not match GCC version. See /etc/arc/nagios/20-dist.ini for debugging hints. *Message:*'' which is going to be ok in the messag, one minute later The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE) It became Ok *at* 2025-03-14T09:17:53Z due to *METRIC* org.nordugrid.ARC-CE-sw-gcc *Summary: *Found GCC version 11.5.0. *Message: *'' Can you please advice where to look for any clue in solving this mystery? Thank you Felix :-- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Slurm version 24.05.7 is now available
by Marshall Garey 13 Mar '25

13 Mar '25

We are pleased to announce the availability of Slurm version 24.05.7. This release fixes some stability issues in 24.05, including a crash in slurmctld after updating a reservation with an empty nodelist. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.7 > ========================== > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix jobs being scheduled on higher weighted powered down > nodes. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Prevent slurmctld deadlock in the assoc mgr.

1 0

SLURM_JOB_ACCOUNT var missing in prolog
by Jonás Arce 13 Mar '25

13 Mar '25

Hi everyone, The Slurm variable SLURM_JOB_ACCOUNT is blank in my prolog, currently i have a TaskProlog and a Prolog in my system, in TaskProlog this variable works just fine (I use TaskProlog to set env variables), but in my Prolog (which I use to create temporal directories), it's just blank, it's strange beacuse other variables such as SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT or SLURM_JOBID work just fine in both Prolog and TaskProlog, I looked into the Slurm doc and it seems that this var should work everywhere, if someone could shed some light into this or tell me another equivalent var i'd appreciate it a lot. I need it because I need to make this type of temporal directories with my Prolog: /scratch-global/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOBID.

2 1

Job Env Vars in Slurm Core
by Bhaskar Chakraborty 13 Mar '25

13 Mar '25

Hi everyone, I have tried my best to extract custom job environment variables inside a custom select plugin.However, I am not able to get the same. I am able to get the env in a custom client filter plugin during job submission (sbatch --export) but Ineed the same in slurmctld as well. Is there any way this can be achieved. I have looked over few of the internal DS pointers (job_record, job_details etc)but none seem to provide the required functionality. Any insight is appreciated. Regards,Bhaskar.

2 1

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 12 Mar '25

12 Mar '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

3 5

SLURM queue/partition/qos configuration
by vittorio.confuorto＠gmail.com 12 Mar '25

12 Mar '25

Hi everyone, I need some advice on properly configuring my HPC cluster, which consists of three compute nodes (each with 256 cores and 800GB of RAM). The end users belong to three distinct groups: internal, external, and collaborator. I would like the internal users to have full (and exclusive) access to the first two compute nodes. That is, by default, if they submit a job, it should be scheduled on Node1 or Node2, depending on availability (or placed in the queue if necessary). Internal users should also have access to 100 cores and 200GB of RAM on Node3, with this limit applying as a hard cap across all their jobs. The remaining 156 cores and 600GB of RAM on Node3 should be exclusively available to external and collaborator users. Specifically, I would like the following setup: - The resources on Node3 should be split as follows: - 100 cores and 400GB of RAM for external users. - 56 cores and 200GB of RAM for collaborator users. - External users should be able to use the resources dedicated to collaborator users but with lowest priority. This means an external user can submit a job requesting the resources reserved for collaborators, but the job will only start if no collaborator jobs are in the queue. Any job submitted by a collaborator—even after an external user's job—should take priority. The same rule should apply in reverse for collaborators trying to use external resources. - There should be a group/partition/queue that includes all the remaining 156 cores and 600GB of RAM on Node3. Both external and collaborator users should be able to submit jobs to this queue, but a job submitted here should only start if there are no pending jobs in the two dedicated partitions. What is the best way to implement such an architecture? Thanks, everyone, for your support!

1 0

Slurm Job Kill intimation
by Bhaskar Chakraborty 10 Mar '25

10 Mar '25

Hello All, We wish to do some cleanup when user kills a pending slurm job using scancel. Our setup has a client filter plugin which sets some internal variables while interacting to a remote process.This needs to be undone during job termination. Is there a way to do this (using submitter credentials) ? Also, if one is to use a custom select plugin is there a way to inform select that the job is terminated.Presently, we don't find that slurmctld informs select plugin when a pending job is killed. Thanks in advance for any pointers ! Regards,Bhaskar.

1 0

2025

2024

slurm-users ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users