- slurm-users - lists.schedmd.com

How limit CPUs per node in a partition
by Gestió Servidors 27 Mar '25

27 Mar '25

Hello, I have a testing partition with only a node. That server has 12 CPUs (it's a very old server) (2 sockets, 6 cores per socket, 1 thread per core). That partition, called "test.q" only has that node, so by default, partition test.q has 12 CPUs (all from testing node). However, now I would like to decrease that CPU number from 12 to 4, for example. "slurmd -C" returns "CPUs=12 Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1" as I have said to you before and if in slurm.conf I configure partition in this way "PartitionName=test.q Nodes=test-node MaxTime=10:00 State=UP AllocNodes=clus-login MaxCPUsPerNode=4", partition continues accepting jobs with 12 CPUs. Is it any wrong in my configuration? How could I get my purpose? Thanks. -- [cid:image001.jpg@01DB9F0E.FDD9A7A0]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image002.jpg@01DB9F0E.FDD9A7A0]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

1 0

Re: Using more cores/CPUs that requested with
by Shunran Zhang 26 Mar '25

26 Mar '25

Ugh I think I did not catch up with the docs. I started with a system that defaults to cgroup v1 but the Slurm doc for that plugin is NOT available at that time. Thus I converted everything to cgroup v2. It appears that they are both supported and that documentation issue is more on the dev side than admin side. Thanks for pointing that out. I misinterpreted the "coming soon" part of cgroup v1 plugin and the "legacy" naming for "do not use". It should be fine. 2025年3月27日(木) 0:48 Williams, Jenny Avis <jenny_williams(a)unc.edu>: > “ … As cgroup is likely not supposed to be used in newer deployments of > Slurm.” > > > > I am curious about this statement. Would someone expand on this, to either > support or counter it? > > > > Jenny Williams > > UNC Chapel Hill > > > > > > *From:* Shunran Zhang via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Wednesday, March 26, 2025 10:52 AM > *To:* Gestió Servidors <sysadmin.caos(a)uab.cat> > *Cc:* Slurm User Community List <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Re: Using more cores/CPUs that requested with > > > > If you are letting systemd taking most things over, you got systemd-cgtop > that work better than top for your case. There is also systemd-cgls for > non-interactive listing. > > > > Also mind to check if you are using cgroup2? A mount to check your cgroup > would suffice. As cgroup is likely not supposed to be used in newer > deployments of Slurm. > > > > > > 2025年3月26日(水) 17:14 Gestió Servidors via slurm-users < > slurm-users(a)lists.schedmd.com>: > > Hello, > > > > Thanks for your answers. I will try now!! One more question: is there any > way to check if Cgroups restrictions is working fine during a “running” job > or during SLURM scheduling process? > > > > Thanks again! > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > >

1 0

Re: Using more cores/CPUs that requested with
by Gestió Servidors 26 Mar '25

26 Mar '25

Hello, Thanks for your answers. I will try now!! One more question: is there any way to check if Cgroups restrictions is working fine during a "running" job or during SLURM scheduling process? Thanks again!

4 3

Using more cores/CPUs that requested with sbatch
by Gestió Servidors 25 Mar '25

25 Mar '25

Hello, I would like to know if there is any mechanism to avoid a user to do "cheating" when he submits a job. For example, a user submits a OpenMP job requesting 1 node (-N 1) and 2 tasks (-n 2). However, inside his script (or compiled in his binany), user uses more than 2 ntasks, for example, all free cores. For SLURM, job is only using 2, but really, node is 100% assigned to that job. SLURM can control that "abuse" in some way? Cgroups? Thanks.

3 2

Re: Using more cores/CPUs that requested with sbatch
by Megan 25 Mar '25

25 Mar '25

Slurm can restrict jobs to their allocated resources using cgroups. To use cgroups you will have to configure the TaskPlugin cgroup option and specify how the cgroups should behave in cgroup.conf. slurm.conf TaskPlugin=task/cgroup Slurm Workload Manager - slurm.conf <https://slurm.schedmd.com/slurm.conf.html#OPT_task/cgroup> cgroup.conf ConstrainCores=yes Slurm Workload Manager - cgroup.conf <https://slurm.schedmd.com/cgroup.conf.html#OPT_ConstrainCores> Regards, --Megan

1 0

slurmrestd equivalent to "srun -n 10 echo HELLO"
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Community, I'm starting to experiment with slurmrestd for a new app we're writing. I'm having trouble understanding one aspect of submitting jobs. When I run something like `srun -n 10 echo HELLO', I get HELLO returned to my console/stdout 10x. When I submit this command as a script to the /jobs/submit route, I get success/200, but *I cannot determine how to get the console output of HELLO 10x in any form*. It's not in my stdout log for that job even though I can verify that the job ran successfully. Any suggestions? -- Thanks, Daniel Healy

3 2

Priority/Top seems to not be working
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Users, I have a newer install (23.11.3) and the priority of most all jobs is 1 and other small numbers. In previous versions, I would see numbers like 2^32. I have the multifactor plugin configured and confirmed it's in-use when I show the config. When I run `scontrol top` for a given job, the priority number doesn't change from 1. I have lots of jobs running at the moment and many queued as well. I've been using `scontrol top` for years and have seen the priority successfully change. I'm repeating this now with the newer version of Slurm and now seeing it work the same. Are you seeing this too? -- Thanks, Daniel Healy

1 0

missing jobs from queue
by Felix 14 Mar '25

14 Mar '25

Hello at site RO-14-ITIM, we have a very strange problem. A ticket was raised with the following: "Hi, The new ticket (Pilots disappearing from local batch system at ITIM) has been created by "*Timo Wilken*". Group: NGIs › NGI_RO Owner: - State: in progress Information: Hi, Pilots seem to be lost by the local batch system at ITIM. >50% of pilots fail: https://apfmon.lancs.ac.uk/CERN_central_A:RO-14-ITIM-arcn-node?state=fault Example log: https://aipanda156.cern.ch/condor_logs_2/25-03-12_13/grid.15478691.1.log Error message: "ARC job failed: LRMS error: (-1) Job missing from SLURM" Cheers, Timo" The problem is I can not find any of the jobs that are failing on the site: for example: the last job: https://aipanda023.cern.ch/condor_logs_2/25-03-14_09/grid.19602200.0.log with stdlog: 000 (19602200.000.000) 2025-03-14 09:28:03 Job submitted from host: <137.138.31.125:38090?addrs=137.138.31.125-38090+[2001-1458-d00-19--75]-38090&alias=aipanda023.cern.ch> ... 027 (19602200.000.000) 2025-03-14 09:28:13 Job submitted to grid resource GridResource: arc arcn-node.itim-cj.ro:443 GridJobId: arc arcn-node.itim-cj.ro:443 PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n ... 001 (19602200.000.000) 2025-03-14 09:40:51 Job executing on host: arc arcn-node.itim-cj.ro:443 ... 012 (19602200.000.000) 2025-03-14 09:40:58 Job was held. ARC job failed: LRMS error: (-1) Job missing from SLURM Code 0 Subcode 0 ... 009 (19602200.000.000) 2025-03-14 11:03:03 Job was aborted. Python-initiated action. (by user atlpan) ... 009 (19602200.000.000) 2025-03-14 11:03:17 Job was aborted. Python-initiated action. (by user atlpan) This name: PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n is nowhere to find on my server: [root@arcn-node log]# updatedb [root@arcn-node log]# locate PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n [root@arcn-node log]# I have on my site this kind of error The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE)It became Critical*at*2025-03-14T09:16:44Z due to *METRIC*org.nordugrid.ARC-CE-sw-gcc *Summary:*Script exited with code 127. Could not match GCC version. See /etc/arc/nagios/20-dist.ini for debugging hints. *Message:*'' which is going to be ok in the messag, one minute later The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE) It became Ok *at* 2025-03-14T09:17:53Z due to *METRIC* org.nordugrid.ARC-CE-sw-gcc *Summary: *Found GCC version 11.5.0. *Message: *'' Can you please advice where to look for any clue in solving this mystery? Thank you Felix :-- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Slurm version 24.05.7 is now available
by Marshall Garey 13 Mar '25

13 Mar '25

We are pleased to announce the availability of Slurm version 24.05.7. This release fixes some stability issues in 24.05, including a crash in slurmctld after updating a reservation with an empty nodelist. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.7 > ========================== > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix jobs being scheduled on higher weighted powered down > nodes. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Prevent slurmctld deadlock in the assoc mgr.

1 0

SLURM_JOB_ACCOUNT var missing in prolog
by Jonás Arce 13 Mar '25

13 Mar '25

Hi everyone, The Slurm variable SLURM_JOB_ACCOUNT is blank in my prolog, currently i have a TaskProlog and a Prolog in my system, in TaskProlog this variable works just fine (I use TaskProlog to set env variables), but in my Prolog (which I use to create temporal directories), it's just blank, it's strange beacuse other variables such as SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT or SLURM_JOBID work just fine in both Prolog and TaskProlog, I looked into the Slurm doc and it seems that this var should work everywhere, if someone could shed some light into this or tell me another equivalent var i'd appreciate it a lot. I need it because I need to make this type of temporal directories with my Prolog: /scratch-global/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOBID.

2 1

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users