- slurm-users - lists.schedmd.com

missing jobs from queue
by Felix 14 Mar '25

14 Mar '25

Hello at site RO-14-ITIM, we have a very strange problem. A ticket was raised with the following: "Hi, The new ticket (Pilots disappearing from local batch system at ITIM) has been created by "*Timo Wilken*". Group: NGIs › NGI_RO Owner: - State: in progress Information: Hi, Pilots seem to be lost by the local batch system at ITIM. >50% of pilots fail: https://apfmon.lancs.ac.uk/CERN_central_A:RO-14-ITIM-arcn-node?state=fault Example log: https://aipanda156.cern.ch/condor_logs_2/25-03-12_13/grid.15478691.1.log Error message: "ARC job failed: LRMS error: (-1) Job missing from SLURM" Cheers, Timo" The problem is I can not find any of the jobs that are failing on the site: for example: the last job: https://aipanda023.cern.ch/condor_logs_2/25-03-14_09/grid.19602200.0.log with stdlog: 000 (19602200.000.000) 2025-03-14 09:28:03 Job submitted from host: <137.138.31.125:38090?addrs=137.138.31.125-38090+[2001-1458-d00-19--75]-38090&alias=aipanda023.cern.ch> ... 027 (19602200.000.000) 2025-03-14 09:28:13 Job submitted to grid resource GridResource: arc arcn-node.itim-cj.ro:443 GridJobId: arc arcn-node.itim-cj.ro:443 PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n ... 001 (19602200.000.000) 2025-03-14 09:40:51 Job executing on host: arc arcn-node.itim-cj.ro:443 ... 012 (19602200.000.000) 2025-03-14 09:40:58 Job was held. ARC job failed: LRMS error: (-1) Job missing from SLURM Code 0 Subcode 0 ... 009 (19602200.000.000) 2025-03-14 11:03:03 Job was aborted. Python-initiated action. (by user atlpan) ... 009 (19602200.000.000) 2025-03-14 11:03:17 Job was aborted. Python-initiated action. (by user atlpan) This name: PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n is nowhere to find on my server: [root@arcn-node log]# updatedb [root@arcn-node log]# locate PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n [root@arcn-node log]# I have on my site this kind of error The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE)It became Critical*at*2025-03-14T09:16:44Z due to *METRIC*org.nordugrid.ARC-CE-sw-gcc *Summary:*Script exited with code 127. Could not match GCC version. See /etc/arc/nagios/20-dist.ini for debugging hints. *Message:*'' which is going to be ok in the messag, one minute later The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE) It became Ok *at* 2025-03-14T09:17:53Z due to *METRIC* org.nordugrid.ARC-CE-sw-gcc *Summary: *Found GCC version 11.5.0. *Message: *'' Can you please advice where to look for any clue in solving this mystery? Thank you Felix :-- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Slurm version 24.05.7 is now available
by Marshall Garey 13 Mar '25

13 Mar '25

We are pleased to announce the availability of Slurm version 24.05.7. This release fixes some stability issues in 24.05, including a crash in slurmctld after updating a reservation with an empty nodelist. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.7 > ========================== > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix jobs being scheduled on higher weighted powered down > nodes. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Prevent slurmctld deadlock in the assoc mgr.

1 0

SLURM_JOB_ACCOUNT var missing in prolog
by Jonás Arce 13 Mar '25

13 Mar '25

Hi everyone, The Slurm variable SLURM_JOB_ACCOUNT is blank in my prolog, currently i have a TaskProlog and a Prolog in my system, in TaskProlog this variable works just fine (I use TaskProlog to set env variables), but in my Prolog (which I use to create temporal directories), it's just blank, it's strange beacuse other variables such as SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT or SLURM_JOBID work just fine in both Prolog and TaskProlog, I looked into the Slurm doc and it seems that this var should work everywhere, if someone could shed some light into this or tell me another equivalent var i'd appreciate it a lot. I need it because I need to make this type of temporal directories with my Prolog: /scratch-global/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOBID.

2 1

Job Env Vars in Slurm Core
by Bhaskar Chakraborty 13 Mar '25

13 Mar '25

Hi everyone, I have tried my best to extract custom job environment variables inside a custom select plugin.However, I am not able to get the same. I am able to get the env in a custom client filter plugin during job submission (sbatch --export) but Ineed the same in slurmctld as well. Is there any way this can be achieved. I have looked over few of the internal DS pointers (job_record, job_details etc)but none seem to provide the required functionality. Any insight is appreciated. Regards,Bhaskar.

2 1

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 12 Mar '25

12 Mar '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

3 5

SLURM queue/partition/qos configuration
by vittorio.confuorto＠gmail.com 12 Mar '25

12 Mar '25

Hi everyone, I need some advice on properly configuring my HPC cluster, which consists of three compute nodes (each with 256 cores and 800GB of RAM). The end users belong to three distinct groups: internal, external, and collaborator. I would like the internal users to have full (and exclusive) access to the first two compute nodes. That is, by default, if they submit a job, it should be scheduled on Node1 or Node2, depending on availability (or placed in the queue if necessary). Internal users should also have access to 100 cores and 200GB of RAM on Node3, with this limit applying as a hard cap across all their jobs. The remaining 156 cores and 600GB of RAM on Node3 should be exclusively available to external and collaborator users. Specifically, I would like the following setup: - The resources on Node3 should be split as follows: - 100 cores and 400GB of RAM for external users. - 56 cores and 200GB of RAM for collaborator users. - External users should be able to use the resources dedicated to collaborator users but with lowest priority. This means an external user can submit a job requesting the resources reserved for collaborators, but the job will only start if no collaborator jobs are in the queue. Any job submitted by a collaborator—even after an external user's job—should take priority. The same rule should apply in reverse for collaborators trying to use external resources. - There should be a group/partition/queue that includes all the remaining 156 cores and 600GB of RAM on Node3. Both external and collaborator users should be able to submit jobs to this queue, but a job submitted here should only start if there are no pending jobs in the two dedicated partitions. What is the best way to implement such an architecture? Thanks, everyone, for your support!

1 0

Slurm Job Kill intimation
by Bhaskar Chakraborty 10 Mar '25

10 Mar '25

Hello All, We wish to do some cleanup when user kills a pending slurm job using scancel. Our setup has a client filter plugin which sets some internal variables while interacting to a remote process.This needs to be undone during job termination. Is there a way to do this (using submitter credentials) ? Also, if one is to use a custom select plugin is there a way to inform select that the job is terminated.Presently, we don't find that slurmctld informs select plugin when a pending job is killed. Thanks in advance for any pointers ! Regards,Bhaskar.

1 0

Slurm upgrade using Debian packages
by Matthias Leopold 10 Mar '25

10 Mar '25

Hi, I'm building Slurm Debian packages from SchedMD sources using this tutorial https://www.schedmd.com/slurm/installation-tutorial/. Now I tried upgrading (minor release upgrade within 24.05) using these packages. https://slurm.schedmd.com/upgrades.html tells me to upgrade (a) slurmdbd (b) slurmctld (c) slurmd separately in this order, stopping each service for the upgrade. How can I follow this when the Debian packages have a dependency between slurmdbd + slurmctld that upgrades both packages at the same time? thx Matthias

4 7

allowing spyder-kernels to an interactive session without pam_slurm_adopt and older version of Slurm from OpenHPC repo; parmiko?
by Robert Kudyba 06 Mar '25

06 Mar '25

So the Spyder IDE has this great feature to connect a local/laptop/workstation to a spyder-kernels session <https://docs.spyder-ide.org/current/panes/ipythonconsole.html#using-externa…>. If a user starts an interactive, e.g., srun session, they end up on a compute node which is on one of our older clusters without the pam_slurm_adopt module. Is there a way with a library like Parmiko or this slurmjob <https://github.com/daangeijs/deepops-slurmjob> package? I see for vscode an admin posted the following option <https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment…> : if [[ -v SSH_AUTH_SOCK ]]; then if [[ ${SSH_AUTH_SOCK} =~ vscode ]]; then [ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash fi fi Is there a way to work around the restriction of non admins to connect to the spyder-kernel session? Is there anything similar to batchspawner for Jupyterhub <https://github.com/jupyterhub/batchspawner>? Unless anyone knows of an archive for Slurm slurm-ohpc-18.08 that includes pam_adopt?

1 0

Slurm versions 24.11.3 is now available
by Marshall Garey 06 Mar '25

06 Mar '25

We are pleased to announce the availability of Slurm version 24.11.3. 24.11.3 fixes the database cluster ID generation not being random, a regression in which slurmd -G gave no output, a long-standing crash in slurmctld after updating a reservation with an empty nodelist, and some other minor to moderate bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.3 > ========================== > -- Fix race condition in slurmrestd that resulted in "Requested > data_parser plugin does not support OpenAPI plugin" error being returned > for valid endpoints. > -- If multiple partitions are requested, set the SLURM_JOB_PARTITION > output environment variable to the partition in which the job is running > for salloc and srun in order to match the documentation and the behavior of > sbatch. > -- Fix regression where slurmd -G gives no output. > -- Don't print misleading errors for stepmgr enabled steps. > -- slurmrestd - Avoid connection to slurmdbd for the following > endpoints: > GET /slurm/v0.0.41/jobs > GET /slurm/v0.0.41/job/{job_id} > -- slurmrestd - Avoid connection to slurmdbd for the following > endpoints: > GET /slurm/v0.0.40/jobs > GET /slurm/v0.0.40/job/{job_id} > -- Significantly increase entropy of clusterid portion of the > sluid by seeding the random number generator > -- Avoid changing process name to "watch" from original daemon name. > This could potentially breaking some monitoring scripts. > -- Avoid slurmctld being killed by SIGALRM due to race condition > at startup. > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix race between task/cgroup cpuset and jobacctgather/cgroup. > The first was removing the pid from task_X cgroup directory causing > memory limits to not be applied. > -- srun - Fixed wrongly constructed SLURM_CPU_BIND env variable > that could get propagated to downward srun calls in certain mpi > environments, causing launch failures. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.40. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.41. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.42.

1 0

2025

2024

slurm-users ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users