- slurm-users - lists.schedmd.com

Preemption question
by Kamil Wilczek 31 Mar '25

31 Mar '25

Dear All, I would like to be able to preempt (SUSPEND) a single QoS of a user that blocks the queue for several days. Currently I have about 100 users on the cluster and it seems that setting the "Preempt" option to each QoS (we have personal QoSes) is not optimal. https://slurm.schedmd.com/sacctmgr.html#OPT_Preempt Is there a way to to set an option to this single problematic QoS, saying that the QoS can be preempted by any other QoS? It would be much more administrator-friendly solution ;) Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

3 4

Slurm 24.05 and OpenMPI
by Matthias Leopold 28 Mar '25

28 Mar '25

Hi, I built a small Slurm 21.08 cluster with NVIDIA GPU hardware and NVIDIA deepops framework a couple of years ago. It is based on Ubuntu 20.04 and makes use of the NVIDIA pyxis/enroot container solution. For operational validation I used the nccl-tests application in a container. nccl-tests is compiled with MPI support (OpenMPI 4.1.6 or 4.1.7) and I used it also for validation of MPI jobs. Slurm jobs use "pmix" and tasks are launched via srun (not mpirun). Some of the GPUs can talk to each other via Infiniband, but MPI is rarely used at our site and I'm fully aware that my MPI knowledge is very limited. Still it worked with Slurm 21.08. Now I built a Slurm 24.05 cluster based on Ubuntu 24.04 and started to move hardware there. When I run my nccl-tests container (also with newer software) I see error messages like this: [node1:21437] OPAL ERROR: Unreachable in file ext3x_client.c at line 111 -------------------------------------------------------------------------- The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. There are several options for building PMI support under SLURM, depending upon the SLURM version you are using: version 16.05 or later: you can use SLURM's PMIx support. This requires that you configure and build SLURM --with-pmix. Versions earlier than 16.05: you must use either SLURM's PMI-1 or PMI-2 support. SLURM builds PMI-1 by default, or you can manually install PMI-2. You must then build Open MPI using --with-pmi pointing to the SLURM PMI library location. Please configure as appropriate and try again. -------------------------------------------------------------------------- *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [node1:21437] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! One simple question: Is this related to https://github.com/open-mpi/ompi/issues/12471? If so: is there some workaround? I'm very grateful for any comments. I know that a lot of detail information is missing, but maybe someone can still already give me a hint where to look. Thanks a lot Matthias

3 10

Doubt with SelectTypeParameters in slurm.conf
by Gestió Servidors 28 Mar '25

28 Mar '25

Hello, I'm running some tests in a very small testing environment (before applying in the real scenario). My environment is only a computer with a old Intel i4 with this "lscpu" configuration: Architecture: x86_64 CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz "slurmd -C" returns: "CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2". My slurm.conf is configured with: SelectType=select/cons_tres SelectTypeParameters=CR_Core AccountingStorageTRES=cpu,mem,node,billing,fs/disk,vmem,pages After running a simple "helloworld" test, I have noticed that if SelectTypeParameters=CR_Core, system always reserves me an even number of CPUs (during "pending" time, I can see the real number I have requested, but when job starts "running", that number is increased to the next even number). Howerver, if I reconfigure slurm with SelectTypeParameters=CR_CPU, system reserves me the correct number of CPUs (during "pending" time and, also, during "running" time). I suppose that my system must be configured with CR_CPU, according https://slurm.schedmd.com/cons_tres.html, but could someone confirm that? Thanks.

2 1

bit_cache_init failure on the second time backup controller tries to take control
by Safdar Iqbal 28 Mar '25

28 Mar '25

Hi, We're running into an issue where slurmctld core-dumps with the following error. This happens on the backup controller, if it needs to take over from the primary, _for a second time_. slurmctld: fatal: bit_cache_init: cannot change size once set Has anyone seen this error before? Also if there are any existing discussions and/or tickets related to this, please let me know. Our slurm version is 24.11.1. ________________ Steps to reproduce: 1. On a healthy cluster, we make the primary controller unavailable. Since we're running our cluster on cloud VMs, we cause this by stopping the primary controller VM. 2. From the logs we can see the backup controller take over, and log the message "Running as primary controller" 3. We then start the primary again, making sure the IP addresses and hostnames stay consistent. Once slurmctld on the primary has started and taken back control, we can see the log "Running as primary controller" on that VM. 4. We then stop the primary controller VM again, causing the backup to try taking the control a second time. This time however the slurmctld on the backup coredumps, with following log entries from journalctl -u slurmctld: slurmctld: fatal: bit_cache_init: cannot change size once set slurmctld.service: Main process exited, code=dumped, status=6/ABRT slurmctld.service: Failed with result 'core-dump'. Thanks! - Safdar

1 0

How limit CPUs per node in a partition
by Gestió Servidors 27 Mar '25

27 Mar '25

Hello, I have a testing partition with only a node. That server has 12 CPUs (it's a very old server) (2 sockets, 6 cores per socket, 1 thread per core). That partition, called "test.q" only has that node, so by default, partition test.q has 12 CPUs (all from testing node). However, now I would like to decrease that CPU number from 12 to 4, for example. "slurmd -C" returns "CPUs=12 Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1" as I have said to you before and if in slurm.conf I configure partition in this way "PartitionName=test.q Nodes=test-node MaxTime=10:00 State=UP AllocNodes=clus-login MaxCPUsPerNode=4", partition continues accepting jobs with 12 CPUs. Is it any wrong in my configuration? How could I get my purpose? Thanks. -- [cid:image001.jpg@01DB9F0E.FDD9A7A0]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image002.jpg@01DB9F0E.FDD9A7A0]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

1 0

Re: Using more cores/CPUs that requested with
by Shunran Zhang 26 Mar '25

26 Mar '25

Ugh I think I did not catch up with the docs. I started with a system that defaults to cgroup v1 but the Slurm doc for that plugin is NOT available at that time. Thus I converted everything to cgroup v2. It appears that they are both supported and that documentation issue is more on the dev side than admin side. Thanks for pointing that out. I misinterpreted the "coming soon" part of cgroup v1 plugin and the "legacy" naming for "do not use". It should be fine. 2025年3月27日(木) 0:48 Williams, Jenny Avis <jenny_williams(a)unc.edu>: > “ … As cgroup is likely not supposed to be used in newer deployments of > Slurm.” > > > > I am curious about this statement. Would someone expand on this, to either > support or counter it? > > > > Jenny Williams > > UNC Chapel Hill > > > > > > *From:* Shunran Zhang via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Wednesday, March 26, 2025 10:52 AM > *To:* Gestió Servidors <sysadmin.caos(a)uab.cat> > *Cc:* Slurm User Community List <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Re: Using more cores/CPUs that requested with > > > > If you are letting systemd taking most things over, you got systemd-cgtop > that work better than top for your case. There is also systemd-cgls for > non-interactive listing. > > > > Also mind to check if you are using cgroup2? A mount to check your cgroup > would suffice. As cgroup is likely not supposed to be used in newer > deployments of Slurm. > > > > > > 2025年3月26日(水) 17:14 Gestió Servidors via slurm-users < > slurm-users(a)lists.schedmd.com>: > > Hello, > > > > Thanks for your answers. I will try now!! One more question: is there any > way to check if Cgroups restrictions is working fine during a “running” job > or during SLURM scheduling process? > > > > Thanks again! > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > >

1 0

Re: Using more cores/CPUs that requested with
by Gestió Servidors 26 Mar '25

26 Mar '25

Hello, Thanks for your answers. I will try now!! One more question: is there any way to check if Cgroups restrictions is working fine during a "running" job or during SLURM scheduling process? Thanks again!

4 3

Using more cores/CPUs that requested with sbatch
by Gestió Servidors 25 Mar '25

25 Mar '25

Hello, I would like to know if there is any mechanism to avoid a user to do "cheating" when he submits a job. For example, a user submits a OpenMP job requesting 1 node (-N 1) and 2 tasks (-n 2). However, inside his script (or compiled in his binany), user uses more than 2 ntasks, for example, all free cores. For SLURM, job is only using 2, but really, node is 100% assigned to that job. SLURM can control that "abuse" in some way? Cgroups? Thanks.

3 2

Re: Using more cores/CPUs that requested with sbatch
by Megan 25 Mar '25

25 Mar '25

Slurm can restrict jobs to their allocated resources using cgroups. To use cgroups you will have to configure the TaskPlugin cgroup option and specify how the cgroups should behave in cgroup.conf. slurm.conf TaskPlugin=task/cgroup Slurm Workload Manager - slurm.conf <https://slurm.schedmd.com/slurm.conf.html#OPT_task/cgroup> cgroup.conf ConstrainCores=yes Slurm Workload Manager - cgroup.conf <https://slurm.schedmd.com/cgroup.conf.html#OPT_ConstrainCores> Regards, --Megan

1 0

slurmrestd equivalent to "srun -n 10 echo HELLO"
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Community, I'm starting to experiment with slurmrestd for a new app we're writing. I'm having trouble understanding one aspect of submitting jobs. When I run something like `srun -n 10 echo HELLO', I get HELLO returned to my console/stdout 10x. When I submit this command as a script to the /jobs/submit route, I get success/200, but *I cannot determine how to get the console output of HELLO 10x in any form*. It's not in my stdout log for that job even though I can verify that the job ran successfully. Any suggestions? -- Thanks, Daniel Healy

3 2

Priority/Top seems to not be working
by Dan Healy 24 Mar '25

24 Mar '25

Hi Slurm Users, I have a newer install (23.11.3) and the priority of most all jobs is 1 and other small numbers. In previous versions, I would see numbers like 2^32. I have the multifactor plugin configured and confirmed it's in-use when I show the config. When I run `scontrol top` for a given job, the priority number doesn't change from 1. I have lots of jobs running at the moment and many queued as well. I've been using `scontrol top` for years and have seen the priority successfully change. I'm repeating this now with the newer version of Slurm and now seeing it work the same. Are you seeing this too? -- Thanks, Daniel Healy

1 0

missing jobs from queue
by Felix 14 Mar '25

14 Mar '25

Hello at site RO-14-ITIM, we have a very strange problem. A ticket was raised with the following: "Hi, The new ticket (Pilots disappearing from local batch system at ITIM) has been created by "*Timo Wilken*". Group: NGIs › NGI_RO Owner: - State: in progress Information: Hi, Pilots seem to be lost by the local batch system at ITIM. >50% of pilots fail: https://apfmon.lancs.ac.uk/CERN_central_A:RO-14-ITIM-arcn-node?state=fault Example log: https://aipanda156.cern.ch/condor_logs_2/25-03-12_13/grid.15478691.1.log Error message: "ARC job failed: LRMS error: (-1) Job missing from SLURM" Cheers, Timo" The problem is I can not find any of the jobs that are failing on the site: for example: the last job: https://aipanda023.cern.ch/condor_logs_2/25-03-14_09/grid.19602200.0.log with stdlog: 000 (19602200.000.000) 2025-03-14 09:28:03 Job submitted from host: <137.138.31.125:38090?addrs=137.138.31.125-38090+[2001-1458-d00-19--75]-38090&alias=aipanda023.cern.ch> ... 027 (19602200.000.000) 2025-03-14 09:28:13 Job submitted to grid resource GridResource: arc arcn-node.itim-cj.ro:443 GridJobId: arc arcn-node.itim-cj.ro:443 PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n ... 001 (19602200.000.000) 2025-03-14 09:40:51 Job executing on host: arc arcn-node.itim-cj.ro:443 ... 012 (19602200.000.000) 2025-03-14 09:40:58 Job was held. ARC job failed: LRMS error: (-1) Job missing from SLURM Code 0 Subcode 0 ... 009 (19602200.000.000) 2025-03-14 11:03:03 Job was aborted. Python-initiated action. (by user atlpan) ... 009 (19602200.000.000) 2025-03-14 11:03:17 Job was aborted. Python-initiated action. (by user atlpan) This name: PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n is nowhere to find on my server: [root@arcn-node log]# updatedb [root@arcn-node log]# locate PPVNDm5VGD7ngvuSSqSAreymYz3jwmOETUEm1dOSDmbfXLDmHXmz9n [root@arcn-node log]# I have on my site this kind of error The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE)It became Critical*at*2025-03-14T09:16:44Z due to *METRIC*org.nordugrid.ARC-CE-sw-gcc *Summary:*Script exited with code 127. Could not match GCC version. See /etc/arc/nagios/20-dist.ini for debugging hints. *Message:*'' which is going to be ok in the messag, one minute later The ENDPOINT affected is arcn-node.itim-cj.ro (ARC-CE) It became Ok *at* 2025-03-14T09:17:53Z due to *METRIC* org.nordugrid.ARC-CE-sw-gcc *Summary: *Found GCC version 11.5.0. *Message: *'' Can you please advice where to look for any clue in solving this mystery? Thank you Felix :-- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Slurm version 24.05.7 is now available
by Marshall Garey 13 Mar '25

13 Mar '25

We are pleased to announce the availability of Slurm version 24.05.7. This release fixes some stability issues in 24.05, including a crash in slurmctld after updating a reservation with an empty nodelist. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.7 > ========================== > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix jobs being scheduled on higher weighted powered down > nodes. > -- Fix memory leak when RestrictedCoresPerGPU is enabled. > -- Prevent slurmctld deadlock in the assoc mgr.

1 0

SLURM_JOB_ACCOUNT var missing in prolog
by Jonás Arce 13 Mar '25

13 Mar '25

Hi everyone, The Slurm variable SLURM_JOB_ACCOUNT is blank in my prolog, currently i have a TaskProlog and a Prolog in my system, in TaskProlog this variable works just fine (I use TaskProlog to set env variables), but in my Prolog (which I use to create temporal directories), it's just blank, it's strange beacuse other variables such as SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT or SLURM_JOBID work just fine in both Prolog and TaskProlog, I looked into the Slurm doc and it seems that this var should work everywhere, if someone could shed some light into this or tell me another equivalent var i'd appreciate it a lot. I need it because I need to make this type of temporal directories with my Prolog: /scratch-global/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOBID.

2 1

Job Env Vars in Slurm Core
by Bhaskar Chakraborty 13 Mar '25

13 Mar '25

Hi everyone, I have tried my best to extract custom job environment variables inside a custom select plugin.However, I am not able to get the same. I am able to get the env in a custom client filter plugin during job submission (sbatch --export) but Ineed the same in slurmctld as well. Is there any way this can be achieved. I have looked over few of the internal DS pointers (job_record, job_details etc)but none seem to provide the required functionality. Any insight is appreciated. Regards,Bhaskar.

2 1

SLURM queue/partition/qos configuration
by vittorio.confuorto＠gmail.com 12 Mar '25

12 Mar '25

Hi everyone, I need some advice on properly configuring my HPC cluster, which consists of three compute nodes (each with 256 cores and 800GB of RAM). The end users belong to three distinct groups: internal, external, and collaborator. I would like the internal users to have full (and exclusive) access to the first two compute nodes. That is, by default, if they submit a job, it should be scheduled on Node1 or Node2, depending on availability (or placed in the queue if necessary). Internal users should also have access to 100 cores and 200GB of RAM on Node3, with this limit applying as a hard cap across all their jobs. The remaining 156 cores and 600GB of RAM on Node3 should be exclusively available to external and collaborator users. Specifically, I would like the following setup: - The resources on Node3 should be split as follows: - 100 cores and 400GB of RAM for external users. - 56 cores and 200GB of RAM for collaborator users. - External users should be able to use the resources dedicated to collaborator users but with lowest priority. This means an external user can submit a job requesting the resources reserved for collaborators, but the job will only start if no collaborator jobs are in the queue. Any job submitted by a collaborator—even after an external user's job—should take priority. The same rule should apply in reverse for collaborators trying to use external resources. - There should be a group/partition/queue that includes all the remaining 156 cores and 600GB of RAM on Node3. Both external and collaborator users should be able to submit jobs to this queue, but a job submitted here should only start if there are no pending jobs in the two dedicated partitions. What is the best way to implement such an architecture? Thanks, everyone, for your support!

1 0

Slurm Job Kill intimation
by Bhaskar Chakraborty 10 Mar '25

10 Mar '25

Hello All, We wish to do some cleanup when user kills a pending slurm job using scancel. Our setup has a client filter plugin which sets some internal variables while interacting to a remote process.This needs to be undone during job termination. Is there a way to do this (using submitter credentials) ? Also, if one is to use a custom select plugin is there a way to inform select that the job is terminated.Presently, we don't find that slurmctld informs select plugin when a pending job is killed. Thanks in advance for any pointers ! Regards,Bhaskar.

1 0

Slurm upgrade using Debian packages
by Matthias Leopold 10 Mar '25

10 Mar '25

Hi, I'm building Slurm Debian packages from SchedMD sources using this tutorial https://www.schedmd.com/slurm/installation-tutorial/. Now I tried upgrading (minor release upgrade within 24.05) using these packages. https://slurm.schedmd.com/upgrades.html tells me to upgrade (a) slurmdbd (b) slurmctld (c) slurmd separately in this order, stopping each service for the upgrade. How can I follow this when the Debian packages have a dependency between slurmdbd + slurmctld that upgrades both packages at the same time? thx Matthias

4 7

allowing spyder-kernels to an interactive session without pam_slurm_adopt and older version of Slurm from OpenHPC repo; parmiko?
by Robert Kudyba 06 Mar '25

06 Mar '25

So the Spyder IDE has this great feature to connect a local/laptop/workstation to a spyder-kernels session <https://docs.spyder-ide.org/current/panes/ipythonconsole.html#using-externa…>. If a user starts an interactive, e.g., srun session, they end up on a compute node which is on one of our older clusters without the pam_slurm_adopt module. Is there a way with a library like Parmiko or this slurmjob <https://github.com/daangeijs/deepops-slurmjob> package? I see for vscode an admin posted the following option <https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment…> : if [[ -v SSH_AUTH_SOCK ]]; then if [[ ${SSH_AUTH_SOCK} =~ vscode ]]; then [ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash fi fi Is there a way to work around the restriction of non admins to connect to the spyder-kernel session? Is there anything similar to batchspawner for Jupyterhub <https://github.com/jupyterhub/batchspawner>? Unless anyone knows of an archive for Slurm slurm-ohpc-18.08 that includes pam_adopt?

1 0

Slurm versions 24.11.3 is now available
by Marshall Garey 06 Mar '25

06 Mar '25

We are pleased to announce the availability of Slurm version 24.11.3. 24.11.3 fixes the database cluster ID generation not being random, a regression in which slurmd -G gave no output, a long-standing crash in slurmctld after updating a reservation with an empty nodelist, and some other minor to moderate bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.3 > ========================== > -- Fix race condition in slurmrestd that resulted in "Requested > data_parser plugin does not support OpenAPI plugin" error being returned > for valid endpoints. > -- If multiple partitions are requested, set the SLURM_JOB_PARTITION > output environment variable to the partition in which the job is running > for salloc and srun in order to match the documentation and the behavior of > sbatch. > -- Fix regression where slurmd -G gives no output. > -- Don't print misleading errors for stepmgr enabled steps. > -- slurmrestd - Avoid connection to slurmdbd for the following > endpoints: > GET /slurm/v0.0.41/jobs > GET /slurm/v0.0.41/job/{job_id} > -- slurmrestd - Avoid connection to slurmdbd for the following > endpoints: > GET /slurm/v0.0.40/jobs > GET /slurm/v0.0.40/job/{job_id} > -- Significantly increase entropy of clusterid portion of the > sluid by seeding the random number generator > -- Avoid changing process name to "watch" from original daemon name. > This could potentially breaking some monitoring scripts. > -- Avoid slurmctld being killed by SIGALRM due to race condition > at startup. > -- Fix slurmctld crash when after updating a reservation with an empty > nodelist. The crash could occur after restarting slurmctld, or if > downing/draining a node in the reservation with the REPLACE or REPLACE_DOWN > flag. > -- Fix race between task/cgroup cpuset and jobacctgather/cgroup. > The first was removing the pid from task_X cgroup directory causing > memory limits to not be applied. > -- srun - Fixed wrongly constructed SLURM_CPU_BIND env variable > that could get propagated to downward srun calls in certain mpi > environments, causing launch failures. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.40. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.41. > -- slurmrestd - Fix possible memory leak when parsing arrays with > data_parser/v0.0.42.

1 0

broken SLURM-PMIX out-of-band communication on v24.11.0 with PMIx v5
by Bertini, Denis Dr. 06 Mar '25

06 Mar '25

Hi there, Using slurm v24.11.0 together with openMPI 5.0.7 built with openpmix v5.0.6 i am facing a systematical crash at process wiring-up phase when launching standard MPI job (OSU benchmarks ) on our new AMD compute nodes ( amd-epyc 9654, 192 phys. cores +HT ) running Rocky Linux 9.4 OS The typical error reads: slurmstepd: error: mpi/pmix_v5: pmixp_p2p_send: ccexe0094 [4]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit slurmstepd: error: mpi/pmix_v5: _slurm_send: ccexe0094 [4]: pmixp_server.c:1581: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.656.0, size = 46979, hostlist: (null) srun: error: Node failure on ccexe0091 after such a error as you can see the node move to state down It looks like the slurmstep pmix_server can not use the local socket at var/spool/slurmd/stepd.slurm.pmix.job_id.0 for inter-node communication . * On one AMD node ( same SLURM version, same cluster setup ) wiring up works smoothly even at core satuation (192 cores used) * On Intel node (intel,xeon,gold6248r, 48 cores ) wiring-up works even with multiple node without any problem * When the problematic AMD nodes are setup as dynamic node<https://slurm.schedmd.com/dynamic_nodes.html> the wiring-up phase with multiple nodes works perfectly, without any issue Has anybody experienced this kind of problem? Any idea what could be the reason for that? I also add that when the problematic AMD nodes are setup as dynamic node<https://slurm.schedmd.com/dynamic_nodes.html> the wiring-up phase with multiple nodes works perfectly, without any issue Cheers, Denis * --------- Denis Bertini Abteilung: CIT Ort: SB3 2.265a Tel: +49 6159 71 2240 Fax: +49 6159 71 2986 E-Mail: d.bertini(a)gsi.de GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing Directors / Geschäftsführung: Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: Ministerialdirigent Dr. Volkmar Dietz

1 0

error in logs, does it mean anything?
by Steven Jones 06 Mar '25

06 Mar '25

[2025-03-06T00:26:06.002] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port [2025-03-06T00:31:06.002] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port [2025-03-06T00:36:06.001] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port [2025-03-06T00:41:07.001] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port regards Steven

1 0

Re: mariadb refusing access
by Steven Jones 05 Mar '25

05 Mar '25

Yes that is set, and my settings are, # slurmDBD info DbdAddr=localhost DbdHost=vuwunicoslurmd3.ods.vuw.ac.nz #DbdHost=localhost DbdPort=6819 SlurmUser=slurm #MessageTimeout=300 DebugLevel=verbose #DefaultQOS=normal,standby LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd/slurmdbd.pid #PluginDir=/usr/lib/slurm #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql #StorageHost=localhost StorageHost=localhost #StoragePort=1234 #StoragePort=3306 StoragePass=xxxxxxxx StorageUser=slurm StorageLoc=slurm_acct_db #ssj regards Steven ________________________________ From: Sarlo, Jeffrey S <JSarlo(a)Central.UH.EDU> Sent: Thursday, 6 March 2025 7:42 am To: Steven Jones <steven.jones(a)vuw.ac.nz> Subject: RE: [slurm-users] Re: mariadb refusing access You don't often get email from jsarlo(a)central.uh.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Do you have the correct password listed in the slurmdbd.conf file? That was toward the bottom of the Wiki that Ole sent. Jeff From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com> Sent: Wednesday, March 5, 2025 12:39 PM To: slurm-users(a)lists.schedmd.com; Ole Holm Nielsen <Ole.H.Nielsen(a)fysik.dtu.dk> Subject: [slurm-users] Re: mariadb refusing access Sorry, I dont follow you, Pass word works fine and grant looks OK. [root@vuwunicoslurmd3 ~]# mysql -u slurm -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 208 Server version: 10.11.10-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show grants; +--------------------------------------------------------------------------------------------------------------+ | Grants for slurm@localhost | +--------------------------------------------------------------------------------------------------------------+ | GRANT USAGE ON *.* TO `slurm`@`localhost` IDENTIFIED BY PASSWORD '*2F4EEC707189D8B5E329829B9303058EF7585569' | | GRANT ALL PRIVILEGES ON `slurm_acct_db`.* TO `slurm`@`localhost` | +--------------------------------------------------------------------------------------------------------------+ 2 rows in set (0.000 sec) MariaDB [(none)]> Isnt there something about password encryption / password hash? regards Steven ________________________________ From: Ole Holm Nielsen via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Sent: Wednesday, 5 March 2025 9:25 pm To: slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Subject: [slurm-users] Re: mariadb refusing access On 3/5/25 02:23, Steven Jones via slurm-users wrote: > In the logs I am seeing, > > root@vuwunicoslurmd3 mariadb]# tail -f mariadb.log > 2025-03-04 19:01:32 12565 [Warning] Access denied for user > 'slurm'@'localhost' (using password: YES) > 2025-03-04 19:06:19 12566 [Warning] Access denied for user > 'slurm'@'localhost' (using password: YES) > > However mysql -u slurm -p works just fine so it seems to be a config > error for slurmdbd It seems that you didn't select a suitable slurm user’s database password, see https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.fysi…<https://urldefense.com/v3/__https:/apc01.safelinks.protection.outlook.com/?…> IHTH, Ole -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 1

mariadb refusing access
by Steven Jones 05 Mar '25

05 Mar '25

In the logs I am seeing, root@vuwunicoslurmd3 mariadb]# tail -f mariadb.log 2025-03-04 19:01:32 12565 [Warning] Access denied for user 'slurm'@'localhost' (using password: YES) 2025-03-04 19:06:19 12566 [Warning] Access denied for user 'slurm'@'localhost' (using password: YES) However mysql -u slurm -p works just fine so it seems to be a config error for slurmdbd Any ideas please? regards Steven

4 6

Tracking costs - variable costs per partition
by Jeffrey A Dusenberry 05 Mar '25

05 Mar '25

Hello - We're in a similar situation as was described here: https://groups.google.com/g/slurm-users/c/eBDslkwoFio where we want to track (and control) costs on a fairly heterogenous system with different billing weights per partition. The solution proposed seems like it would work rather well, except our use of fairshare seems to interfere with the billing values we would want to use to limit usage based on credits granted. We have PriorityDecayHalfLife set on our system, so that billing value (GrpTRESRaw) seems to drop with time. Is there a way to implement something similar on an otherwise fairshare-based system? Thanks, Jeff

1 0

orted errors
by Berg, Stephen P CIV USN NRL DET SSC MS (USA) 04 Mar '25

04 Mar '25

I'm running a small-ish slurm grid, 87 nodes with various hardware. On a few occasions lately users submitting jobs will get an orted error and the job fails. Try again a few hours later or the next day and the same job runs just fine. Google-fu indicated it might be a DNS issue if for whatever reason a node couldn't figure out the address for other nodes in the job. So I populated the /etc/hosts on each node with a complete listing of all the nodes so there wouldn't be any reliance on DNS. And that very afternoon another job failed with orted. So it seems at least in my case DNS isn't the issue. What's the best way to troubleshoot this when orted fails but doesn't give any sort of error to indicate what the root cause of the failure might be? And I also can't predictably induce the failure, just have to wait until it randomly chokes.

2 1

Slurm versions 24.11.2 and 24.05.6 are now available
by Marshall Garey 04 Mar '25

04 Mar '25

We are pleased to announce the availability of Slurm versions 24.11.2 and 24.05.6. 24.11.2 fixes a variety of minor to major bugs. Fixed regressions include loading non-default QOS on pending jobs from pre-24.11 state, pending jobs displaying QOS=(null) when not explicitly requesting a QOS, running jobs that requested multiple partitions potentially having an incorrect partition when slurmctld is restarted, and burst_buffer.lua failing if slurm.conf is in a non-standard location. This release also fixes a few crashes in slurmctld: crashing when a job that can preempt requests --test-only, crasing when the scheduler evaluates a job on nodes with suspended jobs, and crashing due to a long-standing bug causing a job record without job_resrcs. 24.05.6 fixes sattach with auth/slurm, a slurmrestd crash when using data_parser/v0.0.40, a slurmctld crash when using job suspension, a performance regression for RPCs with large amounts of data, and some other moderate severity bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.2 > ========================== > -- Fix segfault when submitting --test-only jobs that can preempt. > -- Fix regression introduced in 23.11 that prevented the following > flags from being added to a reservation on an update: > DAILY, HOURLY, WEEKLY, WEEKDAY, and WEEKEND. > -- Fix crash and issues evaluating job's suitability for running in > nodes with already suspended job(s) there. > -- Slurmctld will ensure that healthy nodes are not reported as > UnavailableNodes in job reason codes. > -- Fix handling of jobs submitted to a current reservation with > flags OVERLAP,FLEX or OVERLAP,ANY_NODES when it overlaps nodes with a > future maintenance reservation. When a job submission had a time limit that > overlapped with the future maintenance reservation, it was rejected. Now > the job is accepted but stays pending with the reason "ReqNodeNotAvail, > Reserved for maintenance". > -- pam_slurm_adopt - avoid errors when explicitly setting > some arguments to the default value. > -- Fix qos preemption with PreemptMode=SUSPEND > -- slurmdbd - When changing a user's name update lineage > at the same time. > -- Fix regression in 24.11 in which burst_buffer.lua does not > inherit the SLURM_CONF environment variable from slurmctld and fails to run > if slurm.conf is in a non-standard location. > -- Fix memory leak in slurmctld if select/linear and the > PreemptParameters=reclaim_licenses options are both set in slurm.conf. > Regression in 24.11.1. > -- Fix running jobs, that requested multiple partitions, from > potentially being set to the wrong partition on restart. > -- switch/hpe_slingshot - Fix compatibility with newer cxi > drivers, specifically when specifying disable_rdzv_get. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace > from any fatal() message. > -- Fix printing invalid address in rate limiting log statement. > -- sched/backfill - Fix node state PLANNED not being cleared from > fully allocated nodes during a backfill cycle. > -- select/cons_tres - Fix future planning of jobs with bf_licenses. > -- Prevent redundant "on_data returned rc: Rate limit exceeded, > please retry momentarily" error message from being printed in > slurmctld logs. > -- Fix loading non-default QOS on pending jobs from pre-24.11 state. > -- Fix pending jobs displaying QOS=(null) when not explicitly > requesting a QOS. > -- Fix segfault issue from job record with no job_resrcs > -- Fix failing sacctmgr delete/modify/show account operations > with where clauses. > -- Fix regression in 24.11 in which Slurm daemons started catching > several SIGTSTP, SIGTTIN and SIGUSR1 signals and ignored them, while before > they were not ignoring them. This also caused slurmctld to not being > able to shutdown after a SIGTSTP because slurmscriptd caught the signal > and stopped while slurmctld ignored it. Unify and fix these situations and > get back to the previous behavior for these signals. > -- Document that SIGQUIT is no longer ignored by slurmctld, > slurmdbd, and slurmd in 24.11. As of 24.11.0rc1, SIGQUIT is identical to > SIGINT and SIGTERM for these daemons, but this change was not documented. > -- Fix not considering nodes marked for reboot without ASAP > in the scheduler. > -- Remove the boot^ state on unexpected node reboot after > return to service. > -- Do not allow new jobs to start on a node which is being rebooted > with the flag nextstate=resume. > -- Prevent lower priority job running after cancelling an ASAP reboot. > -- Fix srun jobs starting on nextstate=resume rebooting nodes. > > * Changes in Slurm 24.05.6 > ========================== > -- data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when > dumping data with v0.0.40+complex data parser. > -- Fix sattach when using auth/slurm. > -- scrun - Add support '--all' argument for kill subcommand. > -- Fix performance regression while packing larger RPCs. > -- Fix crash and issues evaluating job's suitability for running in > nodes with already suspended job(s) there. > -- Fixed a job requeuing issue that merged job entries into the > same SLUID when all nodes in a job failed simultaneously. > -- switch/hpe_slingshot - Fix compatibility with newer cxi > drivers, specifically when specifying disable_rdzv_get. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace > from any fatal() message.

7 6

Re: how to set slurmdbd.conf if using two slurmdb node with HA database?
by taleintervenor＠sjtu.edu.cn 27 Feb '25

27 Feb '25

Do you mean the second configuration scheme? I think configuring `dbdhost=localhost` is the same as configuring ` DbdAddr =nodeC` and ` DbdAddr =nodeD` on the two nodes respectively. The key point is whether we should set the DbdBackupHost option and how it work? 发件人: Daniel Letai <dani(a)letai.org.il> 发送时间: 2025年2月19日 18:21 收件人: slurm-users(a)lists.schedmd.com 主题: [slurm-users] Re: how to set slurmdbd.conf if using two slurmdb node with HA database? I'm not sure it will work, didn't test it, but could you just do `dbdhost=localhost` to solve this? On 18/02/2025 11:59, hermes via slurm-users wrote: The deployment scenario is as follows: nodeA nodeB (slurmctld) (backup slurmctld) | \-------------------------------/ | | / \ | nodeC nodeD (slurmdbd) (backup slurmdbd) (mysql) <--multi master replica--> (mysql) Since the database is multi-master replicated, the slurmdbd should only talk to the mysql on its own node. In such case, how should we set the slurmdbd.conf? The conf file contains options “DbdAddr”, “DbdHost” and “DbdBackupHost”. Should they be consistent between nodeA-2 and nodeB-2? Such as: DbdAddr = nodeC | DbdAddr = nodeC DbdHost = nodeC | DbdHost = nodeC DbdBackupHost = nodeD | DbdBackupHost = nodeD StorageHost = nodeC | StorageHost = nodeD Or maybe just set different conf and don’t use the “DbdBackupHost” like: DbdAddr = nodeC | DbdAddr = nodeD DbdHost = nodeC | DbdHost = nodeD StorageHost = nodeC | StorageHost = nodeD I’m quite confused about the usage of DbdAddr and DbdHost. What is the difference between them and why only DbdHost has the backup one? Another confusing point is how DbdBackupHost work. I guess It is slurmctld that is responsible for selecting the available slurmdbd. Since the slurm.conf already contains “AccountingStorageHost” and “AccountingStorageBackupHost”, why we need set backupdbd again on slurmdbd side?

4 10

Limit CPUs per job (but not per user, partition or node)
by Herbert Fruchtl 26 Feb '25

26 Feb '25

We have a cluster with multi-core nodes (168) that can be shared by multiple jobs at the same time. How do I configure a partition such that it only accepts jobs requesting up to (say) 8 cores, but will run multiple jobs at the same time? The following is apparently not working: PartitionName=debug Nodes=node01 MaxTime=02:00:00 DefMemPerCPU=1000 MaxCPUsPerNode=8 Default=NO It allows one job using 8 cores, but a second one will not start because the limit is apparently for the partition as a whole. Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

3 2

Re: MinTRES in QoS and power saving
by Patryk Bełzak 26 Feb '25

26 Feb '25

Hi, there was this issue raised some time ago: https://www.mail-archive.com/slurm-users@lists.schedmd.com/msg10799.html We're experiencing exactly the same issue now with GPU nodes in power saving, some (but not all) jobs doesn't start because of that, and it's annoying users - badly. Anyone experienced something similiar? Or maybe Stefan has a solution? (we're on version 24.05.5) Best regards Patryk Belzak. -- HPC Administrator Wroclaw Centre for Networking and Supercomputing

1 0

Shard conf weirdness
by Reed Dier 24 Feb '25

24 Feb '25

Hoping someone can help me pin down the weirdness I’m experiencing. There are actually two issues, I’ve run into, the root issue, and then something odd when trying to work around the root issue. v23.11.10 - Ubuntu 22.04 - slurm-smd debs built <https://slurm.schedmd.com/archive/slurm-23.11.10/quickstart_admin.html#debu…> from the tarball. I have 2 slurmctld daemons, and 1 slurmdbd daemon. The slurm.conf is consistent across the cluster, all ctld,dbd,slurmd, have the same shasum hash. The backup slurmctld does not appear to like the gres configured on my gpu nodes, despite the primary slurmctld having no issues. When failing over with scontrol takeover, I get the following log messages on the secondary slurmctld, where it complains about a reported, but not configured typed gres/shard. > [2025-02-21T05:02:02.017] error: Setting node $HOST2 state to INVAL with reason:gres/shard type (p100) reported but not configured > [2025-02-21T05:02:02.018] drain_nodes: node $HOST2 state set to DRAIN > [2025-02-21T05:02:02.018] error: _slurm_rpc_node_registration node=$HOST2: Invalid argument > [2025-02-21T05:02:02.020] error: Setting node $HOST3 state to INVAL with reason:gres/shard type (p40) reported but not configured > [2025-02-21T05:02:02.020] drain_nodes: node $HOST3 state set to DRAIN > [2025-02-21T05:02:02.020] error: _slurm_rpc_node_registration node=$HOST3: Invalid argument > [2025-02-21T05:02:02.023] error: Setting node $HOST1 state to INVAL with reason:gres/shard type (t4) reported but not configured > [2025-02-21T05:02:02.020] drain_nodes: node $HOST1 state set to DRAIN > [2025-02-21T05:02:02.023] error: _slurm_rpc_node_registration node=$HOST1: Invalid argument And looking at those hosts in the slurm.conf, the shards are not typed, but generic. > NodeName=$HOST1 [SNIP] State=UNKNOWN Gres=gpu:t4:2,shard:8 > NodeName=$HOST2 [SNIP] State=UNKNOWN Gres=gpu:p100:2,shard:8 > NodeName=$HOST3 [SNIP] State=UNKNOWN Gres=gpu:p40:2,gpu:p100:2,shard:16 I should also point out at this point that my gres.conf is just AutoDetect=nvml. I am not explicitly mapping any devices. So this is problem 1, the slurmctld behaving differently between daemons, where the primary has zero issue with the configuration, but the secondary balking and draining the nodes consistently. Moving on to problem 2, which is me trying to solve the issue, and running into a different issue. I decided to try to add the typed gres/shards to both the AccountingStorageTRES, and the NodeName lists. > NodeName=$HOST1 [SNIP] State=UNKNOWN Gres=gpu:t4:2,shard:t4:8 > NodeName=$HOST2 [SNIP] State=UNKNOWN Gres=gpu:p100:2,shard:p100:8 > NodeName=$HOST3 [SNIP] State=UNKNOWN Gres=gpu:p40:2,gpu:p100:2,shard:p40:8,shard:p100:8 Now the secondary slurmctld is happy and no longer immediately drains the gpu nodes for the invalid gres. However, the node ($HOST3) with a mixed set of gpus is now complaining. > [2025-02-23T21:41:05.311] gpu/nvml: _get_system_gpu_list_nvml: 4 GPU system device(s) detected > [2025-02-23T21:41:05.311] fatal: _build_shared_list: bad configuration, multiple configurations without "File" No matter what I tried, any line of the NodeName with multiple shard:$type:$num generates the _build_shared_list error above. I tried re-ordering the list as gpu,shard,gpu,shard to no avail, I tried having untyped shards, with typed shards after that, but then it complained about too many shards (double the intended number, $untyped + $typed = too many) So my current solution is to have the slurm conf in sync everywhere BUT the host(s) with multiple gpu models in the same host, where I had to revert to untyped gres on those slurmd’s, but on the slurmctlds they are typed gres. Hopefully I’ve done a decent job of explaining the corner case, enough that someone can point me in the direction of figuring out whats going on, and what is the “correct” way of doing this. I tried increasing the slurmd and slurmctld logging to debug2, to nothing that stood out beyond what was already gathered. > [2025-02-23T21:46:44.895] debug: GRES[shard] Type:p100 Count:8 Cores(88):(null) Links:(null) Flags:HAS_TYPE File:(null) UniqueId:(null) > [2025-02-23T21:46:44.895] debug: GRES[shard] Type:p40 Count:8 Cores(88):(null) Links:(null) Flags:HAS_TYPE File:(null) UniqueId:(null) > [2025-02-23T21:46:44.895] fatal: _build_shared_list: bad configuration, multiple configurations without "File" Any ideas are greatly appreciated, Reed

1 0

Plese help [CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=1]
by Hugo Solís 24 Feb '25

24 Feb '25

I am installing slurm on a small cluster of 3 identical computers at the university. It turns out that I get the message when starting: error: NodeNames=hugo-big-1 CPUs=24 match no Sockets, Sockets*CoresPerSocket or Sockets*CoresPerSocket*ThreadsPerCore. Resetting CPUs. error: NodeNames=hugo-big-2 CPUs=24 match no Sockets, Sockets*CoresPerSocket or Sockets*CoresPerSocket*ThreadsPerCore. Resetting CPUs. error: NodeNames=hugo-big-3 CPUs=24 match no Sockets, Sockets*CoresPerSocket or Sockets*CoresPerSocket*ThreadsPerCore. Resetting CPUs. if I run sudo slurmd -C I get NodeName=hugo-big-1 CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=128586 reading many post it seems there is a bug starting in version 18. However I had modify the slurm.conf file NodeName=hugo-big-1 CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=128586 Gres=gpu:nvidia_geforce_rtx_3090:2 to several different version where I do not specify CPUs or sockets and I failed everytime. I do need help, any idea? warm regards Hugo Solis BTW. if I run lscpu I got that any product of Thread, cores or sockets do not result in 24. Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Vendor ID: GenuineIntel Model name: 12th Gen Intel(R) Core(TM) i9-12900K CPU family: 6 Model: 151 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 CPU max MHz: 5200.0000 CPU min MHz: 800.0000 BogoMIPS: 6374.40 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_ti mer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault cat_l2 cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_e pp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities Virtualization features: Virtualization: VT-x Caches (sum of all): L1d: 640 KiB (16 instances) L1i: 768 KiB (16 instances) L2: 14 MiB (10 instances) L3: 30 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-23

2 2

Please help - Building Slurm-24.11.1 Failed
by Zhang, Yuan 23 Feb '25

23 Feb '25

Hello, I got errors about missing perl modules when building slurm24.11.1 rpm packages. Has anyone seen this error before? And how to fix it? Here are the error messages: -- Processing files: slurm-perlapi-24.11.1-1.el8.x86_64 error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Bitstr.pm error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Constant.pm error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Hostlist.pm error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurm/Slurm.so error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/Slurmdb.so error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/autosplit.ix RPM build errors: Macro expanded in comment on line 31: %_prefix path install path for commands, libraries, etc. Macro expanded in comment on line 240: %define _unpackaged_files_terminate_build 0 File listed twice: /usr/lib/.build-id/49/9394dac8b96530e1a29d0e35608ef9acedec70 File listed twice: /usr/lib/.build-id/88/4e3a854911920ae0c77c4b32d5af70dad538a9 File listed twice: /usr/lib/.build-id/c1/72dbf51fe1c17868a4e0dbf0aa55a7923f6d02 File listed twice: /usr/lib/.build-id/dc/67c93438d36986971444d218350d27b413c2cb File listed twice: /usr/lib/.build-id/ee/e79f70a62bf62e74705a75daa87d26857452ef Deprecated external dependency generator is used! File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Bitstr.pm File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Constant.pm File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Hostlist.pm File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurm/Slurm.so File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/Slurmdb.so File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/autosplit.ix -- Thanks! Yuan -- Yuan Zhang Senior HPC System Programmer Nationwide Children’s Hospital Research Institute Columbus, Ohio 43215

3 5

Running SLURM in a laptop
by Antonio Rius 19 Feb '25

19 Feb '25

For testing purposes I am using SLURM installed in a lenovo laptop running ubuntu 24-04. I am trying this configuration as part of my slurm learning process before using a real HPC facility. In my trial I want to submit an array of 9 independent processes. The CPU information of such machine is: Total Logical CPUs: 16 Physical Cores: 12 Threads per Core: 2 ================================= and the slurm script (attached) that I am using contain the sentences : #SBATCH --job-name=process_pkl #SBATCH --array=1-9 #SBATCH --output=/tmp/process_pkl_%A_%a.out #SBATCH --error=/tmp/process_pkl_%A_%a.err #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=2G =================================== but in practice, using htop, only 4 processes are running concurrently, and there are other 12 CPUs without activity. My question is: What am I missing in my setup?

4 3

slurmrestd health check
by Hagdorn, Magnus Karl Moritz 19 Feb '25

19 Feb '25

Hi there, we use haproxy to distribute SLURM REST API requests to multiple instances of slurmrestd. For the haproxy we need a health check. At the moment we are just checking that we get a 401 status. This works but we are ending up with a lot of noise in the log files. It would be very nice if there was an unauthenticated REST API endpoint that can be used to check the daemon is up. Has anybody solved this issue? Refards magnus -- Dr. Magnus Hagdorn Charité – Universitätsmedizin Berlin Geschäftsbereich IT | Scientific Computing

1 0

/etc/passwd sync?
by mark.w.moorcroft＠ama-inc.com 19 Feb '25

19 Feb '25

If you set up slurm elastic cloud in EC2 without LDAP, what is the recommended method for sync of the passwd/group files? Is this necessary to get openmpi jobs to run. I would swear I had this working last week without synced passwd on two nodes. But thinking about it now I'm not sure how this could have worked. My home directories are in an NFS mount, but the user accounts don't exist on the node AMI. I'm using ansible/packer to manage the AMI's. When I ran OpenHPC / Slurm on bare metal there was a sync process. This is my first AWS Slurm cluster rodeo. I can't use the Amazon Parallel Computing tools because we are forced to be in GovCloud. I started with "ClusterInTheCloud", but it's all 4 years old, and semi-broken out of the box. My manager had me ditch a lot of it (including LDAP). So I'm building out a fork that is getting heavily modded for our situation. An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. This could be caused by a number of factors, including an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity

4 4

how to set slurmdbd.conf if using two slurmdb node with HA database?
by taleintervenor＠sjtu.edu.cn 19 Feb '25

19 Feb '25

The deployment scenario is as follows: nodeA nodeB (slurmctld) (backup slurmctld) | \-------------------------------/ | | / \ | nodeC nodeD (slurmdbd) (backup slurmdbd) (mysql) <--multi master replica--> (mysql) Since the database is multi-master replicated, the slurmdbd should only talk to the mysql on its own node. In such case, how should we set the slurmdbd.conf? The conf file contains options "DbdAddr", "DbdHost" and "DbdBackupHost". Should they be consistent between nodeA-2 and nodeB-2? Such as: DbdAddr = nodeC | DbdAddr = nodeC DbdHost = nodeC | DbdHost = nodeC DbdBackupHost = nodeD | DbdBackupHost = nodeD StorageHost = nodeC | StorageHost = nodeD Or maybe just set different conf and don't use the "DbdBackupHost" like: DbdAddr = nodeC | DbdAddr = nodeD DbdHost = nodeC | DbdHost = nodeD StorageHost = nodeC | StorageHost = nodeD I'm quite confused about the usage of DbdAddr and DbdHost. What is the difference between them and why only DbdHost has the backup one? Another confusing point is how DbdBackupHost work. I guess It is slurmctld that is responsible for selecting the available slurmdbd. Since the slurm.conf already contains "AccountingStorageHost" and "AccountingStorageBackupHost", why we need set backupdbd again on slurmdbd side?

2 1

Run only one time on a node
by John Hearns 19 Feb '25

19 Feb '25

I am running single node tests on a cluster. I can select named nodes using the -2 flag with sbatch. However - if I want to submit perhaps 20 test jobs is there any smart way to run only one time on a node? I know I could touch a file with the hostname and test for that file. I am just wondering if there is a smarter way to do this.\ I should point out that the tests take a few minutes to run, so if a node finishes a test run it could become idle and run the tests again.

3 2

Create filenames based on slurm hosts
by John Hearns 15 Feb '25

15 Feb '25

I am working on power logging of a GPU cluster I am working with. I am running jobs on multiple hosts. I wanst to create a file , one for each host, which has a unique filename containing the host name. Something like clush -w $SLURM_JOB_NODELIST "touch file$(hostname)" My foo is weak today. Help me Ole Wan Neilsen or any $Jedi

3 3

avoid using same GPU by the interactive job
by navin srivastava 13 Feb '25

13 Feb '25

hi, facing an issue in my environment where the batch job and the interactive job use the same gpu. Each server has 2 gpu. When 2 batch jobs are running it works fine and use the 2 different gpu's. but if one batch job is running and another job is submitted interactively then it uses the same GPU . Is there a way to avoid this? GresTypes=gpu NodeName=node[01-02] NodeAddr=node[01-02] CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=24 ThreadsPerCore=1 TmpDisk=6000000 RealMemory=515634 Feature=A100 Gres=gpu:2 PartitionName=onprem Nodes=node[01-10] Default=YES MaxTime=21-00:00:00 DefaultTime=3-00:00:00 State=UP Shared=YES OverSubscribe=NO gres.conf: Name=gpu File=/dev/nvidia0 Name=gpu File=/dev/nvidia1 Any suggestions on this. Regards Navin

3 3

error no error
by Ricardo Román-Brenes 13 Feb '25

13 Feb '25

Hello. Could someone enlighten me as to what this error message is? Feb 13 10:02:00 gpu1 slurmd[573705]: slurmd: error: slurm_msg_sendto: address:port=192.168.9.1:36698 msg_type=8001: No error

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 13 Feb '25

13 Feb '25

I made some progress without need for the /etc/passwd sync. Sbatch is working fine on multi-node jobs it appears. Now only salloc runs fail, which I guess is expected behavior without user account sync and ssh key setup. On bare metal OpenHPC, warewulf handles all that for me. So presumably I have MPI keys, but not user account sync and keys. Again, on OpenHPC they populated keys on account creation, and passwd was synced to the boot image. Can anyone tell me what this is about? slurmctld[46449]: slurmctld: error: failed to generate json for resume job/node list Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 12 Feb '25

12 Feb '25

The only service found in gov cloud with a search for “parallel” is Batch. And the awscli commands are absent there as well. In commercial cloud “parallel” yields many services. Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates e. mark.w.moorcroft(a)ama-inc.com<mailto:mark.w.moorcroft@ama-inc.com> w. ama-inc.com<https://ama-inc.com/> t. +1 408-290-5455 _____________________________________ Confidentiality Notice: This e-mail is intended only for the addressee named above. It contains information that is privileged, confidential or otherwise protected from use and disclosure. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, or dissemination of this transmission, or taking of any action in reliance on its contents, or other use is strictly prohibited. If you have received this transmission in error, please reply to the sender listed above immediately and permanently delete this message from your inbox. Thank you for your cooperation.

2 1

Force a refresh for sreport
by Jackson, Gary L. 10 Feb '25

10 Feb '25

As I understand it, slurmdbd will compile statistics for sreport once an hour. Is there any way I can force that to happen immediately? Restarting slurmdbd doesn’t seem to do anything. I don’t want to use this for anything operational. Just testing. -- Gary

1 0

jobs getting stuck in CG
by Ricardo Román-Brenes 10 Feb '25

10 Feb '25

Hello everyone. I have a cluster composed of 16 nodes, with 4 of them having GPUs with no particular configuration to manage them. The filesystem is gluster, authentication via slapd/munge. My problem is that very frequently, let's say at least a job daily, gets stuck in CG. I have no idea why this happens. Manually killing the slurmstep process releases the node but this is in no way a manageable solution. Has anyone experienced this (and fixed it?) Thank you. -Ricardo

4 5

MemSpecLimit Error
by Chase Schuette 07 Feb '25

07 Feb '25

After a Bright Computing/Base Command update. I'm encountering a slurm.conf error as seen below. I tried removing the MemSpecLimit parameter from the node names but the changes don't seem to be taking affect. Even after restarting slurmd across compute nodes and restarting slurmctld on the head node. I'm also suspicious of the RealMemory being set to zero. Any insight? Open to suggestions. Thanks ahead of time! sbatch: error: NodeNames=ch3lahpccn[001-032] MemSpecLimit=0 is invalid, reset to 0 sbatch: error: NodeNames=ch3lahpcgpu1 MemSpecLimit=0 is invalid, reset to 0 sbatch: error: Memory specification can not be satisfied sbatch: error: Batch job submission failed: Requested node configuration is not available Caterpillar: Confidential Green

2 1

Issue running slurm commands as normal account but work as root.
by Kent L. Hanson 05 Feb '25

05 Feb '25

Hello all! I have an odd question. On my headnode, also my login nodes, I can ping my outside network and inside network using any DNS name I choose. It is dual homed. When I run squeue, sinfo, or any slurm command as my regular account I get the below error. I can run srun, squeue, sinfo, or any slurm command and it works as expected. Does anyone know what is going on? I feel like this is a permissions issue on something. On the compute nodes I have no problem running slurm commands as my regular account or root. Everything works well on my compute nodes. Error: [user.name@headnode ~]$ squeue squeue: error: resolve_ctls_from_dns_srv: res_nsearch error: Hostname lookup failure squeue: error: fetch_config: DNS SRV lookup failed squeue: error: _establish_config_source: failed to fetch config squeue: fatal: Could not establish a configuration source Thanks! Kent Hanson

2 2

Submit multiple jobs odd info on squeue
by Vitorio Cargnini 04 Feb '25

04 Feb '25

Hi, I submit 2 or more jobs to test the scheduler and how it is handling multiple job requests and overall info. Now I submit two or more jobs I get the following odd output: [vitorio@dev35 verif_dev]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2 sim run.sh vitorio PD 0:00 1 (InvalidAccount) 1 sim run.sh vitorio R 15:35 1 dev35 [vitorio@dev35 verif_dev]$ ^C What about this InvalidAccount ?? All subsequent jobs show up like that Regards, Vitorio

1 0

RHEL8.10 V slurmctld
by Steven Jones 04 Feb '25

04 Feb '25

I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz uid=1204805830(xxxjoness(a)xxx.ac.nz) gid=1204805830(xxxjoness(a)xxx.ac.nz) groups=1204805830(xxxjoness(a)xxx.ac.nz) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven

8 24

System limits propagation
by Fatih Ertinaz 31 Jan '25

31 Jan '25

Hi everyone, I saw similar discussions in the archives, however I could not see a clear explanation therefore decided to open up a new thread. The problem I have is that default ulimits defined in the compute node OS image are overridden by Slurm and somehow changes in slurm.conf does not resolve the problem. This is specifically about max memory size, but the same issue applies to stack size and user processes as well. From the head node or from one of the compute nodes if I bypass slurm: max memory size (kbytes, -m) unlimited But going through salloc / sbatch it becomes: max memory size (kbytes, -m) 1024 As an initial attempt I tried setting `PropagateResourceLimitsExcept=NONE` but it didn't help. Then I tried `PropagateResourceLimits=ALL` also with no luck. So in theory PropagateResourceLimits should propagate all limits, but I'm not sure if that's really the case. I'm open to suggestions, Fatih Ertinaz Note: Slurm version is 24.05.1 OS is SLES SP5

2 1

Issues starting the SLURMCTLD on Rocky
by Vitorio Cargnini 31 Jan '25

31 Jan '25

Hello, I just deployed the latest SLURM and I am getting some odd issues restating it. Anyone saw this before, how can I fix it? Jan 30 17:22:09 slurmctrl01 systemd[1]: slurmctld.service: Main process exited, code=dumped, status=6/ABRT Jan 30 17:22:09 slurmctrl01 systemd[1]: slurmctld.service: Failed with result 'core-dump'. Jan 30 17:22:09 slurmctrl01 systemd[1]: Failed to start Slurm controller daemon. Jan 30 17:22:09 slurmctrl01 systemd[1]: systemd-coredump(a)8-92126-0.service: Succeeded. journalctl -xe #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92119: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92112: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92122: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92111: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92115: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92123: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92104: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) -- Subject: Process 92054 (slurmctld) dumped core -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: man:core(5) -- -- Process 92054 (slurmctld) crashed and dumped core. -- -- This usually indicates a programming error in the crashing program and -- should be reported to its vendor as a bug. Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: slurmctld.service: Main process exited, code=dumped, status=6/ABRT Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: slurmctld.service: Failed with result 'core-dump'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The unit slurmctld.service has entered the 'failed' state with result 'core-dump'. Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: Failed to start Slurm controller daemon. -- Subject: Unit slurmctld.service has failed -- Defined-By: systemd Regards, Vitorio

2 1

Re: RHEL8.10 V slurmctld
by Steven Jones 30 Jan '25

30 Jan '25

Hi, Already done as part of the build process. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu> Sent: Friday, 31 January 2025 9:06 am To: Steven Jones <steven.jones(a)vuw.ac.nz>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com <slurm-users(a)schedmd.com> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> First I’d verify munge functionality in the updated environment – https://github.com/dun/munge/wiki/Installation-Guide#troubleshooting From: Steven Jones <steven.jones(a)vuw.ac.nz> Sent: Thursday, January 30, 2025 2:55 PM To: Williams, Jenny Avis <jenny_williams(a)unc.edu>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com Subject: Re: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Hmmm, yes I am using munge. [root@node1 ~]# strings `which slurmd` |egrep -i munge [root@node1 ~]# Does not return anything on the nodes, but worked fine for RHEL9.5 [root@xxxunidrslurmd2 munge]# scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge [root@vuwunidrslurmd2 munge]# Munge logs are 0 length ============= slurmd2 slurm]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:46 AM UTC Group : Unspecified Size : 320124 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon 27 Apr 2020 11:43:24 PM UTC, Key ID 199e2f91fd431d51 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Fri 24 Apr 2020 07:37:08 AM UTC Build Host : x86-vm-02.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. ========== [root@node1 ~]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:14:28 2025 Group : Unspecified Size : 319876 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon Apr 12 06:46:59 2021, Key ID 15af5dac6d745a60 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Mon Apr 12 05:07:29 2021 Build Host : ord1-prod-x86build003.svc.aws.rockylinux.org Relocations : (not relocatable) Packager : infrastructure(a)rockylinux.org<mailto:infrastructure@rockylinux.org> Vendor : Rocky URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. [root@node1 ~]# ========== [root@node1 ~]# rpm -qi slurm-slurmd Name : slurm-slurmd Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:15:34 2025 Group : Unspecified Size : 517922 License : GPLv2 and BSD Signature : RSA/SHA256, Fri May 6 00:03:49 2022, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu May 5 23:33:30 2022 Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm compute node daemon Description : Slurm compute node daemon. Used to launch jobs on compute nodes [root@node1 ~]# ======= slurmd2 ~]# rpm -qi slurm-slurmctld Name : slurm-slurmctld Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:48 AM UTC Group : Unspecified Size : 1097306 License : GPLv2 and BSD Signature : RSA/SHA256, Fri 06 May 2022 12:03:49 AM UTC, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu 05 May 2022 11:33:30 PM UTC Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm controller daemon Description : Slurm controller daemon. Used to manage the job queue, schedule jobs, and dispatch RPC messages to the slurmd processon the compute nodes to launch jobs. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu<mailto:jenny_williams@unc.edu>> Sent: Friday, 31 January 2025 8:36 am To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>>; John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu<mailto:jenny_williams@unc.edu>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> On both a compute node and the controller rpm -qi slurm-slurmctld rpm -qi slurm-slurmd check what the auth type is – for example, we still use munge, which in my compile is also the default auth type. : # strings `which slurmd` |egrep -i munge DEFAULT_AUTH_TYPE "auth/munge" DEFAULT_CRED_TYPE "cred/munge" #scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Sent: Thursday, January 30, 2025 2:07 PM To: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> Subject: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Yes, even ssh works OK. [root@xxxunicobuildt1 warewulf]# ssh xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> (xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1>) Password: Last login: Wed Jan 29 01:26:21 2025 from 130.195.87.12 [xxxjonesst@xxx.ac.nz@node1 ~]$ xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> ~]$ whoami | id uid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) gid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) tail -f /var/log/secure ========= Jan 30 18:19:56 node1 sshd[15443]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.195.87.12 user=xxxjonesst(a)xxx.ac.nz<mailto:user=xxxjonesst@xxx.ac.nz> Jan 30 18:19:56 node1 sshd[15440]: Accepted keyboard-interactive/pam for xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz> from 130.195.87.12 port 59402 ssh2 Would there be any relevant changes between RHEL8's slurm and RHEL9's slurm? [root@node1 ~]# rpm -qa |grep slurm slurm-libs-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 [root@node1 ~]# I would have to go back and check but I do not think I hit this on RHEL9 what I did get was srun ver22 on the RHEL9 server didnt like srun ver20 on the rocky8 node. Can I compile / rpm build srun ver22 to run on rocky8? or is that part of slurmd? regards Steven ________________________________ From: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Sent: Thursday, 30 January 2025 10:53 pm To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: Re: [slurm-users] RHEL8.10 V slurmctld You don't often get email from hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Have you run id on a computer node? On Wed, Jan 29, 2025, 6:47 PM Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> wrote: I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz> uid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) gid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) groups=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 0

Re: RHEL8.10 V slurmctld
by Steven Jones 30 Jan '25

30 Jan '25

Hi, Hmmm, yes I am using munge. [root@node1 ~]# strings `which slurmd` |egrep -i munge [root@node1 ~]# Does not return anything on the nodes, but worked fine for RHEL9.5 [root@xxxunidrslurmd2 munge]# scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge [root@vuwunidrslurmd2 munge]# Munge logs are 0 length ============= slurmd2 slurm]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:46 AM UTC Group : Unspecified Size : 320124 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon 27 Apr 2020 11:43:24 PM UTC, Key ID 199e2f91fd431d51 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Fri 24 Apr 2020 07:37:08 AM UTC Build Host : x86-vm-02.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. ========== [root@node1 ~]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:14:28 2025 Group : Unspecified Size : 319876 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon Apr 12 06:46:59 2021, Key ID 15af5dac6d745a60 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Mon Apr 12 05:07:29 2021 Build Host : ord1-prod-x86build003.svc.aws.rockylinux.org Relocations : (not relocatable) Packager : infrastructure(a)rockylinux.org Vendor : Rocky URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. [root@node1 ~]# ========== [root@node1 ~]# rpm -qi slurm-slurmd Name : slurm-slurmd Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:15:34 2025 Group : Unspecified Size : 517922 License : GPLv2 and BSD Signature : RSA/SHA256, Fri May 6 00:03:49 2022, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu May 5 23:33:30 2022 Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm compute node daemon Description : Slurm compute node daemon. Used to launch jobs on compute nodes [root@node1 ~]# ======= slurmd2 ~]# rpm -qi slurm-slurmctld Name : slurm-slurmctld Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:48 AM UTC Group : Unspecified Size : 1097306 License : GPLv2 and BSD Signature : RSA/SHA256, Fri 06 May 2022 12:03:49 AM UTC, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu 05 May 2022 11:33:30 PM UTC Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm controller daemon Description : Slurm controller daemon. Used to manage the job queue, schedule jobs, and dispatch RPC messages to the slurmd processon the compute nodes to launch jobs. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu> Sent: Friday, 31 January 2025 8:36 am To: Steven Jones <steven.jones(a)vuw.ac.nz>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com <slurm-users(a)schedmd.com> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> On both a compute node and the controller rpm -qi slurm-slurmctld rpm -qi slurm-slurmd check what the auth type is – for example, we still use munge, which in my compile is also the default auth type. : # strings `which slurmd` |egrep -i munge DEFAULT_AUTH_TYPE "auth/munge" DEFAULT_CRED_TYPE "cred/munge" #scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com> Sent: Thursday, January 30, 2025 2:07 PM To: John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com Subject: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Yes, even ssh works OK. [root@xxxunicobuildt1 warewulf]# ssh xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> (xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1>) Password: Last login: Wed Jan 29 01:26:21 2025 from 130.195.87.12 [xxxjonesst@xxx.ac.nz@node1 ~]$ xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> ~]$ whoami | id uid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) gid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) tail -f /var/log/secure ========= Jan 30 18:19:56 node1 sshd[15443]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.195.87.12 user=xxxjonesst(a)xxx.ac.nz<mailto:user=xxxjonesst@xxx.ac.nz> Jan 30 18:19:56 node1 sshd[15440]: Accepted keyboard-interactive/pam for xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz> from 130.195.87.12 port 59402 ssh2 Would there be any relevant changes between RHEL8's slurm and RHEL9's slurm? [root@node1 ~]# rpm -qa |grep slurm slurm-libs-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 [root@node1 ~]# I would have to go back and check but I do not think I hit this on RHEL9 what I did get was srun ver22 on the RHEL9 server didnt like srun ver20 on the rocky8 node. Can I compile / rpm build srun ver22 to run on rocky8? or is that part of slurmd? regards Steven ________________________________ From: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Sent: Thursday, 30 January 2025 10:53 pm To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: Re: [slurm-users] RHEL8.10 V slurmctld You don't often get email from hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Have you run id on a computer node? On Wed, Jan 29, 2025, 6:47 PM Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> wrote: I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz> uid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) gid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) groups=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 0

Behavior of 'afterok' in cloud clusters
by Thompson, Hoot (GSFC-606.0)[ADNET SYSTEMS INC] 30 Jan '25

30 Jan '25

The behavior of slurm jobs using the ‘afterok’ dependency seems to have issues with ephemeral compute nodes such as those in a cloud cluster. If the specified jobid dependency is associated with a compute node that has already spun down then a subsequent job that requires successful completion of the prior job will fail with a “Job dependency problem”. This occurs when the subsequent job is tied to a node that must spin up before beginning execution. This phenomenon does not occur if the ‘afterany’ dependency is used. It seems that job completion status is retained when a node is spun down but no information as to whether the job was successfully executed is saved. There are perhaps other scenarios that coud cause the same issue. Has anyone else witnessed this problem? How can it be avoided? afterany This job can begin execution after the specified jobs have terminated. aftercorr A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully afternotok This job can begin execution after the specified jobs have terminated in some failed state afterok This job can begin execution after the specified jobs have successfully executed

1 0

Cloud elastic help
by mark.w.moorcroft＠ama-inc.com 29 Jan '25

29 Jan '25

I have a new Slurm setup in AWS gov cloud that is not quite working. I will list a few factoids and maybe someone can suggest where to look next. The Troubleshooting page really has nothing relevant for elastic cloud deployments. The nodes are getting set to DOWN+CLOUD+POWERED_DOWN. Running a job does not launch a node in this state. I can force the nodes to launch with scontrol POWER_UP. The jobs will claim to run, then re-queue, but never complete. When the nodes boot I see them appear in slurmctl, but it soon reports connection lost. The node slurmd claims to be healthy, but the controller eventually just terminates them. You can ping in both directions with the hostnames. I've done 4 clusters. The first was torque/maui and the rest were slurm, but all were bare metal. This is my first attempt at cloud. We have ITAR data, so I can't use Amazon Parallel Computing because it is not offered in GovCloud. https://cluster-in-the-cloud.readthedocs.io/en/latest/running.html I had to fork this project because so much is obsolete, but it's mostly working for me now. https://github.com/mntbighker [root@mgmt ~]# journalctl -fu slurmctld Jan 29 19:16:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: Updating partition uid access list Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: purge_old_job: job file deletion is falling behind, 1 left to remove Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:16:50 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sackd_mgr_dump_state: saved state of 0 nodes Jan 29 19:17:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 29 19:17:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:17:26 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: POWER: Power save mode: 4 nodes [root@mgmt ~]# scontrol show node many-antelope-c5n-2xlarge-0001 NodeName=many-antelope-c5n-2xlarge-0001 CoresPerSocket=4 CPUAlloc=0 CPUEfctv=8 CPUTot=8 CPULoad=0.00 AvailableFeatures=shape=c5n.2xlarge,ad=None,arch=x86_64 ActiveFeatures=shape=c5n.2xlarge,ad=None,arch=x86_64 Gres=(null) NodeAddr=many-antelope-c5n-2xlarge-0001 NodeHostName=many-antelope-c5n-2xlarge-0001 RealMemory=20034 AllocMem=0 FreeMem=N/A Sockets=1 Boards=1 State=DOWN+CLOUD+POWERED_DOWN ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=production,debug,batch,long BootTime=None SlurmdStartTime=None LastBusyTime=Unknown ResumeAfterTime=None CfgTRES=cpu=8,mem=20034M,billing=8 AllocTRES= CurrentWatts=0 AveWatts=0 Reason=Not responding [slurm@2025-01-29T17:49:26] [root@mgmt ~]# scontrol show jobs JobId=30 JobName=test.sl UserId=mwmoorcroft(1106) GroupId=nssam(1101) MCS_label=N/A Priority=1 Nice=0 Account=nssam QOS=(null) JobState=PENDING Reason=Nodes_required_for_job_are_DOWN,_DRAINED_or_reserved_for_jobs_in_higher_priority_partitions Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2025-01-29T18:58:26 EligibleTime=2025-01-29T18:58:26 AccrueTime=2025-01-29T18:58:26 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-01-29T19:19:15 Scheduler=Backfill:* Partition=production AllocNode:Sid=ip-172-16-2-14.us-gov-east-1.compute.internal:7090 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=20034M,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/mnt/shared/home/mwmoorcroft/test.sl WorkDir=/mnt/shared/home/mwmoorcroft StdErr=/mnt/shared/home/mwmoorcroft/slurm-30.out StdIn=/dev/null StdOut=/mnt/shared/home/mwmoorcroft/slurm-30.out

2 2

Adding a permanent node Feature to slurm in AWS parallelcluster
by luigi＠scorzato.it 28 Jan '25

28 Jan '25

I need to add a node Feature to my SLURM AWS parallel-cluster I understand from here: https://groups.google.com/g/slurm-users/c/LgL6hq3XxzE that to add a node feature permanently I have to edit slurm.conf, because the effects of the command: scontrol update NodeName=[node(s)] AvailableFeatures=[comma separated list of features] will not persist after restart (besides, I could not find how to ADD a feature without deleting the existing AvailableFeatures). However, AWS parallelcluster slurm.conf already includes files that contain lines like: NodeName=nn77[1-100] CPUs=2 RealMemory=186777 State=CLOUD Feature=dynamic,g4dn.large,gpu Weight=1000 Gres=gpu:t4:1 which is automatically generated and should not be edited. What is the recommended way to add a feature without breaking the parallelcluster set up? My I put somewhere something like NodeName=nn77[1-100] Feature+=cascadelake ? thank you!

1 0

Automate job lmod environment setup based on node
by Daniel UB 27 Jan '25

27 Jan '25

Greetings ! We're using Slurm 23.11.8 in a small cluster . The control node shares via NFS the directory /clusterprograms with the compute nodes (using the same name as mountpoint) to provide access to available software. The users are instructed to use lmod<https://lmod.readthedocs.io/en/latest/060_locating.html> (module load/avail/purge) to setup their job's runtime environment. We have the software and modules at /clusterprograms distributed like this: /clusterprograms/ ├── common ├── gpu/ │ ├── cuda8/ │ └── cuda10/ └── cpu/ ├── intel/ └── amd/ We configured the /etc/profile.d/ directory on each node so ,when we access them directly (via ssh) the paths used by module (MODULEPATH) are only these: * /home/<user>/modulefiles * /clusterprograms/common/modules * /clusterprograms/<processor vendor or cuda version>/modules The exception is the control/login node, which only has /home/<user>/modulefiles and /clusterprograms/common/modules The issue we have now is this: We are looking for a safe and practical way to automate the reconfiguration of MODULEPATH<https://lmod.readthedocs.io/en/latest/077_ref_counting.html> in the compute nodes when a job is submitted. For now, the only method that is "working" is the following TaskProlog script, but it forces the users to load the modules inside the scripts that may be called by srun steps, rather than being able to load them in the job base step: #!/bin/bash if [[ -n "${SLURM_JOB_CONSTRAINTS}" ]]; then echo "export ORIGINAL_MPATH=${MODULEPATH}" IFS=',' read -ra JOB_ARCH_CONSTRAINTS <<< "$SLURM_JOB_CONSTRAINTS" for constraint in "${JOB_ARCH_CONSTRAINTS[@]}"; do case "${constraint}" in cuda*) echo "export MODULEPATH=/clusterprograms/gpu/${constraint}:${MODULEPATH}" ;; intel-*) echo "export MODULEPATH=/clusterprograms/cpu/intel/${constraint}:${MODULEPATH}" ;; amd-*) echo "export MODULEPATH=/clusterprograms/cpu/amd/${constraint}:${MODULEPATH}" ;; esac done fi Is there any better way to do this ? How is that it doesn't work if we use Prolog ? I tried using the --export option but it doesn't work, since the control node has MODULEPATH set via the main shell profiles. Many thanks !!! -- Daniel Garrapucho Lévy Técnic informàtic Departament de Física de la Matèria Condensada Facultat de Física Martí i Franquès, 1 08028 Barcelona, SPAIN Despatx V302 Email: daniel.garrapucho(a)ub.edu<mailto:daniel.garrapucho@ub.edu> [https://estatics.web.ub.edu/image/company_logo?img_id=2946262&t=17001439433…] Aquest missatge, i els fitxers adjunts que hi pugui haver, pot contenir informació confidencial o protegida legalment i s’adreça exclusivament a la persona o entitat destinatària. Si no consteu com a destinatari final o no teniu l’encàrrec de rebre’l, no esteu autoritzat a llegir-lo, retenir-lo, modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si l’heu rebut per error, informeu-ne el remitent i elimineu del sistema tant el missatge com els fitxers adjunts que hi pugui haver. Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener información confidencial o legalmente protegida y está exclusivamente dirigido a la persona o entidad destinataria. Si usted no consta como destinatario final ni es la persona encargada de recibirlo, no está autorizado a leerlo, retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si lo ha recibido por error, informe de ello al remitente y elimine del sistema tanto el mensaje como los ficheros adjuntos que pueda contener. This email message and any attachments it carries may contain confidential or legally protected material and are intended solely for the individual or organization to whom they are addressed. If you are not the intended recipient of this message or the person responsible for processing it, then you are not authorized to read, save, modify, send, copy or disclose any part of it. If you have received the message by mistake, please inform the sender of this and eliminate the message and any attachments it carries from your account.

1 0

Problems with slurmdbd on Ubuntu 24.04
by karl＠lovink.net 27 Jan '25

27 Jan '25

Hello, Last weekend, I updated my server to ubuntu 24.04 and instead of using my own compiled version of slurm, I switched to the version (23.11.4) that comes with Ubuntu 24.04. Since then, I have been having problems with slurmdbd. I get the following error message: slurmdbd: error: issue making get_lineage procedure slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin failed slurmdbd: debug2: wsrep_on=0 slurmdbd: error: Couldn't load specified plugin name for accounting_storage/mysql: Plugin init() callback failed slurmdbd: error: cannot create accounting_storage context for accounting_storage/mysql slurmdbd: fatal: Unable to initialize accounting_storage/mysql accounting storage plugin My /etc/slurm/slurmdbd.conf looks like this: AuthType=auth/munge DebugLevel=debug5 DbdHost=localhost DbdPort=3306 StorageLoc=slurm_acct_db StoragePass=<password> StorageType=accounting_storage/mysql StorageUser=slurm SlurmUser=admina PluginDir=/usr/lib/x86_64-linux-gnu/slurm-wlm ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no AuthInfo=/var/run/munge/munge.socket.2 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd.pid NOTE: Removing the PluginDir directive makes no difference. And the Plugin directory contains the accounting/MySQL file: ls -al /usr/lib/x86_64-linux-gnu/slurm-wlm/*mysql* -rw-r--r-- 1 root root 3145992 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.a -rw-r--r-- 1 root root 1071 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.la -rw-r--r-- 1 root root 501848 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.so I checked already the threat on: https://groups.google.com/g/slurm-users/c/wRV8o7qqGi4 Any idea what is going on here? Regards, Karl

3 4

Re: Problems with slurmdbd on Ubuntu 24.04
by karl＠lovink.net 27 Jan '25

27 Jan '25

Hi Stpehan, It is there: root@vrfcanalyzer:/usr/lib/x86_64-linux-gnu/slurm-wlm# ls -al *mysql* -rw-r--r-- 1 root root 3145992 Apr 1 2024 accounting_storage_mysql.a -rw-r--r-- 1 root root 1071 Apr 1 2024 accounting_storage_mysql.la -rw-r--r-- 1 root root 501848 Apr 1 2024 accounting_storage_mysql.so -rw-r--r-- 1 root root 213700 Apr 1 2024 jobcomp_mysql.a -rw-r--r-- 1 root root 994 Apr 1 2024 jobcomp_mysql.la -rw-r--r-- 1 root root 44240 Apr 1 2024 jobcomp_mysql.so dpkg -S jobcomp_mysql.so slurm-wlm-mysql-plugin: /usr/lib/x86_64-linux-gnu/slurm-wlm/jobcomp_mysql.so root@vrfcanalyzer:/usr/lib/x86_64-linux-gnu/slurm-wlm# -----Original Message----- From: Stephan Roth <stephan.roth(a)isg.ee.ethz.ch> Sent: Monday, 27 January 2025 13:23 To: karl(a)lovink.net; slurm-users(a)lists.schedmd.com Subject: Re: [slurm-users] Problems with slurmdbd on Ubuntu 24.04 Hello Karl, I quickly downloaded slurm-wlm-mysql-plugin_23.11.4-1.2ubuntu5_amd64.deb, unpacked it and saw these contents:ä usr/ ├── lib │ └── x86_64-linux-gnu │ └── slurm-wlm │ ├── accounting_storage_mysql.so │ └── jobcomp_mysql.so └── share ├── doc │ └── slurm-wlm-mysql-plugin │ ├── changelog.Debian.gz -> ../slurm-wlm-basic-plugins/changelog.Debian.gz │ └── copyright └── lintian └── overrides └── slurm-wlm-mysql-plugin So it seems to me you're somehow missing /usr/lib/x86_64-linux-gnu/slurm-wlm/jobcomp_mysql.so Does that help? Best, Stephan On 27.01.25 12:00, karl--- via slurm-users wrote: > Hello, > > Last weekend, I updated my server to ubuntu 24.04 and instead of using > my own compiled version of slurm, I switched to > > the version (23.11.4) that comes with Ubuntu 24.04. Since then, I have > been having problems with slurmdbd. > > I get the following error message: > > slurmdbd: error: issue making get_lineage procedure > > slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL > plugin failed > > slurmdbd: debug2: wsrep_on=0 > > slurmdbd: error: Couldn't load specified plugin name for > accounting_storage/mysql: Plugin init() callback failed > > slurmdbd: error: cannot create accounting_storage context for > accounting_storage/mysql > > slurmdbd: fatal: Unable to initialize accounting_storage/mysql > accounting storage plugin > > My /etc/slurm/slurmdbd.conf looks like this: > > AuthType=auth/munge > > DebugLevel=debug5 > > DbdHost=localhost > > DbdPort=3306 > > StorageLoc=slurm_acct_db > > StoragePass=<password> > > StorageType=accounting_storage/mysql > > StorageUser=slurm > > SlurmUser=admina > > PluginDir=/usr/lib/x86_64-linux-gnu/slurm-wlm > > ArchiveEvents=yes > > ArchiveJobs=yes > > ArchiveResvs=yes > > ArchiveSteps=no > > ArchiveSuspend=no > > ArchiveTXN=no > > ArchiveUsage=no > > AuthInfo=/var/run/munge/munge.socket.2 > > PurgeEventAfter=1month > > PurgeJobAfter=12month > > PurgeResvAfter=1month > > PurgeStepAfter=1month > > PurgeSuspendAfter=1month > > PurgeTXNAfter=12month > > PurgeUsageAfter=24month > > LogFile=/var/log/slurm/slurmdbd.log > > PidFile=/var/run/slurmdbd.pid > > NOTE: Removing the PluginDir directive makes no difference. > > And the Plugin directory contains the accounting/MySQL file: > > ls -al /usr/lib/x86_64-linux-gnu/slurm-wlm/*mysql* > > -rw-r--r-- 1 root root 3145992 Apr 1 2024 /usr/lib/x86_64-linux-gnu/ > slurm-wlm/accounting_storage_mysql.a > > -rw-r--r-- 1 root root 1071 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm- > wlm/accounting_storage_mysql.la > > -rw-r--r-- 1 root root 501848 Apr 1 2024 /usr/lib/x86_64-linux-gnu/ > slurm-wlm/accounting_storage_mysql.so > > I checked already the threat on: https://groups.google.com/g/slurm- > users/c/wRV8o7qqGi4 <https://groups.google.com/g/slurm-users/c/wRV8o7qqGi4> > > Any idea what is going on here? > > Regards, > > Karl > > > -- ETH Zurich Stephan Roth Systems Administrator IT Services Group (ISG) D-ITET ETF D 104 Sternwartstrasse 7 8092 Zurich Phone +41 44 632 30 59 stephan.roth(a)isg.ee.ethz.ch www.isg.ee.ethz.ch Working days: Mon, Tue, Thu, Fri

1 0

Slurm version 24.11.1 is now available
by Marshall Garey 24 Jan '25

24 Jan '25

We are pleased to announce the availability of Slurm version 24.11.1. This fixes a few possible crashes of the slurmctld and slurmrestd; a regression in 24.11 which caused file transfers to a job with sbcast to not join the job container namespace; mpi apps using Intel OPA, PSM2 and OMPI 5.x when ran through srun; and various minor to moderate bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.1 > ========================== > -- With client commands MIN_MEMORY will show mem_per_tres if specified. > -- Fix errno message about bad constraint > -- slurmctld - Fix crash and possible split brain issue if the > backup controller handles an scontrol reconfigure while in control > before the primary resumes operation. > -- Fix stepmgr not getting dynamic node addrs from the controller > -- stepmgr - avoid "Unexpected missing socket" errors. > -- Fix `scontrol show steps` with dynamic stepmgr > -- Deny jobs using the "R:" option of --signal if PreemptMode=OFF > globally. > -- Force jobs using the "R:" option of --signal to be preemptable > by requeue or cancel only. If PreemptMode on the partition or QOS is off > or suspend, the job will default to using PreemptMode=cancel. > -- If --mem-per-cpu exceeds MaxMemPerCPU, the number of cpus per > task will always be increased even if --cpus-per-task was specified. This > is needed to ensure each task gets the expected amount of memory. > -- Fix compilation issue on OpenSUSE Leap 15 > -- Fix jobs using more nodes than needed when not using -N > -- Fix issue with allocation being allocated less resources > than needed when using --gres-flags=enforce-binding. > -- select/cons_tres - Fix errors with MaxCpusPerSocket partition > limit. Used cpus/cores weren't counted properly, nor limiting free ones > to avail, when the socket was partially allocated, or the job request > went beyond this limit. > -- Fix issue when jobs were preempted for licenses even if there > were enough licenses available. > -- Fix srun ntasks calculation inside an allocation when nodes are > requested using a min-max range. > -- Print correct number of digits for TmpDisk in sdiag. > -- Fix a regression in 24.11 which caused file transfers to a job > with sbcast to not join the job container namespace. > -- data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when > dumping data with v0.0.40+complex data parser. > -- Remove logic to force lowercase GRES names. > -- data_parser/v0.0.42 - Prevent the association id from always > being dumped as NULL when parsing in complex mode. Instead it will now > dump the id. This affects the following endpoints: > GET slurmdb/v0.0.42/association > GET slurmdb/v0.0.42/associations > GET slurmdb/v0.0.42/config > -- Fixed a job requeuing issue that merged job entries into the > same SLUID when all nodes in a job failed simultaneously. > -- When a job completes, try to give idle nodes to reservations with > the REPLACE flag before allowing them to be allocated to jobs. > -- Avoid expensive lookup of all associations when dumping or > parsing for v0.0.42 endpoints. > -- Avoid expensive lookup of all associations when dumping or > parsing for v0.0.41 endpoints. > -- Avoid expensive lookup of all associations when dumping or > parsing for v0.0.40 endpoints. > -- Fix segfault when testing jobs against nodes with invalid gres. > -- Fix performance regression while packing larger RPCs. > -- Document the new mcs/label plugin. > -- job_container/tmpfs - Fix Xauthoirty file being created > outside the container when EntireStepInNS is enabled. > -- job_container/tmpfs - Fix spank_task_post_fork not always > running in the container when EntireStepInNS is enabled. > -- Fix a job potentially getting stuck in CG on permissions > errors while setting up X11 forwarding. > -- Fix error on X11 shutdown if Xauthority file was not created. > -- slurmctld - Fix memory or fd leak if an RPC is recieved that > is not registered for processing. > -- Inject OMPI_MCA_orte_precondition_transports when using PMIx. This fixes > mpi apps using Intel OPA, PSM2 and OMPI 5.x when ran through srun. > -- Don't skip the first partition_job_depth jobs per partition. > -- Fix gres allocation issue after controller restart. > -- Fix issue where jobs requesting cpus-per-gpu hang in queue. > -- switch/hpe_slingshot - Treat HTTP status forbidden the same as > unauthorized, allowing for a graceful retry attempt.

3 2

Slurmd cannot find cgroup plugin
by Webb, Jordan 20 Jan '25

20 Jan '25

Hi all, When trying to start slurmd, it is failing with cgroup issues. Any suggestions on where to troubleshoot this issue? x8000c0s0b0n0:~ # slurmd -V slurm 24.11.0 x8000c0s0b0n0:~ # slurmd -D -vvv slurmd: debug: Log file re-opened slurmd: debug2: hwloc_topology_init slurmd: debug2: hwloc_topology_load slurmd: debug2: hwloc_topology_export_xml slurmd: debug: CPUs:288 Boards:1 Sockets:4 CoresPerSocket:72 ThreadsPerCore:1 slurmd: error: Couldn't find the specified plugin name for cgroup/v2 looking at all files slurmd: error: cannot find cgroup plugin for cgroup/v2 slurmd: error: cannot create cgroup context for cgroup/v2 slurmd: error: Unable to initialize cgroup plugin slurmd: error: slurmd initialization failed x8000c0s0b0n0:~ # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) x8000c0s0b0n0:~ # grep cgroup /etc/slurm/slurm.conf ProctrackType=proctrack/cgroup TaskPlugin=task/affinity,task/cgroup Thanks! Jordan

1 1

Re: Slurm "showpartitions" tool has been updated
by Ole Holm Nielsen 17 Jan '25

17 Jan '25

Hi Robert, What's your version of Slurm? We run 24.05.5. The showpartitions tools compacts the list of nodenames using "scontrol show hostlistsorted <nodelist>". Can you please check your scontrol manual page if "hostlistsorted" is documented? Does the command work without -N? Thanks, Ole On 17-01-2025 19:54, Robert Kudyba wrote: > Hello Ole, > > If you'd like me to post this on the Git page LMK. Any idea why this > error invalid encode argument with -N? > ./showpartitions -N > Print partitions nodelists > Partition statistics for cluster axon at Fri Jan 17 13:51:43 EST 2025 > Partition #Nodes #CPU_cores Cores_pending Job_Nodes > MaxJobTime Cores Mem/Node NODELIST > Name State Total Idle Total Idle Resorc Other Min Max Day- > hr:mn /node (GB) > nklab up 4 0 352 125 0 0 1 infin > 10-00:00 48 191+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > ctn up 3 1 168 152 0 0 1 infin > 10-00:00 56 128 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > issa up 5 1 384 128 0 0 1 infin > 10-00:00 64 191+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > naplab up 3 1 320 231 0 0 1 infin > 10-00:00 64 515+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > lkumar up 3 2 192 191 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > u19moc3 up 1 1 96 96 0 0 1 infin > 10-00:00 96 257 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > abbott up 3 2 192 187 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > miller up 3 2 192 190 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > burst up 25 10 1896 1300 0 0 1 infin > 5-00:00 48 128+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > zrc up 6 1 376 190 0 0 1 infin > 10-00:00 48 128+invalid encode argument > > On Fri, Jan 17, 2025 at 7:31 AM Ole Holm Nielsen via slurm-users <slurm- > users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: > > Dear Slurm users, > > The "showpartitions" tool prints a Slurm cluster partition status > overview > with 1 line per partition. Color coding helps locating under- or > over-utilized partitions. > > The Slurm "showpartitions" tool has now been updated with a new "-N" > flag > which prints the nodename list for each partition. > > You can download this tool from > https://urldefense.proofpoint.com/v2/url? > u=https-3A__github.com_OleHolmNielsen_Slurm-5Ftools_tree_master_partitions&d=DwICAg&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=SvjV7A7QvPs0H4AmOiKvni-pI3DeHZmNgrCPf9ije3ftTN6STgomGCGHyuspdUFz&s=e2VRHqxuaDlLYggVv8eW9tkQqfXU332ucgmnqiU3Toc&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_OleHolmNiel…> > > If you have any issues or suggestions, please write to > Ole.H.Nielsen(a)fysik.dtu.dk <mailto:Ole.H.Nielsen@fysik.dtu.dk> or > open an issue on GitHub. > > Best regards, > Ole > > -- > Ole Holm Nielsen > PhD, Senior HPC Officer > Department of Physics, Technical University of Denmark > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > <mailto:slurm-users@lists.schedmd.com> > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > <mailto:slurm-users-leave@lists.schedmd.com> >

2 1

Slurm "showpartitions" tool has been updated
by Ole Holm Nielsen 17 Jan '25

17 Jan '25

Dear Slurm users, The "showpartitions" tool prints a Slurm cluster partition status overview with 1 line per partition. Color coding helps locating under- or over-utilized partitions. The Slurm "showpartitions" tool has now been updated with a new "-N" flag which prints the nodename list for each partition. You can download this tool from https://github.com/OleHolmNielsen/Slurm_tools/tree/master/partitions If you have any issues or suggestions, please write to Ole.H.Nielsen(a)fysik.dtu.dk or open an issue on GitHub. Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Unable to receive password reminder
by Loris Bennett 14 Jan '25

14 Jan '25

Hi, Over a week ago I sent the message below to the address I found for the list owner, but have not received a response. Does anyone know how to proceed in this case? Cheers, Loris -------------------- Start of forwarded message -------------------- From: Loris Bennett <loris.bennett(a)fu-berlin.de> To: <slurm-users-owner(a)lists.schedmd.com> Subject: Unable to receive password reminder Date: Mon, 6 Jan 2025 08:35:42 +0100 Dear list owner, I have recently switched from reading the list via mail to using the mail to news gateway at news.gmane.io. Therefore I would like change my mailman settings in order to stop delivery of postings via mail. As I have forgotten my list password, I requested a reminder. However I get the reply that no user with the given email address was found in the user database. The addresses I tried were loris.bennett(a)fu-berlin.de loris(a)zedat.fu-berlin.de the former being an alias for the latter. This is the email account which to which emails from the list are sent, so I am somewhat confused as to why the neither of the addresses is recognised. Could you please help me to resolve this issue? Regards Loris Bennett -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin -------------------- End of forwarded message -------------------- -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

2 1

Environment Variables per Queue
by Vitorio Cargnini 14 Jan '25

14 Jan '25

Hello, Can environment variables be set up based on a specific queue? How do I set up the environment variables for a given queue? Like we are deploying SLURM to control our jobs of simulation using VCS, and we don't want to make public the LM_LICENSE environment variable to all users. So we would like to set it on the queue once a job is started on that queue would have that environment variable. Regards, Vitorio

2 1

Discrepancy in TmpDisk values between scontrol and sinfo in Slurm24.05.3
by Gizo Nanava 11 Jan '25

11 Jan '25

There seems to be an issue with the TmpDisk value reporting in Slurm 24.05.3. While the correct value is displayed using the scontrol show nodes command, sinfo appears to report an incorrect value under certain conditions. For example, the TmpDisk parameter for my compute nodes is configured as 21800000 (21.8 TB) in slurm.conf, and this value is shown correctly with > scontrol show nodes ai-n007 ...... State=MIXED ThreadsPerCore=1 TmpDisk=21800000 Weight=1 Owner=N/A MCS_label=N/A .... However, when using the sinfo command with the -O Disk:20 option, the displayed value is 2180000, which is an order of magnitude lower: > sinfo -hO Disk:20 -n ai-n007 2180000 It appears that sinfo might be truncating or misinterpreting values with 8 or more digits. Has anyone encountered a similar issue? Is this a known bug in Slurm 24.05.3? Any suggestions or insights would be greatly appreciated. Best regards, Gizo

2 1

Power saving is getting disabled in Slurm
by Suman Deb 10 Jan '25

10 Jan '25

Hello, We are enabling global power saving for slurm. This is our config. But we are experiencing an issue where Slurm is talking nodes out of power saving mode. We are using config less and dynamic nodes. Slurmctrld logs 2025-01-08T09:54:18.096] Cleared POWER_SAVE flag from nodes dev3-cbf-debug-e2std4-s0-[0-1],dev3-cbf-infra-e2std4-s0-[0-1],dev3-cbf-slurmd-e2std4-[0-4] [2025-01-08T09:54:18.097] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:05:09.683] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:12:00.447] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:17:27.077] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:20:06.735] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:26:16.663] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:26:19.017] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T10:46:25.078] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T15:58:54.080] debug: power_save module disabled, SuspendTime < 0 [2025-01-08T15:58:56.452] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:08:01.993] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:08:02.157] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:14:26.817] debug: power_save module disabled, SuspendTime < 0 [2025-01-09T00:14:26.971] debug: power_save module disabled, SuspendTime < 0 We manage the infra through Puppet and I can confirm is SuspendTime=3600. Slurm version:- slurm 24.05.2 # SLURM POWER SAVING FEATURS SuspendProgram=/etc/slurm/suspend_nodes_slurm.par ResumeProgram=/etc/slurm/resume_nodes_slurm.par SuspendTimeout=600 ResumeTimeout=900 ResumeRate=100 SuspendRate=100 SuspendTime=3600 SlurmctldParameters=enable_configless,cloud_dns,idle_on_node_suspend DebugFlags=Power

1 0

Unexpected node got allocation
by Dan Healy 09 Jan '25

09 Jan '25

Hello there and good morning from Baltimore. I have a small cluster with 100 nodes. When the cluster is completely empty of all jobs, the first job gets allocated to node 41. In other clusters, the first job gets allocated to mode 01. If I specify node 01, the allocation works perfectly. I have my partition NodeName set as node[01-99], so having node41 used first is a surprise to me. We also have many other partitions which start with node41, but the partition being used for the allocation starts with node01. Does anyone know what would cause this? Thanks, Daniel Healy

3 3

need help with seff script on ubuntu (slurm 21.08)
by Gérard Henry (AMU) 09 Jan '25

09 Jan '25

Hello all and happy new year, i have installed slurm 21.08 on ubuntu 22 LTS, and database for accounting on a remote machine running same release. I got seff from: https://support.schedmd.com/show_bug.cgi?id=1611 but, there are errors: Use of uninitialized value $ncpus in numeric eq (==) at /usr/bin/seff line 72, <DATA> line 602. Use of uninitialized value $lmem in numeric lt (<) at /usr/bin/seff line 109, <DATA> line 602. Job ID: 14 Cluster: cathena2 Use of uninitialized value $user in concatenation (.) or string at /usr/bin/seff line 126, <DATA> line 602. does anyone have a more recent version? i try to enable debug uncommenting the line: 60c60 < #print Dumper($job); --- > print Dumper($job); but it's strange, because if you run the seff command several times in a row, the dump result is very different, even though the job has been completed. It doesn't change the most important values, but how is it possible not to have the same output? Thanks in advance for help, -- Gérard HENRY Institut Fresnel - UMR 7249 +33 413945457 Aix-Marseille Université - Campus Etoile, BATIMENT FRESNEL, Avenue Escadrille Normandie Niemen, 13013 Marseille Site : https://fresnel.fr/ Afin de respecter l'environnement, merci de n'imprimer cet email que si nécessaire.

4 3

launch failed requeued held
by sportlecon sportlecon 08 Jan '25

08 Jan '25

slurm 24.11 - squeue displays reason "launch failed requeued held"

2 2

formatting node names
by Davide DelVento 07 Jan '25

07 Jan '25

Hi all, I remember seeing on this list a slurm command to change a slurm-friendly list such as gpu[01-02],node[03-04,12-22,27-32,36] into a bash friendly list such as gpu01 gpu02 node03 node04 node12 etc I made a note about it but I can't find my note anymore, nor the relevant message. Can someone please refresh my memory? I'll be more careful with such a note this time, I promise! Thanks and happy new year!

6 11

Change config file location of slurmdbd temporarily
by Sven Schulze 07 Jan '25

07 Jan '25

Hey all, Is there a way to change the location of the slurmdbd.conf temporarily when starting slurmdbd? For all other daemons I can specify "-f", but this doesn't seem to work for slurmdbd, is there a way to edit the build files to achieve this? Kind Regards, Sven

3 4

Permission denied for slurmdbd.conf
by sportlecon＠gmail.com 07 Jan '25

07 Jan '25

ls -ls /usr/local/slurm/etc/slurmdbd.conf 4 -rw------- 1 slurm slurm 497 Dec 28 16:34 /usr/local/slurm/etc/slurmdbd.conf sudo -u slurm /usr/local/slurm/sbin/slurmdbd -Dvvv slurmdbd: error: s_p_parse_file: unable to read "/usr/local/slurm/etc/slurmdbd.conf": Permission denied slurmdbd: fatal: Could not open/read/parse slurmdbd.conf file /usr/local/slurm/etc/slurmdbd.conf

3 2

Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions
by sportlecon sportlecon 07 Jan '25

07 Jan '25

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 26 cpu myscript user1 PD 0:00 4 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions) Anyone can help to fix this?

4 3

All GPUs are Usable if no Gres is Defined
by Jacob Gordon 04 Jan '25

04 Jan '25

Hello, We have a two node GPU cluster with 8 NVidia GPUs. GRES is currently configured and works if a user defines it within their sbtach/interactive job submission (--gres=gpu:3). Users only have access to the GPUs they request. However, when they omit “--gres=gpu:n”, they can use every GPU, which interferes with running jobs that used the gres option. I’m at a loss as to why this is happening. Can someone please look at our configuration to see if anything stands out? SLURM Version = 21.08.5 *Slurm.conf* ClusterName=ommit SlurmctldHost=headnode ProctrackType=proctrack/cgroup ReturnToService=2 SlurmdPidFile=/run/slurmd.pid SlurmdSpoolDir=/var/lib/slurm/slurmd StateSaveLocation=/var/lib/slurm/slurmctld SlurmUser=slurm TaskPlugin=task/cgroup SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory AccountingStorageType=accounting_storage/slurmdbd # AccountingStorageType for other resources # AccountingStorageTRES=gres/gpu #DebugFlags=CPU_Bind,gres JobCompType=jobcomp/none JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log DefMemPerCPU=4000 #NodeName=n01 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 NodeName=n01 Gres=gpu:nvidia-l40:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 NodeName=n02 Gres=gpu:nvidia-l40:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1000000 #Gres config for GPUs GresTypes=gpu PreemptType=preempt/qos PreemptMode=REQUEUE # reset usage after 1 week PriorityUsageResetPeriod=WEEKLY # The job's age factor reaches 1.0 after waiting in the # queue for 2 weeks. PriorityMaxAge=14-0 # This next group determines the weighting of each of the # components of the Multifactor Job Priority Plugin. # The default value for each of the following is 1. PriorityWeightAge=1000 PriorityWeightFairshare=10000 PriorityWeightJobSize=1000 PriorityWeightPartition=1000 PriorityWeightQOS=1500 # Primary partitions PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=all Nodes=n01,n02 Default=YES MaxTime=01:00:00 DefaultTime=00:30:00 State=UP PartitionName=statds Nodes=n01 Default=NO MaxTime=48:00:00 State=UP Priority=100 State=UP OverSubscribe=FORCE AllowAccounts=statds PartitionName=phil Nodes=n02 Default=NO MaxTime=48:00:00 State=UP Priority=100 State=UP OverSubscribe=FORCE AllowAccounts=phil #Set up condo mode # Condo partitions PartitionName=phil_condo Nodes=n02 Default=NO MaxTime=48:00:00 DefaultTime=00:01:00 State=UP Priority=50 OverSubscribe=FORCE AllowQos=normal PartitionName=statds_condo Nodes=n01 Default=NO MaxTime=48:00:00 DefaultTime=00:01:00 State=UP Priority=50 OverSubscribe=FORCE AllowQos=normal JobSubmitPlugins=lua *Gres.conf* NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia0 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia1 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia2 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia3 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia4 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia5 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia6 NodeName=n01 Name=gpu Type=nvidia-l40 File=/dev/nvidia7 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia0 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia1 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia2 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia3 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia4 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia5 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia6 NodeName=n02 Name=gpu Type=nvidia-l40 File=/dev/nvidia7 *Cgroup.conf* CgroupMountpoint="/sys/fs/cgroup" CgroupAutomount=yes CgroupReleaseAgentDir="/etc/slurm/cgroup" AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf" ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes *cgroup_allowed_devices_file.conf* /dev/null /dev/urandom /dev/zero /dev/sda* /dev/cpu/*/* /dev/pts/* /dev/nvidia*

3 2

Resource guarantees
by christopher.furbee＠jhuapl.edu 23 Dec '24

23 Dec '24

Hello, Long time SGE admin, new SLURM admin here. I recently started the transition of all my clusters from SGE to SLURM and everything was great until I hit the "Taco Bell" cluster (fake name). Taco Bell supports 4 projects and under SGE we had a priority system setup using projects to balance the queue. For the life of me I have been unable to replicate this in SLURM. We are looking to configure guaranteed resources based on the project. I had thought we could accomplish this with QOS and accounts but so far we have failed. What we would like to end up with is; When project Gordita is running uncontested 100% of the cluster is available. While Gordita is running, if Crunchwrap submits their jobs we want the scheduler to prioritize those jobs until a 75% Gordita, 25% Crunchwrap balance of jobs is reached. No preempting or priority overriding, just as a Gordita job finishes, if Crunchwrap is less than 25%, start a Crunchwrap job. And then maintain that balance until one of the projects jobs are 100% completed. Any assistance or guidance is greatly appreciated.

1 0

Slurm plugin for custom hardware allocation
by Laura Zharmukhametova 23 Dec '24

23 Dec '24

Hello, Is there an existing Slurm plugin for FPGA allocation? If not, can someone please point me in the right direction for how to approach it. Many thanks

2 1

sending mails wit smail on rocky9
by Marcus Wagner 19 Dec '24

19 Dec '24

Hi all, I have a problem with sending mails on rocky 9 via Slurm. One needs to install s-nail to have "/bin/mail" being available. There are some caveats in smail. In the second part (for the message, when the job began) one need to pipe ( eg echo "") into $MAIL, even in a script with no input, s-nail wants to be interactive. but it suffices to echo an empty text to snail. Nonetheless, I don't get any mail through. it seems the, mailprog for some reason gets killed or errors out for some other reason. While it is perfectionally working if run from the console :/ I all the time get in the slurmctld.log the following: 27212:[2024-12-19T15:54:54.935] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27213:[2024-12-19T15:54:54.945] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27214:[2024-12-19T15:54:54.945] error: MailProg returned error, it's output was '' 27395:[2024-12-19T15:55:55.540] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27396:[2024-12-19T15:55:55.551] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27397:[2024-12-19T15:55:55.551] error: MailProg returned error, it's output was '' 27438:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27439:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27440:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27441:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27442:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27443:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27450:[2024-12-19T15:56:58.849] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27451:[2024-12-19T15:56:58.859] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 0 any hints? Best Marcus -- Dipl.-Inf. Marcus Wagner stellv. Gruppenleitung IT Center Gruppe: Server, Storage, HPC Abteilung: Systeme und Betrieb RWTH Aachen University Seffenter Weg 23 52074 Aachen Tel: +49 241 80 24383 wagner(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social-Media-Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/c/ITCenterRWTHAachen

1 0

lua, glib, gtk, kafka not found for slurm 24.11 and Alma 8
by Bernd Melchers 17 Dec '24

17 Dec '24

Dear all, i tried to rpmbuild slurm-24.11.0 for Alma Linux 8. Build failed because some installed Packages are not found by slurms configure script: rdkafka, glib, gtp and lua But all these packages are installed and they are found by slurm-24.05.x: librdkafka-1.6.1-1.el8.x86_64 librdkafka-devel-1.6.1-1.el8.x86_64 lua-5.3.4-12.el8.x86_64 lua-devel-5.3.4-12.el8.x86_64 glib2-2.56.4-165.el8_10.x86_64 glib2-devel-2.56.4-165.el8_10.x86_64 gtk2-2.24.32-5.el8.x86_64 gtk2-devel-2.24.32-5.el8.x86_64 gtk3-3.22.30-12.el8_10.x86_64 gtk3-devel-3.22.30-12.el8_10.x86_64 Mit freundlichen Grüßen Bernd Melchers -- Archiv- und Backup-Service | fab-service(a)zedat.fu-berlin.de Freie Universität Berlin | Tel. +49-30-838-55905

2 3

AllocNode:Sid in scontrol but not sacct?
by Chris Taylor 17 Dec '24

17 Dec '24

Does the accounting database keep this? Maybe I'm missing something but I don't see a way to query for it in sacct. Chris

1 1

job_container/tmpfs, TmpFS and TmpDisk usage
by Paul Musset 16 Dec '24

16 Dec '24

Hello, I have multiple questions about the usage of job_container/tmpfs, and the TmpFS and TmpDisk variables 1) If my job_container.conf files contains: ``` BasePath=/mnt/slurm_tmp Shared=true ``` is it important what I set TmpFS to in slurm.conf ? Should I set it to '/mnt/slurm_tmp' or '/tmp' ? 2) What size should I put in TmpDisk ? the size advertised by df ? 3) Finally, is there any recommended file system for the partition used as the job_container/tmpfs BasePath ? Best regards, Paul Musset Max Planck Institute for Brain Research

1 0

Difference between sreport's cpu time usage and sacct -j usage
by KK 14 Dec '24

14 Dec '24

Hi all, I have observed a significant discrepancy in CPU usage time calculations between sreport and sacct, and I would like to understand the underlying cause. Let me share the specific case I encountered when calculating CPU usage time for user zt23132881r from November 1, 2024, to November 30, 2024. 1. sreport Results (995,171 minutes): -------------------------------------------------------------------------------- *[root@master ~]# sreport Cluster UserUtilizationByAccount user=zt23132881r start=2024-11-01 end=2024-11-30--------------------------------------------------------------------------------Cluster/User/Account Utilization 2024-11-01T00:00:00 - 2024-11-29T23:59:59 (2505600 secs)Usage reported in CPU Minutes-------------------------------------------------------------------------------- Cluster Login Proper Name Account Used Energy--------- --------- --------------- --------------- -------- ---------djhpc-po+ zt231328+ zt23132881r zt+ zt23132881r_ba+ 995171 6294875* 2. sacct Results: # Without truncate (1,019,927 minutes / 61,195,668 seconds) *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'61195668* # With truncate (967,165 minutes / 58,029,908 seconds) *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" -X --truncate |awk 'BEGIN{total=0}{total+=$6}END{print total}'58029908* # No -X *[root@master ~]# sacct -u zt23132881r -S 2024-11-01 -E 2024-11-30 -o "jobid,partition,account,user,alloccpus,cputimeraw,state" |awk 'BEGIN{total=0}{total+=$6}END{print total}'61195668* The results show three different values: - *sreport: 995,171 minutes* - *sacct (without truncate): 1,019,927 minutes* - *sacct (with truncate): 967,165 minutes* I would appreciate if someone could explain: - Which of these results is more accurate? - How does sreport calculate CPU usage time? - Why does the --truncate option in sacct lead to different results? Thank you for your assistance in clarifying these discrepancies. Best regards

2 1

Node configuration unavailable when using --mem-per-gpu , for specific GPU type
by Matthew R. Baney 13 Dec '24

13 Dec '24

Hi all, I'm seeing some odd behavior when using the --mem-per-gpu flag instead of the --mem flag to request memory when also requesting all available CPUs on a node (in this example, all available nodes have 32 CPUs): $ srun --ntasks-per-node=8 --cpus-per-task=4 --gpus-per-node=gtx1080ti:1 --mem-per-gpu=1g --pty bash srun: error: Unable to allocate resources: Requested node configuration is not available $ srun --ntasks-per-node=8 --cpus-per-task=4 --gpus-per-node=gtx1080ti:1 --mem=1g --pty bash srun: job 3479971 queued and waiting for resources srun: job 3479971 has been allocated resources $ The nodes in this partition have a mix of gtx1080ti and rtx2080ti GPUs, but only one type of GPU is in any one node. The same behavior does not occur when requesting a (node with a) rtx2080ti instead. Is there something I'm missing that would cause the --mem-per-gpu flag to not be working in this example? Thanks, Matthew

1 0

Adding gres usage to the accounting database
by Mark Dixon 13 Dec '24

13 Dec '24

Hello! Our jobs can ask for dedicated per-node disk space, e.g. "--gres=tmp:1G", where an ephemeral directory is managed by the site prolog/epilog and usage is capped using an xfs project quota. This works well, although we really need to look at job_container/tmpfs. I note that slurm already periodically polls jobs for their memory usage, which finds its way into the accounting database tres_usage fields. What would be really nice is if we could extend this polling so that we feed our xfs project quota utilisation into the accounting database, too: users get feedback on what their jobs need, and we notice people who ask for a fraction of a node and all of the disk space, but not use it. Slurm is famous for being extendible. Is there some sort of plugin hook someone can point me at to have a go with, or am I missing a simpler method, please? Thanks! Mark

1 0

Slurm version 24.05.5 is now available
by Marshall Garey 12 Dec '24

12 Dec '24

We are pleased to announce the availability of Slurm version 24.05.5. This release fixes a few potential crashes, several stepmgr bugs, compatibility for sstat and sattach with newer version steps, and some other minor bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.5 > ========================== > -- Fix issue signaling cron jobs resulting in unintended requeues. > -- Fix slurmctld memory leak in implementation of HealthCheckNodeState=CYCLE. > -- job_container/tmpfs - Fix SLURM_CONF env variable not being properly set. > -- sched/backfill - Fix job's time_limit being overwritten by time_min for job > arrays in some situations. > -- RoutePart - fix segfault from incorrect memory allocation when node doesn't > exist in any partition. > -- slurmctld - Fix crash when a job is evaluated for a reservation after > removal of a dynamic node. > -- gpu/nvml - Attempt loading libnvidia-ml.so.1 as a fallback for failure in > loading libnvidia-ml.so. > -- slurmrestd - Fix populating non-required object fields of objects as '{}' in > JSON/YAML instead of 'null' causing compiled OpenAPI clients to reject > the response to 'GET /slurm/v0.0.40/jobs' due to validation failure of > '.jobs[].job_resources'. > -- Fix sstat/sattach protocol errors for steps on higher version slurmd's > (regressions since 20.11.0rc1 and 16.05.1rc1 respectively). > -- slurmd - Avoid a crash when starting slurmd version 24.05 with > SlurmdSpoolDir files that have been upgraded to a newer major version of > Slurm. Log warnings instead. > -- Fix race condition in stepmgr step completion handling. > -- Fix slurmctld segfault with stepmgr and MpiParams when running a job array. > -- Fix requeued jobs keeping their priority until the decay thread happens. > -- slurmctld - Fix crash and possible split brain issue if the > backup controller handles an scontrol reconfigure while in control > before the primary resumes operation. > -- Fix stepmgr not getting dynamic node addrs from the controller > -- stepmgr - avoid "Unexpected missing socket" errors. > -- Fix `scontrol show steps` with dynamic stepmgr > -- Support IPv6 in configless mode

1 0

node3 not working - down
by Steven Jones 10 Dec '24

10 Dec '24

Hi, As suggested, 8><--- Stop their services, start them manually one by one (ctld first), then watch whether they talk to each other, and if they don't, learn what stops them from doing so - then iterate editing the config, "scontrol reconfig", lather, rinse, repeat. 8><--- Error logs, On node3, ======== [2024-12-09T21:43:10.694] slurmd started on Mon, 09 Dec 2024 21:43:10 +0000 [2024-12-09T21:43:10.694] CPUs=20 Boards=1 Sockets=20 Cores=1 Threads=1 Memory=48269 TmpDisk=23308 Uptime=3703 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) [2024-12-09T23:38:56.645] error: Munge decode failed: Rewound credential [2024-12-09T23:38:56.645] auth/munge: _print_cred: ENCODED: Tue Dec 10 23:38:30 2024 [2024-12-09T23:38:56.645] auth/munge: _print_cred: DECODED: Mon Dec 09 23:38:56 2024 [2024-12-09T23:38:56.645] error: Check for out of sync clocks [2024-12-09T23:38:56.645] error: slurm_receive_msg_and_forward: g_slurm_auth_verify: REQUEST_PING has authentication error: Invalid authentication credential [2024-12-09T23:38:56.645] error: slurm_receive_msg_and_forward: Protocol authentication error [2024-12-09T23:38:56.655] error: service_connection: slurm_receive_msg: Protocol authentication error [2024-12-10T01:13:18.454] Slurmd shutdown completing ====== I have checked the time, this is fine munge.key I ran md5sum over it and get the same output. When I reboot the warewulf nodes its a common Container file so 1~2 working but not 3 is just wierd. On the controller, ========= [2024-12-10T01:30:35.108] Node node2 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.109] Node node1 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.109] Node node6 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.109] Node node7 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.110] Node node3 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.110] Node node5 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:35.110] Node node4 appears to have a different version of Slurm than ours. Please update at your earliest convenience. [2024-12-10T01:30:36.104] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2 ========== So the controller is RHEL9 and the nodes Rocky8 However 1~2 and 4~7 work OK. When I run, [root@vuwunicoslurmd1 log]# scontrol show node node3 NodeName=node3 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUEfctv=20 CPUTot=20 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=node3 NodeHostName=node3 Version=20.11.9 OS=Linux 4.18.0-553.30.1.el8_10.x86_64 #1 SMP Tue Nov 26 18:56:25 UTC 2024 RealMemory=48000 AllocMem=0 FreeMem=43378 Sockets=20 Boards=1 State=DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=debug BootTime=2024-12-09T20:41:27 SlurmdStartTime=2024-12-10T01:13:25 LastBusyTime=2024-12-10T01:30:31 CfgTRES=cpu=20,mem=48000M,billing=20 AllocTRES= CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Reason=Node unexpectedly rebooted [slurm@2024-12-09T20:40:27] [root@vuwunicoslurmd1 log]# I get a reason, its like the node is held down / off, does this need to be / can this be cleared? regards Steven

2 2

Job not starting
by Diego Zuccato 10 Dec '24

10 Dec '24

Hello all. An user reported that a job wasn't starting, so I tried to replicate the request and I get: -8<-- [root@ophfe1 root.old]# scontrol show job 113936 JobId=113936 JobName=test.sh UserId=root(0) GroupId=root(0) MCS_label=N/A Priority=1 Nice=0 Account=root QOS=long JobState=PENDING Reason=Priority Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2024-12-06T13:19:36 EligibleTime=2024-12-06T13:19:36 AccrueTime=2024-12-06T13:19:36 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-12-06T13:21:32 Scheduler=Backfill:* Partition=m3 AllocNode:Sid=ophfe1:855189 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=96 NumTasks=96 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=96,mem=95000M,node=1,billing=1296 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=95000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/home/root.old/test.sh WorkDir=/home/root.old StdErr=/home/root.old/%N-%J.err StdIn=/dev/null StdOut=/home/root.old/%N-%J.out Power= [root@ophfe1 root.old]# scontrol sho partition m3 PartitionName=m3 AllowGroups=ALL DenyAccounts=formazione AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=mtx20 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=CANCEL State=UP TotalCPUs=192 TotalNodes=1 SelectTypeParameters=CR_SOCKET_MEMORY JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=192,mem=1150000M,node=1,billing=2592 TRESBillingWeights=CPU=13.500,Mem=2.2378G [root@ophfe1 root.old]# scontrol show node mtx20 NodeName=mtx20 Arch=x86_64 CoresPerSocket=24 CPUAlloc=0 CPUEfctv=192 CPUTot=192 CPULoad=0.00 AvailableFeatures=ib,matrix,intel,avx ActiveFeatures=ib,matrix,intel,avx Gres=(null) NodeAddr=mtx20 NodeHostName=mtx20 Version=22.05.6 OS=Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 RealMemory=1150000 AllocMem=0 FreeMem=1156606 Sockets=4 Boards=1 MemSpecLimit=2048 State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=8 Owner=N/A MCS_label=N/A Partitions=m3 BootTime=2024-12-06T10:01:42 SlurmdStartTime=2024-12-06T10:02:54 LastBusyTime=2024-12-06T10:51:58 CfgTRES=cpu=192,mem=1150000M,billing=2592 AllocTRES= CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s -8<-- So the node is free, the partition does not impose extra limits (used only for accounting factors) but the job does not start. Any hints? Tks -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786

3 5

TRES cpu vs tasks
by Miriam Olmi 10 Dec '24

10 Dec '24

Hi all, I cannot understand the true difference and definition of "core", "task" and "cpu" within the limits associated to a partition via the TRES variable of a QOS. More precisely I have 2 partitions defined as follows: PartitionName=lprod AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=lprod_part DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=r037c01s[01-12] PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=432 TotalNodes=12 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=432,mem=12M,node=12,billing=432 PartitionName=bprod AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=bprod_part DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=r037c01s[01-12] PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=432 TotalNodes=12 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=432,mem=12M,node=12,billing=432 with the two qos defined as: Name Priority GraceTime PreemptMode UsageFactor MaxJobsPU MaxSubmitPU MaxTRES MinTRES Flags -------------------- ---------- ---------- ----------- ----------- --------- ----------- -------------------- -------------------- -------------------------------------------------- lprod_part 0 00:00:00 cluster 1.000000 100 120 cpu=32,mem=366G cpu=1 DenyOnLimit,PartitionMaxNodes,PartitionMinNodes bprod_part 0 00:00:00 cluster 1.000000 100 120 cpu=64,mem=366G cpu=33 DenyOnLimit,OverPartQOS If I try to submit a job on the lprod partition with the directives: #SBATCH --nodes=12 #SBATCH --ntasks-per-node=1 the job is executed correctly while it is not executed if submitted on the bprod partition due to the error: sbatch: error: QOSMinCpuNotSatisfied sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) I understand that this is related to the limit of cpu associated to the partition via the qos: lprod->[1-32]cpus, bprod->[33-64]cpus but I would like to have a more proper explanation since the options I am using are not referring to "cpu" but to "tasks" and I could not find a proper definition of "cpu" in the framework of the TRES. Many thanks in advance. Best, Miriam

2 2

slurm nodes showing down*
by Steven Jones 09 Dec '24

09 Dec '24

I have just rebuilt all my nodes and I see Only 1 & 2 seem available? While 3~6 are not 3's log, [root@node3 log]# tail slurmd.log [2024-12-08T21:45:51.250] CPU frequency setting not configured for this node [2024-12-08T21:45:51.251] slurmd version 20.11.9 started [2024-12-08T21:45:51.252] slurmd started on Sun, 08 Dec 2024 21:45:51 +0000 [2024-12-08T21:45:51.252] CPUs=20 Boards=1 Sockets=20 Cores=1 Threads=1 Memory=48269 TmpDisk=23324 Uptime=30 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) [root@node3 log]# And 7 doesnt want to talk to the controller. [root@node7 slurm]# sinfo slurm_load_partitions: Zero Bytes were transmitted or received [root@node7 slurm]# These are all rebuilt and 1~3 are identical and 4~7 are identical. 7's log keep saying, [2024-12-08T21:49:17.246] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:18.263] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:19.278] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:20.294] error: Unable to register: Zero Bytes were transmitted or received [2024-12-08T21:49:21.310] error: Unable to register: Zero Bytes were transmitted or received [root@vuwunicoslurmd1 slurm]# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST debug* up infinite 2 idle* node[1-2] debug* up infinite 4 down* node[3-6] [root@vuwunicoslurmd1 slurm]# regards Steven

2 4

error and output files
by michaelmorgan937＠gmail.com 09 Dec '24

09 Dec '24

Hi all, I have a program (for example x.x). When I run it as "x.x -I input -o output", I will get an "output" file, as well as output and error files from slurm. However, all the content of slurm output file is in the "output" file (and there are some extra content), so it is a waste to print the slurm output. Is it possible to set directives so that slurm only print error file? Thank you very much. Michael Morgan

2 1

Why is my job killed when ResumeTimeout is reached instead of it being requeued?
by Xaver Stiensmeier 09 Dec '24

09 Dec '24

Dear Slurm-user list, when a job fails because the node startup fails (cloud scheduling), the job should be re-queued: Resume Timeout Maximum time permitted (in seconds) between when a node resume request is issued and when the node is actually available for use. Nodes which fail to respond in this time frame will be marked DOWN and the jobs scheduled on the node requeued. however, instead of requeuing the job, it is killed. [2024-11-18T10:41:52.003] node bibigrid-worker-wubqboa1z2kkgx0-0 not resumed by ResumeTimeout(1200) - marking down and power_save [2024-11-18T10:41:52.003] Killing JobId=1 on failed node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.046] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:52.046] power down request repeating for node bibigrid-worker-wubqboa1z2kkgx0-0 Our ResumeProgram does not change the state of the underlying workers, I think we should set the nodes to DOWN explicitly if the startup fails given: *ResumeProgram* is unable to restore a node to service with a responding slurmd and an updated BootTime, it should set the node state to DOWN, which will result in a requeue of any job associated with the node - this will happen automatically if the node doesn't register within ResumeTimeout but in any case as we can see in the log the job should be requeued based on it reaching the ResumeTimeout alone. I am unsure why that is not happening. The power down request is sent by the ResumeFailProgram. We have SlurmctldParameters=idle_on_node_suspend activated, but that shouldn't affect Resume, I guess. My Slurm version is slurm 23.11.5 Best regards, Xaver # More context ## Slurmctld from submitting job to failure [2024-11-18T10:21:45.490] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=bibigrid-worker-wubqboa1z2kkgx0-0 usec=1221 [2024-11-18T10:21:45.499] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:21:58.387] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:21:58.387] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:22:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:23:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:23:20.009] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:23:23.003] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:23:23.398] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:23:23.398] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:23:53.398] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:23:53.398] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:24:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:24:21.484] slurmscriptd: error: _run_script: JobId=0 resumeprog exit status 1:0 [2024-11-18T10:25:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:26:02.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:26:02.417] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:26:02.417] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:26:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:26:32.417] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:26:32.417] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:27:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:28:20.003] debug: Updating partition uid access list [2024-11-18T10:28:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:28:20.008] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:29:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:29:22.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:29:22.448] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:29:22.448] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:30:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:31:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:32:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:32:42.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:32:42.478] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:32:42.478] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:33:12.479] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:33:12.479] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:33:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:33:20.010] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:34:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:35:20.007] debug: sched: Running job scheduler for full queue. [2024-11-18T10:36:01.004] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:36:01.504] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:36:01.504] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:36:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:36:31.505] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:36:31.505] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:37:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:38:20.008] debug: Updating partition uid access list [2024-11-18T10:38:20.008] debug: sched: Running job scheduler for full queue. [2024-11-18T10:38:20.017] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:39:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:39:21.003] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:39:21.530] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:39:21.530] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:39:51.531] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:39:51.531] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:40:21.000] debug: sched: Running job scheduler for full queue. [2024-11-18T10:41:20.003] debug: sched: Running job scheduler for full queue. [2024-11-18T10:41:52.003] node bibigrid-worker-wubqboa1z2kkgx0-0 not resumed by ResumeTimeout(1200) - marking down and power_save [2024-11-18T10:41:52.003] Killing JobId=1 on failed node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.046] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:52.046] power down request repeating for node bibigrid-worker-wubqboa1z2kkgx0-0 [2024-11-18T10:41:52.047] debug: sackd_mgr_dump_state: saved state of 0 nodes [2024-11-18T10:41:52.549] debug: sched/backfill: _attempt_backfill: beginning [2024-11-18T10:41:52.549] debug: sched/backfill: _attempt_backfill: no jobs to backfill [2024-11-18T10:41:52.736] _slurm_rpc_complete_job_allocation: JobId=1 error Job/step already completing or completed [2024-11-18T10:41:53.000] debug: Spawning ping agent for bibigrid-master-wubqboa1z2kkgx0 [2024-11-18T10:41:53.000] debug: sched: Running job scheduler for default depth. [2024-11-18T10:41:53.014] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 reason set to: FailedStartup [2024-11-18T10:41:53.014] update_node: node bibigrid-worker-wubqboa1z2kkgx0-0 state set to IDLE

1 1

getting slurm going
by Steven Jones 08 Dec '24

08 Dec '24

What tests can I do to prove that slurm is talking to the nodes pls? regards Steven

3 5

Non-Standard Mail Notification in Job
by Daniel Miliate 05 Dec '24

05 Dec '24

Newbie here, I'm trying to send myself an email notification with a custom body/subject for notification of important events. Is this possible? I've tried a few things in the script below. Thanks in advance and apologies if this is not how this listing works! - Daniel ---- Script ---- .... #SBATCH --mail-user= example(a)example.example #SBATCH --export=ALL #SBATCH --mail-type=ALL # Both of these work which mail which mailx EMAIL="example(a)example.example" # Attempt 1 echo "Test email body" | mail -s "Test Subject" "$EMAIL" # Attempt 2 mail -s "hello" "$EMAIL" <<EOF hello world EOF # Attempt 3 echo "Specific Information" | mail -s "SLURM Job information" $EMAIL ---- End ----

3 2

slurm not running on a warewulf node
by Steven Jones 04 Dec '24

04 Dec '24

Hi, I have set a log creation/location in slurm.conf as, SlurmdLogFile=/var/log/slurm/slurmd.log But it is 0 length. Slurm will not run, what else do I need to do to log why its failing pls? regards Steven

4 6

slurmd on a warwwulf node - not running
by Steven Jones 04 Dec '24

04 Dec '24

Hi, I have munge running on the controller and nodes fine, all tests passed. I have slurmctld running on the controller ok after checking the logs /var/spool/slurmctld was not created which I assume should have happened via the rpm install? Anyway I cant get slurmd to run on the warewulf nodes and cant find any log to check? How to fault find this? regards Steven

3 3

How can I make sure my user have only one job per node (Job array --exclusive=user,)
by Oren 03 Dec '24

03 Dec '24

Hi, I have a cluster of 20-nodes, and I want to run a jobarray on that cluster, but I want each node to get one job per node. When I do the following: #!/bin/bash #SBATCH --job-name=process_images_train # Job name #SBATCH --time=50:00:00 # Time limit hrs:min:sec #SBATCH --tasks=1 #SBATCH --cpus-per-task=4 #SBATCH --mem=50000 #SBATCH --array=0-19# 19 # Job array with 20 jobs (0 to 19) I get 10 jobs in node #1 and 10 jobs in node #20, I want a job in each node. I've tried: #SBATCH --exclusive=user Also #SBATCH --spread-job #SBATCH --distribution=cyclic Nothing changes, node#1 got 10 jobs and node#2 got 10 jobs. Thanks

2 6

multiple conf-server entries for sackd
by Brian Andrus 03 Dec '24

03 Dec '24

Not sure anyone would know, but... If you are running slurm in HA mode (multiple SlurmctldHost entries) is it possible to point sackd to more than one using the --conf-server option? So either specify --conf-server more than once, or have a comma-delimited list of them? The docs are a little light about that. Brian Andrus

1 0

sinfo not listing any partitions
by Kent L. Hanson 02 Dec '24

02 Dec '24

I am doing a new install of slurm 24.05.3 I have all the packages built and installed on headnode and compute node with the same munge.key, slurm.conf, and gres.conf file. I was able to run munge and unmunge commands to test munge successfully. Time is synced with chronyd. I can't seem to find any useful errors in the logs. For some reason when I run sinfo no nodes are listed. I just see the headers for each column. Has anyone seen this or know what a next step of troubleshooting would be? I'm new to this and not sure where to go from here. Thanks for any and all help! The odd output I am seeing [username@headnode ~] sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST (Nothing is output showing status of partition or nodes) Slurm.conf ClusterName=slurmkvasir SlurmctldHost=kadmin2 MpiDefault=none ProctrackType=proctrack/cgroup PrologFlags=contain ReturnToService=2 SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/cgroup MinJobAge=600 SchedulerType=sched/backfill SelectType=select/cons_tres PriorityType=priority/multifactor AccountingStorageHost=localhost AccountingStoragePass=/var/run/munge/munge.socket.2 AccountingStorageType=accounting_storage/slurmdbd AccountingStorageTRES=gres/gpu,cpu,node JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmLogFile=/var/log/slurm/slurmd.log nodeName=k[001-448] PartitionName=default Nodes=k[001-448] Default=YES MaxTime=INFINITE State=up Slurmctld.log Error: Configured MailProg is invalid Slurmctld version 24.05.3 started on cluster slurmkvasir Accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Regisetering slurmctld at port 8617 Error: read_slurm_conf: default partition not set. Revovered state of 448 nodes Down nodes: k[002-448] Recovered information about 0 jobs Revovered state of 0 reservations Read_slurm_conf: backup_controller not specified Select/cons_tres; select_p_reconfigure: select/cons_tres: reconfigure Running as primary controller Slurmd.log Error: Node configuration differs from hardware: CPUS=1:40(hw) Boards=1:1(hw) SocketsPerBoard=1:2(hw) CoresPerSocket=1:20(hw) ThreadsPerCore:1:1(hw) CPU frequency setting not configured for this node Slurmd version 24.05.3started Slurmd started on Wed, 27 Nov 2024 06:51:03 -0700 CPUS=1 Boards=1 Cores=1 Threads=1 Memory=192030 TmpDisk=95201 uptime 166740 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) Error: _forward_thread: failed to k019 (10.142.0.119:6818): Connection timed out (Above line repeated 20 or so times for different nodes.) Thanks, Kent Hanson

6 9

Slurm version 24.11 is now available
by Tim Wickberg 29 Nov '24

29 Nov '24

We are pleased to announce the availability of the Slurm 24.11 release. To highlight some new features in 24.11: - New gpu/nvidia plugin. This does not rely on any NVIDIA libraries, and will build by default on all systems. It supports basic GPU detection and management, but cannot currently identify GPU-to-GPU links, or provide usage data as these are not exposed by the kernel driver. - Add autodetected GPUs to the output from "slurmd -C". - Added new QOS-based reports to "sreport". - Revamped network I/O with the "conmgr" thread-pool model. - Added new "hostlist function" syntax for management commands and configuration files. - switch/hpe_slingshot - Added support for hardware collectives setup through the fabric manager. (Requires SlurmctldParameters=enable_stepmgr) - Added SchedulerParameters=bf_allow_magnetic_slot configuration option to allow backfill planning for magnetic reservations. - Added new "scontrol listjobs" and "liststeps" commands to complement "listpids", and provide --json/--yaml output for all three subcommands. - Allow jobs to be submitted against multiple QOSes. - Added new experimental "oracle" backfill scheduling support, which permits jobs to be delayed if the oracle function determines the reduced fragmentation of the network topology is sufficiently advantageous. - Improved responsiveness of the controller when jobs are requeued by replacing the "db_index" identifier with a slurmctld-generated unique identifier. ("SLUID") - New options to job_container/tmpfs to permit site-specific scripts to modify the namespace before user steps are launched, and to ensure all steps are completely captured within that namespace. The Slurm documentation has also been updated to the 24.11 release. (Older versions can be found in the archive, linked from the main documentation page.) Slurm can be downloaded from https://www.schedmd.com/download-slurm/ . - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0